systemd Operations Manual

systemd is the init system and service manager for the Linux user space. Understanding unit types (service/timer/socket/mount), dependencies, the logging system, and the override mechanism is fundamental for operations.

Process viewing: btm (bottom, a replacement for top — CPU/memory/disk/temperature TUI), procs (a replacement for ps — colored, auto column width, process tree)

Check Status

# Check the current status of a unit, including the last 10 lines of logs, cgroup tree, and main PID
systemctl status nginx
# Active: active (running) since ... — Running
# Active: inactive (dead) — Not running
# Active: failed — Startup failed (check exit code)

# List all units, filtered by status
systemctl list-units                    # All loaded units
systemctl list-units --state=failed    # Only failed ones
systemctl list-units --type=service    # Only services
systemctl list-units --type=timer      # Only timers

# Check if a unit is enabled (starts on boot)
systemctl is-enabled nginx
systemctl is-active nginx              # Is it running?

# View the dependency tree of a unit
systemctl list-dependencies nginx
systemctl list-dependencies nginx --reverse  # What depends on it?

# View the unit file content (including all drop-ins)
systemctl cat nginx

# List all timers and their next trigger times
systemctl list-timers
systemctl list-timers --all            # Including inactive ones

Management

# Start/Stop/Restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx                # Process will briefly interrupt
systemctl reload nginx                 # Hot reload (requires service support, e.g., nginx -s reload)
# Difference: reload does not interrupt connections, restart does
# If the service does not support reload, this command will fail

systemctl enable nginx                 # Start on boot (creates symlink)
systemctl disable nginx                # Disable start on boot
systemctl reenable nginx               # Disable then enable (fixes broken symlinks)

systemctl mask nginx                   # Completely prevent startup (symlink → /dev/null)
systemctl unmask nginx                 # Restore
# mask vs disable: after disable, you can still manually start; after mask, even manual start is rejected

# Must run after modifying unit files
systemctl daemon-reload                # Reload all unit files

# Clear failed status (otherwise status will always show failed)
systemctl reset-failed nginx

User-level Services

# User services are independent of system services: ~/.config/systemd/user/
# User services stop by default after logout (unless lingering is enabled)
systemctl --user status myservice
systemctl --user start myservice
systemctl --user enable myservice

# Enable lingering (service continues running after logout):
loginctl enable-linger <username>
# Suitable for: rathole-watchdog, local llama-server, and other user services that need to run persistently

Timer

Timers trigger the corresponding service at a specified time (paired by name: .timer/.service):

# View timer definition
systemctl cat mytask.timer

# Manually trigger (don't wait for next scheduled time)
systemctl start mytask.service         # Run the service directly; timer is unaffected

# /etc/systemd/system/mytask.timer
[Timer]
# Trigger every 5min after last activation
OnUnitActiveSec=5min
# Trigger first time 2min after boot
OnBootSec=2min
# Random delay of 30s (avoid thundering herd when multiple machines run simultaneously)
RandomizedDelaySec=30
# Compensate for missed triggers (e.g., if the machine was asleep)
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/mytask.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/mytask.sh
User=myuser

Logs

journalctl -u nginx                     # All logs for the specified unit
journalctl -u nginx -f                  # Follow (tail -f)
journalctl -u nginx --since "10 min ago"
journalctl -u nginx --since "2025-01-01" --until "2025-01-02"
journalctl -u nginx -n 50               # Last 50 lines
journalctl -u nginx -o json-pretty | head -100  # Structured output (includes PID, UID, cmdline)

journalctl -b                           # Logs from the current boot
journalctl -b -1                        # Logs from the previous boot (troubleshoot issues before reboot)

journalctl _PID=12345                   # Filter by PID

# Space management
journalctl --disk-usage                  # Current disk usage
journalctl --vacuum-size=500M            # Limit maximum size to 500MB
journalctl --vacuum-time=7d              # Keep logs from the last 7 days

Override

To modify unit configuration, do not directly edit /usr/lib/systemd/system/ (as it will be overwritten by package management). Instead, create a drop-in:

# Open editor to create override
systemctl edit nginx
# → /etc/systemd/system/nginx.service.d/override.conf

# Or manually
mkdir -p /etc/systemd/system/nginx.service.d
cat > /etc/systemd/system/nginx.service.d/override.conf <<EOF
[Service]
# Add environment variable
Environment=DEBUG=true
# Modify startup command (clear first, then set new)
ExecStart=
ExecStart=/usr/bin/nginx -g "daemon off;"
# Set restart policy
Restart=on-failure
RestartSec=5s
EOF
systemctl daemon-reload
systemctl restart nginx

Note: The first line ExecStart= must exist to clear the previous value. If not cleared first, the new ExecStart will be appended rather than replaced.

Troubleshooting

# Service fails to start
systemctl status nginx                  # Check Active: failed and exit code
journalctl -u nginx -e --no-pager      # Last 1000 lines (check stderr)

# Analyze startup bottlenecks
systemd-analyze blame | head -20        # Initialization time for each unit
systemd-analyze critical-chain nginx    # Which link in the dependency chain is slowest?

# Verify unit file
systemd-analyze verify /etc/systemd/system/myservice.service

# A unit is masked, causing WantedBy to fail
ls -la /etc/systemd/system/*.target.wants/ | grep /dev/null