---
title: systemd Operations Manual
url: https://doc.liz6.com/en/devops/systemd-ops
locale: en
area: devops
tags:
- devops
date: 2026-06-30
modified: 2026-07-16
description: systemd Operations Manual systemd is the init system and service manager for the Linux user space. Understanding unit types (service/timer/socket/mount), depend…
---

# systemd Operations Manual

systemd is the init system and service manager for the Linux user space. Understanding unit types (service/timer/socket/mount), dependencies, the logging system, and the override mechanism is fundamental for operations.

Process viewing: `btm` (bottom, a replacement for top — CPU/memory/disk/temperature TUI), `procs` (a replacement for ps — colored, auto column width, process tree)

## Check Status

```bash
# Check the current status of a unit, including the last 10 lines of logs, cgroup tree, and main PID
systemctl status nginx
# Active: active (running) since ... — Running
# Active: inactive (dead) — Not running
# Active: failed — Startup failed (check exit code)

# List all units, filtered by status
systemctl list-units                    # All loaded units
systemctl list-units --state=failed    # Only failed ones
systemctl list-units --type=service    # Only services
systemctl list-units --type=timer      # Only timers

# Check if a unit is enabled (starts on boot)
systemctl is-enabled nginx
systemctl is-active nginx              # Is it running?

# View the dependency tree of a unit
systemctl list-dependencies nginx
systemctl list-dependencies nginx --reverse  # What depends on it?

# View the unit file content (including all drop-ins)
systemctl cat nginx

# List all timers and their next trigger times
systemctl list-timers
systemctl list-timers --all            # Including inactive ones
```

## Management

```bash
# Start/Stop/Restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx                # Process will briefly interrupt
systemctl reload nginx                 # Hot reload (requires service support, e.g., nginx -s reload)
# Difference: reload does not interrupt connections, restart does
# If the service does not support reload, this command will fail

systemctl enable nginx                 # Start on boot (creates symlink)
systemctl disable nginx                # Disable start on boot
systemctl reenable nginx               # Disable then enable (fixes broken symlinks)

systemctl mask nginx                   # Completely prevent startup (symlink → /dev/null)
systemctl unmask nginx                 # Restore
# mask vs disable: after disable, you can still manually start; after mask, even manual start is rejected

# Must run after modifying unit files
systemctl daemon-reload                # Reload all unit files

# Clear failed status (otherwise status will always show failed)
systemctl reset-failed nginx
```

### User-level Services

```bash
# User services are independent of system services: ~/.config/systemd/user/
# User services stop by default after logout (unless lingering is enabled)
systemctl --user status myservice
systemctl --user start myservice
systemctl --user enable myservice

# Enable lingering (service continues running after logout):
loginctl enable-linger <username>
# Suitable for: rathole-watchdog, local llama-server, and other user services that need to run persistently
```

## Timer

Timers trigger the corresponding service at a specified time (paired by name: .timer/.service):

```bash
# View timer definition
systemctl cat mytask.timer

# Manually trigger (don't wait for next scheduled time)
systemctl start mytask.service         # Run the service directly; timer is unaffected
```

```ini
# /etc/systemd/system/mytask.timer
[Timer]
# Trigger every 5min after last activation
OnUnitActiveSec=5min
# Trigger first time 2min after boot
OnBootSec=2min
# Random delay of 30s (avoid thundering herd when multiple machines run simultaneously)
RandomizedDelaySec=30
# Compensate for missed triggers (e.g., if the machine was asleep)
Persistent=true

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/mytask.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/mytask.sh
User=myuser
```

## Logs

```bash
journalctl -u nginx                     # All logs for the specified unit
journalctl -u nginx -f                  # Follow (tail -f)
journalctl -u nginx --since "10 min ago"
journalctl -u nginx --since "2025-01-01" --until "2025-01-02"
journalctl -u nginx -n 50               # Last 50 lines
journalctl -u nginx -o json-pretty | head -100  # Structured output (includes PID, UID, cmdline)

journalctl -b                           # Logs from the current boot
journalctl -b -1                        # Logs from the previous boot (troubleshoot issues before reboot)

journalctl _PID=12345                   # Filter by PID

# Space management
journalctl --disk-usage                  # Current disk usage
journalctl --vacuum-size=500M            # Limit maximum size to 500MB
journalctl --vacuum-time=7d              # Keep logs from the last 7 days
```

## Override

To modify unit configuration, do not directly edit `/usr/lib/systemd/system/` (as it will be overwritten by package management). Instead, create a drop-in:

```bash
# Open editor to create override
systemctl edit nginx
# → /etc/systemd/system/nginx.service.d/override.conf

# Or manually
mkdir -p /etc/systemd/system/nginx.service.d
cat > /etc/systemd/system/nginx.service.d/override.conf <<EOF
[Service]
# Add environment variable
Environment=DEBUG=true
# Modify startup command (clear first, then set new)
ExecStart=
ExecStart=/usr/bin/nginx -g "daemon off;"
# Set restart policy
Restart=on-failure
RestartSec=5s
EOF
systemctl daemon-reload
systemctl restart nginx
```

Note: The first line `ExecStart=` must exist to clear the previous value. If not cleared first, the new `ExecStart` will be appended rather than replaced.

## Troubleshooting

```bash
# Service fails to start
systemctl status nginx                  # Check Active: failed and exit code
journalctl -u nginx -e --no-pager      # Last 1000 lines (check stderr)

# Analyze startup bottlenecks
systemd-analyze blame | head -20        # Initialization time for each unit
systemd-analyze critical-chain nginx    # Which link in the dependency chain is slowest?

# Verify unit file
systemd-analyze verify /etc/systemd/system/myservice.service

# A unit is masked, causing WantedBy to fail
ls -la /etc/systemd/system/*.target.wants/ | grep /dev/null
```