systemd: A Practical Field Reference for Linux Operators

Most Linux distributions since 2015 ship with systemd as the default init system. It replaced the SysV init scripts that came before — and brought a unified configuration model, dependency handling, logging, and a toolchain for managing user services.

This is a working reference. Not the full architecture tour — just the bits you actually touch when running a server.

The Mental Model

systemd’s core concept is the unit. Every thing it manages is a unit:

• Service (.service) — long-running processes, the thing you’ll touch 90% of the time
• Socket (.socket) — listens on a port; triggers a service when a connection arrives
• Mount/Automount (.mount, .automount) — filesystem mount points
• Path (.path) — watches a file or directory; runs a service on change
• Timer (.timer) — cron replacement
• Target (.target) — groups of units; milestones like multi-user.target
• Slice/Scope (.slice, .scope) — cgroup resource grouping

The unit files themselves live in three directories, and their order is significant (lower-numbered directories override higher ones):

/lib/systemd/system/        # the OS vendor puts files here — don't edit
/etc/systemd/system/         # admin overrides (your custom units)
/run/systemd/system/        # runtime, ephemeral — survives reboot? no.

Rule of thumb: vendor files stay in /lib, your changes go in /etc. If a vendor file needs editing, copy it to /etc first, then edit the copy. Otherwise your changes get clobbered by the next package upgrade.

The Cheat Sheet

80% of the time you only need four commands:

systemctl status <unit>      # what's the current state, last few log lines
systemctl start   <unit>     # start it now (does NOT survive reboot)
systemctl stop    <unit>
systemctl restart <unit>
systemctl reload  <unit>     # ask the daemon to reload config without restart
sudo systemctl enable  <unit>   # start on boot — creates symlinks
sudo systemctl disable <unit>   # remove from boot chain
journalctl -u <unit> -f      # tail logs for that unit

The cardinal sin: editing a unit file and forgetting to systemctl daemon-reload. systemd caches parsed unit files. Until you tell it the cache is stale, you’re running the old version.

sudo systemctl daemon-reload   # after any unit file edit
sudo systemctl restart <unit>  # daemon-reload alone doesn't restart

Anatomy of a Unit File

A service file looks like this. The structure is fixed; only the directives inside change.

[Unit]
Description=My custom app on port 8080
Documentation=https://example.com/docs
After=network-online.target
Requires=network-online.target
Wants=

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/srv/myapp
ExecStart=/usr/bin/python3 /srv/myapp/main.py
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
Environment=NODE_ENV=production
EnvironmentFile=/etc/myapp/env
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Section-by-section

After= does not create a hard dependency — just ordering. If network.target isn’t ready, After= will still try to start your service, it’ll just wait first. Use Requires= or Wants= for actual start ordering with failure semantics.

• Requires= — hard dependency. If the required unit fails, this one fails too.
• Wants= — soft dependency. This unit runs whether or not the wanted one succeeds. Preferred for optional services.

WantedBy= in [Install] is what makes enable work. WantedBy=multi-user.target means “when the system reaches multi-user (non-graphical) boot, this should be in the chain.” 99% of services use this.

The Service Types

Type= controls how systemd understands “started.” Choose the right one or you’ll get weird behavior.

Type=simple    # default. Process must stay foreground. Most use this.
Type=forking   # forks daemonizes. systemd considers started when parent exits.
Type=oneshot   # runs to completion and exits. Good for setup scripts.
Type=notify    # service tells systemd it's ready via sd_notify().
Type=dbus      # service gets a D-Bus name when ready.
Type=idle      # like simple, but delays until other jobs finish.

Type=simple is the default and works for almost everything. Use Type=forking for legacy daemons that fork into the background (old nginx binary, traditional postgres, etc.). Use Type=notify if the daemon supports sd_notify(3) — that’s the modern, correct way for new services.

A common pitfall: using Type=forking for a process that doesn’t fork. systemd will wait for the parent to exit, parent never exits, service appears “activating (start)” forever.

Restart and Watchdog

Restart=no
Restart=on-success
Restart=on-failure
Restart=on-abnormal
Restart=always
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60

Restart=on-failure is what you usually want. systemd will not restart a service that exited cleanly with code 0 — that’s an indication the service shut down on purpose.

RestartSec= is the sleep between attempts. Without it, a crashing service hits a tight loop and fills logs. 5 seconds is a reasonable default for most cases.

StartLimitBurst= and StartLimitIntervalSec= together define the rate limit: don’t restart more than 5 times in 60 seconds. After that, the service is left in failed state and you have to investigate.

Security Hardening (the Easy Wins)

systemd ships a sandbox; you turn it on per unit. Most production services need only a few lines:

[Service]
PrivateTmp=true          # service can't see the system /tmp
ProtectSystem=strict     # /usr and /boot read-only; only /var, /etc/myapp writable
ProtectHome=true         # /home, /root invisible to this service
NoNewPrivileges=true     # service can't escalate via SUID binaries
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

That’s a sane default for any network-facing service you don’t fully trust. Test with systemd-analyze security myapp.service — it scores your unit from 1.0 (loose) to 9.0 (paranoid). Aim for 6+.

The hardening directives are cumulative — each one removes a capability or syscall. The order doesn’t matter. They all degrade gracefully if the service needs something: Allow= directives to grant back specific bits.

Resource Limits

systemd runs everything in cgroups. You can cap CPU, memory, IO per unit:

[Service]
CPUQuota=50%                # 50% of one CPU
MemoryMax=512M              # OOM kill if exceeded
MemoryHigh=384M             # soft pressure threshold (slow down earlier)
IOWeight=500                # 1-10000, default 100, this is medium
IOReadBandwidthMax=/dev/sda 50M
TasksMax=100                # max number of threads/processes
LimitNOFILE=65536

Useful for noisy-neighbor containment. If you’re running multi-tenant workloads on a single box, set resource caps on each service.

Timers: The Modern Cron Replacement

systemd timers do everything cron does, better:

[Unit]
Description=Run myapp-cleanup every 15 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=15min
Persistent=true

[Service]
Type=oneshot
ExecStart=/usr/local/bin/myapp-cleanup

sudo systemctl enable --now myapp-cleanup.timer

The pattern: a .timer unit triggers a .service unit. The .service for a timer usually has Type=oneshot — it runs to completion, not stays running.

OnCalendar= for cron-style: OnCalendar=Mon..Fri 09:00 America/Los_Angeles. Both can be combined with calendar + relative time.

Best practice for scheduled jobs: always use a randomized delay to avoid thundering herd:

[Timer]
RandomizedDelaySec=120

Otherwise at 09:00 sharp every box hits your shared backend at once.

User Services

systemd manages user-level services as well, each user gets their own systemd instance. Useful for desktop apps that need to run at boot, or for server admins who want to keep their personal daemons out of /etc.

systemctl --user status    # user instance
systemctl --user enable myapp.timer

User services are in ~/.config/systemd/user/. Survive logout only with loginctl enable-linger <user>.

On a headless server, the user manager (PID 1 of user systemd) only runs while the user is logged in. To keep user services running without an active login, enable lingering:

sudo loginctl enable-linger myuser

After that, --user services run from boot. This is how to deploy a long-running service that doesn’t need root access.

Targets (Boot Milestones)

A target is just a group of units. The boot sequence uses these:

sysinit.target        → early kernel/userspace
basic.target          → sysinit + some boot plumbing
multi-user.target     → networking, login, all the rest (default)
graphical.target      → multi-user + display manager

Override the boot default:

sudo systemctl set-default multi-user.target   # console boot
sudo systemctl set-default graphical.target    # default
sudo systemctl isolate rescue.target           # jump to single-user

Use isolate for emergency recovery. It kills all units not in the requested target tree. Useful when fixing a broken login manager.

debug and Recovery

When a service won’t start:

sudo systemctl status myapp.service        # see why it failed
sudo journalctl -u myapp.service --since "1 hour ago"
sudo journalctl -u myapp.service -n 200    # last 200 lines
sudo journalctl -f -u myapp.service        # follow new
sudo systemctl show myapp.service          # all properties as key=value
sudo systemd-analyze verify myapp.service  # static analysis, catches typos
sudo systemd-analyze blame                 # what took the longest at boot
sudo systemd-analyze critical-chain myapp.service  # full dependency tree

systemd-analyze verify is underused — it catches missing binaries, bad environment variables, and inconsistent dependencies before you ever try to start the service.

For services that keep failing:

sudo systemctl reset-failed myapp.service   # clear failure state, allow enable again

For deep boot issues, add systemd.log_level=debug to the kernel command line via GRUB_CMDLINE_LINUX and reboot. The kernel log shows every unit transition.

Templates (One File, Many Instances)

Some services have many similar instances — database shards, per-user app instances. Use unit templates:

# Generic file: myapp@.service
[Service]
ExecStart=/usr/local/bin/myapp --id=%i
WorkingDirectory=/var/lib/myapp/%i

# Spawn concrete instances:
sudo systemctl enable myapp@1.service
sudo systemctl enable myapp@2.service

%i gets the instance name. Common uses: per-user XDG dirs (systemd user services), PostgreSQL replication slots, RabbitMQ nodes. The Postgres deb package ships postgresql@12-main.service and friends using this pattern.

Migration From SysV init

If you’re holding onto old distro knowledge:

# See a runlevel's services (legacy views)
sudo systemctl list-units --type=service --all
systemctl list-dependencies multi-user.target   # what runs at boot

# Convert: systemctl mask prevents ALL start including manual
sudo systemctl mask ntp.service    # symlinks to /dev/null

# Disable old /etc/rc.local
sudo systemctl status rc-local.service

Modern distros still ship /etc/rc.local as a rc-local.service for nostalgia. Don’t use it. Write a real .service file or .timer.

A Working Recipe: Deploy a Custom App

Putting it together — a small but real service file you can adapt:

# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp production server
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=notify
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
ExecStart=/srv/myapp/bin/myapp --config=/etc/myapp/config.toml
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5

# Hardening
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true

# Resources
CPUQuota=80%
MemoryMax=1G
LimitNOFILE=65535

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# Environment
EnvironmentFile=/etc/myapp/env

[Install]
WantedBy=multi-user.target

Deploy:

sudo install -m644 /tmp/myapp.service /etc/systemd/system/myapp.service
sudo install -d -omyapp -gmyapp -m750 /srv/myapp
sudo install -d -omyapp -gmyapp -m750 /etc/myapp
sudo install -m640 /tmp/env /etc/myapp/env      # mode 640, not 644
sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status myapp.service
sudo journalctl -u myapp.service -f

That’s the deploy flow. No forking init scripts, no service myapp start, no update-rc.d.

Quick Glossary

• Unit — anything systemd manages (.service, .socket, .timer, etc.)
• Target — a group of units, like a runlevel
• Active/Enabled — Active is “running right now,” Enabled is “starts at boot”
• Mask — prevent start via mask — even if the unit file exists
• Cgroup — kernel feature systemd uses for resource limits and containment
• journal — systemd’s structured log
• Slice — a cgroup-defined hierarchy; useful for grouping related services
• Scope — a transient cgroup for foreign processes (Docker, libvirt)

Resources:

• systemd official docs — Lennart’s project site
• systemd.directives — every directive with one-line explanation
• systemd-analyze — the security, verify, blame, and critical-chain subcommands
• Practical systemd — Red Hat Enterprise Linux 9 docs covering the production defaults