systemd Unit Files & journalctl

Reading unit files, writing simple ones, controlling services, and finding log output with journalctl.

systemd basics

systemd is the init system and service manager on all modern RHEL/Debian-based distributions. It starts and stops services, manages boot ordering, handles system targets (similar to old runlevels), and collects logs via journald.

Everything systemd manages is a unit. Services are .service units. There are also .timer, .socket, .target, and .mount units. When you say "start the nginx service", you are telling systemd to activate the nginx.service unit.

Essential systemctl commands

# Start / stop / restart a service
systemctl start nginx
systemctl stop nginx
systemctl restart nginx

# Reload config without full restart (if the service supports it)
systemctl reload nginx

# Enable at boot / disable at boot
systemctl enable nginx
systemctl disable nginx

# Enable AND start in one command
systemctl enable --now nginx
systemctl disable --now nginx

# Check status
systemctl status nginx

# Is it running?
systemctl is-active nginx        # prints: active or inactive

# Is it enabled at boot?
systemctl is-enabled nginx       # prints: enabled or disabled

# List all running services
systemctl list-units --type=service --state=running

# List failed services
systemctl --failed

# Clear the "failed" state after you have investigated and fixed
systemctl reset-failed nginx
systemctl reset-failed     # clear all failed units

# Mask a unit — prevents it from being started at all (even manually)
# Use when a package installs a service you never want running
systemctl mask postfix
# To re-enable it:
systemctl unmask postfix

# disable stops it starting at boot; mask makes it impossible to start
# disable: "don't start on boot"   mask: "block completely"

# View the unit file
systemctl cat nginx

Unit file anatomy

A service unit file has three sections: [Unit], [Service], and [Install].

[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Key directives explained:

• After= — start this service after these units (ordering, not requirement)
• Requires= — if this service fails, stop mine too (hard dependency)
• Wants= — start those units too, but continue even if they fail (soft dependency)
• Type= — how systemd knows the service is started:
  • simple — the ExecStart process IS the main process (most common)
  • forking — the process forks and the parent exits; systemd tracks the child via PIDFile
  • oneshot — runs once and exits; systemd waits for it to finish
  • notify — service tells systemd when it is ready via sd_notify
• ExecStart= — the command to run to start the service
• ExecStartPre= — commands to run before ExecStart (e.g. config validation)
• Restart= — when to automatically restart:
  • on-failure — restart if the process exits with a non-zero code
  • always — restart regardless of how it exited
  • no — never restart (useful for one-shot tasks)
• WantedBy=multi-user.target — enable this service in normal (non-graphical) boot

Resource limits in unit files

[Service]
# Limit CPU to 50% of one core
CPUQuota=50%

# Limit RAM to 512 MB (kills the process if exceeded)
MemoryMax=512M
MemoryHigh=400M   # soft limit — systemd starts throttling before killing

# Limit open file descriptors (common for databases and web servers)
LimitNOFILE=65536

# Limit number of processes this service can create
LimitNPROC=512

Resource limits are useful for multi-tenant servers and for preventing runaway services from taking down the whole host. Check current limits for a running service with systemctl show nginx | grep -i limit. Changes take effect after a service restart.

Where unit files live

/lib/systemd/system/        # package-installed units (do not edit these)
/usr/lib/systemd/system/    # same, on RHEL
/etc/systemd/system/        # your overrides and custom units (edit here)
~/.config/systemd/user/     # user-level units (for non-root services)

Files in /etc/systemd/system/ take precedence over files with the same name in /lib/systemd/system/. This is how you override a package-provided unit without modifying it.

Writing a simple service unit

Creating a systemd service for a custom application:

# /etc/systemd/system/myapp.service

[Unit]
Description=My Application
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml
Restart=on-failure
RestartSec=5s

# Resource limits
LimitNOFILE=65536

# Security hardening (optional but recommended)
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# After creating the file, reload systemd and start the service
systemctl daemon-reload
systemctl enable --now myapp
systemctl status myapp

Override without editing the original

To change a setting in a package-provided unit file without editing the original (which would be overwritten on package update):

# The easy way — opens the correct override file in an editor
systemctl edit nginx

# This creates /etc/systemd/system/nginx.service.d/override.conf
# Add ONLY the directives you want to change:
[Service]
Restart=always
RestartSec=10s
Environment="EXTRA_OPTS=-p /var/run/nginx_custom.pid"

systemctl daemon-reload
systemctl restart nginx

To see the effective unit file after overrides are applied:

systemctl cat nginx

The override file only needs to contain the sections and directives you are changing. systemd merges it with the original. You do not need to copy the entire unit file.

Service dependencies and ordering

[Unit]
# Start after these (ordering only — they can fail)
After=network.target rsyslog.service

# Require these to be running (if they fail, stop me too)
Requires=postgresql.service

# Start those alongside me if possible (soft want)
Wants=optional-helper.service

# Tightly couples this unit's lifecycle to another — if the bound
# unit stops or fails, this unit stops too (stronger than Requires)
BindsTo=required-service.service

journalctl — reading logs

journalctl reads the systemd journal — the central log store that captures output from all systemd services.

# View logs for a specific service
journalctl -u nginx

# Follow logs in real time (like tail -f)
journalctl -u nginx -f

# Show last 50 lines
journalctl -u nginx -n 50

# Show logs since last boot
journalctl -u nginx -b

# Show logs from previous boot (when a service crashed at boot)
journalctl -u nginx -b -1

# Show errors and above only
journalctl -u nginx -p err

# Show all kernel messages (useful for OOM / hardware issues)
journalctl -k

# Show all logs since a specific time
journalctl --since "2024-10-01 08:00:00"
journalctl --since "1 hour ago"

Useful journalctl flags

-u nginx            # filter by unit name
-f                  # follow (live)
-n 100              # last 100 lines
-b                  # current boot
-b -1               # previous boot
-p err              # errors and above (emerg, alert, crit, err)
-p warning          # warnings and above
--since "1h ago"    # time filter
--until "10 min ago"
--no-pager          # don't page output (useful in scripts)
-o short            # short format (default)
-o json             # JSON format
-o cat              # just the message, no metadata

# Show all failed service logs from the current boot
journalctl -b -p err --no-pager

# Combine filters
journalctl -u nginx -b --since "30 min ago" -p warning

Making logs persistent

By default on some systems, the journal does not persist across reboots (it lives in /run/log/journal/ which is tmpfs).

# Check where journal is stored
ls /var/log/journal/    # exists = persistent
ls /run/log/journal/    # exists only = volatile

# Enable persistence
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

# Or set in config
echo "Storage=persistent" >> /etc/systemd/journald.conf
systemctl restart systemd-journald

# Control how much disk space the journal uses
# In /etc/systemd/journald.conf:
SystemMaxUse=2G       # max disk space
SystemKeepFree=500M   # always keep this free
MaxRetentionSec=30d   # delete logs older than 30 days

# Vacuum old logs manually
journalctl --vacuum-size=2G
journalctl --vacuum-time=30d

timedatectl — time and timezone

timedatectl is the systemd tool for checking and configuring the system clock and timezone. It replaces the older date and hwclock workflows on systemd-based systems.

# Check current time, timezone, and NTP sync status
timedatectl
# Output shows: Local time, Universal time, RTC time, Time zone, NTP active, Synchronized

# Set timezone
timedatectl set-timezone Europe/London
timedatectl set-timezone America/New_York
timedatectl set-timezone UTC

# List available timezones (pipe through grep)
timedatectl list-timezones
timedatectl list-timezones | grep Australia

# Check if NTP is synchronized
timedatectl show --property=NTPSynchronized
timedatectl show --property=NTPService

# Enable NTP sync (uses systemd-timesyncd or chrony, whichever is configured)
timedatectl set-ntp true

If Chrony is installed, it takes over NTP duties and timedatectl will report its sync status. See the Chrony page for diagnosing time sync issues. On RHEL, chronyd is the default; on Debian/Ubuntu, systemd-timesyncd is the lightweight default but Chrony is preferred for servers.