Linux & DevOps Quick Reference
Study notes and cheatsheets for Linux, SSH, Git, Ansible, certificates, and common services. Built to be used in order — read Start Here first.
- What is the symptom exactly?
- Is the service running?
- What do the logs say?
- Is the config valid?
- Is the network working?
- Is DNS working?
- Is auth, a cert, or time involved?
- New to the site — Start Here explains the foundations (01–18), the Ansible practice path, and how the rest of the material fits around that spine.
- Learn Ansible by doing — YAML and Ansible foundations, then Learn Ansible (Tutorial) and Best practices when you are ready to ship playbooks to production.
- On call right now — Generic, Troubleshooting, Service troubleshooting, First 15 minutes, and the Cheatsheet index. Use Ctrl+K or Cmd+K or / to search the whole site from any page.
The site map lists every page by group (kept in step with the sidebar). Below: quick entry points for tracks that are easy to miss if you only scroll the numbered path.
Observability
Observability Overview
Metrics, logs, traces, cardinality, SLI/SLO framing, and the minimum signal set every service should emit.
Prometheus & node_exporter
Scrape model, service discovery, recording rules, and host-level metrics.
Grafana Basics
Dashboards, data sources, alert rules, and mapping metrics to SLOs.
Loki Logs
Log aggregation, shippers, LogQL, and deriving metrics from logs.
OpenTelemetry Traces
Trace context, OTLP, sampling, and tying spans to services and SLOs.
SLOs & On-Call
SLIs, SLOs, error budgets, burn rates, on-call handover, and post-incident review.
Backup & DR
Backup & Restore
Strategy, 3-2-1, restore drills, and runbook patterns.
Borg & Borgmatic
Encrypted deduplicated archives, retention, key handling, and restore drills on Linux.
Postgres Backup
Physical vs logical, pgbackrest, PITR, and verification.
MySQL Backup
mysqldump, XtraBackup, binlogs, PITR, restore drills, and common failures.
DR Runbook Template
Structured template for disaster recovery playbooks.
Containers
Containers 101
Images, registries, namespaces, and how containers differ from VMs.
Podman Basics
Rootless containers, pods, and systemd integration on Linux.
Docker Compose
Multi-container apps, networks, volumes, and compose workflows.
Kubernetes Light
Deployments, Services, probes, resources, ConfigMaps, Secrets, Ingress, and a kubectl debug checklist.
IaC
Terraform Basics
State, plan/apply, modules, and working safely with real infra.
Terraform + Cloudflare
DNS, Pages, WAF, and API tokens with the Cloudflare provider and CI-safe patterns.
Packer Images
Reproducible machine images and provisioners.
Core Skills
Linux CLI
Navigation, files, permissions, processes, services, networking, DNS, packages, and one-liners — with explanations.
SSH Keys
What SSH keys are, how to generate and use them, the ssh-agent, config file, and common errors.
Git Basics
What Git is, branching, staging, committing, pushing, rebasing, undoing mistakes.
GitLab Basics
SSH setup, the merge request workflow, pipelines, protected branches, common problems.
YAML Basics
What YAML is, why indentation matters, dicts, lists, nested data, booleans, common mistakes.
Ansible
What Ansible is, inventory, playbooks, modules, handlers, idempotency, lint, best practices.
Jinja2
Template language used by Ansible. Variables, conditionals, loops, filters, and examples.
Certificates
Private keys, CSRs, CAs, openssl commands, checking expiry, converting formats, common issues.
FreeIPA
Users, groups, hosts, services, Kerberos, HBAC, client enrolment, troubleshooting.
Services
Chrony
Time sync. Why time matters, config, commands, service checks.
Rsyslog
Local logging and remote forwarding. Config, facilities, severities, service checks.
Postfix
Mail transfer. main.cf, relay, queue, useful commands, service checks.
Dovecot
IMAP/POP3 and mailbox access. Useful commands and service checks.
Squid
Proxy server. What a proxy and reverse proxy are, ACL example, service checks.
Nginx
Web server and reverse proxy. Server blocks, proxy setup, config test, service checks.
Apache
Web server. VirtualHosts, config test, apachectl, service checks.
Workflow & Reference
Troubleshooting Workflow
A repeatable step-by-step process: symptom → service → config → network → DNS → auth.
Glossary
Key terms across Linux, Git, YAML, Ansible, Jinja2, certificates, and FreeIPA.
Ansible Collection
Production-ready Ansible collection structure for a standard Linux stack.
Ansible Deep Dives
Quickstart (For Dummies)
Zero-to-playbook in ten minutes. Plain language, no jargon. Install Ansible, write an inventory, run your first playbook.
Learn Ansible (Tutorial)
Hands-on, progressively harder labs from a one-file playbook to a multi-role stack with tags, environments, and CI.
Best Practices & Refactoring
Naming, layout, idempotency, error handling, performance, security, refactoring recipes, anti-patterns, review checklist.
Roles in Practice
Role directory layout, defaults vs vars, tasks, handlers, templates, and calling a role from a playbook.
Project Structure
Production repo layout: inventories, group_vars, host_vars, ansible.cfg, and how to navigate a real repo.
Variable Precedence
The full priority chain from role defaults to extra vars, and how to debug variable values.
Debugging
Verbose mode, --check --diff, the debug module, common errors, and ansible-lint.
Testing
ansible-lint, yamllint, --syntax-check, --check --diff, Molecule scenarios, ansible-test, GitLab CI skeletons.
Error Handling
block/rescue/always, failed_when with complex rc sets, any_errors_fatal, sanity checks before destructive tasks.
Performance
Fact caching, pipelining, ControlMaster, strategies, async/poll, forks, when to use delegation.
Inventory Patterns
Static INI vs YAML, group_vars as directories, dynamic inventory plugins, lab vs prod layout.
Custom Modules
When to write one, minimal Python module skeleton, return contract, check_mode, collections, action/filter plugins.
Deploy Flow
End-to-end trace from variable → template → deployed file → handler → service restart.
Handlers & Templates
Multiple handlers, listen, flush_handlers, OS-aware templates, and whitespace control.
Tags
How tags work, --tags/--skip-tags, tag inheritance through imports, and CI-friendly tagging patterns.
Cheatsheet
Dense single-page reference: CLI flags, ad-hoc modules, task/handler boilerplate, loops, precedence, Jinja filters, vault.
GitLab & CI
CI/CD Pipelines
.gitlab-ci.yml structure, running Ansible in CI, SSH keys, vault in CI, reading failed job logs.
Merge Requests
Creating MRs, reviewing infra diffs, approvals, responding to comments, squash and merge.
Git for Infra
Branch conventions, finding what to edit, clean commits, hotfixes, and reverting bad merges.
GitLab Runner Setup
Install, register, executors, tags, protected runners, autoscaling, and gotchas.
CI for Ansible
Pragmatic pipeline: lint, Molecule, OIDC to Vault, junit artifacts, protected deploy.
Secrets & OIDC
CI variables, protected/masked vars, and OIDC to Vault, AWS, and GCP with troubleshooting.
Platform
SELinux
Modes, contexts, reading denials, booleans, file context fixes, and Ansible management.
firewalld
Zones, services, ports, rich rules, runtime vs permanent, and Ansible management.
systemd & journalctl
Unit file anatomy, writing services, overrides, and reading logs with journalctl.
Linux Networking
ip, routes, DNS tools, ss, nc, tcpdump, nmcli, and the networking troubleshoot checklist.
LVM
Physical volumes, volume groups, logical volumes, online extend, snapshots, thin provisioning, and LVM RAID.
SELinux Debugging
AVCs, audit2allow, semanage fcontext, restorecon, booleans, and custom policy modules.
firewalld Rich Rules
Syntax, priorities, rate limits, forward-port, logging, and IPv6 patterns.
systemd Unit Authoring
Unit files, Restart=, drop-ins, hardening, timers, sockets, and full examples.
Bonding & Bridges
NetworkManager bonds, LACP, bridges for VMs, VLANs, MTU, and failover.
LVM Thin & Snapshots
Thin pools, overprovisioning, autoextend, and avoiding thin-LVM data loss.
Identity & Auth
SSSD & Auth Flow
The Linux auth stack: PAM, NSS, SSSD, sssd.conf, cache, and login failure diagnosis.
Kerberos
Tickets, TGTs, kinit/klist/kdestroy, keytabs, clock requirements, and common errors.
FreeIPA HBAC & Sudo
HBAC rules, testing access with hbactest, and centrally managed sudo rules.
Keycloak
Realms, clients, roles, identity federation, admin CLI, topology, and production ops.
OpenLDAP & 389DS
Schema, replication, indexes, ACLs, TLS, migration, and troubleshooting.
Config Literacy
Deep, line-by-line config walkthroughs for Nginx and Postfix as exemplars. Other services use intro pages plus service deep dives in the sidebar; the same “read the config like a pro” pattern shows up there where it matters.
Nginx Config
Context hierarchy, server blocks, location matching, proxy directives, TLS, and logging.
Postfix Config
main.cf directives, relay, TLS, SASL auth, maps, queue management, and annotated example.
File Sharing
This section is SMB / Samba–first (Windows-friendly shares from Linux). NFS and other protocols are not covered as first-class pages here.
Linux Tools
Bash Scripting
Variables, conditionals, loops, functions, set -euo pipefail, error handling, and common production patterns.
cron & Timers
Schedule recurring tasks with cron or the modern systemd timer alternative. Syntax, logging, and Ansible.
rsync
Efficient file sync and backup — SSH transport, --delete, excludes, dry run, and deploy patterns.
User & Group Mgmt
useradd, usermod, groupadd, passwd, visudo, /etc/sudoers.d/, and service accounts.
lsof & strace
Find what's listening on a port, debug open file handles, and trace system calls to diagnose hard problems.
Wireshark & tshark
Capture vs display filters, reading pcaps, follow TCP streams, export objects, and workflows for TLS and HTTP debugging.
NMAP
Host discovery, port scans, service detection, NSE scripts, and reading open/filtered/closed results.
Identity & Auth Deep Dives
Windows 11 + FreeIPA
How a non-AD Windows 11 host talks directly to FreeIPA — CA trust, krb5.ini, DNS SRV, time sync, verification.
FreeIPA CLI on Windows
Running the ipa command from Windows 11: WSL2 + launcher script, SSH, or native Python + REST.
Keycloak on Windows + LDAPS
Keycloak via kc.bat on Windows 11, wiring LDAPS User Federation to FreeIPA with CA trust, group mapper, SSO.
FreeIPA Replication
Replica topology, ipa-replica-manage, promoting CA replicas, segment design.
FreeIPA Dogtag Certs
IPA CA vs sub-CAs, issuing service certs, ACME on IPA, renewing the IPA CA itself.
Keytab Rotation
ipa-getkeytab, KVNO bumps, ktutil splicing, service principal cleanup.
Guides
Infra Change Lifecycle
End-to-end: understand → branch → lint → dry-run → MR → CI → review → merge → verify.
Service Troubleshooting
Concrete diagnosis steps for nginx, postfix, SSH, NTP, auth, DNS, SELinux, and disk full.
Generic
Reusable infrastructure troubleshooting cheat sheet covering systemd, Ansible, TLS, identity, mail, logging, rollout, and rollback.
Change Window Runbook
Pre-change checklist, comms, rollback criteria, post-change verification table.
Incident: First 15 Minutes
Stabilise, communicate, gather, decide — a playbook for the start of an on-call page.