lsof & strace

Two essential production debugging tools — lsof shows open files and network connections; strace shows every system call a process makes.

What lsof is

lsof stands for "list open files." In Linux, almost everything is a file — regular files, sockets, pipes, devices. lsof shows you which process has which file open, which ports are in use, and who is connected where.

# Install if not present
dnf install lsof    # RHEL
apt install lsof    # Debian

lsof: network connections

# Show all listening TCP/UDP ports (most common use)
lsof -i -P -n          # -P = show port numbers, -n = no hostname resolution
lsof -i -P -n | grep LISTEN

# Find what is listening on a specific port
lsof -i :80
lsof -i :443
lsof -i :8080

# Show all connections to/from a remote IP
lsof -i @192.168.1.50

# Show all TCP connections
lsof -i tcp

# Show all UDP connections
lsof -i udp

# Show connections for a specific service
lsof -i :5432    # PostgreSQL
lsof -i :3306    # MySQL
lsof -i :6379    # Redis

lsof: open files

# Who has a specific file open?
lsof /var/log/nginx/access.log

# Who has files open under a directory?
lsof +D /var/log/nginx/

# Find deleted files still held open (causing disk space not to be freed)
lsof | grep deleted
lsof | grep "(deleted)"

# Common situation: log file was rotated/deleted but service still writes to it
# The disk space won't be freed until the process closes the file descriptor
# Fix: reload/restart the service so it opens the new log file

lsof: by process

# Show all files opened by a specific PID
lsof -p 1234

# Show all files opened by a user
lsof -u nginx
lsof -u alice

# Show all files opened by a command name
lsof -c nginx
lsof -c python3

# Combine: user AND command
lsof -u alice -c vim

Reading lsof output

# Example output:
# COMMAND   PID   USER   FD   TYPE  DEVICE SIZE/OFF  NODE NAME
# nginx    1234   root   cwd   DIR   8,1    4096     1234 /
# nginx    1234   root   txt   REG   8,1   1234567   2345 /usr/sbin/nginx
# nginx    1234   root   6u  IPv4  12345    0t0   TCP *:80 (LISTEN)
# nginx    1234   root   7u  IPv4  12346    0t0   TCP web01:80->client:54321 (ESTABLISHED)

# Column meanings:
# COMMAND  — process name
# PID      — process ID
# USER     — user running the process
# FD       — file descriptor type:
#            cwd = current working directory
#            txt = program text (the binary itself)
#            mem = memory-mapped file
#            [0-9]+r = read, [0-9]+w = write, [0-9]+u = read+write
# TYPE     — file type: REG=regular file, DIR=directory, IPv4/IPv6=socket
# NAME     — file path, or for sockets: address:port->remote:port (STATE)

What strace is

strace intercepts and records all system calls a process makes — file opens, network calls, memory allocations, signal handling. It is the last line of debugging when logs reveal nothing. You see exactly what the process asks the kernel to do and what the kernel returns.

# Install if not present
dnf install strace    # RHEL
apt install strace    # Debian

Attaching to a running process

# Attach to an already-running process by PID
strace -p 1234

# Attach and write output to a file (useful — strace output floods the terminal)
strace -p 1234 -o /tmp/strace.out

# Follow child processes too (important for multi-process services like nginx)
strace -f -p 1234

# Run a command under strace from the start
strace /usr/bin/nginx -t

# Attach to all processes of a service name
strace -p $(pgrep -d, nginx)   # comma-separate PIDs for -p

Filtering system calls

# Trace only specific syscall categories
strace -e trace=network -p 1234    # only network calls
strace -e trace=file    -p 1234    # only file-related calls
strace -e trace=process -p 1234    # only process calls (fork, exec)

# Trace a single syscall
strace -e trace=open,openat,read,write -p 1234

# Useful category shortcuts:
# network — connect, bind, send, recv, socket, etc.
# file    — open, openat, read, write, close, stat, etc.
# signal  — kill, sigaction, etc.
# ipc     — shmget, msgget, etc.

# Show timestamps (absolute time)
strace -t -p 1234

# Show relative time between calls (performance profiling)
strace -r -p 1234

# Show time spent in each syscall (summary at end)
strace -c /usr/bin/myapp arg1 arg2

Reading strace output

# Each line shows: syscall(args) = return_value [error if any]

# File open example
openat(AT_FDCWD, "/etc/nginx/nginx.conf", O_RDONLY) = 5
# openat opened /etc/nginx/nginx.conf for reading, returned fd=5

# File read
read(5, "user nginx;\nworker_processes aut"..., 4096) = 512
# read from fd=5, buffer size 4096, read 512 bytes

# Failed open (file not found)
openat(AT_FDCWD, "/etc/nginx/missing.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
# return value -1 with errno ENOENT — file doesn't exist

# Network connect
connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.5")}, 16) = 0
# connected socket fd=4 to 10.0.0.5:5432 (PostgreSQL) — success

# Failed connect (connection refused)
connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.5")}, 16) = -1 ECONNREFUSED
# nothing listening on that port

Common strace patterns

# Which config file is a service trying to read?
strace -e trace=openat /usr/sbin/myapp 2>&1 | grep "openat"

# Why won't a service start? (check for permission errors, missing files)
strace -e trace=file /usr/sbin/myapp 2>&1 | grep "ENOENT\|EACCES\|EPERM"

# What network connections is a process making?
strace -e trace=network -p 1234 2>&1 | grep "connect\|bind"

# What files is a process writing to?
strace -e trace=write,openat -p 1234 2>&1

# Is a process stuck in a syscall?
strace -p 1234
# If it immediately shows one call and hangs there, the process is blocked
# e.g. poll([...], 1, -1) <unfinished ...> — waiting for I/O indefinitely

Practical workflows

Port conflict — "Address already in use"

# Service fails to bind to a port — find what's using it
lsof -i :8080
ss -tlnp | grep :8080

# If it's a ghost process:
kill -9 $(lsof -ti :8080)

Disk space not freed after log deletion

# Large file was deleted but disk is still full
df -h              # disk shows full
du -sh /var/log/*  # but files look small

# Find the process still holding the deleted file open
lsof | grep deleted

# The FD column shows the file descriptor
# Restart the service to close the old file descriptor
systemctl reload nginx   # or restart if reload isn't enough

Process can't read a config file — is it permissions or path?

# Run the service under strace, filter for file errors
strace -e trace=openat,read /usr/sbin/myapp 2>&1 | grep -E "ENOENT|EACCES|EPERM"

# Common output patterns:
# ENOENT — file doesn't exist (check the path)
# EACCES — permission denied (check file permissions and ownership)
# EPERM  — operation not permitted (check capabilities or SELinux)

# For a running service, attach and watch:
strace -f -e trace=file -p $(pgrep myapp) 2>&1 | grep -E "ENOENT|EACCES"

Database connection failures

# App says "connection refused" — verify what it's actually trying to connect to
strace -e trace=network -p $(pgrep myapp) 2>&1 | grep connect

# Look for the IP and port in the connect() call
# If IP or port is wrong, it's an app config issue (wrong DB host in config)
# If IP/port is right, confirm something is listening:
lsof -i :5432
ss -tlnp | grep 5432

Advanced strace flags (-ff, -s, -y)

Three flags that significantly improve strace output quality for real-world multi-process services.

-ff -o /path — separate file per child process

When tracing a service that forks workers (nginx, Apache, PostgreSQL, gunicorn), using -f alone produces interleaved output from all processes in one stream — almost unreadable. Use -ff -o to write each process's trace to a separate file named file.PID.

# Trace nginx master + all worker processes, separate file per PID
strace -ff -o /tmp/strace.nginx -p $(pgrep -d, nginx)

# Output files created:
# /tmp/strace.nginx.1234   ← master process
# /tmp/strace.nginx.1235   ← worker 1
# /tmp/strace.nginx.1236   ← worker 2

# Inspect a specific worker
less /tmp/strace.nginx.1235

# Grep across all worker traces
grep "ENOENT" /tmp/strace.nginx.*

-ff requires -o — it won't work with stdout. This is the correct way to trace any multi-process service. Always use it instead of bare -f for services like nginx, Apache, PostgreSQL, or Celery workers.

-s 512 — longer string output

By default, strace truncates strings at 32 characters. File paths, SQL queries, and HTTP headers get cut off. Increase it with -s.

# Default: paths get truncated
openat(AT_FDCWD, "/etc/nginx/conf.d/defa"..., O_RDONLY) = 5

# With -s 512: full path
strace -s 512 -p 1234
openat(AT_FDCWD, "/etc/nginx/conf.d/default.conf", O_RDONLY) = 5

# Useful values:
# -s 128  — for most paths and short strings
# -s 512  — for long config paths and URLs
# -s 4096 — for SQL queries, HTTP request bodies

-y — annotate file descriptor numbers with paths

By default, strace shows raw file descriptor numbers (e.g. read(5, ...). The -y flag annotates each FD with its path, so you immediately know which file is being read.

# Without -y: FD numbers are opaque
read(5, "server {\n    listen 80;\n", 4096) = 24

# With -y: FD annotated with file path
strace -y -p 1234
read(5, "server {\n    listen 80;\n", 4096) = 24
write(7, "127.0.0.1 - ...", 98) = 98

# Combine all three for maximum clarity
strace -ff -s 512 -y -o /tmp/strace.nginx -p $(pgrep -d, nginx)

-y is available in strace 4.7+ (2011). Should be available on any modern RHEL 7+ or Debian 9+ system. If the flag is not recognized, upgrade strace with dnf update strace.