# Advanced Linux System Administration: Application Behavior Analysis

## Educational Module for Junior System Administrators

This comprehensive guide teaches systematic approaches to understanding how performance-critical applications interact with Linux at a low level—essential skills for debugging, optimization, and professional system administration.

---

## 1. Dynamic Library Dependencies and Loading

### Understanding the Dynamic Linker

**Core Concepts:**
- The dynamic linker (`ld-linux.so`) resolves and loads shared libraries at runtime
- Library search paths follow a specific hierarchy: `RPATH` → `LD_LIBRARY_PATH` → `RUNPATH` → `/etc/ld.so.cache` → system defaults
- Version conflicts occur when multiple library versions exist or ABI incompatibilities arise

### Practical Investigation Techniques

**Static Dependency Analysis:**
```bash
# List direct library dependencies
ldd /usr/bin/application

# Show library dependencies with full paths
ldd -v /usr/bin/application

# Alternative: use readelf for more detailed information
readelf -d /usr/bin/application | grep NEEDED

# Check RPATH/RUNPATH settings
readelf -d /usr/bin/application | grep -E 'RPATH|RUNPATH'
```

**Dynamic Loading Analysis:**
```bash
# Trace library loading in real-time with verbose output
LD_DEBUG=libs /usr/bin/application 2>&1 | less

# Detailed binding information
LD_DEBUG=bindings /usr/bin/application 2>&1 | less

# All debug information (very verbose)
LD_DEBUG=all /usr/bin/application 2>&1 | tee debug_output.log

# Alternative: use ltrace to see library calls
ltrace -C /usr/bin/application
```

**Version Conflict Detection:**
```bash
# Find all versions of a library on the system
locate libname.so | sort

# Check which version is being loaded
ldd /usr/bin/application | grep libname

# Verify library compatibility
objdump -p /usr/lib/libname.so.1 | grep SONAME
```

**Advanced: Using strace for dlopen() calls:**
```bash
# Track dynamically loaded libraries (plugins, modules)
strace -e openat,open -f /usr/bin/application 2>&1 | grep '\.so'
```

---

## 2. Process-Level Resource Consumption

### Comprehensive Resource Monitoring Strategy

**Real-Time Monitoring:**

```bash
# Interactive process monitoring with thread view
htop -p $(pgrep application)

# Top consumers sorted by CPU
top -b -n 1 -o %CPU | head -20

# Detailed process information
ps aux | grep application
ps -eLf | grep application  # Include threads
```

**CPU Analysis:**

```bash
# Per-thread CPU usage
top -H -p $(pgrep application)

# CPU affinity and scheduling
taskset -cp $(pgrep application)

# Detailed CPU time breakdown
/usr/bin/time -v /usr/bin/application

# CPU profiling with perf
perf stat -p $(pgrep application) sleep 10
perf record -p $(pgrep application) -g -- sleep 30
perf report
```

**Memory Analysis:**

```bash
# Detailed memory map
pmap -x $(pgrep application)

# Memory usage breakdown
smem -P application -k

# Real-time memory monitoring
watch -n 1 'ps -o pid,vsz,rss,comm -p $(pgrep application)'

# Check for memory leaks over time
valgrind --leak-check=full --track-origins=yes /usr/bin/application

# Alternative: massif for heap profiling
valgrind --tool=massif /usr/bin/application
ms_print massif.out.*
```

**I/O Analysis:**

```bash
# Real-time I/O statistics per process
iotop -p $(pgrep application)

# File descriptor information
lsof -p $(pgrep application)

# I/O statistics from /proc
cat /proc/$(pgrep application)/io

# Disk I/O patterns
iostat -x 1

# Trace file operations
strace -e trace=file -p $(pgrep application)
```

**Network Analysis:**

```bash
# Active network connections
netstat -tulpn | grep application
ss -tupn | grep application

# Network I/O per process
nethogs

# Bandwidth usage
iftop -P -p

# Packet-level analysis
tcpdump -i any -n port 8080 -w capture.pcap
```

**Comprehensive Monitoring:**

```bash
# systemd resource accounting (if service runs under systemd)
systemctl status application.service
systemd-cgtop

# cgroups resource limits and usage
cat /sys/fs/cgroup/system.slice/application.service/memory.current
cat /sys/fs/cgroup/system.slice/application.service/cpu.stat
```

---

## 3. Inter-Process Communication (IPC) Analysis

### Understanding IPC Mechanisms

**Common IPC Types:**
- Pipes (named and unnamed)
- Unix domain sockets
- Network sockets (TCP/UDP)
- Shared memory segments
- Message queues
- Signals
- D-Bus (higher-level IPC)

### Systematic IPC Discovery

**File Descriptor Analysis:**
```bash
# List all open file descriptors
ls -l /proc/$(pgrep application)/fd/

# Detailed FD information with types
lsof -p $(pgrep application)

# Filter for specific IPC types
lsof -p $(pgrep application) | grep -E 'PIPE|unix|TCP|UDP'
```

**Socket Analysis:**
```bash
# Unix domain sockets
ss -x -p | grep application

# Network sockets with process info
ss -tunapw | grep application

# Alternative with netstat
netstat -anp | grep application
```

**Shared Memory Investigation:**
```bash
# List shared memory segments
ipcs -m

# Show which processes are using shared memory
ipcs -m -p

# Detailed information about specific segment
ipcs -m -i <shmid>

# Check memory mappings
grep -E 'shared|shmem' /proc/$(pgrep application)/maps
```

**Message Queue Analysis:**
```bash
# List message queues
ipcs -q

# Process associations
ipcs -q -p

# POSIX message queues
ls -la /dev/mqueue/
```

**D-Bus Monitoring:**
```bash
# Monitor system bus
dbus-monitor --system

# Monitor session bus
dbus-monitor --session

# List services on D-Bus
busctl list

# Introspect specific service
busctl introspect org.freedesktop.ServiceName
```

**Real-Time IPC Tracing:**
```bash
# Trace IPC-related system calls
strace -e trace=ipc -p $(pgrep application)

# Trace network operations
strace -e trace=network -p $(pgrep application)

# Comprehensive IPC system calls
strace -e trace=read,write,sendto,recvfrom,sendmsg,recvmsg,connect,accept -p $(pgrep application)
```

---

## 4. Application-Specific Configuration and Runtime Environment

### Systematic Configuration Discovery

**Environment Variables:**
```bash
# View process environment
cat /proc/$(pgrep application)/environ | tr '\0' '\n'

# Alternative
strings /proc/$(pgrep application)/environ

# Launch with modified environment for testing
ENV_VAR=value /usr/bin/application

# Common important variables to check
env | grep -E 'PATH|LD_LIBRARY_PATH|HOME|XDG_|DISPLAY'
```

**Configuration File Locations:**

**Standard locations to investigate:**
```bash
# User-specific configuration
~/.config/application/
~/.application/
~/.applicationrc

# System-wide configuration
/etc/application/
/etc/application.conf
/usr/share/application/

# Check what files application actually opens
strace -e openat,open /usr/bin/application 2>&1 | grep -E '\.(conf|cfg|ini|json|yaml|toml|xml)'

# Monitor configuration file access
inotifywait -m -r ~/.config/application/ -e access,modify,open
```

**Runtime Directories:**
```bash
# Working directory
readlink /proc/$(pgreg application)/cwd

# Runtime data directories
ls -la /run/user/$(id -u)/application/
ls -la /var/run/application/

# XDG Base Directory specification
echo $XDG_CONFIG_HOME  # defaults to ~/.config
echo $XDG_DATA_HOME    # defaults to ~/.local/share
echo $XDG_CACHE_HOME   # defaults to ~/.cache
echo $XDG_RUNTIME_DIR  # defaults to /run/user/UID

# Temporary files
ls -la /tmp/ | grep application
lsof -p $(pgrep application) | grep /tmp
```

**Command-Line Arguments Analysis:**
```bash
# View exact command used to start process
cat /proc/$(pgreg application)/cmdline | tr '\0' ' '; echo

# Alternative
ps -fp $(pgrep application)

# Help/documentation for configuration options
/usr/bin/application --help
man application
```

**Package-Provided Files:**
```bash
# On Arch Linux, see what files a package provides
pacman -Ql package-name

# Find which package owns a file
pacman -Qo /usr/bin/application

# View package information
pacman -Qi package-name
```

---

## 5. Event Tracing and System Call Monitoring

### Low-Level Behavioral Analysis

**System Call Tracing with strace:**

```bash
# Basic system call trace
strace /usr/bin/application

# Attach to running process
strace -p $(pgrep application)

# Filter specific system calls
strace -e trace=open,read,write -p $(pgreg application)

# Time spent in system calls
strace -c /usr/bin/application

# Detailed timing information
strace -T -tt -p $(pgrep application)

# Follow forks and threads
strace -f -p $(pgrep application)

# Output to file for analysis
strace -o trace.log -p $(pgreg application)

# String size for arguments
strace -s 256 -p $(pgrep application)
```

**Common strace Patterns:**

```bash
# File I/O bottlenecks
strace -e trace=file -T /usr/bin/application 2>&1 | grep -E '<[0-9]+\.[0-9]+>'

# Network operations
strace -e trace=network -p $(pgrep application)

# Memory operations
strace -e trace=memory -p $(pgreg application)

# Signal handling
strace -e trace=signal -p $(pgrep application)
```

**Kernel Event Tracing with perf:**

```bash
# Record system-wide events
sudo perf record -a -g -- sleep 30

# Record specific process
sudo perf record -p $(pgrep application) -g -- sleep 30

# View recorded events
sudo perf report

# Real-time event monitoring
sudo perf top

# Trace specific events
sudo perf stat -e cycles,instructions,cache-misses -p $(pgrep application)

# Function-level tracing
sudo perf probe --add function_name
sudo perf record -e probe:function_name -p $(pgrep application)
```

**eBPF-Based Tracing (Modern Approach):**

```bash
# Install bpftrace (if not already installed)
sudo pacman -S bpftrace

# Trace file opens
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Monitor process execution
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s\n", str(args->filename)); }'

# Track memory allocations
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @bytes = hist(args->bytes_alloc); }'

# Custom scripts for specific applications
sudo bpftrace application_trace.bt
```

**ftrace (Kernel Function Tracer):**

```bash
# Enable ftrace
echo 1 | sudo tee /sys/kernel/debug/tracing/tracing_on

# Set up function tracer
echo function | sudo tee /sys/kernel/debug/tracing/current_tracer

# Filter for specific functions
echo '*networking*' | sudo tee /sys/kernel/debug/tracing/set_ftrace_filter

# Read trace
sudo cat /sys/kernel/debug/tracing/trace

# Disable when done
echo 0 | sudo tee /sys/kernel/debug/tracing/tracing_on
```

**Performance Counter Monitoring:**

```bash
# Hardware performance counters
perf list  # See available events

# Count cache misses
perf stat -e cache-misses,cache-references -p $(pgrep application)

# Branch prediction analysis
perf stat -e branches,branch-misses -p $(pgrep application)

# Memory access patterns
perf mem record -p $(pgreg application)
perf mem report
```

---

## Practical Workflow: Complete Application Analysis

### Step-by-Step Diagnostic Methodology

**Phase 1: Initial Assessment**
```bash
# 1. Identify the process
ps aux | grep application
pgrep -a application

# 2. Check basic resource usage
top -b -n 1 -p $(pgrep application)

# 3. View environment and configuration
cat /proc/$(pgrep application)/environ | tr '\0' '\n'
cat /proc/$(pgrep application)/cmdline | tr '\0' ' '; echo
```

**Phase 2: Dependency Analysis**
```bash
# 4. Check library dependencies
ldd /usr/bin/application
LD_DEBUG=libs /usr/bin/application 2>&1 | less

# 5. Verify library versions
pacman -Q | grep relevant-lib
```

**Phase 3: Resource Profiling**
```bash
# 6. CPU profiling
perf record -p $(pgrep application) -g -- sleep 30
perf report

# 7. Memory analysis
valgrind --tool=massif /usr/bin/application

# 8. I/O monitoring
iotop -p $(pgrep application)
```

**Phase 4: Communication Analysis**
```bash
# 9. Check IPC mechanisms
lsof -p $(pgreg application)
ss -xp | grep application

# 10. Monitor network activity
nethogs
tcpdump -i any -n -p $(pgrep application)
```

**Phase 5: Behavioral Tracing**
```bash
# 11. System call analysis
strace -c -p $(pgrep application)  # Summary
strace -T -tt -p $(pgrep application)  # Detailed

# 12. Event tracing
sudo perf trace -p $(pgrep application)
```

---

## Best Practices and Safety Considerations

### Debugging Without Disruption

1. **Use read-only tools first**: `ldd`, `pmap`, `lsof` don't affect application behavior
2. **Tracing has overhead**: Tools like `strace` can slow applications by 10-100x
3. **Test in non-production**: Always validate techniques in development environments first
4. **Document baseline behavior**: Capture normal operation before investigating issues
5. **Incremental analysis**: Start with lightweight tools, escalate to invasive ones only if needed

### Performance Impact Hierarchy

**Minimal impact:**
- `ps`, `top`, `htop`
- `ldd`, `readelf`
- `/proc` filesystem reads
- `lsof`, `ss`

**Moderate impact:**
- `perf stat` (sampling mode)
- `iotop`, `nethogs`
- `ltrace` (with filtering)

**High impact:**
- `strace` (especially without filtering)
- `valgrind`
- `perf record` (high-frequency sampling)

---

## Additional Resources for Continued Learning

### Documentation
```bash
# Essential man pages
man strace
man perf
man ldd
man proc  # /proc filesystem documentation
man ld.so  # Dynamic linker documentation

# Arch Wiki resources
# Visit: https://wiki.archlinux.org/title/Debugging
# Visit: https://wiki.archlinux.org/title/Performance
```

### Recommended Tools to Explore

- **SystemTap**: Advanced tracing and profiling
- **BCC**: eBPF-based performance tools collection
- **gdb**: Source-level debugging
- **rr**: Record and replay debugging
- **bpftrace**: High-level eBPF tracing language

---

This module provides a comprehensive foundation for understanding application behavior on Linux systems. The techniques described are universally applicable to any software and form essential skills for professional system administration, performance engineering, and DevOps roles.