Crash forensics for a cloud VM in one Ansible run

The problem

A production VM fails its instance status check. Someone reboots it, it comes back, the alert clears. Two questions remain: why did it fail, and will it happen again tonight?

On AWS, an instance status check failure (as opposed to a system status check, which covers the provider's hardware) usually means something inside the guest: memory exhaustion, a full disk, a kernel panic, a filesystem remounted read-only, a broken network configuration, or a package upgrade that went wrong. A small Docker host adds two more: container logs filling the disk, and a burstable instance that ran out of CPU credits.

The evidence for each sits in a different place: dmesg, the journal, rotated syslog files, Docker's state, the instance metadata, CloudWatch. After a reboot some of it is already gone. Checking by hand over SSH at 2 a.m. means you check what you remember.

We hit this on a client's production server. We wrote one read-only playbook that walks the usual causes in order and ends with a summary. Its run pointed to CPU-credit throttling on a burstable instance, and the instance's resources were then set so the workload runs correctly. The excerpts below are illustrations adapted from that playbook, with the corrections we would make on a second pass.

The pattern

• One play, one host, read-only. Every task issues a read command and declares changed_when: false. Reading the logs needs root, so it runs with become; tell whoever approves the run.
• The crash date is a variable. Every time window derives from it.
• Sections in order of likelihood. Memory, disk, kernel, containers, the crash window, CPU, network, filesystem, swap, access, failed units and upgrades.
• Every check prints something, including "nothing found", so an empty section never passes for a clean one.
• It ends with a summary that maps each section to a cause.
• You write down what it found, next to the tool.

The play header (illustration):

- name: VM crash diagnosis
  hosts: prod_app_server
  gather_facts: true
  become: true
  vars:
    crash_date: "YYYY-MM-DD"        # set per incident
    separator: "{{ '=' * 80 }}"

The twelve sections

The sections below do the most work.

Memory: four places to look

An OOM kill shows up in the kernel ring buffer, but after a reboot dmesg holds only the current boot. Look in the persistent sources too.

OOM events from the journal and the rotated files (illustration):

    - name: OOM events in the journal for the crash date
      shell: |
        set -o pipefail
        journalctl -k --since '{{ crash_date }}' --until '{{ crash_date }} 23:59:59' --no-pager \
          | grep -iE 'oom|out of memory|killed process' \
          || echo 'No OOM events in the journal for the crash date'
      args: { executable: /bin/bash }
      register: oom_journal
      changed_when: false

    - name: OOM events in rotated syslog and kern.log
      shell: |
        set -o pipefail
        zgrep -ihE 'oom|out of memory|killed process' /var/log/syslog* /var/log/kern.log* 2>/dev/null \
          | tail -50 || echo 'No OOM events in syslog or kern.log'
      args: { executable: /bin/bash }
      register: oom_files
      changed_when: false

zgrep reads the .gz rotations that plain grep skips. And journalctl -k for a past date works only if the journal is persistent: check that /var/log/journal exists, or -b -1 (the previous boot) will come back empty.

Disk: container logs

A common cause on small Docker hosts is mundane. Docker's default json-file log driver has no size limit, so one chatty container can fill the root disk over a few months.

Log sizes and whether rotation was ever configured (illustration):

    - name: Container log sizes
      shell: "du -sh /var/lib/docker/containers/*/ 2>/dev/null | sort -rh | head -20"
      register: docker_log_sizes
      changed_when: false

    - name: Docker log rotation config
      shell: "cat /etc/docker/daemon.json 2>/dev/null || echo 'No daemon.json: no log rotation configured'"
      register: docker_daemon_json
      changed_when: false

Check inodes as well as bytes (df -i). A host can run out of inodes with gigabytes free, and the symptoms look the same. If there is no daemon.json, the fix is "log-driver": "json-file" with "log-opts": {"max-size": "10m", "max-file": "3"}, and it applies only to containers created after the daemon restart.

Containers: restarts, limits and events

Restart counts and memory limits for every container, stopped ones included (illustration):

    - name: Container restart counts
      shell: >-
        docker inspect --format
        '{{ '{{' }}.Name{{ '}}' }} restarts={{ '{{' }}.RestartCount{{ '}}' }} mem={{ '{{' }}.HostConfig.Memory{{ '}}' }}'
        $(docker ps -aq) 2>/dev/null || echo 'No containers'
      register: container_state
      changed_when: false

The {{ '{{' }} escaping passes a Go template through Jinja. mem=0 means no limit: one container can take the host's memory, and the OOM killer then picks a victim that may not be the one at fault.

docker events for the crash date is usually empty after a reboot, because the daemon keeps only a small in-memory buffer of recent events. The journal's docker.service and containerd.service entries are the better record.

CPU: a burstable instance out of credits

This is the section that answered our case. A T-series instance that spends its CPU credits is throttled to its baseline, and a throttled host running PHP and a database can stop answering health checks.

The playbook asks the instance metadata service for the instance type. Use the IMDSv2 form, which fetches a session token first; the token-less call fails on instances that enforce IMDSv2.

The burstable check (illustration, IMDSv2):

    - name: Instance type from metadata (IMDSv2)
      shell: |
        TOKEN=$(curl -s -X PUT --connect-timeout 2 http://169.254.169.254/latest/api/token \
          -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
        curl -s --connect-timeout 2 -H "X-aws-ec2-metadata-token: $TOKEN" \
          http://169.254.169.254/latest/meta-data/instance-type || echo 'unknown'
      register: instance_type
      changed_when: false

    - name: Flag burstable types
      debug:
        msg: "{{ instance_type.stdout }}{{ ' (BURSTABLE: check CPU credits)' if instance_type.stdout is match('t[234]') else '' }}"

Inside the VM you see the type, not the credit balance. If it is burstable, open the CPUCreditBalance metric in CloudWatch for the crash window. A balance at or near zero before the failure, with CPU pinned at the baseline, points to throttling. The fix is a sizing decision: a larger or non-burstable type, unlimited credit mode, or less work on the host.

The closing summary

The playbook does not guess at a verdict. It ends by telling the reader where to look.

The closing task (illustration, condensed):

    - name: "=== DIAGNOSIS SUMMARY ==="
      debug:
        msg: |
          Instance: {{ ansible_hostname }} ({{ instance_type.stdout }})
          CPUs: {{ ansible_processor_vcpus }} | RAM: {{ ansible_memtotal_mb }} MB | Swap: {{ ansible_swaptotal_mb }} MB
          Crash date: {{ crash_date }}
          Review: 1 OOM (2), 2 disk and Docker logs (3), 3 kernel panic (4),
                  4 container restarts (5), 5 CPU credits (7),
                  6 filesystem errors (9), 7 unattended upgrades (12)

The traps

cmd | tail || echo "none found" never prints "none found". A pipeline's exit status is its last command's. tail succeeds on empty input, so the fallback never runs and the section prints nothing. Our first version had this pattern in several places. Use set -o pipefail (with executable: /bin/bash), or put the grep last.

Hard-coded dates. Our first version grepped syslog for one literal date, with the double space syslog uses for single-digit days. That works for one incident and matches nothing for the next. Drive every window from crash_date, and prefer journalctl --since/--until, which does not depend on how a release formats syslog timestamps.

Non-persistent journals. If /var/log/journal does not exist, the journal lives in memory and the previous boot is gone. As standard practice, make the journal persistent on every server, so the previous boot is there when an incident needs it.

Output you cannot read later. Twelve sections of debug output scroll past fast. Run it with ansible-playbook ... | tee crash-$(date +%F).log, or render the registered results to a local file in a second play.

Read-only has to stay true. changed_when: false only tells Ansible not to report a change. The guarantee is that every task is a read command, which a reviewer can check. Put any fix in a separate playbook.

We used this in Infrastructure for a medical-device company's clinical platform.