Automating application support on a legacy Java estate

The situation

We joined a European manufacturer's application management team to support one of its financial reporting applications. The application was a Java web tool on Tomcat with mutual TLS, a SQL Server backend split across several schemas, a managed file-transfer server exchanging files with outside parties, and an NFS share for each stage. There were two stages, production and QA, and work arrived as tickets through the service desk.

Every morning someone checked both stages by hand: uptime, disk, load, memory, Tomcat errors, the application's job logs, the file-transfer monitor and certificate expiry. The data lived in about eight places, and questions about yesterday afternoon could not be answered from the live system at all.

These were managed servers, so installing a monitoring agent was not an option. Whatever we built had to use what was already on the boxes. The application itself was inherited: we operated it and automated the work around it.

What we did

A morning report from history the servers already kept

sysstat was already recording. sar -f reads any day's file and -s / -e cut a time window, so one loop turned existing history into hourly load and memory figures for the previous day, with nothing new installed. The same script collected error counts from the application's logs, file-transfer errors and certificate details from the keystores, and mailed the result.

One alert per outage

A cron check that mails on every failed run floods the inbox while something stays down, and people learn to filter it. Cron keeps no memory between runs, so each component got a small state file: the first failure sends mail and sets the latch, later failures stay quiet, and the first success clears it.

Checks proven by breaking what they watch

The scripts were first tested by flipping each condition, so the alert fired and the mail arrived. That proves the alert path. It says nothing about whether the check can detect a failure. When we tested each check against a real fault instead (a closed port, a stopped process, a dead PID), four of them failed the test:

• The Tomcat check read the exit status of the wrong stage in its pipe, a grep -v that succeeds whenever any process exists, so it could not report Tomcat down.
• The database check tested the printed output of a connection attempt, which is empty whether the port is open or not, so it always reported the database up.
• The load check compared numbers as text inside [[ ]], where "10.5" < 8 is true.
• The file-share latch read a misnamed variable, so it mailed on every run.

We caught these before any of them hid a real outage. The corrected pattern judges every check by an exit status, compares numbers as numbers, and sends all components through one latch that also announces recovery. A simplified version:

# Exit status decides health; one latch per component; recovery is announced.
report() {                           # $1 component, $2 exit status (0 = healthy)
  local latch="${state_dir}/$1.down"
  if (( $2 == 0 )); then
    [[ -f $latch ]] && { rm -f "$latch"; notify "$1" recovered; }
  elif [[ ! -f $latch ]]; then
    touch "$latch"; notify "$1" down
  fi
}

check_app()  { local pid; pid=$(cat "$pid_file" 2>/dev/null)
               [[ -n $pid ]] && kill -0 "$pid" 2>/dev/null; report app $?; }
check_db()   { timeout 5 bash -c "</dev/tcp/${db_host}/${db_port}" 2>/dev/null; report db $?; }
check_load() { local l; read -r l _ < /proc/loadavg
               awk -v l="$l" 'BEGIN { exit !(l < 8) }'; report load $?; }

Each check is then proven by breaking its target. Pointing the database check at an address where nothing listens, and the process check at a PID file holding a dead PID, must produce one "down" message and then one "recovered".

Migrations without a second copy of the password

A release could carry SQL for several schemas, each with its own Flyway configuration. Credentials stayed where the application already kept them: a wrapper takes migrate, info or validate plus a schema name, reads that schema's password from there at run time and exports it for Flyway, which resolves the password from the environment. No second copy was written.

We also encoded the release procedure as a script with two deliberate pauses: one for a manual database backup with typed confirmation, and one for the release's SQL scripts. It backs up before replacing anything and carries environment-specific configuration across.

Results

The morning check became reading an email. An outage produced one message and a recovery produced another. Every availability check on the estate was tested against a real fault, and the four that failed were fixed before any of them masked an outage. We kept no figures for time saved or alerts avoided, so this page reports none.

Related reading

• Silent zeros: four times a pipeline said success and was not
• Alerts that can never fire: judge by the evaluator, not the preview