Edge hardening for a small health API

The situation

The client runs a Laravel portal for logged-in users and a public API that partners call, including webhook deliveries, on a single small EC2 instance. The portal kept going down overnight. We put the API and portal behind Cloudflare on the free plan and measured before changing anything else.

In the first 24 hours we measured, 902 requests arrived and 814 of them went straight to the server. Only 28 were stopped at the edge and none were served from it. One burst of about 340 requests in fifteen minutes was enough to take the server down.

The free plan allows one rate-limit rule, and its bot protection applies to the whole zone with no way to skip it by path. Partners and webhooks are machines, which cannot solve a browser challenge, and the portal's pages are per user, so they cannot be cached.

What we did

Read the traffic before writing rules

"The site goes down overnight" invites a login-flood story. The sampled logs told a different one. The /login traffic was mostly uptime monitors running from a cloud provider's IPv6 ranges. The rest was a scanner probing for /.env and legitimate webhook deliveries. Blocking "the attackers" would have blocked the client's own monitoring and its partners, so the goal became keeping bursts off the server without touching machine traffic.

Wrote a rule set that leaves machines alone

We used three custom rules, evaluated in order, and one cache rule for fingerprinted static assets matched on URI path. A pattern like this, with placeholder paths:

# evaluated top to bottom; the first matching Skip wins
1  monitor      http.request.uri.path eq "/<health-path>"           Skip
2  api paths    starts_with(http.request.uri.path, "/<api-prefix>/")  Skip
3  everything   not starts_with(http.request.uri.path, "/<api-prefix>/")  Managed Challenge

# cache: static assets only, never per-user pages
starts_with(http.request.uri.path, "/<asset-dir>/")
  or http.request.uri.path.extension in {"js" "css" "png" "svg" "woff2"}

The decisions behind it: on the free plan, leave Bot Fight Mode off when machine clients must get through, because no rule can exempt a path from it. No "cache everything", which on an authenticated app without a cookie bypass can serve one user's page to another. Per-IP rate limiting is weak against IPv6, where Cloudflare counts clients by /64. And a CAPTCHA widget would not help, because the server that renders it is the one falling over.

Our first version allowed the uptime monitor by its User-Agent, which anyone can copy. We found that and replaced it, so a copied User-Agent no longer skips the challenge.

In the next 24 hours, 757 requests arrived: 225 were stopped at the edge, 40 were served from it, and 492 reached the server.

Sized the server instead of adding rules

The rules cut the load, but the instance was undersized, so we resized it. That was a capacity problem edge rules could not solve, so we stopped adding rules.

Traced certificate renewal to the edge

The server's certificate came from Let's Encrypt, renewed by a nightly job that proves domain ownership by serving a file under /.well-known/acme-challenge/. Rule 3 challenges every path outside the API prefix, and a certificate authority cannot run JavaScript, so in September 2026 the renewal stopped succeeding and the certificate expired. Cloudflare, set to Full (strict), refuses an expired origin certificate and serves error 526.

The server itself served the challenge file correctly. We pinned the block to the edge with one token, asked of the server and then of the edge:

# place a token on the server, then fetch it both ways
echo ok | sudo tee /var/www/acme/.well-known/acme-challenge/probe
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: app.example.com' \
  http://127.0.0.1/.well-known/acme-challenge/probe          # server: 200
curl -sI http://app.example.com/.well-known/acme-challenge/probe | grep -iE '^HTTP|cf-mitigated'
# edge: HTTP/1.1 403, cf-mitigated: challenge

Took renewal out of the picture

The quick fix was a skip rule for the ACME path, which would have kept a 90-day clock, a renewal job and a certbot container running, each able to fail again. Instead, nginx now serves a Cloudflare Origin CA certificate, valid until 2041, which Cloudflare trusts under Full (strict). No public certificate authority needs to reach the server. The trade-off is that the DNS record must stay proxied, and that was already the design: the server's security group accepts web traffic only from Cloudflare's ranges. We applied the change live that morning, with backups of every file we touched.

Made the fix permanent in the deploy

A live change only lasts until the next deploy, so we read the production deploy job end to end. It copied a folder that staging did not use over the live directory, which meant the origin certificate also had to land in the production path of the repository. The same review found a deploy runner on the production server running as root, which made write access to the repository equal to root on production.

We handed the repository side to the client's developer that morning. The same day, he gave production its own branch and workflow, applied the origin certificate there, and removed certbot and the renewal job from the repository and the host. The runner no longer runs as root.

Results