Keeping an ITSM integration alive when its token lives 30 minutes

Many alerting tools deliver to an ITSM system through a webhook with a fixed Authorization header. Many ITSM teams, reasonably, require OAuth. The two usually meet as a bearer token pasted into the webhook, which works until the token expires. After that, alerts stop becoming incidents and no error appears anywhere. The alerting tool shows the destination as configured, and the ITSM tool simply sees no traffic.

This article covers a relay that keeps such a token fresh, the measurements that set its schedule, and the traps that cost us time. The examples use New Relic and ServiceNow. The pattern fits any pair where one side stores a static header and the other issues short-lived tokens.

Confirm the destination cannot refresh

Before designing anything, ask the alerting tool's API. Asked for an OAuth flow on webhook or ServiceNow destination types, New Relic answers OAUTH_NOT_SUPPORTED. That one call turns "we think it cannot refresh" into a fact you can show the ITSM team.

Measure the token before you schedule anything

The integration design said the token lived 100 days, which would make a weekly refresh comfortable. We measured it:

• A password grant returned expires_in: 1799, the stock ServiceNow lifespan of 30 minutes.
• Refreshed every 240 seconds for 32 minutes, the token was still accepted at 2,102 seconds.
• Left idle, the same token answered 200 at 1,500 seconds and 401 at 1,800.

The 100 days was the refresh token's lifespan. The access token, which the webhook actually sends, lived half an hour. A weekly schedule would have left alert routing dead for about 6 days 23.5 hours of every 7.

The same hour of testing produced two more facts that shaped the design:

1. The refresh grant extends the same token string. It does not mint a new one. Three consecutive password grants also returned the identical token. The string changes only after a lapse.
2. The service account could not read its own OAuth configuration (403). The token response is the only authority on the lifespan, so the relay reads expires_in on every run.

The relay

A scheduled function, plus an HTTP trigger for runs on demand. Each run:

1. Reads the stored refresh token.
2. Tries the refresh grant, and falls back to the password grant when there is no stored token or the refresh is refused.
3. Stores the new refresh token before calling the alerting tool.
4. Writes the access token into the destination only when the string changed, or once a day.
5. Emits an audit event on success and on failure.

Keep the schedule in configuration, and keep the interval it is checked against as a separate setting. Parsing the interval out of a cron expression adds a parser that can break the one check that catches a lifespan mismatch:

import os

# Both values follow the ITSM instance's token lifespan, which can change.
SCHEDULE = os.environ.get("REFRESH_CRON", "0 */20 * * * *")
INTERVAL_S = int(os.environ.get("REFRESH_INTERVAL_SECONDS", "1200"))

# The refresh grant extends the same string, so write on change or on this floor.
WRITE_FLOOR_S = int(os.environ.get("WRITE_FLOOR_SECONDS", str(24 * 3600)))

With a 30-minute token we run every 20 minutes. If the ITSM team raises the lifespan later, those two settings are the only change.

Refresh first, password second

A human runbook usually starts with the password grant and refreshes when a call fails. A scheduled job never sees that failure, because by the time a notification is refused the alert is already lost. So the order inverts:

def get_token(cfg, saved_refresh):
    """Refresh grant when we hold a refresh token; password grant otherwise."""
    if saved_refresh:
        try:
            return request_token(cfg, grant_type="refresh_token",
                                 refresh_token=saved_refresh) | {"grant": "refresh"}
        except TokenRefused as err:
            # Normal after the refresh token's own lifespan, or after a rotation.
            log.warning("refresh grant refused (%s), trying password grant", err)
    return request_token(cfg, grant_type="password",
                         username=cfg.user, password=cfg.password) | {"grant": "password"}

Store before the risky call, and report on the failure path

Rotating refresh tokens punish one mistake: exchange a token, fail to store its replacement, and the chain breaks. The next run drops to the password grant, which is the path that fails first when the service account is locked or moved to single sign-on. So the relay stores the new refresh token first, with retries, and fails the run if it cannot.

The failure path needs the same care. Our first version built its audit event after minting. When the ITSM side began refusing both grants, the function raised before reporting anything, and the outage of about 17 hours showed only as missing events. The fixed run records the failure, then raises:

def run_once(cfg, store, now):
    state = store.load() or {}
    try:
        token = get_token(cfg, state.get("refresh_token"))
    except TokenRefused as err:
        # Record the failure first, or the outage looks like an idle relay.
        emit(cfg, {"eventType": "TokenRelay", "ok": False, "stage": "mint",
                   "heldRefreshToken": bool(state.get("refresh_token")),
                   "error": str(err)[:300]})          # error body only, never credentials
        raise
    state["refresh_token"] = token.get("refresh_token", state.get("refresh_token"))
    store.save(state)                                 # before touching the destination
    lifespan_ok = int(token.get("expires_in", 0)) > INTERVAL_S
    if not lifespan_ok:
        log.error("token lives %ss, relay runs every %ss", token.get("expires_in"), INTERVAL_S)
    # ... decide whether to write, update the destination, emit the audit event

Write only when it matters

Because the token string rarely changes, writing it every 20 minutes would mean about 26,000 destination updates a year for no benefit. The relay keeps a hash prefix of the last token it wrote (never the token) and records why it writes:

def why_write(state, token_hash, now):
    """Return a reason to write the destination, or None to skip this run."""
    if state.get("written_hash") != token_hash:
        return "token-changed"
    last = state.get("written_at")
    if last is None:
        return "never-written"
    age = (now - datetime.fromisoformat(last)).total_seconds()
    return "daily-floor" if age >= WRITE_FLOOR_S else None

The write replaces only the destination's auth block and leaves its URL and sync settings alone. Judge the result by an allowlist of healthy statuses (DEFAULT, CONFIGURED, TESTING in New Relic) rather than by the absence of an error. The mutation's error field is a GraphQL union, so a plain error { description } selection fails the whole request. Use inline fragments:

mutation SetToken($account: Int!, $id: ID!, $prefix: String!, $token: SecureValue!) {
  aiNotificationsUpdateDestination(
    accountId: $account
    destinationId: $id
    destination: { auth: { type: TOKEN, token: { prefix: $prefix, token: $token } } }
  ) {
    destination { id status }
    error {
      __typename
      ... on AiNotificationsResponseError { description type }
      ... on AiNotificationsDataValidationError { fields { field message } }
    }
  }
}

Alert on the relay itself

The relay is now part of the alerting path, so it needs alerts of its own. Give the audit event one attribute per failure mode and one condition per attribute:

-- 1. The relay stopped reporting (use loss of signal on this one)
FROM TokenRelay SELECT count(*) WHERE ok IS TRUE
-- 2. A run failed at mint or at the destination write
FROM TokenRelay SELECT count(*) WHERE ok IS FALSE
-- 3. The schedule no longer fits inside the token lifespan
FROM TokenRelay SELECT count(*) WHERE lifespanOk IS FALSE
-- 4. A refresh token we held was refused, so the chain broke
FROM TokenRelay SELECT count(*) WHERE heldRefreshToken IS TRUE AND grant = 'password'

The first query needs loss of signal rather than a threshold, because a dead relay produces no events at all.

Do not use the destination's "last updated" time as a health signal. With write-on-change, a timestamp 20 hours old is the designed steady state. We misread it once as a stopped relay; the audit event showed otherwise.

The traps

The wrong URL looks like a wrong password. ServiceNow's token endpoint is /oauth_token.do. The generic /oauth/token returns an HTML 404, which a client reads as an invalid response and a person reads as a credential problem. Put the path in a comment next to the code that builds it.

One 401 does not prove the auth method is wrong. Our first live delivery was refused with a header asking for an API key, and we first recommended switching to one. Then we sent the same token to three endpoints: 200 from two table endpoints, 401 from the integration's scripted endpoint only. The cause was one API access policy on one resource, fixed by a one-line change on the ITSM side. Test one credential against several endpoints before changing the design.

Every bad credential returns the same 401. A token endpoint usually returns one access_denied body whether the password is wrong, the client secret is wrong, or a stored secret has a trailing newline. To tell these apart without logging secrets, fingerprint each credential the function received and compare it with the same fingerprint computed where it was issued:

import hashlib

def fingerprint(value: str) -> str:
    """Length and a short hash, plus flags for common storage mistakes."""
    flags = [name for name, bad in (
        ("whitespace", value != value.strip()),
        ("unresolved-vault-ref", value.startswith("@Microsoft.KeyVault")),
        ("non-ascii", not value.isascii()),
    ) if bad]
    digest = hashlib.sha256(value.encode()).hexdigest()[:12]
    return f"{len(value)}:{digest}" + (":" + ",".join(flags) if flags else "")

The vault-reference flag catches a common Azure case: an app setting that still holds the literal Key Vault reference because the platform has not resolved it, or an old cached value after a rotation.

Change detection against your own state misses other writers. If the relay compares the new token with what it last wrote, a person or test script that edits the destination leaves it stale until the daily floor. Calling the on-demand trigger mints the same string and skips again. Read the destination back, or give the HTTP trigger a force parameter.

A stopped relay falls out of the refresh chain. Off for longer than the refresh token's lifespan, the next run needs the password grant. Keep the fallback for that reason.

Repeated failed logins can lock the account. While the ITSM side refuses grants, a 20-minute schedule makes three failed password attempts an hour against a service account. If the outage will last, stop the function until the credential is fixed.

What it gave us

Over 17.8 days in September 2026 the relay ran 1,284 times, every 20 minutes without a gap, and every run was healthy. It wrote the destination 18 times (one token change, 17 daily writes) and skipped the rest, with no failed write. After the ITSM-side refusal described above, the relay now records ok: false on a refused grant, and the four alerts above watch it.

We used this in Self-service observability for a consumer-health company.