← All projects

Production incident · ASP.NET Core · July 2026

Eliminating a cross-request HttpClient header race in production

Under real concurrency a live multi-tenant ASP.NET Core platform started failing in ways nobody could reproduce alone: sidebar errors, HTTP 500s, redirects to login. I traced the failures through Azure telemetry and the HTTP stack to shared mutable headers on a singleton HttpClient, replaced them with request-scoped messages, covered the race with tests and shipped the fix without changing any interface.

My role
Diagnosis, fix, regression tests and release documentation, end to end.
Stack
  • ASP.NET Core
  • C#
  • HttpClient
  • Polly
  • Azure Application Insights
  • NUnit

At a glance

  • Root cause found from Azure telemetry and stack traces, not from a local reproduction
  • 12 targeted regression tests, including 2 users making 40 overlapping calls through one client
  • Failures stopped recurring after deployment (operational observation)

Context

The web host of a multi-tenant ASP.NET Core platform calls a separate API on behalf of the logged-in user, forwarding that user's JWT together with a tenant/environment header. The service classes shared one singleton HttpClient and mutated its default headers before every call.

Under real concurrency — several users logged in, overlapping calls — production began showing sidebar failures, HTTP 500 responses and unexpected redirects to the login page. Application Insights recorded failed dependencies with result code 0 and a NullReferenceException inside the HTTP connection's header-writing routine.

What I did

  • Correlated Azure telemetry, stack traces, historical code and the dependency-injection setup until the shared mutable state stood out as the common factor.
  • Identified concurrent mutation of the client's default headers as the cause, and named the security risk it implied: an overlapping call could in theory carry another request's bearer token or tenant values.
  • Replaced shared-header mutation with a fresh HttpRequestMessage for every attempt, sent through SendAsync, with the authorization, tenant, accept and API-key headers attached to that individual message.
  • Re-created the request body for every Polly retry, corrected the environment value propagation that was consumed once and then lost, and stopped a failed page-catalogue lookup from being written to cache — successful results now carry an explicit five-minute TTL.
  • Wrote the targeted regression suite and documented the change for the release.

Decisions

Request-scoped messages, not per-request clients

A client per call would have removed the race but invited socket exhaustion and a rewrite of the registration. Keeping the singleton and moving every per-user value onto the HttpRequestMessage left the public interfaces and the DI setup untouched.

No interface changes

Frontend, BFF and API contracts stayed identical, so the fix shipped in a normal release with a trivial rollback.

Fail closed

The API keeps returning 401 when authentication is missing or invalid; the fix adds no fallback that could mask a missing token, and tokens stay out of error logs.

Never cache a failure

A temporary failure of the page catalogue used to be stored like a valid result and poison the sidebar for later requests. Only successful results are cached now, with a bounded TTL.

The hard part

The failure was nondeterministic. It needed real overlap between users and never appeared in a single-user session, so the evidence had to be assembled from telemetry patterns and the internals of the HTTP stack rather than from a debugger.

The tests had to force the race on purpose: two simulated users issuing 40 concurrent calls through the same singleton client, asserting that every call carried its own bearer and tenant values, plus sibling-task and cache-behaviour cases.

Result

  • 12 of 12 targeted regression tests passed: the 2-user × 40-call scenario, bearer and environment isolation, sibling tasks and cache behaviour.
  • Release build with zero errors; the full suite reported 443 passing tests and 10 pre-existing baseline failures out of 453, unchanged by this work.
  • No public frontend, BFF or API contract changed.
  • After deployment the sidebar and login failures stopped recurring and the matching Application Insights errors were no longer observed.

Numbers, with their limits

  • Targeted regression tests12 / 12 passed
  • Concurrency scenario2 users × 40 overlapping calls
  • Full suite (Release build)443 / 453 passing10 pre-existing baseline failures, unchanged
  • Public contracts changed0

What these numbers do not say

  • Post-deployment success is an operational observation: the user-facing failures stopped and the telemetry errors disappeared. No exported monitoring metric was retained.
  • The cross-user token exposure was a credible risk created by shared headers, not a documented incident: no actual credential crossover was observed.
  • Test counts are the historical results of July 2026, not a re-run made for this page.

What it proves

I can trace a production failure across telemetry, framework internals and legacy code, fix it without breaking a contract, and prove the fix with tests that force the race.

Attribution

Diagnosis, fix and tests were my work. Deployment followed the team's normal release process.