Live $CHIPS market data from DexScreener. Opens the chart on defined.fi.
← Back to Newsroom

Incident Report

INC-0047 — Elevated Sogginess, EU-WEST

Approximately 12% of units produced in EU-WEST exceeded the 0.3 ms crunch latency SLO following a humidity controller replacement.

SeveritySev 2StatusResolvedDuration4h 12m

Summary

Between 03:12 and 07:24 UTC on 22 July 2026, approximately 12% of units produced in EU-WEST exceeded the 0.3 ms crunch latency objective. No units left the facility in a non-conforming state.

Timeline

All times UTC.

  • 03:12 — Humidity controller HC-31 replaced in Foundry 31 during scheduled maintenance. Replacement passed post-install self-test.
  • 03:47 — Crunch verification begins flagging units in the 0.31–0.44 ms range. Flag rate remains below the paging threshold.
  • 04:05 — Flag rate crosses 8%. On-call Salt Architect paged.
  • 04:38 — Thermal grid and salt deposition ruled out. Both operating within tolerance.
  • 05:30 — Ambient humidity in Foundry 31 measured at 94.7% RH against a 12% ceiling. Root cause identified.
  • 05:44 — HC-31 reverted to Cleanroom profile. Humidity begins falling.
  • 07:24 — Ambient humidity returns below 12% RH. Flag rate returns to baseline. Incident resolved.

Root cause

The replacement HC-31 unit shipped with firmware 4.2, whose default operating profile is Bakery.

Bakery targets 95% RH in order to prevent products from drying out. Our process requires the exact opposite. The profile was never changed after installation because the post-install self-test validates that the unit can reach its target humidity, not that the target is correct.

Resolution

The unit was reverted to the Cleanroom profile. Affected inventory was quarantined, reclassified as "Chip Style", and released through the discount channel.

What went well

  • Crunch verification detected the deviation within 35 minutes of onset.
  • No non-conforming units reached customers.

What went wrong

  • No alerting existed on ambient humidity. We were monitoring the output and not the environment.
  • Bakery was reachable in production firmware. It should never have been.

Action items

  • Alert when ambient humidity exceeds 12% RH in any foundry
  • Remove Bakery from the shipped firmware mode enum
  • Audit remaining HC units across Foundries 1–47 for profile drift
  • Extend post-install self-test to validate target setpoints, not just convergence
  • Review with the Salt Architecture team