New York Stock Exchange
When a failed health check was marked successful, what remained untested?
An SEC order found that NYSE staff manually marked a failed disaster-recovery health check as successfully resolved without full investigation, escalation or approval. The next morning, 2,824 securities opened without their required auctions and more than 4,000 trades were later busted.
Documented impact
Authoritative findings
The documented event
During planned maintenance on 23 January 2023, NYSE staff mistakenly ran shutdown commands against an already-shut production system, leaving the Pillar disaster-recovery system active. A health check against Pillar DR failed because maintenance was incomplete and the DR system was still operating. The SEC order finds that staff investigated only the first reason and manually marked the failed check as successfully resolved without full investigation, escalation or supervisory approval. Data from the still-active DR system later returned through market-data infrastructure and caused production to treat opening auctions as completed for 2,824 securities. NYSE issued a 'STATUS: NORMAL' update at 9:48am and did not identify missing auctions until about 10:09am. The event led to 84 trading pauses and more than 4,000 busted trades. The SEC imposed a $9 million penalty.
Hypothetical institutional scenario
How might the same control pattern appear?
A disaster-recovery environment is activated for maintenance. Its automated health check fails for more than one reason, but staff explain the familiar exception, manually mark the control successful and proceed. The backup environment remains active and sends state data that production trusts. Dashboards show components as available, while no control verifies whether the critical business outcome - a complete opening, settlement, calculation or customer process - actually occurred.
Stress-test questions
Questions for challenge and assurance
-
Technology
Can a failed automated control be manually marked successful, and what evidence and second-line approval are required before that override?
-
Operations
What control confirms that recovery environments are inactive and cannot send authoritative state into production after tests or maintenance?
-
Audit
Do monitoring procedures verify completion of the critical business process, or only component availability and job status?
-
Risk committee
How quickly would abnormal outcomes overturn a 'normal' systems assessment, and who can declare the original assessment unreliable?
NFRisk practitioner interpretation
Control implication
A control result should not be converted from failed to passed by narrative alone. Overrides need complete causal investigation, independent approval and evidence that every failed condition has cleared. Resilience monitoring must also verify business outcomes, not only component health. A technically available system can still omit the process it exists to perform, and authoritative state can be corrupted when recovery and production environments share uncontrolled feedback paths.
Framework relevance
Explicitly labelled analytical mappings
Regulation SCI lens: monitor critical systems for potential SCI events
The SEC found that NYSE's written policies and procedures did not establish monitoring to determine whether opening auctions had occurred, as required for SCI systems supporting the opening of trading.
Regulation SCI monitoring requirements as applied to NYSE · US Securities and Exchange CommissionSEC order: outcome monitoring and controlled override
The order records remedial measures including monitoring whether auctions ran, a second approval before a failed health check can be marked successful, and controls to confirm the DR environment is inactive.
NYSE remediation recorded in SEC Release No. 34-104934 · US Securities and Exchange CommissionEvidence register
Primary and supporting sources
-
US Securities and Exchange Commission
In the Matter of New York Stock Exchange LLC, Release No. 104934 (opens in a new tab) 6 March 2026 · Authoritative primary source
Publication note
A documented external event—not an NFRisk client engagement.
The named organisations are included because authoritative sources document the event. Their inclusion does not imply that they are or were NFRisk clients, that they endorse this analysis, or that NFRisk participated in the event or response. Framework relevance and NFRisk practitioner interpretation are analytical layers applied after the event.
Return to the Risk Scenario LibraryFrom scenario to mandate
Test the equivalent control assumption in your environment.
NFRisk can use this scenario as a starting point for a focused structural diagnostic, risk-architecture review or delivery-assurance discussion.