The race condition occurred when one DNS Enactor experienced "unusually high delays" while the DNS Planner continued generating new plans. A second DNS Enactor began applying the newer plans and executed a clean-up process just as the first Enactor completed its delayed run. This clean-up deleted the older plan as stale, immediately removing all IP addresses for the regional endpoint and leaving the system in an inconsistent state that prevented further automated updates applied by any DNS Enactors.
Someone definitely unplugged something they shouldn't have. This morning the worldwide internet saw some major outages after Amazon's AWS server farms suffered a whoopsie, taking down big-name games like Fortnite and Roblox, and as Eurogamer spotted, Palworld, Clash Royale and various Nintendo online services. Most of these have since recovered. Update: Fortnite has recovered, with multiplayer matches working once again. Nintendo is also back online, after both Switch and Switch 2 network services went offline for about four hours this morning.
We can confirm significant error rates for requests made to the DynamoDB endpoint in the US-EAST-1 Region. This issue also affects other AWS Services in the US-EAST-1 Region as well. During this time, customers may be unable to create or update Support Cases. Engineers were immediately engaged and are actively working on both mitigating the issue, and fully understanding the root cause.