Everyone, we're experiencing an outage related to signing into Defender. We're investigating the issue and will be sharing updates as it progresses.
1 Like
Hey @dan_oz,
Our Optimism / Arbitrum / Avalanche relayers are down too.
Upon making a transaction request, we're getting being returned 500 status code with "Internal Server Error" error message.
AWS has reported an outage in API Gateway in the us-west-2 region and this is impacting numerous Defender Services. We are monitoring the situation and will provide updates on this thread.
1 Like
In our internal testing, Defender performance is still severely degraded (500 responses, increased latency), but AWS has indicated that the root cause has been identified and they are working to resolve the issue:
[11:33 AM PDT] We continue to work on resolving the elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. We continue to see a significant improvement in error rates, starting at 10:40 AM PDT, but are not seeing full recovery yet. The issue is caused by contention within the subsystem that is responsible for request processing within the API Gateway service. Engineers are engaged and have applied traffic filters as a precautionary measure, while they work to identify the root cause and resolve the issue. Engineers continue to work to reduce contention within the affected subsystem, which we believe will resolve the elevated error rates and latencies. Customers with applications that use API Gateway, or customers invoking Lambda functions via API Gateway, will be experiencing elevated error rates and latencies as a result of this issue...
2 Likes
Hi all,
AWS is still categorizing the API Gateway outage as a Disruption as of this writing, but they have noted significant reductions in error rates and our internal testing indicates that services have vastly improved.
The maintenance page has been removed from the Defender Console and it is available for use once again. The API is also operational. Please note as long as API Gateway issues persist, you may need to refresh the page (or retry an API request) in order for the request to process successfully. AWS has indicated they expect to be fully operational within about an hour.
Sorry for the disruption this has caused. If there are any further updates on this incident, we'll provide them here.
1 Like
AWS now considers the API Gateway issue resolved and Defender Services appear to be restored to pre-event levels.
1 Like