Randomly API key was de-authorized and throwing errors

Our environment was working great, processed around 5,000 NFTs, and then about an hour ago we started receiving this error:

2022-09-07T01:41:54.798551884Z 01:41:54 0|Workers  | Failed to get a token for the API key [----redacted-----] {
2022-09-07T01:41:54.798560985Z 01:41:54 0|Workers  |   code: 'NotAuthorizedException',
2022-09-07T01:41:54.798566085Z 01:41:54 0|Workers  |   name: 'NotAuthorizedException',
2022-09-07T01:41:54.798570685Z 01:41:54 0|Workers  |   message: 'Incorrect username or password.'
2022-09-07T01:41:54.798575485Z 01:41:54 0|Workers  | }

:computer: Environment
Node server hosted on Azure

:memo:Details
We tried switching the API key to a new one but still receive the same issues

Hey @Andrew_Wiskus, I don't see any Relayers in the Defender account registered under your email. Could you let me know the relayer address or ID (it's shown in the URL when you go to that relayer in the app), so I can look into this?

Also, we're seeing multiple sign-in throttles for Relayer API keys from 3am UTC to 11.30am UTC today, but nothing out of that period. Any chance you experienced those issues during that time window? We'll work on increasing our sign-in capacity in the meantime.

Yes that was the time window, we lowered the amount of batch process that we did in parallel and seemed to have gotten within your threshold for sign-in. 28k NFTs minted now from our users (yay)

If there is a hard limit around this it would be nice to know for us & for our future projects so we can avoid this happening in the future. Not sure if this is an account issue either since we're kind of a custom solution. (I don't have access to the ID right now, but if you need it still let me know)

There is a rate limit of 10txs per second (in addition to the hourly quota), and a limit of 120 sign in requests per second. It's odd that you had hit the latter, but we'll follow up with our respective infra provider.

Does this 10txs per second apply to the 10k tx/hr surge plan? We're sending 15tx simultaneously at the moment and it seems to be working fine, before when it went down we were sending 30~ simultaneously.

It does apply by default, but we can manually increase it for the surge plans if needed. And the throttling has some leeway to go over the 10txs/s, which seems to be the case here. Also, note that the throttling is per relayer.

Still, the 10txs/s kicks in after the client has authenticated itself, so it's not related to the error you were seeing originally.