The incident is over. A release has made it to prod that fixes the bug.
Posted May 06, 2021 - 17:13 UTC
Update
A total of 83 login attempts were rejected as a result of this incident. Several of those were attempts tried by CockroachCloud staff in the process of identifying and resolving the issue.
We will update further before 16:00 UTC.
Posted May 06, 2021 - 14:57 UTC
Update
We are actively alerting on this condition, and the error is not occuring.
Posted May 06, 2021 - 14:26 UTC
Monitoring
The issue has been identified, a fix is in the works, and the problem has been mitigated. Currently users can log in onto the CockroachCloud normally. Until the fix is rolled out to production services, the problem may reoccur sporadically. This will manifest in some users again receiving the error: "Login failed, ResourceExhausted: Login attempt throttled". We are monitoring these errors and can re-apply mitigation steps to return the system to normal, until the definitive fix be applied.
We will update before 16:00 UTC.
Posted May 06, 2021 - 14:16 UTC
Investigating
We are currently investigating this issue. We will update before 10:30 EDT (14:30 UTC).