Some users are unable to login onto the CockroachCloud console.

Incident Report for CockroachDB Cloud

Resolved

The incident is over. A release has made it to prod that fixes the bug.
Posted May 06, 2021 - 17:13 UTC

Update

A total of 83 login attempts were rejected as a result of this incident. Several of those were attempts tried by CockroachCloud staff in the process of identifying and resolving the issue.

We will update further before 16:00 UTC.
Posted May 06, 2021 - 14:57 UTC

Update

We are actively alerting on this condition, and the error is not occuring.
Posted May 06, 2021 - 14:26 UTC

Monitoring

The issue has been identified, a fix is in the works, and the problem has been mitigated. Currently users can log in onto the CockroachCloud normally. Until the fix is rolled out to production services, the problem may reoccur sporadically. This will manifest in some users again receiving the error: "Login failed, ResourceExhausted: Login attempt throttled". We are monitoring these errors and can re-apply mitigation steps to return the system to normal, until the definitive fix be applied.

We will update before 16:00 UTC.
Posted May 06, 2021 - 14:16 UTC

Investigating

We are currently investigating this issue. We will update before 10:30 EDT (14:30 UTC).
Posted May 06, 2021 - 14:03 UTC
This incident affected: CockroachDB Cloud Console.