Resolved
High latency for some historical read queries
Started 27 Aug at 07:26pm UTC, last updated 28 Aug at 03:15am UTC.
Updated
We have identified the root cause of the issue as database index imbalances, and have been running a script to remove some excess data in the imbalanced trees. We are seeing latency and database throughput improve, and believe that the incident is now resolved.
Created
Final Update (04:25UTC): we see read latency now back in acceptable levels.
Update (03:15UTC): we are seeing some improvement in latency of historical lookups as we continue to apply mitigations. We will aim to provide another update by 04:30 UTC.
Starting around 17:15 UTC, some reads from the Rekor service for older data have been experiencing high latency or timing out. We are currently investigating the issue, which appears to be affecting certain MySQL queries.