Resolved

High latency for some historical read queries

Final Update (04:25UTC): we see read latency now back in acceptable levels.

Update (03:15UTC): we are seeing some improvement in latency of historical lookups as we continue to apply mitigations. We will aim to provide another update by 04:30 UTC.

Starting around 17:15 UTC, some reads from the Rekor service for older data have been experiencing high latency or timing out. We are currently investigating the issue, which appears to be affecting certain MySQL queries.

Resolved

Temporary Disruption of Fulcio/CT Logs

On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved.