Resolved

Rekor and CTLog disruptions caused by Trillian issues

We've experienced some issues on Trillian which caused disruptions on Rekor and CTLog services. The issue required manual intervention by the on-call engineers and has been resolved.

Maintenance

Database migration

We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary.

Resolved

High latency for some historical read queries

Final Update (04:25UTC): we see read latency now back in acceptable levels.

Update (03:15UTC): we are seeing some improvement in latency of historical lookups as we continue to apply mitigations. We will aim to provide another update by 04:30 UTC.

Starting around 17:15 UTC, some reads from the Rekor service for older data have been experiencing high latency or timing out. We are currently investigating the issue, which appears to be affecting certain MySQL queries.

Resolved

Temporary Disruption of Fulcio/CT Logs

On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved.