Incidents reported on status page for sigstore https://status.sigstore.dev/ https://d1lppblt9t2x15.cloudfront.net/logos/a363dea5b1a161e2fab96a3b45d5064b.png https://status.sigstore.dev/ en https://status.sigstore.dev/incident/701149 Wed, 06 Aug 2025 04:16:00 -0000 https://status.sigstore.dev/incident/701149#39df74309446099ac6053817bb2368d1e27e72206fb87487502ff026c6cf0bc1 Requests are now succeeding, the GCP issue appears to have been resolved. We will continue to monitor. https://status.sigstore.dev/incident/701149 Wed, 06 Aug 2025 04:03:00 -0000 https://status.sigstore.dev/incident/701149#290a2cbaaa8e8c20090bfb340d2177e4fda4d042b193c15a5845a1e6ed7b647b Rekor is currently unavailable. The root cause appears to be due to a GCP outage. Writes are currently failing, while reads may sporadically succeed. https://status.sigstore.dev/incident/601807 Thu, 12 Jun 2025 18:24:00 -0000 https://status.sigstore.dev/incident/601807#d2852da5976695f9c5075e4b606057fa6980dfd016e12e653bc2fd760c4471f1 The root cause of availability issues has been resolved by our cloud provider. https://status.sigstore.dev/incident/601807 Thu, 12 Jun 2025 18:24:00 -0000 https://status.sigstore.dev/incident/601807#d2852da5976695f9c5075e4b606057fa6980dfd016e12e653bc2fd760c4471f1 The root cause of availability issues has been resolved by our cloud provider. https://status.sigstore.dev/incident/594517 Sat, 31 May 2025 05:05:00 -0000 https://status.sigstore.dev/incident/594517#4004e6969d2937ab67bc69256c3ce5dc1c715912f9976a0013ba1fe504440faa We have identified the root cause and resolved the incident by restarting the database. Please retry entry uploads now. https://status.sigstore.dev/incident/594517 Sat, 31 May 2025 04:39:00 -0000 https://status.sigstore.dev/incident/594517#2732c3c1e0029154962fecca8aadd41c6af95e49b3f3d8bf6a12c59b2778377f Rekor is temporarily degraded. https://status.sigstore.dev/incident/436845 Mon, 30 Sep 2024 06:04:00 -0000 https://status.sigstore.dev/incident/436845#84ecc556e05e7e906133bfe2b5617d6e785cf52ebb931ce6ab8003a266cac0b9 We've experienced some issues on Trillian which caused disruptions on Rekor and CTLog services. The issue required manual intervention by the on-call engineers and has been resolved. https://status.sigstore.dev/incident/436845 Mon, 30 Sep 2024 06:04:00 -0000 https://status.sigstore.dev/incident/436845#84ecc556e05e7e906133bfe2b5617d6e785cf52ebb931ce6ab8003a266cac0b9 We've experienced some issues on Trillian which caused disruptions on Rekor and CTLog services. The issue required manual intervention by the on-call engineers and has been resolved. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 -0000 https://status.sigstore.dev/incident/423676#bbad3861f4f82e98ffd58d4665d209d899d9b12bbbdb32ebb6f2ea27bcd4701c Maintenance completed https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 -0000 https://status.sigstore.dev/incident/423676#bbad3861f4f82e98ffd58d4665d209d899d9b12bbbdb32ebb6f2ea27bcd4701c Maintenance completed https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 -0000 https://status.sigstore.dev/incident/423676#bbad3861f4f82e98ffd58d4665d209d899d9b12bbbdb32ebb6f2ea27bcd4701c Maintenance completed https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/420250 Wed, 28 Aug 2024 03:15:00 -0000 https://status.sigstore.dev/incident/420250#1e1b16788e5089003a615ef231cdec8ddfd877039431ffb73819b3455d2c25b3 We have identified the root cause of the issue as database index imbalances, and have been running a script to remove some excess data in the imbalanced trees. We are seeing latency and database throughput improve, and believe that the incident is now resolved. https://status.sigstore.dev/incident/420250 Tue, 27 Aug 2024 19:26:00 -0000 https://status.sigstore.dev/incident/420250#de6903468c0ffb971ac334f5573cddcd6f6f2b72b1eeb0a6c85304efd4d65a11 Final Update (04:25UTC): we see read latency now back in acceptable levels. Update (03:15UTC): we are seeing some improvement in latency of historical lookups as we continue to apply mitigations. We will aim to provide another update by 04:30 UTC. Starting around 17:15 UTC, some reads from the Rekor service for older data have been experiencing high latency or timing out. We are currently investigating the issue, which appears to be affecting certain MySQL queries. https://status.sigstore.dev/incident/410910 Thu, 08 Aug 2024 15:16:00 -0000 https://status.sigstore.dev/incident/410910#ac44dd69f53a5ea99278793e212a3ea0b570c99ba829b883053e1b16ca28ba31 On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved. https://status.sigstore.dev/incident/410910 Thu, 08 Aug 2024 15:16:00 -0000 https://status.sigstore.dev/incident/410910#ac44dd69f53a5ea99278793e212a3ea0b570c99ba829b883053e1b16ca28ba31 On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved. https://status.sigstore.dev/incident/410910 Thu, 08 Aug 2024 15:16:00 -0000 https://status.sigstore.dev/incident/410910#ac44dd69f53a5ea99278793e212a3ea0b570c99ba829b883053e1b16ca28ba31 On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved. https://status.sigstore.dev/incident/303035 Wed, 20 Dec 2023 08:00:00 -0000 https://status.sigstore.dev/incident/303035#bb45a3bf9a7efb57d28b0741c85c01c31e5b0dc54ca674a047b630ee13510ab3 Cosign signing using our Rekor public instance have experienced some disruptions starting at 8am UTC during approximately 1 hour. The issue has been caused by networking problems between Trillian and our MySQL instance. This provoked our Trillian healthchecks to fail and consequently a restart of the pods. During this pod restart, one of the Trillian pods could not start again having issues to bind the address again. Once we bounced the affected Trillian pods, everything started to work again without experiencing networking issues