Incidents reported on status page for sigstore https://status.sigstore.dev/ https://d1lppblt9t2x15.cloudfront.net/logos/a363dea5b1a161e2fab96a3b45d5064b.png https://status.sigstore.dev/ en https://status.sigstore.dev/ Thu, 12 Jun 2025 20:10:00 +0000 https://status.sigstore.dev/#bfbbad3355d8ebd5bc55ac41c1e426134b94d2a66a6cef9ecd5cb3a72de383c6 Rekor Log recovered https://status.sigstore.dev/ Thu, 12 Jun 2025 18:31:48 +0000 https://status.sigstore.dev/#bfbbad3355d8ebd5bc55ac41c1e426134b94d2a66a6cef9ecd5cb3a72de383c6 Rekor Log went down https://status.sigstore.dev/incident/601807 Thu, 12 Jun 2025 18:24:00 -0000 https://status.sigstore.dev/incident/601807#d2852da5976695f9c5075e4b606057fa6980dfd016e12e653bc2fd760c4471f1 The root cause of availability issues has been resolved by our cloud provider. https://status.sigstore.dev/incident/601807 Thu, 12 Jun 2025 18:24:00 -0000 https://status.sigstore.dev/incident/601807#d2852da5976695f9c5075e4b606057fa6980dfd016e12e653bc2fd760c4471f1 The root cause of availability issues has been resolved by our cloud provider. https://status.sigstore.dev/ Thu, 12 Jun 2025 18:12:53 +0000 https://status.sigstore.dev/#870f0a8baae8ea089ee7793f3fd288eb0c96edadd95016d388867dfaa4b4aee4 Rekor Log recovered https://status.sigstore.dev/ Thu, 12 Jun 2025 18:10:47 +0000 https://status.sigstore.dev/#870f0a8baae8ea089ee7793f3fd288eb0c96edadd95016d388867dfaa4b4aee4 Rekor Log went down https://status.sigstore.dev/incident/594517 Sat, 31 May 2025 05:05:00 -0000 https://status.sigstore.dev/incident/594517#4004e6969d2937ab67bc69256c3ce5dc1c715912f9976a0013ba1fe504440faa We have identified the root cause and resolved the incident by restarting the database. Please retry entry uploads now. https://status.sigstore.dev/ Sat, 31 May 2025 04:58:32 +0000 https://status.sigstore.dev/#0364f7a9fdfd30fc5ce989685682c570cf1cb99d25236aa908e830f6ee9225f5 CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 04:57:43 +0000 https://status.sigstore.dev/#cba3fb271a73a4adf273ccd3c725fe5c9538fbd0b65e9981df645c297b4f4f34 Rekor Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 04:55:25 +0000 https://status.sigstore.dev/#0364f7a9fdfd30fc5ce989685682c570cf1cb99d25236aa908e830f6ee9225f5 CT Log went down https://status.sigstore.dev/ Sat, 31 May 2025 04:54:43 +0000 https://status.sigstore.dev/#cba3fb271a73a4adf273ccd3c725fe5c9538fbd0b65e9981df645c297b4f4f34 Rekor Log went down https://status.sigstore.dev/ Sat, 31 May 2025 04:49:23 +0000 https://status.sigstore.dev/#6fd0f6fa50e2f8abc7441da8c6c4e7c14bcf7b04be870c8907e25a683bf8a384 CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 04:46:42 +0000 https://status.sigstore.dev/#6fd0f6fa50e2f8abc7441da8c6c4e7c14bcf7b04be870c8907e25a683bf8a384 CT Log went down https://status.sigstore.dev/incident/594517 Sat, 31 May 2025 04:39:00 -0000 https://status.sigstore.dev/incident/594517#2732c3c1e0029154962fecca8aadd41c6af95e49b3f3d8bf6a12c59b2778377f Rekor is temporarily degraded. https://status.sigstore.dev/ Sat, 31 May 2025 04:37:31 +0000 https://status.sigstore.dev/#9334e14f5333a1c1a53e1f595ef44e3dfd1e5cc57032c9b4521dc74e55a0fa2d CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 04:34:42 +0000 https://status.sigstore.dev/#9334e14f5333a1c1a53e1f595ef44e3dfd1e5cc57032c9b4521dc74e55a0fa2d CT Log went down https://status.sigstore.dev/ Sat, 31 May 2025 04:19:32 +0000 https://status.sigstore.dev/#352b4dda9bf02a310d2bfa9caacc3656b49e7149d3e82d4b22532774f758d1ff CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 04:16:42 +0000 https://status.sigstore.dev/#352b4dda9bf02a310d2bfa9caacc3656b49e7149d3e82d4b22532774f758d1ff CT Log went down https://status.sigstore.dev/ Sat, 31 May 2025 04:07:36 +0000 https://status.sigstore.dev/#615a4987875fdf16a4043840ce8bdaeef768ff3e43c13567c2f8d2b246d93fd2 CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 04:01:43 +0000 https://status.sigstore.dev/#615a4987875fdf16a4043840ce8bdaeef768ff3e43c13567c2f8d2b246d93fd2 CT Log went down https://status.sigstore.dev/ Sat, 31 May 2025 03:51:43 +0000 https://status.sigstore.dev/#503020c4925ae92fceb8825b4066f6c3df81da3228272e6ecdf9206fcc6d18f0 Rekor Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 03:49:23 +0000 https://status.sigstore.dev/#503020c4925ae92fceb8825b4066f6c3df81da3228272e6ecdf9206fcc6d18f0 Rekor Log went down https://status.sigstore.dev/ Sat, 31 May 2025 03:40:24 +0000 https://status.sigstore.dev/#91245e9fff700fa838e09da16f4009ef5f3596b300401f00326c208b8f8865de CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 03:38:01 +0000 https://status.sigstore.dev/#91245e9fff700fa838e09da16f4009ef5f3596b300401f00326c208b8f8865de CT Log went down https://status.sigstore.dev/ Sat, 31 May 2025 03:31:25 +0000 https://status.sigstore.dev/#f0bfa7c76728a010417cd0f19752a1ba96bf04fbf764c26d30dfb79e0a9971bf CT Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 03:30:55 +0000 https://status.sigstore.dev/#d86f25ae26d2806ab730cebb97d5599c941a2a979c306a5e6ac74b02c724ac63 Rekor Log recovered https://status.sigstore.dev/ Sat, 31 May 2025 03:28:43 +0000 https://status.sigstore.dev/#f0bfa7c76728a010417cd0f19752a1ba96bf04fbf764c26d30dfb79e0a9971bf CT Log went down https://status.sigstore.dev/ Sat, 31 May 2025 03:28:33 +0000 https://status.sigstore.dev/#d86f25ae26d2806ab730cebb97d5599c941a2a979c306a5e6ac74b02c724ac63 Rekor Log went down https://status.sigstore.dev/incident/436845 Mon, 30 Sep 2024 06:04:00 -0000 https://status.sigstore.dev/incident/436845#84ecc556e05e7e906133bfe2b5617d6e785cf52ebb931ce6ab8003a266cac0b9 We've experienced some issues on Trillian which caused disruptions on Rekor and CTLog services. The issue required manual intervention by the on-call engineers and has been resolved. https://status.sigstore.dev/incident/436845 Mon, 30 Sep 2024 06:04:00 -0000 https://status.sigstore.dev/incident/436845#84ecc556e05e7e906133bfe2b5617d6e785cf52ebb931ce6ab8003a266cac0b9 We've experienced some issues on Trillian which caused disruptions on Rekor and CTLog services. The issue required manual intervention by the on-call engineers and has been resolved. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:02:26 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:02:26 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:02:26 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 +0000 https://status.sigstore.dev/incident/423676#bbad3861f4f82e98ffd58d4665d209d899d9b12bbbdb32ebb6f2ea27bcd4701c Maintenance completed https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 +0000 https://status.sigstore.dev/incident/423676#bbad3861f4f82e98ffd58d4665d209d899d9b12bbbdb32ebb6f2ea27bcd4701c Maintenance completed https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 +0000 https://status.sigstore.dev/incident/423676#bbad3861f4f82e98ffd58d4665d209d899d9b12bbbdb32ebb6f2ea27bcd4701c Maintenance completed https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 20:00:24 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 18:37:16 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 18:37:16 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 18:37:16 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/423676 Sun, 08 Sep 2024 16:00:00 -0000 https://status.sigstore.dev/incident/423676#60378930aea68e197bfab1befac09dfdc087ea26102432ecc3000a7efdecb44f We will be migrating our production database infrastructure to a newer version of MySQL; this requires some downtime and validation testing as we promote a backup to primary. https://status.sigstore.dev/incident/420250 Wed, 28 Aug 2024 03:15:00 -0000 https://status.sigstore.dev/incident/420250#1e1b16788e5089003a615ef231cdec8ddfd877039431ffb73819b3455d2c25b3 We have identified the root cause of the issue as database index imbalances, and have been running a script to remove some excess data in the imbalanced trees. We are seeing latency and database throughput improve, and believe that the incident is now resolved. https://status.sigstore.dev/incident/420250 Tue, 27 Aug 2024 19:26:00 -0000 https://status.sigstore.dev/incident/420250#de6903468c0ffb971ac334f5573cddcd6f6f2b72b1eeb0a6c85304efd4d65a11 Final Update (04:25UTC): we see read latency now back in acceptable levels. Update (03:15UTC): we are seeing some improvement in latency of historical lookups as we continue to apply mitigations. We will aim to provide another update by 04:30 UTC. Starting around 17:15 UTC, some reads from the Rekor service for older data have been experiencing high latency or timing out. We are currently investigating the issue, which appears to be affecting certain MySQL queries. https://status.sigstore.dev/incident/410910 Thu, 08 Aug 2024 15:16:00 -0000 https://status.sigstore.dev/incident/410910#ac44dd69f53a5ea99278793e212a3ea0b570c99ba829b883053e1b16ca28ba31 On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved. https://status.sigstore.dev/incident/410910 Thu, 08 Aug 2024 15:16:00 -0000 https://status.sigstore.dev/incident/410910#ac44dd69f53a5ea99278793e212a3ea0b570c99ba829b883053e1b16ca28ba31 On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved. https://status.sigstore.dev/incident/410910 Thu, 08 Aug 2024 15:16:00 -0000 https://status.sigstore.dev/incident/410910#ac44dd69f53a5ea99278793e212a3ea0b570c99ba829b883053e1b16ca28ba31 On 2024-08-08, the Trillian database server encountered an unexpected table lock which disrupted normal operations. Users were unable to issue new Fulcio certificates or do online verification of CT log inclusion. The issue required manual intervention by the on-call engineer and has been resolved. https://status.sigstore.dev/incident/303035 Wed, 20 Dec 2023 08:00:00 -0000 https://status.sigstore.dev/incident/303035#bb45a3bf9a7efb57d28b0741c85c01c31e5b0dc54ca674a047b630ee13510ab3 Cosign signing using our Rekor public instance have experienced some disruptions starting at 8am UTC during approximately 1 hour. The issue has been caused by networking problems between Trillian and our MySQL instance. This provoked our Trillian healthchecks to fail and consequently a restart of the pods. During this pod restart, one of the Trillian pods could not start again having issues to bind the address again. Once we bounced the affected Trillian pods, everything started to work again without experiencing networking issues