Secrets Live in More Places Than Your Repository

You get the alert: a production API key is sitting in a commit. You delete the line, push, close the ticket and go back to work. You removed one copy of a string. You did nothing to the credential, and it still works.
The mistake is treating a secret as a file problem. A secret is a live credential. Where it got written matters less than whether it still opens the door, so my position is simple: a leak is closed when the old value stops working.
Why deleting it fails
GitHub says more than 39 million secrets were leaked across GitHub in 2024. That is the platform's own count, and it does not say how many were live or exploited.
What happens after a leak was measured earlier. Meli, McNiece and Reaves scanned public GitHub in real time for about six months and published at NDSS in 2019. They found over 100,000 affected repositories and thousands of new, unique secrets leaking every day. When they pushed a test string and queried the search API until it appeared, the median wait was 20 seconds. Only 19% of the secrets they discovered were removed within roughly two weeks. Among the repositories that removed a secret and stayed up, none rewrote history, so the value was still reachable through Git history.
Those numbers describe that period, not today's GitHub. They still kill "we will clean it up on Monday". Treat a pushed secret as already read.
Rewriting history does not rescue you either. GitHub's own guide says "as a first step you need to revoke and/or rotate that secret", and that a rewrite does not reach forks, clones, cached views or pull requests. The same guide says rotation "may be sufficient to solve your problem" and a rewrite "may not be warranted", then lists a rewrite's side effects. My position follows from that: if you do rewrite, it comes second.
The same mistake in other places
Git is not the only store that refuses to forget. Docker says image layers, "once created, are immutable". A layer is a set of filesystem changes, so a later deletion is only another change on top. It follows that a file copied in one Dockerfile instruction and deleted in the next is still in the first layer.
Dahlmanns and colleagues analysed 337,171 images from Docker Hub and 8,076 other private registries, and reported in 2023 that 8.5% included secrets (9.0% of Docker Hub images, 6.3% of private registry images): 52,107 private keys and 3,158 API secrets. They also found 275,269 TLS and SSH hosts authenticating with leaked private keys. But 30% of those private keys came from layers that installed the OpenSSH server, which generates host keys automatically, and those are not credentials in the API key sense.
In that study, most leaked API secrets arrived through file operations, such as a copy from the creator's host. That is your own build step, and it is why I would scan the artifact you ship, every layer of it, as well as the source tree.
Build arguments fail the same way. Docker's documentation calls build arguments and environment variables "inappropriate for passing secrets" because they persist in the final image. Docker's answer is a secret mount, which exposes the value to build containers as a file or environment variable without persisting it in the final image.
OWASP notes that CI/CD tooling consumes high-privilege credentials regularly, and GitHub warns that secrets can get printed when a workflow runs. If one lands in a log, OWASP wants a process for removing it and says exposed keys "should undergo immediate revocation". Your CI pipeline is production, and its logs are part of it.
The pipeline can also leak with no mistake on your side. Codecov's Bash Uploader was modified without permission from January 31, 2021, exposing credentials passed through customers' CI runners. A Travis CI flaw meant that, for certain builds from 3 to 10 September 2021, secrets could be revealed to someone who forked a public repository and printed files during a build. Travis CI's bulletin limits this to public repositories that were forked and says it found no evidence of misuse. At CircleCI, an attacker exfiltrated customer environment variables, tokens and keys in December 2022. These were platform failures, not developer errors, and the advice each time was to re-roll or rotate.
| Where it lands | What cleanup can reach | What cleanup cannot reach |
|---|---|---|
| Git history | Your hosted copy, after a rewrite | Clones and forks; cached views and pull requests unless GitHub Support removes them |
| Image layer | Future builds | Layers already created |
| Build argument or ENV | The next build, using a secret mount | Final images already built |
| CI log | The log itself, through a removal process | Anyone who already read it |
| CI platform | Nothing on your side | Values held in the vendor's systems |
The one action that covers every row is making the old value useless.
Finding is the cheap part
A scanner tells you a string exists. It does not tell you the risk has gone. Verizon's 2025 DBIR reports a median of 94 days to remediate leaked secrets discovered in a GitHub repository. That rests on 141 events from one data contributor, so it is a signal and not an industry figure.
Google Cloud's threat intelligence write-up of the Snowflake data theft campaign shows how long a credential stays useful. Those credentials came mainly from infostealer malware on non-Snowflake systems, so this was not a repository or image leak. Some had not been rotated for as long as four years and MFA was not in place.
So change what closes the ticket: not "the string is gone", but "the old value is rejected". Rotate first, confirm, then clean up.
Where this position stops applying
Rotation is the minimum, not the ideal. It can break production, and it needs you to know everything that uses the credential. Better to have less to rotate: OWASP says dynamic secrets should be used where possible. A credential that expires on its own deals with the leak you never heard about.
Cleanup can still matter, but weigh it against the side effects GitHub lists, such as changed commit hashes and the sensitive data returning when someone pushes from an old clone. And the repository remains a major leak site. The GitHub, NDSS and container image figures above measure different things, so do not compare their counts.
Try it on one credential
All of this starts with knowing what you ship.
Pick one long-lived credential your deploy pipeline uses. Write down every place it could have landed: commits and their history, each layer of the last image you shipped, build arguments, CI logs. Then rotate it, on a normal working day, and time it. Record how long it took, what broke, and how you confirmed the old value was rejected. If you could not list everything that uses it, you have found your real gap on a quiet day, not the day a vendor tells you to rotate everything.
References
- GitHub found 39M secret leaks in 2024. Here's what we're doing to help, GitHub Blog, April 1, 2025
- How Bad Can It Git? Characterizing Secret Leakage in Public GitHub Repositories, NDSS Symposium, February 2019
- Secrets Revealed in Container Images: An Internet-wide Study on Occurrence and Impact, ACM ASIA CCS 2023 via arXiv, July 2023
- 2025 Data Breach Investigations Report, Verizon, April 2025
- Bash Uploader Security Update, Codecov, April 15, 2021
- Security Bulletin, Travis CI Community, September 13, 2021
- CVE-2021-41077, NVD, published September 14, 2021
- Incident report for January 4, 2023 security incident, CircleCI, January 12, 2023
- UNC5537 Targets Snowflake Customer Instances for Data Theft and Extortion, Google Cloud Blog, June 10, 2024
- Removing sensitive data from a repository, GitHub Docs, undated
- Using secrets in GitHub Actions, GitHub Docs, undated
- Build secrets, Docker Docs, undated
- Understanding the image layers, Docker Docs, undated
- Secrets Management Cheat Sheet, OWASP Cheat Sheet Series, undated
