Code Scanning Is Not Enough: Scan the Artifact You Ship

Your pull request is green. Static analysis passed, the dependency check read your lockfile, and then a build turned that commit into an image with an operating system, a language runtime and whatever the build steps fetched. Those checks never looked at that image, and the image is what runs in production.
The repository is the input
Scanning the repository is the obvious place to start, and I would not give it up. But a repository describes how to make the thing you ship. It is not the thing.
NIST SP 800-190 describes an image as an immutable snapshot of the language runtime, third-party libraries, init scripts and OS tools. It also names the upside: with containers, "the developer, build system, and production all run the same immutable artifact". There is one object to test, so test that object.
What only the artifact can tell you
| What ships | What the repository shows | What the artifact shows |
|---|---|---|
| Base image packages | A line naming the base image | The packages installed, stale ones included |
| Build-time downloads | The command that fetches something | The bytes fetched, if the build kept them |
| Vendored and bundled binaries | A file source analysis cannot read, or a step that fetches it | The binary, where it will run |
| Secrets in layers | Nothing, if the secret came from the CI environment | Whatever a build step wrote into a layer |
NIST warns that "components within an image may be missing critical security updates or are otherwise outdated", and names third-party base layers as a possible source of embedded malware. The build-download and binary rows are my reasoning, not a measurement: the repository holds a command or a file it cannot explain, and the image holds whatever the build kept.
Secrets survive the step that deleted them
Suppose one build step copies a deploy key into the image, the next uses it, and a third deletes it. The finished filesystem has no key. But the OCI image spec defines a layer as a changeset packaged as a tar archive, and records a removal as a whiteout: "an empty file with a special filename that signifies a path should be deleted". The spec does not say in these words that the earlier tar still holds the key. That is my reading of the format, and it is why I treat a deleted secret as a leaked one.
Docker's documentation says build arguments and environment variables "are inappropriate for passing secrets to your build, because they persist in the final image". It recommends secret mounts or SSH mounts instead.
Dahlmanns et al. analysed 337,171 images from Docker Hub and 8,076 other private registries and report that 8.5% of images include secrets, along with 275,269 TLS and SSH hosts using leaked private keys. That population is not your registry, and you will not know your own rate until you look.
Codecov's 2021 notice shows the cost. It attributes the compromise of its Bash Uploader to "an error in Codecov's Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script". The defect it names sits in how an image was made.
The release can differ from the repository
The XZ Utils maintainer's page says the 5.6.0 and 5.6.1 release tarballs contain a backdoor, that Jia Tan created and signed them, and that the trigger code "was never included in the Git repository". CISA's alert on CVE-2024-3094 confirms both versions. Reading the repository would not have shown you the complete payload.
An artifact vulnerability scanner would not have flagged the backdoor by CVE match before the CVE existed. In my view, what could have helped earlier is comparing the tarball with git or using build provenance. A maintainer's signature would not have helped, because the tarballs were signed by the person who created them. The artifact scan earns its keep after the alert: an inventory of your images answers "do we ship 5.6.0 or 5.6.1?", and if xz arrived through a base image, your application repository never mentions it.
Periodic alterations to Codecov's uploader began on 31 January 2021, Codecov learned of them on 1 April, and they could "potentially export information stored in our users' continuous integration (CI) environments". In my view, a script that runs in a pipeline and leaves nothing behind is invisible to both scans. That needs the integrity validation OWASP describes in CICD-SEC-9.
Where artifact scanning is weak
It can miss application packages. A metadata match cannot see code that is not a declared package. That is my reasoning, not a measurement, so test your scanner on an image you understand before trusting its silence.
It matches rather than understands. Ponta et al. describe common approaches that "depend on metadata to identify vulnerable OSS dependencies" and propose code-centric reachability analysis. A package match cannot tell you whether the vulnerable function is ever called, so you need better ways to rank the list than a severity score.
It does not read your logic. A package inventory does not look for injection paths or insecure design in code you wrote. NIST's SSDF keeps analysis of human-readable code (PW.7) separate from executable code testing (PW.8), which it frames as finding vulnerabilities "not identified by previous reviews, analysis, or testing".
It is a snapshot. NIST says an image current at build time will later have vulnerabilities found in its components, so a build gate alone goes stale.
And the fix lives elsewhere: in source, a lockfile or a Dockerfile, so someone must trace each finding to an owner. Much of this evidence is also about containers. The logic carries to any built package, but the layer mechanics do not.
The position
Scan source before merge, while the author can still fix it. Scan the artifact, gate release on that result, and rescan what is stored. Connect the two with SLSA provenance, an attestation that "a particular build platform produced a set of software artifacts", so a finding on an image leads back to a build and its source.
Dwaar works on this: discovering what you ship from source control, registries and cloud accounts, scanning it, and keeping the results in one findings list with EPSS and KEV signals.
Try it on one image
Take the image behind your most exposed service from the registry, not a fresh local build. Generate a package inventory with a tool you already have and put it beside the lockfile. For every entry the lockfile does not explain, write down who would patch it. Then search each layer, not just the final filesystem, for private keys and tokens. Any blank owner and any key in any layer is your first piece of work.
References
- NIST, SP 800-190 Application Container Security Guide, September 2017
- NIST, SP 800-218 Secure Software Development Framework Version 1.1, February 2022
- arXiv / ACM ASIA CCS 2023, Secrets Revealed in Container Images: An Internet-wide Study on Occurrence and Impact, July 2023
- arXiv, Beyond Metadata: Code-centric and Usage-based Analysis of Known Vulnerabilities in Open-source Software, June 2018
- CISA, Reported Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094, 29 March 2024
- Lasse Collin (tukaani.org), XZ Utils backdoor, March 2024, last updated 17 January 2025
- Codecov, Security update on the Bash Uploader, 15 April 2021, updated 29 April 2021
- Docker Docs, Build secrets, page undated, source on main last changed 24 February 2026 before this post
- Open Container Initiative, Image Layer Filesystem Changeset, image-spec v1.0.0, 19 July 2017
- SLSA / OpenSSF, SLSA v1.0 Provenance, v1.0 released April 2023
- OWASP, CICD-SEC-9: Improper Artifact Integrity Validation, page undated, checked against the Internet Archive capture of 3 March 2026
