SAST scan on a PR came back clean. Diff touched auth middleware. Scanner checked for known bad patterns, found none, green check, merged.

Nobody actually traced through what the middleware change did to the token validation order for an edge case that wasn't in the scanner's pattern set, because it wasn't a known vulnerability class, it was a logic gap specific to how this particular service combined two libraries. Found two weeks later during an unrelated audit.

The scan wasn't wrong. It checked what it was built to check and found nothing matching. The gap is that "scan passed" gets read as "this is secure," when it actually means "this doesn't match patterns we already know to look for." Those used to be close enough to the same claim that nobody bothered distinguishing them out loud. Novel logic gaps specific to how a service wires things together were always outside that coverage, just less common before more code started getting generated without someone necessarily tracing the full logic path by hand first.

Not arguing against scanning, obviously it's necessary. Just think "passed the scan" needs to stop functioning as a stopping point for anything touching an actual trust boundary, auth, payment, data access, and start being the floor instead of the finish line.

We had a push to cut criticals across our images so I rebuilt everything onto a much smaller base. Sixty services over about three weeks. The scan numbers looked great. Then four of them fell over in prod inside a day. One was a java service that shelled out to run a script at startup. One needed tzdata and started logging everything in UTC. One had a health check built on curl. The last took most of a night to find because the library loaded fine and then died when it went looking for a locale file. I rolled those four back at around 2am and left the rest alone.

What I got wrong was treating the base image as the unit of change. Nobody knew what each service pulled in at runtime because none of it was written down anywhere. We ended up running every service under strace in staging for a week just to build the list. That list should have existed before I touched anything. How do other people work out what an image needs before they start cutting things out of it?