Worked example: hardening upload validation when extensions lie

Note: this is a composite, illustrative scenario assembled from common upload-validation problems. It does not describe a specific organisation or incident. The technical details (file signatures, the checks, the failure modes) are the point.

The scenario

Example scenario: an internal ticketing tool lets staff attach "screenshots". Someone uploads invoice.jpg, and a server-side thumbnail worker crashes on it. The logs show the file is mostly ASCII text with a .jpg extension. It has no JPEG start-of-image marker (FF D8 FF), so it was never an image; it was probably an exported text file renamed by hand or by a misconfigured tool.

The obvious first fix is blunt: reject anything whose extension does not match the sniffed type. In this scenario that fix causes a new problem. Some legitimate images (for example photos converted between formats by mobile apps, or files whose extension says .jpg while the content is actually PNG or HEIC) are now rejected, and support tickets move from "thumbnail broken" to "can't upload my photo".

Testing approach

The useful move is to build a small matrix of synthetic files (all generated locally, none containing real data):

File	First bytes	Expected
valid.jpg (real JPEG)	FF D8 FF	Accepted, thumbnail created
png-as.jpg (real PNG, .jpg name)	89 50 4E 47	Accepted, stored as image/png (policy choice)
text-as.jpg (text padded to 200 KB)	ASCII	Rejected or quarantined; worker never sees it
pdf-as.png (PDF with .png name)	25 50 44 46	Rejected for image fields
truncated.jpg (JPEG cut in half)	FF D8 FF	Header passes; decode fails safely

The valid images come from the generator (padded JPG and PNG files remain decodable); the mismatches are made by renaming. The magic-byte sniffer confirms what each file really is before it is uploaded. A command-line check does the same:

for f in *.jpg *.png; do
  printf '%-20s %s  ' "$f" "$(file -b --mime-type "$f")"
  xxd -l 4 -p "$f"
done

Each file is then run through every stage, not just the first request: the upload API, the asynchronous virus scan, the thumbnail queue, and the CDN fetch of the stored object. Pass or fail is recorded per stage. A 201 from the upload API says nothing about whether the thumbnail worker will crash ten seconds later.

Layered defences

1. Treat client-reported types as hints. The browser's Content-Type comes from the file extension and the operating system's mapping. It is useful for UX, never for security decisions.
2. Magic-byte gate before any image processing. Require FF D8 FF for JPEG, 89 50 4E 47 0D 0A 1A 0A for PNG, 47 49 46 38 for GIF, and RIFF ... WEBP for WebP before a file is queued for thumbnails. Store the detected type, not the claimed one.
3. A real decode in a sandbox. A correct header does not make a valid image (see truncated.jpg). Decode in an isolated worker with a time and memory limit, and treat failure as "not an image" rather than an exception.
4. Quarantine instead of hard failure. Mismatches go to a non-public storage prefix. The user gets a clear message and a reference ID, not a stack trace.
5. Audited human override. Support can reclassify a quarantined file after review, and that action is logged.
6. Serve safely. Send the stored (detected) Content-Type with X-Content-Type-Options: nosniff, and serve user files from a separate domain where possible.

Avoiding false positives

The overly strict fix in this scenario failed because it compared long header sequences against a fixed table. JPEGs in particular vary after the first three bytes: the next marker may be an APP0 (JFIF), APP1 (Exif) or another segment depending on the camera or app that wrote the file. Checking only FF D8 FF and then relying on a sandboxed decode is both more permissive for real photos and stricter for fakes than a long signature match.

For files where the extension and content disagree but the content is a valid, allowed type (a PNG named .jpg), accepting the file and correcting the stored type is usually the friendliest policy. Whatever you choose, write it in the expected-results table so the behaviour is tested, not accidental.

Try it yourself

1. Generate a JPG and a PNG from the generator, and a TXT padded to 200 KB (/generator?ext=txt&bytes=204800).
2. Make renamed copies as in the table above and inspect them with the magic-byte sniffer and MIME lookup.
3. Upload them to a staging environment you control, and record the result at each processing stage.

The full walkthrough is in the extension versus sniffed MIME scenario. Only run mismatch probes against systems you own or are authorised to test.