Build an Image Processing Pipeline
28 Jul 2026Don’t resize images on every request. That’s the single most common mistake in a homegrown image pipeline — reprocessing the same 4000px photo into a 200px thumbnail on every page load because nobody cached the result. Process once, cache the derivative, serve that forever.
Process on upload, not on request
When an image is uploaded, generate the sizes you actually use — thumbnail, card, full — right then, and store all of them:
import sharp from 'sharp'
const sizes = { thumb: 200, card: 600, full: 1600 }
for (const [name, width] of Object.entries(sizes)) {
const buffer = await sharp(original)
.resize(width, null, { withoutEnlargement: true })
.webp({ quality: 80 })
.toBuffer()
await s3.putObject({ Bucket: 'images', Key: `${id}/${name}.webp`, Body: buffer })
}
sharp is the right default here — it’s fast (backed by libvips, not pure JS), handles the common formats, and this is exactly the kind of well-trodden problem where you don’t write your own resize/crop logic.
On-demand resizing, if you must
Sometimes you genuinely don’t know every size you’ll need ahead of time — a CMS where editors pick arbitrary crop dimensions. In that case, resize on first request and cache the result so the second request never reprocesses:
app.get('/img/:id/:width.webp', async (req, res) => {
const cacheKey = `${req.params.id}/${req.params.width}.webp`
const cached = await s3.headObject({ Bucket: 'images', Key: cacheKey }).catch(() => null)
if (!cached) {
const original = await s3.getObject({ Bucket: 'images', Key: `${req.params.id}/original` })
const resized = await sharp(original.Body).resize(Number(req.params.width)).webp().toBuffer()
await s3.putObject({ Bucket: 'images', Key: cacheKey, Body: resized })
}
res.redirect(`https://cdn.example.com/${cacheKey}`)
})
The request path branches on whether a derivative already exists:
flowchart TD
A["Request: /img/:id/:width.webp"] --> B{"Cached derivative exists?"}
B -->|"Yes"| C["Redirect to CDN"]
B -->|"No"| D["Fetch original from storage"]
D --> E["Resize with sharp"]
E --> F["Store derivative"]
F --> C
The first request per size is slow; every one after it is a cache hit served straight from your CDN, never touching your app server again. Put a CDN in front of the bucket regardless — even a cache-hit request to your app server is unnecessary latency compared to an edge cache hit.
Format matters more than most people think
Serve WebP or AVIF instead of JPEG/PNG by default — both give meaningfully smaller files at the same visual quality, and every browser that matters supports WebP now. If you need to support ancient clients, generate a JPEG fallback alongside it and pick based on the Accept header, but don’t let that hold back the default for everyone else.
Accept: image/webp,image/avif,image/*
What breaks if you skip caching
Without cached derivatives, resize cost scales with traffic, not with uploads — a photo that gets resized once at upload time costs you compute once, forever. The same photo reprocessed on every page view costs you compute on every page view, and on a popular page that’s the difference between a $20/month image bill and a five-figure one, plus a CPU-bound endpoint that falls over under load nobody planned for.
Original files
Always keep the original, untouched, in cold or infrequent-access storage even after you’ve generated derivatives. You will eventually want a size you didn’t generate up front — a new device pixel density, a new crop the design team wants — and reprocessing from a full-resolution original beats reprocessing from an already-downscaled derivative, which compounds quality loss.
The short version
Generate the sizes you use at upload time. Cache anything generated on demand so it’s generated exactly once. Serve WebP by default. Put a CDN in front of the bucket. That’s the whole pipeline for the vast majority of products — reach for a dedicated image CDN service (Cloudflare Images, imgix) only once you’re generating enough variants, or serving enough traffic, that managing this yourself is costing more engineering time than the service would cost in dollars.