Streaming makes an LLM response feel instant instead of making the user stare at a spinner for eight seconds. The mechanics are simple. The edge cases are what break in production.

The full path, including the close-tab case:

sequenceDiagram
    participant Browser
    participant Server
    participant LLM
    Browser->>Server: GET /chat stream request
    Server->>LLM: stream true
    loop each token
        LLM-->>Server: token chunk
        Server-->>Browser: SSE data event
    end
    Browser--xServer: tab closed
    Server->>LLM: abort upstream call
  

The server side: Server-Sent Events

SSE is the right transport for one-directional token streams — simpler than WebSockets, works over plain HTTP, and every browser has EventSource built in. A minimal Express endpoint proxying an OpenAI stream:

app.get('/chat', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream')
  res.setHeader('Cache-Control', 'no-cache')
  res.setHeader('Connection', 'keep-alive')

  const stream = await client.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: req.query.prompt }],
    stream: true,
  })

  for await (const chunk of stream) {
    const token = chunk.choices[0]?.delta?.content || ''
    if (token) res.write(`data: ${JSON.stringify({ token })}\n\n`)
  }

  res.write('data: [DONE]\n\n')
  res.end()
})

The client side

const evtSource = new EventSource(`/chat?prompt=${encodeURIComponent(prompt)}`)
evtSource.onmessage = (e) => {
  if (e.data === '[DONE]') { evtSource.close(); return }
  const { token } = JSON.parse(e.data)
  appendToOutput(token)
}

EventSource only supports GET requests with query params, no custom headers, no request body. That’s fine for a prompt in a query string, not fine if you need to send auth headers or a large payload. For those cases, use fetch with a ReadableStream reader instead — more code, but no transport limitation.

Backpressure: the part people skip

If the client is slow to consume — a throttled connection, a background tab — and your server keeps writing chunks as fast as the LLM produces them, Node buffers those writes in memory. On one connection this is nothing. On a few thousand concurrent streams to slow clients, it’s a real memory leak that only shows up under load you didn’t test with.

Respect res.write()’s return value. It returns false when the internal buffer is full:

const ok = res.write(`data: ${JSON.stringify({ token })}\n\n`)
if (!ok) {
  await new Promise(resolve => res.once('drain', resolve))
}

This pauses your loop until the client’s buffer has room, instead of piling tokens up in server memory faster than the network can send them.

What to do when the user closes the tab

This is the case that actually costs you money if you skip it: the user navigates away mid-response, but if you don’t detect that, your server keeps calling the LLM API and burning tokens for a response nobody will ever see.

req.on('close', () => {
  stream.controller?.abort()
})

Listen for the request’s close event and abort the upstream LLM call. Most SDK clients (OpenAI’s included) accept an AbortController signal on the request — wire it through so closing the tab actually stops billing, not just stops rendering.

Reconnection

Plain EventSource auto-reconnects on a dropped connection, but it starts the request over from scratch — there’s no built-in resume-from-token. If a flaky connection matters for your use case, track the last received token index server-side per request ID, and let the client send that index on reconnect so you can resume rather than restart the whole generation.

The one thing to test before shipping

Throttle your own connection (Chrome DevTools → Network → Slow 3G) and watch memory on the server while streaming. If it climbs and doesn’t come back down after the response ends, you’re missing the backpressure handling above, not something more exotic.