Server Sent Events(SSE): The Backbone of LLM Token Streaming

Search for a command to run...

No comments yet. Be the first to comment.
Here's a full deep-dive on the SWE-bench paper: PAPER X-RAY β Title : SWE-bench: Can Language Models β β Resolve Real-World GitHub β

This is a Claude-generated summary! PAPER X-RAY β Title : SWE-agent: Agent-Computer β β Interfaces Enable Automated β β

Paper: Zhang, Kraska & Khattab β MIT CSAIL, January 2026Code: github.com/alexzhang13/rlm TLDR; From first principles β before RLMs, performance degradation over large contexts was a known issue. RLM

The core question is: what makes zero-shot retrieval fail, and what would fix it? Let me build up the intuition step by step.The root problem: A user query like "how do I fix a leaky pipe?" and a docu
From first principles to production-grade architecture.
Covers: protocol internals, Express server setup, multi-client forwarding, debugging, CORS, buffering pitfalls, and Windows quirks.
Server-Sent Events (SSE) is an HTTP/1.1 feature that lets a server push a continuous stream of UTF-8 text events to a client over a single, persistent connection β without the client polling.
The browser exposes this through the EventSource API. Once you new EventSource('/endpoint'), the browser fires onmessage every time the server calls res.write(...).
You need server β client push only (live feeds, notifications, progress bars, logs).
You want automatic reconnection for free (browser EventSource handles it).
You need something simpler than WebSockets β SSE rides plain HTTP, works through corporate proxies, requires no upgrade handshake, and doesn't need a special library on the client.
You need bidirectional real-time communication β use WebSockets.
You need to push binary data (images, audio) β use WebSockets or chunked HTTP.
You have >6 tabs open on the same domain over HTTP/1.1 (browser cap) β upgrade to HTTP/2 or use WebSockets.
SSE is dead simple β it's plain text over a persistent HTTP connection. You only need to know four things:
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
These tell the browser: "Don't buffer this, don't cache it, and keep the TCP socket open."
Every event is one or more field lines, terminated by a blank line (\n\n):
data: your payload here\n\n
Multi-line payloads:
data: line one\n
data: line two\n\n
Named events (the client must use addEventListener, not onmessage):
event: update\n
data: {"status":"ok"}\n\n
Event IDs (allow the client to resume from a specific point on reconnect):
id: 42\n
data: {"msg":"checkpoint"}\n\n
Retry hint (tells the browser how long to wait before reconnecting, in ms):
retry: 3000\n
data: {"msg":"reconnect in 3s"}\n\n
| Rule | Correct | Broken |
|---|---|---|
| Space after colon | data: payload |
data:payload |
| Double newline to end block | ...\n\n |
...\n |
| No binary content | UTF-8 strings only | Buffer objects |
Last-Event-ID HeaderOn reconnect, the browser automatically sends:
Last-Event-ID: 42
Your server can read req.headers['last-event-id'] and resume streaming from that point β enabling lossless reconnection.
import express from "express";
const app = express();
app.get("/stream-content", (req, res) => {
// Step 1: Set the three mandatory SSE headers
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
let counter = 1;
// Step 2: Push an event every second
const interval = setInterval(() => {
res.write(`data: chunk object ${counter++}\n\n`);
}, 1000);
// Step 3: Clean up when the client disconnects
req.on("close", () => {
clearInterval(interval);
res.end();
});
});
app.listen(4000, () => console.log("SSE Server running on port 4000"));
Why req.on('close') not res.on('close')?req represents the incoming connection. When the browser navigates away or closes the tab, Node.js emits close on the request object. Using req.on('close') is the canonical pattern β always clean up your timers and database subscriptions here to prevent memory leaks.
const interval = setInterval(() => {
const payload = JSON.stringify({
timestamp: new Date().toISOString(),
value: Math.random() * 100,
});
res.write(`data: ${payload}\n\n`);
}, 1000);
On the client:
es.onmessage = (event) => {
const { timestamp, value } = JSON.parse(event.data);
console.log(timestamp, value);
};
// Server
res.write(`event: priceUpdate\ndata: {"symbol":"BTC","price":67000}\n\n`);
res.write(`event: alert\ndata: {"level":"warn","msg":"High load"}\n\n`);
// Client β onmessage will NOT fire for named events
es.addEventListener("priceUpdate", (e) => {
const { symbol, price } = JSON.parse(e.data);
});
es.addEventListener("alert", (e) => {
const { level, msg } = JSON.parse(e.data);
});
Common trap: If your server sends
event: update\n, but your client only hases.onmessage, nothing logs.onmessageonly fires for unnamed (default) events.
app.get("/stream-content", (req, res) => {
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
// Check if client is resuming
const lastId = parseInt(req.headers["last-event-id"] ?? "0", 10);
let counter = lastId + 1;
const interval = setInterval(() => {
res.write(`id: \({counter}\ndata: event \){counter}\n\n`);
counter++;
}, 1000);
req.on("close", () => {
clearInterval(interval);
res.end();
});
});
EventSource API)Works in all modern browsers with zero dependencies:
const es = new EventSource("http://localhost:4000/stream-content");
es.onopen = () => console.log("Connection established");
es.onmessage = (event) => {
// event.data is always a string
try {
const parsed = JSON.parse(event.data);
console.log("Received:", parsed);
} catch {
console.log("Raw string:", event.data);
}
};
es.onerror = (err) => {
// EventSource AUTOMATICALLY reconnects β onerror is informational only
console.warn("SSE error (will reconnect):", err);
};
// Shutdown when done
// es.close();
eventsource npm PackageFor a Node.js backend that needs to consume another server's SSE stream:
npm install eventsource
import EventSource from "eventsource";
const es = new EventSource("https://api.example.com/events", {
headers: {
Authorization: "Bearer YOUR_TOKEN",
"X-Client-ID": "my-service",
},
});
es.onopen = () => console.log("Stream connected");
es.onmessage = (event) => {
const data = JSON.parse(event.data);
console.log("Received:", data);
};
es.addEventListener("update", (event) => {
console.log("Named event 'update':", event.data);
});
es.onerror = (err) => {
if (es.readyState === EventSource.CLOSED) {
console.error("Connection closed permanently");
}
};
fetch + Web Streams (Node 18+ / Browser)No external dependencies. Ideal when you need fine-grained control:
async function consumeSSENatively(url, token) {
const response = await fetch(url, {
headers: {
Authorization: `Bearer ${token}`,
Accept: "text/event-stream",
},
});
if (!response.ok || !response.body) {
throw new Error(`HTTP ${response.status}`);
}
const reader = response.body
.pipeThrough(new TextDecoderStream())
.getReader();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
const lines = buffer.split("\n");
buffer = lines.pop() ?? ""; // keep last incomplete line
for (const line of lines) {
const trimmed = line.trim();
if (trimmed.startsWith("data:")) {
const raw = trimmed.replace(/^data:\s*/, "");
try {
console.log("Parsed:", JSON.parse(raw));
} catch {
console.log("Raw:", raw);
}
}
}
}
}
Why the buffer trick? TCP delivers data in arbitrary chunks. A single reader.read() call may give you half an event block. The buffer accumulates bytes until a complete \n-terminated line exists before parsing.
| Feature | Browser EventSource | eventsource npm |
Native fetch |
|---|---|---|---|
| Auto-reconnect | β Built-in | β Built-in | β Must write your own |
| Named events | β
addEventListener |
β
addEventListener |
β Must parse event: line manually |
| Custom headers | β Not supported | β Supported | β Supported |
| Auth tokens | β (use cookies/query param) | β | β |
| Dependencies | None | 1 package | None |
| Node.js support | β | β | β (v18+) |
A real-world pattern: your backend subscribes to an upstream SSE stream and fans it out to many browser clients. This is a SSE proxy.
import express from "express";
import EventSource from "eventsource";
const app = express();
// Registry of connected browser clients
const clients = new Set();
// 1. Connect once to the upstream SSE source
const upstream = new EventSource("https://upstream-api.com/events", {
headers: { Authorization: "Bearer UPSTREAM_TOKEN" },
});
upstream.onmessage = (event) => {
// 2. Fan out every upstream event to all connected clients
const message = `data: ${event.data}\n\n`;
for (const client of clients) {
client.write(message);
}
};
upstream.onerror = (err) => {
console.error("Upstream SSE error:", err);
};
// 3. Expose the fan-out endpoint to browsers
app.get("/events", (req, res) => {
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
clients.add(res);
console.log(`Client connected. Total: ${clients.size}`);
req.on("close", () => {
clients.delete(res);
console.log(`Client disconnected. Total: ${clients.size}`);
});
});
app.listen(3000);
Key insight: The upstream connection is created once at startup, not per browser client. This prevents N clients from opening N upstream connections β a common and expensive bug.
When events are not logging in your client, work through this list in order:
# Linux / macOS
curl -N -v http://localhost:4000/stream-content
# Windows PowerShell (must use curl.exe, not the built-in alias)
curl.exe -N -v http://localhost:4000/stream-content
Why -N? This disables curl's internal output buffer. Without it, curl queues all received bytes until the connection closes β your terminal stays blank even though the server is sending data perfectly.
Expected output:
< HTTP/1.1 200 OK
< Content-Type: text/event-stream
< Cache-Control: no-cache
<
data: chunk object 1
data: chunk object 2
If data streams here but not in your app β the server is fine; the problem is in your client code (wrong event name, JSON parse crash, CORS).
If the terminal stays blank β the server is not writing, or a proxy is buffering.
// Server sends:
res.write(`event: priceUpdate\ndata: {"p":100}\n\n`);
// Client must use addEventListener, NOT onmessage:
es.addEventListener("priceUpdate", handler); // β
es.onmessage = handler; // β never fires for named events
// Server sends plain text: "chunk object 1"
// Client tries:
const parsed = JSON.parse(event.data); // β SyntaxError β crashes silently
// Fix: always guard
es.onmessage = (event) => {
try {
const parsed = JSON.parse(event.data);
console.log(parsed);
} catch {
console.log("Raw:", event.data); // β
logs the plain string
}
};
If your browser client is on localhost:3000 and your server is on localhost:4000, the browser blocks the request unless the server sends CORS headers:
Access-Control-Allow-Origin: http://localhost:3000
See Section 7 for the full fix.
Browsers cap persistent connections per origin at 6 over HTTP/1.1. If you have 6+ tabs open to the same SSE endpoint, any new tab will open the EventSource, appear to connect (no error), but receive zero events.
Fix: Use HTTP/2 (which multiplexes unlimited streams over one TCP connection), or close extra tabs during development.
npm install cors
import express from "express";
import cors from "cors";
const app = express();
// Allow any origin (development only)
app.use(cors());
// Production: restrict to your frontend domain
app.use(cors({
origin: "https://your-frontend.com",
methods: ["GET"],
}));
app.get("/stream-content", (req, res) => {
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
// ... your event loop
});
Note: The
cors()middleware must come before your route handlers, not after.
If your client passes an Authorization header (which browser EventSource cannot do β use the eventsource npm package or fetch instead), you also need:
app.use(cors({
origin: "https://your-frontend.com",
allowedHeaders: ["Authorization"],
}));
compression MiddlewareIf you use the compression package, it buffers output waiting for enough data to compress efficiently. Your events get stuck in that buffer.
Fix: manually flush after each write:
import compression from "compression";
app.use(compression());
app.get("/stream-content", (req, res) => {
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
const interval = setInterval(() => {
res.write(`data: ping\n\n`);
if (typeof res.flush === "function") res.flush(); // β force flush
}, 1000);
req.on("close", () => {
clearInterval(interval);
res.end();
});
});
Nginx buffers proxy responses by default. Add these headers from your Express server to opt out:
res.setHeader("Cache-Control", "no-cache, no-transform");
res.setHeader("X-Accel-Buffering", "no"); // disables nginx proxy_buffering
Or configure Nginx directly:
location /stream-content {
proxy_pass http://localhost:4000;
proxy_buffering off;
proxy_cache off;
proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding on;
}
Cloudflare buffers responses shorter than a few KB before forwarding to clients. SSE events are tiny β they'll be held indefinitely.
Fix: Disable Cloudflare's cache for your SSE route using a Page Rule that sets Cache Level: Bypass, or deploy your SSE endpoint on a subdomain that bypasses the CDN (orange-cloud off in Cloudflare DNS).
Windows PowerShell has a built-in alias curl that points to Invoke-WebRequest β Microsoft's own cmdlet β which does not understand Unix flags like -N or -v.
# β Fails β PowerShell intercepts this as Invoke-WebRequest
curl -N http://localhost:4000/stream-content
# β
Force the real curl executable
curl.exe -N -v http://localhost:4000/stream-content
Alternatively, check if the built-in curl alias is shadowing:
Get-Command curl
# If this returns "Alias" β you're using Invoke-WebRequest
# If it returns "Application" pointing to curl.exe β you're fine
You can also remove the alias permanently in your PowerShell profile:
Remove-Item Alias:curl -Force
Browser β Express SSE Server β setInterval / DB query
Best for: dashboards, log tails, progress indicators. No extra infrastructure.
ββββββββ Browser client 1
Producer β Redis/MQ β SSE Server βββ Browser client 2
ββββββββ Browser client 3
Your Express server subscribes to a Redis pub/sub channel or a message queue (BullMQ, RabbitMQ). When a message arrives, it broadcasts to all connected SSE clients. This lets you scale horizontally β multiple SSE server instances all subscribe to the same queue.
import { createClient } from "redis";
const subscriber = createClient();
await subscriber.connect();
await subscriber.subscribe("events", (message) => {
for (const client of clients) {
client.write(`data: ${message}\n\n`);
}
});
Upstream SSE API β Your Node.js SSE Proxy β Many browsers
See Section 5. Use when you need to add auth, filter events, enrich payloads, or rate-limit before forwarding to clients.
| Dimension | SSE | WebSockets | Long Polling |
|---|---|---|---|
| Direction | Server β Client only | Bidirectional | Server β Client (inefficient) |
| Protocol | Plain HTTP/1.1 or HTTP/2 | HTTP Upgrade β WS | Plain HTTP |
| Auto-reconnect | β Built into browser | β Must implement | β Must implement |
| Binary data | β | β | β |
| Proxy/firewall friendly | β High | β οΈ Moderate | β High |
| Setup complexity | Low | Medium | Low |
| Browser support | All modern | All modern | All |
| Multiplexed streams | β With HTTP/2 | Per connection | β |
| Good for | Feeds, notifications, logs, AI streaming | Chat, gaming, collaborative editing | Legacy fallback |
Decision rule:
Need push only + want simplicity? β SSE
Need two-way real-time + binary? β WebSockets
Legacy browser support or strict proxy env? β Long Polling
Every time you see an AI assistant typing out its response word by word β that is SSE. It's not a UI trick. The model genuinely cannot produce the full answer at once: it generates one token per forward pass through its neural network, and SSE is what carries each token to your screen the instant it exists.
A large language model is autoregressive β each token it produces becomes part of the input context used to generate the next one. There is no completed answer to send until the last token is generated. Waiting for completion before responding means the user stares at a blank screen for 10β30 seconds on a long output. SSE solves this by making latency invisible: the first token arrives in ~300ms, and the rest stream in continuously.
The Anthropic API (and OpenAI, Mistral, Gemini β all major LLM providers) sends each token as a JSON-encoded SSE event. A real Anthropic streaming response on the wire:
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"!"}}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"}}
event: message_stop
data: {"type":"message_stop"}
OpenAI-compatible APIs use a simpler format:
data: {"choices":[{"delta":{"content":"Hello"},"index":0}]}
data: {"choices":[{"delta":{"content":"!"},"index":0}]}
data: [DONE]
The [DONE] sentinel: OpenAI-style APIs terminate streams with
data: [DONE]β not valid JSON. Your parser must check for this before callingJSON.parse(), otherwise you get a silent crash and miss the end-of-stream signal.
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
const stream = await client.messages.stream({
model: "claude-sonnet-4-6",
max_tokens: 1024,
messages: [{ role: "user", content: "Explain SSE in one paragraph." }],
});
// Method 1: Higher-level helper (recommended)
for await (const chunk of stream) {
if (
chunk.type === "content_block_delta" &&
chunk.delta.type === "text_delta"
) {
process.stdout.write(chunk.delta.text);
}
}
Your Express server acts as an authenticated SSE proxy, keeping your API key server-side while streaming tokens directly to browser clients.
import express from "express";
import Anthropic from "@anthropic-ai/sdk";
const app = express();
app.use(express.json());
const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env
app.post("/chat", async (req, res) => {
const { message } = req.body;
// Step 1: Set SSE headers on your outgoing response
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
res.setHeader("X-Accel-Buffering", "no"); // disable Nginx buffering
try {
const stream = client.messages.stream({
model: "claude-sonnet-4-6",
max_tokens: 1024,
messages: [{ role: "user", content: message }],
});
// Step 2: Forward each token as its own SSE event
stream.on("text", (text) => {
res.write(`data: ${JSON.stringify({ token: text })}\n\n`);
if (res.flush) res.flush();
});
// Step 3: Signal end of stream
stream.on("finalMessage", () => {
res.write("data: [DONE]\n\n");
res.end();
});
stream.on("error", (err) => {
res.write(`data: ${JSON.stringify({ error: err.message })}\n\n`);
res.end();
});
} catch (err) {
res.write(`data: ${JSON.stringify({ error: "Stream failed" })}\n\n`);
res.end();
}
});
app.listen(4000);
// Browser client
const es = new EventSource("/chat-stream");
es.onmessage = (event) => {
if (event.data === "[DONE]") {
es.close();
hideCursor();
return;
}
const { token } = JSON.parse(event.data);
appendTokenToUI(token);
};
POST vs GET: Browser
EventSourceonly supports GET. For chat interfaces that need to send a message body, usefetchwith a streaming reader, or the@microsoft/fetch-event-sourcelibrary.
| Concern | SSE | WebSocket |
|---|---|---|
| Setup complexity | 3 headers + res.write | Upgrade handshake, ws library, message framing |
| CDN / proxy support | Works with Cloudflare, Nginx (with config) | Requires special proxy config |
| Auto-reconnect | Built into browser | Must implement manually |
| Data direction | Server β client only (correct for LLM output) | Bidirectional (unnecessary complexity) |
| HTTP/2 multiplexing | Many streams over one TCP connection | One connection per stream |
| Adoption | Anthropic, OpenAI, Mistral, Gemini, Cohere | Rare for pure token streaming |
β
Check data === "[DONE]" before JSON.parse() β it is not valid JSON
β
Filter by event: field β only content_block_delta carries text tokens
β
Buffer partial JSON lines in a string before parsing β TCP may split event blocks
β
Set X-Accel-Buffering: no when behind Nginx β prevents token batching
β
Handle onerror and show a "reconnectingβ¦" state β mid-stream drops are common
β
Check stop_reason β max_tokens truncation looks identical to normal completion
β
res.setHeader("Content-Type", "text/event-stream")
β
res.setHeader("Cache-Control", "no-cache")
β
res.setHeader("Connection", "keep-alive")
β
res.write(`data: payload\n\n`) β two newlines mandatory
β
req.on("close", cleanup)
β
if (res.flush) res.flush() β if compression middleware is used
β
res.setHeader("X-Accel-Buffering", "no") β if behind Nginx
β
new EventSource(url) for browser / eventsource pkg for Node
β
es.onmessage for unnamed events
β
es.addEventListener("name", handler) for named events
β
JSON.parse inside try/catch
β
es.onerror to detect connection issues
β
es.close() on component unmount
# Test the raw SSE stream (skip the client entirely)
curl.exe -N -v http://localhost:4000/stream-content # Windows
curl -N -v http://localhost:4000/stream-content # macOS/Linux