Streams in the Wild: Handling Data at Scale
In the early days of web development, servers would often read an entire file into memory before sending it to the client. This works fine for small text files or icons. But what happens when thousands of users try to download a 4GB movie file simultaneously? If your server tries to buffer that into RAM, it will crash immediately.
The Solution: Streaming
Streams allow applications to process data piece by piece (chunks), rather than all at once. This is analogous to streaming video on YouTube: you don't wait for the whole movie to download before you start watching. You watch the chunks as they arrive.
Buffers: The Atoms of Node.js
Before understanding streams, you must understand Buffers. A Buffer is a temporary storage spot for a chunk of data that is being transferred from one place to another. It is the raw binary data representation.
const buf = Buffer.from('Hi');
console.log(buf);
// Output: <Buffer 48 69>
// (Hexadecimal representation of 'H' and 'i')The Four Types of Streams
- Readable: A source of data (e.g., `fs.createReadStream`).
- Writable: A destination for data (e.g., `fs.createWriteStream`, `res` in HTTP).
- Duplex: Can be both read from and written to (e.g., TCP sockets).
- Transform: A Duplex stream that modifies data as it is written and read (e.g., `zlib.createGzip`).
✔️ Good Practice (Piping)
const src = fs.createReadStream('big.file');
src.pipe(res);Handles backpressure automatically. Memory efficient.
❌ Bad Practice (Buffering)
fs.readFile('big.file', (err, data) => {
res.end(data);
});Loads entire file into RAM. High crash risk.
Key Takeaway: Always prefer `pipe()` over manual event handling for standard data transfers. It manages the speed difference between the source and the destination (Backpressure) so your memory doesn't overflow.
