Streams are a fundamental concept not only in Node.js but in computer science in general. A stream represents a continuous flow. For example, when we talk about a continuous flow of water, we call it a water stream. In computer science, just like water, data flows through complex networks. So yes (we can say), flowing data is called a stream in CS.

Why do we need streams?

Let’s say you have a 5 GB file that you want to upload to S3 from your frontend to your backend, and then to S3.

One way to do it would be to load the entire 5 GB file into memory on the backend and then push it to S3. But here lies the problem: this solution cannot scale. What if 10 people try to upload at once? You would need a huge server, and it could still fail.

The solution would be to use streams.
Get a stream of the file from the frontend and pipe it from the backend directly to S3.

How does this actually work?

In this example, data is flowing from the frontend to the backend. In Node.js, the incoming HTTP request (req) is a readable stream. On the backend, we just change its direction using pipe() to send it to S3. So you are basically streaming the file to S3 without storing the whole thing in memory.

Streams in Node.js

Node.js provides an abstract API to work with streams of data. It basically provides 4 types of streams:

1. Writable stream – You can use this stream to write data. A good example would be storing an uploaded file from the frontend to the backend.

2. Readable stream – You can use this stream to read streamed data. For example, if you have a large JSON file and you want to perform certain operations on it but cannot load it fully into memory, you use a readable stream.

3. Duplex stream – This stream implements both writable and readable behavior, which makes it very powerful. A good example would be a TCP client-server implementation.

4. Transform stream – This is a special type of duplex stream that takes input and produces output. For example, in a read stream we just read, but in a transform stream we can modify the data while reading and writing it.

There are helper APIs that Node.js provides to help with streaming data.

• pipeline – This is a top-level function that allows you to pipe one stream to another safely and handle errors and backpressure properly.

After reading so far, you might have an idea about streams. But you might be wondering: how does it actually manage everything internally?

Internally, data is handled as buffers (binary data). In Node.js, when you use read or write streams, they store data in an internal buffer. The size of this internal buffer is controlled by the highWaterMark value. For many streams it is 16 KB by default, but for file streams like fs.createReadStream() it is usually 64 KB.

Let’s take the same example where we send a 5 GB file from the frontend to the backend and then to S3. Everything works using streams.

First, you get a stream from the frontend. This is a readable stream because the browser does not send all the data at once. When Node.js receives data, it stores chunks in its internal buffer. It emits a data event when chunks are available (in flowing mode), and you can listen to that event.

If the writable destination is slower and the internal buffer reaches its limit, Node.js applies backpressure and waits until the buffer is drained before continuing. In real-world scenarios, when you pipe the incoming stream directly to S3, data is continuously drained and forwarded, so memory usage stays controlled.

If you use pipeline(), you don’t need to manage this manually — it handles backpressure and errors for you automatically.

There are many interesting things about streams, and we only talked briefly about some of them. You can read more about streams in the official Node.js documentation as well.

If you find anything wrong or if I made any mistakes, I’m open to suggestions. Feel free to point them out.

• Buffer is a Node.js core module that works with raw binary data.
• We can directly deal with the buffer using the Buffer module.

Why Node.js needs Buffers for binary data?

• Everything on the internet is in binary format, so we need something to deal with binary data. Here, the buffer comes into play and saves the day.

Creating Buffer

We can create the buffer using a 3-way, which are as follows:-

• Buffer.from()

We can create the new buffer using a string, an array, or the buffer itself as parameters to Buffer.from() object.

// 1. From a string

const buf1 = Buffer.from('Hello, Buffer')
console.log(buf1) // <Buffer 45 78 6c ..>
console.log(buf1.toString()) // Hello, Buffer

// 2. From an array

const buf2 = Buffer.from([72, 101, 108, 108, 111])
console.log(buf2.toString()) // Hello

// 3. From another buffer (copying it)

const buf3 = Buffer.from(buf1);
console.log(buf3.toString()) // Hello, Buffer

• Buffer.alloc() and Buffer.allocUnsafe()

We use this function to create a new Buffer of a given size.

1. From Buffer.alloc()

const buf4 = Buffer.alloc(10)
console.log(buf4) // <Buffer 00 00 00  00 00 ...>

2. From Buffer.allocUnsafe()
const buf5 = Buffer.allocUnsafe(10)
console.log(buf5); // <Buffer e8 91 ... random data>

When to use alloc() or allocUnsafe()?

• alloc()
  • Slower because it initializes memory with zeros.
  • Safer, since no old data leaks (NodeJS clears/cleans memory before assigning).
• allocUnsafe()
  • Faster because it skips initialization (it skips the zero-filling step).
  • Unsafe, because the buffer may contain old memory values until overwritten (NodeJS does not clear/clean memory before assigning).

It depends on the situation, what to use if you need speed and don't need to care about safety use allocUnsafe or use alloc

NOTE:
Don't use new Buffer() – it’s deprecated and unsafe.
Use Buffer.from(), Buffer.alloc(), or Buffer.allocUnsafe() instead.

Reading & Writing Data in Buffers

• Accessing Bytes

Each element in the buffer is a byte (0-255).

const buf = Buffer.from("Hello");
console.log(buf[0]);  // 72 (ASCII for 'H')
console.log(buf[1]);  // 101 (ASCII for 'e')

• Converting Buffer to String

Buffers can easily convert back and forth with strings, using encodings.

const buf = Buffer.from("Hello, world!", "utf8");

console.log(buf.toString("utf8"));   // "Hello, world!"
console.log(buf.toString("hex"));    // "48656c6c6f2c20776f726c6421"
console.log(buf.toString("base64")); // "SGVsbG8sIHdvcmxkIQ=="


const buf1 = Buffer.from("Hello", "utf8");
const buf2 = Buffer.from("48656c6c6f", "hex");
const buf3 = Buffer.from("SGVsbG8=", "base64");

console.log(buf1.toString()); // Hello
console.log(buf2.toString()); // Hello
console.log(buf3.toString()); // Hello

Tips:

• You can perform array-like operations on a buffer, such as slice(), copy(), concat(), etc.
• Buffers are widely used in networking (TCP sockets), file I/O, and exchanging data across the web where raw binary data is required.

Conclusion

That’s all for Buffers in Node.js.
We learned how to create buffers, read/write data, and convert between different encodings.

This is just the beginning! In the upcoming articles, we’ll gonna learn about Node.js core modules and eventually put everything together into a real-world project.

Stay tuned, and follow along for more advanced stuff.
Till then, happy coding & goodbye.

Tech Stack

• Turborepo – for monorepo orchestration

• NestJS – backend and APIs (Auth, Upload, Queue Management)

• React – frontend (upload form, progress viewer)

• PostgreSQL + Prisma – database and ORM

• BullMQ – job queue and worker system

• Docker – isolated video processing environment

• AWS S3 – for video storage (input & output)

Architecture Overview

The overall architecture follows this flow:

1. User uploads a video via the frontend → backend receives it via NestJS.

2. The backend uploads the raw video to AWS S3 and saves metadata in PostgreSQL.

3. A new job is enqueued in BullMQ for video processing.

4. A worker service picks the job and spins up a Docker container with:

   • S3 video URL

   • AWS credentials

5. Inside the Docker container:

   • The video is downloaded from S3

   • Converted to .m3u8 using FFmpeg with multiple formats and auto-bitrate

   • The processed folder is uploaded back to S3

   • master.m3u8 URL is logged via stdout

6. The worker listens to Docker logs, extracts the master.m3u8 URL, and updates the database.

Everything is designed to be fully decoupled, scalable, and cloud-native.

Challenges Faced

1. Choosing the Right Queue System

At first, choosing which queue system to use was frustrating. I didn’t want the overhead of Kafka or RabbitMQ just to manage basic jobs. I needed a simple, reliable, and Node.js-friendly solution.

I chose BullMQ — it offers Redis-based queues with good developer experience and async/await support.

2. Video Processing Inside Docker

Running FFmpeg inside Docker was a challenge. Some public images worked partially, but weren’t customizable or too heavy.

I built my own lightweight Docker image optimized specifically for FFmpeg and S3 integration. This allowed full control, faster spin-up, and smaller footprint.

3. Uploading from Inside Docker & Updating the Database

Uploading to S3 inside Docker is straightforward — but there's a twist:

• We didn’t want to download the video on the main server

• Docker doesn’t have access to the DB directly

• We couldn’t easily "return" data from Docker

Solution: Instead of returning the processed URL via API or database, I made the Docker container log the master.m3u8 URL.
The worker listens to stdout, parses logs, and when a specific log is found (e.g., HLS_READY: <URL>), it updates the DB.
This lightweight pattern was clean, effective, and flexible.

Key Lessons Learned

• Turborepo helped me manage shared types, interfaces, and services across multiple apps (frontend, backend, workers).

• Docker is powerful but can be tricky when communicating with services outside of its context.

• FFmpeg is a beast — combining formats, bitrates, and stream maps takes time and testing.

• Streaming logs and designing your own communication protocols (like log-based status updates) can be extremely useful in decoupled systems.

• BullMQ is enough for most video processing workloads unless you hit extreme scale.

What's Next?

Here are a few future improvements I’m planning:

• Add retry & failure queue handling in BullMQ

• Better job status dashboard with real-time updates

• CDN integration for fast HLS delivery

• Auth + token-based video access control

• Support for more formats (e.g., audio-only, 4K rendering)

Project Links

• Main Project Repository: GitHub – video_streaming

• Custom Docker Image Source: GitHub – ffmpeg_docker

• Docker Image (Public): Docker Hub – abhishekshivale21/ffmpeg