Nuances of Reading Files in Node.js

Manipulating files using the fs module is a very common operation in Node.js. However, in high-throughput scenarios, you should be extremely careful with any I/O operations. For example, here is a very common piece of code for reading a file:

const fs = require('fs')
fs.readFile('./text.txt', (err, data) => {
  if (!err) {
    console.log(data.toString())
  }
})

This approach works fine for small files, but if the file is too large, it places a heavy burden on the memory footprint. In node.js, the maximum Buffer size is determined based on the platform’s integer pointer size.

console.log(buffer.constants.MAX_LENGTH)
// 4294967296 = 4GB

This means that if a file exceeds 4GB, the code above runs into trouble and node.js will throw an error immediately.

Handling Files with Streams

More experienced developers usually use fs.createReadStream to avoid issues caused by file size. The biggest difference compared to readFile is that streaming lets us break a massive file into multiple chunks, each only tens of kilobytes in size. For web services, transferring data via streams is second nature; streaming even allows browsers to start rendering as soon as partial HTML content arrives, without having to wait for the entire response to finish downloading.

Rewriting it with createReadStream looks like this:

const fs = require('fs')
let data = ''

const stream = fs.createReadStream('./text.txt')

stream.on('data', chunk => {
  data += chunk
})

stream.on('end', () => {
  console.log(data)
})

At first glance, this seems totally fine, and the code runs properly—it might even work for most file-handling tasks. But if you take a closer look at data += chunk, you’ll notice something suspicious. Some developers naturally treat chunk as a string, but the data emitted by the stream is actually a Buffer. Therefore, data += chunk is secretly performing chunk.toString() before concatenating. Reading this, alarm bells should probably be going off for some developers.

That’s right! When dealing with strings, character encoding is paramount. By default, converting a buffer to a string uses UTF-8. As a result, writing data += chunk can lead to corrupted results because UTF-8 uses anywhere from 1 to 4 bytes to represent a single character. To easily demonstrate this, I set highWaterMark to 5 bytes.

// text.txt
這是一篇部落格PO文

const fs = require('fs')
let data = ''
const stream = fs.createReadStream('./text.txt', { highWaterMark: 5 })
stream.on('data', chunk => {
	data += chunk
})
stream.on('end', () => {
	console.log(data)
})

The output is:

這一部落PO

Because each chunk contains at most 5 bytes, chunk.toString() can produce garbled characters since the full byte sequence for a character might not have arrived yet.

The Proper Concatenation Method: Buffer.concat

If you want to handle Buffers correctly, it’s best to use the APIs provided by node.js before converting to a string, avoiding encoding issues altogether.

const fs = require('fs')
let data = []
let size = 0
const stream = fs.createReadStream('./text.txt', { highWaterMark: 5 })

stream.on('data', chunk => {
	data.push(chunk)
  size += chunk.length
})

stream.on('end', () => {
  Buffer.concat(data, size).toString()
})

This avoids any encoding issues. Admittedly, writing it this way is rather cumbersome. If you’re just doing simple analysis, there’s nothing wrong with using readFile or readFileSync directly. However, if you need to analyze large files or achieve high throughput, these nuances are essential to keep in mind. (Side note: then again, at that point, you might just write it in another language.)

Conclusion

Avoid loading entire large files in and out of memory at once; stick to streams whenever possible. When transferring data, operate on Buffer instances directly to increase throughput and avoid unnecessary encoding overhead, while remaining mindful of how encoding operations actually work.