Read Very Large File (7+ GB file) in Nodejs

S ometimes, we need to read the very large file in our Node app, at that time it's a very big deal about this large file. We need to write the very optimized code to read this file. In this article, we will see how we can read very large files using NodeJs.

Let download the large file first:

You can download the large file (name planet-latest_geonames.gz) from this download file link. This file is approx 1.6 GB large. After the download, extract the file.

Understand the file:

The downloaded file planet-latest_geonames.gz contains the geo details. Let's see the file details first. Let's open the file and see:

Read Very Large File (7+ GB file) in Nodejs

In the above gif, you can see the file size is more than 7 GB. Let's have a look at what is inside the file planet-latest_geonames?
In the above file, you can see, there are so many data contains following things:

                          name	alternative_names	osm_type	osm_id	class	type	lon	lat	place_rank	importance	street	
                          city	county	state	country	country_code	display_name	west	south	east	
                          north	wikidata	wikipedia	housenumbers
                          

As you can see in the below snapshot:

Read Very Large File (7+ GB file) in Nodejs

If you observe the file (planet-latest_geonames.tsv) carefully, you can see, the data are separated with '/t', so we can distribute the data by '/t' programmatically. It means, we can get many sets of data by distributing with '\t'.
This file is a CSV file, and the data are comma-separated, we can parse this data with any CSV parser module. We will use an npm module called "csv-parser". Now it's time to write the code and read and get the file content.

Write the code to read the large file:

Now, we need to write the optimized code to read the very large file. we'll use the fs module to read the file and we will use csv-parser to parse the file. Because our file containes data in CSV format. We will read this large file using stream. We will create file stream and through this file stream, we will read this large file.

Let's create a file with name read_large_file.js and start write the code and then we will understand the codes.

read_large_file.js:

                            const _ = require('highland');
                            const fs = require('fs');
                            const csv = require('csv-parser');
                            
                            const start = () => {
                                let currentIndex = 0;
                                const stream = _(
                                  fs.createReadStream('./planet-latest_geonames.tsv').pipe(
                                    csv({
                                      separator: '\t',
                                    })
                                  )
                                )
                            .map(data => ({
                                  name:data.name,
                                }))
                            .batch(100)
                            .each(entries => {
                                  stream.pause();
                                  currentIndex += 100;
                                  console.log('Created index :', currentIndex,entries);
                                  stream.resume();
                            })
                            .on('end', () => {
                                console.log('done');
                                process.exit();
                               });
                            };
                            start();     
                           

Install the dependencies:

After creating the file read_large_file.js, we need to install the dependencies by using below command:

npm install fs highland csv-parser

Understand the codes:

Now its time to understand the codes:

                    const _ = require('highland');
                    const fs = require('fs');
                    const csv = require('csv-parser');
                    

In the above codes, we have imported the required dependencies to read the very large file in nodejs.

                      const stream = _(
                        fs.createReadStream('./planet-latest_geonames.tsv').pipe(
                          csv({
                            separator: '\t',
                          })
                        )
                      )
                      

In the above code, we have created a file stream to read the file. It is a must if you are reading a very large file. This is the optimized way to read the very large file in nodejs.
We are using the CSV module to break the content of the file. It required here to break the line and get the data in chunk. Also, we are storing this stream in a variable called stream for further operation like stream.pause() and stream.resume().

                    
                    .map(data => ({
                      name:data.name,
                    }))
                    
                    

In the above codes, we are creating an object with the property "name", and storing the name data from the file in this object. This code will return the array of objects.

                      .batch(100)
                      

The "batch" method will be used to combine the result in a batch, here we are combining the 100 records at a time to process further. You can give it to any number.

                      .each(entries => {
                        stream.pause();
                        currentIndex += 100;
                        console.log('Created index :', currentIndex,entries);
                        stream.resume();
                      })
                      

In the above codes, we are iterating over data, In these codes, we are getting our data and logging this data on the console. This code will print the data. Here we can write our logic to operate the data as per the requirements.

                      .on('end', () => {
                        console.log('done');
                        process.exit();
                       });
                      };
                      

The above codes will be executed at the end of the file read.

Run the code:

Now, we understand the codes, Now it's time to run the code and see the result. To run the above code, run the below command:

node read_large_file.js

And see the result:

Read Very Large File (7+ GB file) in Nodejs

Now you can see in the above gif, how our code is reading the file and getting the data:

Read Very Large File (7+ GB file) in Nodejs

Get the more data from this file:

The file planet-latest_geonames.tsv contains a lot of data, we are fetching only names from this file. We can fetch the following JSON using following codes:

                    .map(data => ({
                      ...data,
                      alternative_names: data.alternative_names.split(','),
                      lon_num: parseFloat(data.lon),
                      lat_num: parseFloat(data.lat),
                      place_rank_num: parseInt(data.place_rank, 10),
                      importance_num: parseFloat(data.importance),
                    }))
                    

instead

                      .map(data => ({
                        name:data.name,
                      }))
                    

After the above code update, you will have the following kind of array of JSON data:

                        {
                          name: 'Saint-Martin-du-Vieux-Bellême',
                          alternative_names: '',
                          osm_type: 'relation',
                          osm_id: '2755211',
                          class: 'boundary',
                          type: 'administrative',
                          lon: '0.5457148',
                          lat: '48.3700295',
                          place_rank: '16',
                          importance: '0.552497198923526',
                          street: '',
                          city: 'Saint-Martin-du-Vieux-Bellême',
                          county: 'Mortagne-au-Perche',
                          state: 'France métropolitaine',
                          country: 'France',
                          country_code: 'fr',
                          display_name: 'Saint-Martin-du-Vieux-Bellême, Mortagne-au-Perche, Orne, Normandie, France métropolitaine, France',
                          west: '0.5149159',
                          south: '48.3328400',
                          east: '0.5785180',
                          north: '48.4072262',
                          wikidata: 'Q846105',
                          wikipedia: 'fr:Saint-Martin-du-Vieux-Bellême',
                          housenumbers: ''
                        }
                    
                      

So you have seen, how we can read the very large file in nodejs.

Conclusion:

In this article, we learned about the very large file and How we can read and get the data from a very large file.