shopify - How to read JSONL line-by-line after hitting url in Node.JS?

Solution:

usingaxios you can set the response to be astream, and then using a buildin readline module, you can process your data line by line.

import axios from 'axios'
import { createInterface } from 'node:readline'

const response = await axios.get('https://raw.githubusercontent.com/zaibacu/thesaurus/master/en_thesaurus.jsonl', {
  responseType: 'stream'
})

const rl = createInterface({
  input: response.data
})

for await (const line of rl) {
  // do something with the current line
  const { word, synonyms } = JSON.parse(line)
  console.log('word, synonyms: ', word, synonyms);
}
Copy code

testing this there is barely any memory usage

Didn't find the answer?

Our community is visited by hundreds of Shopify development professionals every day. Ask your question and get a quick answer for free.