Do the children in a Shopify bulk operation JSONL with nested connections come right after their parent?

Question

493 votes

1 answers

Do the children in a Shopify bulk operation JSONL with nested connections come right after their parent?

When parsing a bulk operation JSONL file with nested items from top to bottom line by line, when I reach a new top level parent object, does that mean I've gone through all children of the previous parent?

Context

When processing a bulk operation JSONL file, I do some processing that requires having a parent and all of their children. I'd like to keep my memory requirements as small as possible, so I need to know when I'm done processing an object and all of its children.

Example for clarification

Using the documentation page's (https://shopify.dev/api/usage/bulk-operations/queries#the-jsonl-data-format) JSONL example:

{"id":"gid://shopify/Product/1921569226808"}
{"id":"gid://shopify/ProductVariant/19435458986123","title":"52","__parentId":"gid://shopify/Product/1921569226808"}
{"id":"gid://shopify/ProductVariant/19435458986040","title":"70","__parentId":"gid://shopify/Product/1921569226808"}
{"id":"gid://shopify/Product/1921569259576"}
{"id":"gid://shopify/ProductVariant/19435459018808","title":"34","__parentId":"gid://shopify/Product/1921569259576"}
{"id":"gid://shopify/Product/1921569292344"}
{"id":"gid://shopify/ProductVariant/19435459051576","title":"Default Title","__parentId":"gid://shopify/Product/1921569292344"}
{"id":"gid://shopify/Product/1921569325112"}
{"id":"gid://shopify/ProductVariant/19435459084344","title":"36","__parentId":"gid://shopify/Product/1921569325112"}
{"id":"gid://shopify/Product/1921569357880"}
{"id":"gid://shopify/ProductVariant/19435459117112","title":"47","__parentId":"gid://shopify/Product/1921569357880"}

If I'm reading the file line by line from top to bottom and I hit Product with idgid://shopify/Product/1921569259576 on line 4, does this mean that I've already seen all of the previous product's (gid://shopify/Product/1921569226808) product variants the JSONL file contains?

Undefined asked

2022-11-13

Write quick answer

Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.

Undefined answered · Answer 1 · 2022-11-13

Answer

Solution:

What a lot of people do is shell out and use tac to reverse the file. Then when you parse the file you end up nice processing the children first and then knowing when you hit the parent, you have everything and you can move.

Obviously this is nicer than getting the parent and then the children, and then wondering, have I hit all the children or are there more.

Try it! It works!

My pseudo code (which you can convert to whatever scripting language you want looks like this:

inventory_file = Tempfile.new
inventory_file.binmode
uri = URI(result.data.node.url)
IO.copy_stream(uri.open, inventory_file) # store large amount of JSON Lines data in a tempfile
inventory_file.rewind # move from EOF to beginning of file
y = "#{Rails.root}/tmp/#{shop_domain}.reversed.jsonl"
`tac #{inventory_file.path} > #{y}`
puts "Completed Reversal of file using tac, so now we can quickly iterate the inventory"
f = File.foreach(y)
variants = {}
f.each_entry do |line|
  data = JSON.parse(line)
  # play with my data
end

Undefined answered

2022-11-13

Link to answer

Source

Didn't find the answer?

Our community is visited by hundreds of Shopify development professionals every day. Ask your question and get a quick answer for free.

Ask a Question

Do the children in a Shopify bulk operation JSONL with nested connections come right after their parent?

Answer

Solution:

Didn't find the answer?

Similar questions

Write quick answer