jq for all...?
If you've not heard of or used `jq`, then count yourself lucky, as it is the tool of choice for data scientists up a creek with no paddle... If you find yourself wresting a 2GB JSON file then you'll probably end up using jq to help. See more details of the jq command-line tool at https://jqlang.org/
Using jq can be really quick and simple, but sometimes finding the right incantation to return a subset of the data can leave you feeling a bit like you've been learning FFMPEG (another great tool where I found myself deep in a tutorial one weekend).
Don't do it yourself, get AI to write that query!
Fortunately we live in the days of Large Language Models ("AI" LLMs), which like nothing more than answering humans need for the correct FFMPEG/jq arguments to solve some "critical" problem.
So don't feel like you need to learn the syntax, just throw your data and desired output format at the machines and ask for the jq syntax to achieve that!
Here's an example where I threw a slightly truncated version of the 80kb json at chatGPT and asked for the output format I wanted: https://chatgpt.com/share/67c1c7a6-bf5c-8000-823c-d422593d2ed8
There's also a jq playground website, allowing you to submit your JSON and "query" or fetch the data from a website URL, and it runs entirely in the browser (WASM version of jq) at https://play.jqlang.org/
It's great, but not what I was looking for... I needed something that would allow a microcontroller with very little RAM (random access memory) to get some values from large JSON payloads, without having to download and process all the JSON data just to find a couple of values.
Microcontrollers are usually too underpowered to load a full web browser with javascript and WASM support, so ideally we need something more like a web-service that would allow a simple web request to trigger all the work and return the subset of data we're looking for.
What about `adafruit_json_stream` on CircuitPython
While testing some AirLift related (PyPortal and MatrixPortal) projects I found the M4 Matrix Portal couldn't consume the New York Times' API for Top Stories due to running out of memory.
I was loosely aware of a new library to take a response "stream" and request data from the json version of that data stream, without loading all the data into memory first. It's called `adafruit_json_stream and should be just what I wanted.
Unfortunately it didn't immediately resolve the out of memory error, and introduced a couple of strange things related to streaming data like the data should be in order. This ordered data is not a strict requirement, but makes it easier to facilitate requesting data from the stream forwards of the current position (not in the past section of the response stream).
Community Member to the rescue ๐ฅ๐๐งโ๐
Justin Myers (@justmobilize on GitHub) has been reworking a lot of networking things this year, and when I bemoaned the situation and json_stream library in discord he took the opportunity to cast his eye over the json_stream library too, never one to let a coding challenge sit by unanswered.
Over that week the json_steam library was fixed up to be more useful and less restrictive, along with squashing a discovered bug. Yay for open-source (and thanks Justin)!
Impatience is a virtue?
I was initially not content with something that may still struggle with certain JSON data (badly ordered or complex nesting), and the fact that microcontrollers have really slow download speeds (technically reasonable maximums - but never realistically experienced) means that no matter how much data you can swallow it still takes too long.
I'd been convinced in a previous situation that I would really like something like jq as a webservice (REST API) so I could just give a server the URL and jq filter (query). It would fetch the mega JSON using it's blisteringly fast internet connection, and then cut it down using it's mega processor and large amount of RAM, finally sending that filtered data back to me in a nice tidy small package.
Another thing that changed recently was that replit.org (easy way to try a REPL in various languages) had started emailing me about trying their AI assistant to "build an app", and claimed it to be very effective.
So that day the stars aligned, and I chucked my problem at the AI agent and let it have a go.
Did it work?
LLM's care less than you do, preferring patterns to the correct / desired result (but my brain's similar)
Well like most LLM's output, sort of...
Yes it understood the task (win for my communication), but no it chose to combine a nodejs library (jq-web) and webserver (express) with a load of python libraries that caused it no end of compiler warnings.
I tried rolling back the conversation and amending the instruction to avoid the issue, but had to tell it about the python stupidity and ask it 3 times to avoid that happening again.
Long story short, I wasted ten minutes guiding an LLM through it's mistakes after it wrote some perfectly reasonable code and then cocked up the config rendering it all useless.
Of course nothings perfect, and I had a few extra ideas that I added, along with correcting some assumptions / code choices I disagreed with. Then testing it on Replit was a success! Finally I chucked some data and desired output at chatGPT and it gave me the jq "filter" (query) I needed:
{results: (.results | map({title}))}
(This will create a new results object with the old results mapped to just their title field)
Cheap / Fast / Reliable Hosting
Ah the triangle of cheap/fast/good, a famous impossible-to-optimise problem. Sure you can optimise for two out of three, but something always suffers...
Replit does hosting, yay for them, and they give you free hosting while you work to test on, but it turns itself off after a short period unless you pay to deploy it in a permanent or auto-scaling solution.
The cost per month seemed prohibitive (starting at $15/month), considering I have a server with 6gb RAM for $10/month hosting lots of insanity and another for $40/year mostly unused, I thought I probably should just use one of them.
So I did! Quickly setup a different user account (avoid admin/root), test the script starts and server responds to jq requests, add a service file to run the script at boot, test, repeat. Job done?
Not quite... I have a couple of domains, wanting to use a subdomain of jq for the api, I setup the DNS on the server for the short domain (gdenu.fi), tested that, and then went to setup DNS for the longer more official domain (good-enough.cloud) in cloudflare and thought that would be enough.
Of course the server wasn't ready to expect requests for jq.good-enough.cloud yet, as cloudflare just proxies you on to my server, so I had to also define jq.good-enough.cloud as an alias of jq.gdenu.fi
Why do both domains / all this?
Well firstly the cloudflare proxied one is more likely to remain correctly configured, whereas the gdenu.fi domain has it's DNS hosted on itself (and could therefore be wiped out easily - ignoring backups).
Secondly, HTTPS takes an incredible amount of additional memory compared to HTTP.
Unfortunately the internet world has decided no-one should rely on HTTP anymore and as a result provided a mechanism to automatically upgrade connections from HTTP to HTTPS in most situations (like cloudflare does automatically).
So having one domain still support HTTP insecure requests seems like a great idea for low-memory microcontrollers, and that is available at http://jq.gdenu.fi/ [Yes this allows HTTP to HTTPS proxy]
It's also true that mega long urls can be problematic for flash space consumption, and don't fit in QRcodes well, so having the shorter one work for HTTP is the best of both worlds.
โฉEnd Result - https://jq.good-enough.cloud/
It could be pretty, it could support HTTP POST (not just GET) requests, and it could allow custom headers to be added to the request, but for now it's good enough.
The interface takes a URL, then a simple comma separated list of fields to remove/ignore.
Alternatively the jq query / filter can be specified, (use an LLM to work out the filter) with a linked tutorial, and then the final option of whether to sort the fields/keys or not.
Url Syntax is: /filter?apiKey=DEMO&url=URL&jqQuery=filter like . or use ignoredFields&ignoredFields=thing1,thing2
I use the form at the main root page https://jq.gdenu.fi/, or get a url encoded version some other way
e.g.
Original Url: https://api.github.com/repos/adafruit/circuitpython/commits?per_page=10
Cut down using this filter: [.[] | {name: .commit.author.name, message: .commit.message}]
Becomes:
http://jq.gdenu.fi/filter?apiKey=DEMO&url=https%3A%2F%2Fapi.github.com%2Frepos%2Fadafruit%2Fcircuitpython%2Fcommits%3Fper_page%3D10&jqQuery=[.[]%20|%20{name:%20.commit.author.name,%20message:%20.commit.message}] [Add &sort=1 if desired]
Here's a screenshot of the form (remember I said it could be prettier):
Assets:
Code available at:
https://replit.com/@TyethGundry/SecureJsonFilter
https://github.com/tyeth/jq-service [needs updating from server copy to include sort]
This page (jq - now as a webservice! ๐๐ฃ๐ธ๏ธ Filtering large JSON data) was last updated on March 01, 2025.