[{"element_type":"alert","content":"\u003cp\u003eAs per usual, this is a story as much as a bit of advice and details of a project.\nIf you just want to use the JQ Webservice, it's here: \u003ca href=\"http://jq.gdenu.fi/\"\u003ehttp://jq.gdenu.fi/\u003c/a\u003e or here: \u003ca href=\"https://jq.good-enough.cloud/\"\u003ehttps://jq.good-enough.cloud/\u003c/a\u003e. Alternatively skip to the example \u003ca href=\"https://adafruit-playground.com/u/tyeth/pages/jq-now-as-a-webservice-filtering-large-json-data#end-result-https-slash-slash-jq-dot-good-enough-dot-cloud-slash-3194187\"\u003e[here 🔗]\u003c/a\u003e\u003c/p\u003e","metadata":{"class":"element row-fluid alert-element build-alert alert-warning","markdown":"\u003cTLDR\u003e As per usual, this is a story as much as a bit of advice and details of a project.\nIf you just want to use the JQ Webservice, it's here: [http://jq.gdenu.fi/](http://jq.gdenu.fi/) or here: [https://jq.good-enough.cloud/](https://jq.good-enough.cloud/). Alternatively skip to the example [[here 🔗]](https://adafruit-playground.com/u/tyeth/pages/jq-now-as-a-webservice-filtering-large-json-data#end-result-https-slash-slash-jq-dot-good-enough-dot-cloud-slash-3194187)","alert_type":"warning"}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/648/original/DALL%C2%B7E_2025-02-28_14.05.23_-_A_concept_illustration_of_jq_%28JSON_processor%29_as_a_funnel_filter__pulling_in_raw_JSON_data_streams_from_the_internet._The_data_flows_through_the_jq_fi.jpg?1740752269","metadata":{"caption":"Image generated by Dall-E, representing 'jq' filtering the internet into a microcontroller"}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003ejq for all...?\u003c/h1\u003e\n\u003cp\u003e\u003cbr\u003eIf you've not heard of or used `jq`, then count yourself lucky, as it is the tool of choice for data scientists up a creek with no paddle... If you find yourself wresting a 2GB JSON file then you'll probably end up using jq to help. See more details of the jq command-line tool at\u0026nbsp;\u003ca href=\"https://jqlang.org/\" target=\"_blank\"\u003ehttps://jqlang.org/\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eUsing jq can be really quick and simple, but sometimes finding the right incantation to return a subset of the data can leave you feeling a bit like you've been learning FFMPEG (another great tool where I found myself deep in a tutorial one weekend).\u003c/p\u003e\n\u003ch1\u003eDon't do it yourself, get AI to write that query!\u003c/h1\u003e\n\u003cp\u003e\u003cbr\u003eFortunately we live in the days of Large Language Models (\"AI\" LLMs), which like nothing more than answering humans need for the correct FFMPEG/jq arguments to solve some \"critical\" problem.\u003cbr\u003eSo don't feel like you need to learn the syntax, just throw your data and desired output format at the machines and ask for the jq syntax to achieve that!\u003c/p\u003e\n\u003cp\u003eHere's an example where I threw a slightly truncated version of the 80kb json at chatGPT and asked for the output format I wanted: \u003ca href=\"https://chatgpt.com/share/67c1c7a6-bf5c-8000-823c-d422593d2ed8\" target=\"_blank\"\u003ehttps://chatgpt.com/share/67c1c7a6-bf5c-8000-823c-d422593d2ed8\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThere's also a jq playground website, allowing you to submit your JSON and \"query\" or fetch the data from a website URL, and it runs entirely in the browser (WASM version of jq) at \u003ca href=\"https://play.jqlang.org/\" target=\"_blank\"\u003ehttps://play.jqlang.org/\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/642/original/image.png?1740747287","metadata":{"caption":""}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003cp\u003eIt's great, but not what I was looking for... I needed something that would allow a microcontroller with very little RAM (random access memory) to get some values from large JSON payloads, without having to download and process all the JSON data just to find a couple of values.\u003c/p\u003e\n\u003cp\u003eMicrocontrollers are usually too underpowered to load a full web browser with javascript and WASM support, so ideally we need something more like a web-service that would allow a simple web request to trigger all the work and return the subset of data we're looking for.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003eWhat about `adafruit_json_stream` on CircuitPython\u003c/h1\u003e\n\u003cp\u003e\u003cbr\u003eWhile testing some AirLift related (PyPortal and MatrixPortal) projects I found the M4 Matrix Portal couldn't consume the New York Times' API for Top Stories due to running out of memory.\u003c/p\u003e\n\u003cp\u003eI was loosely aware of a new library to take a response \"stream\" and request data from the json version of that data stream, without loading all the data into memory first. It's called `\u003ccode\u003eadafruit_json_stream\u003c/code\u003e and should be just what I wanted.\u003c/p\u003e\n\u003cp\u003eUnfortunately it didn't immediately resolve the out of memory error, and introduced a couple of strange things related to streaming data like the data should be in order. This ordered data is not a strict requirement, but makes it easier to facilitate requesting data from the stream forwards of the current position (not in the past section of the response stream).\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003eCommunity Member to the rescue 🔥🚒🧑‍🚒\u003c/h1\u003e\n\u003cp\u003e\u003cbr\u003eJustin Myers (\u003ca href=\"https://github.com/justmobilize\" target=\"_blank\"\u003e@justmobilize on GitHub\u003c/a\u003e) has been reworking a lot of networking things this year, and when I bemoaned the situation and json_stream library in discord he took the opportunity to cast his eye over the json_stream library too, never one to let a coding challenge sit by unanswered.\u003c/p\u003e\n\u003cp\u003eOver that week the json_steam library was fixed up to be more useful and less restrictive, along with squashing a\u0026nbsp;\u003cspan\u003ediscovered \u003c/span\u003e\u003cspan\u003ebug. Yay for open-source (and thanks Justin)!\u003c/span\u003e\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003eImpatience is a virtue?\u003c/h1\u003e\n\u003cp\u003e\u003cbr\u003eI was initially not content with something that may still struggle with certain JSON data (badly ordered or complex nesting), and the fact that microcontrollers have really slow download speeds (technically reasonable maximums - but never realistically experienced) means that no matter how much data you can swallow it still takes too long.\u003c/p\u003e\n\u003cp\u003eI'd been convinced in a previous situation that I would really like something like \u003ccode\u003ejq\u003c/code\u003e as a webservice (REST API) so I could just give a server the URL and jq filter (query). It would fetch the mega JSON using it's blisteringly fast internet connection, and then cut it down using it's mega processor and large amount of RAM, finally sending that filtered data back to me in a nice tidy small package.\u003c/p\u003e\n\u003cp\u003eAnother thing that changed recently was that replit.org (easy way to try a REPL in various languages)\u0026nbsp; had started emailing me about trying their AI assistant to \"build an app\", and claimed it to be very effective.\u003c/p\u003e\n\u003cp\u003eSo that day the stars aligned, and I chucked my problem at the AI agent and let it have a go.\u003c/p\u003e\n\u003cp\u003eDid it work?\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003eLLM's care less than you do, preferring patterns to the correct / desired result (but my brain's similar)\u003c/h1\u003e\n\u003cp\u003e\u0026nbsp;\u003cbr\u003eWell like most LLM's output, sort of...\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eYes it understood the task (win for my communication), but no it chose to combine a nodejs library (jq-web) and webserver (express) with a load of python libraries that caused it no end of compiler warnings.\u003c/p\u003e\n\u003cp\u003eI tried rolling back the conversation and amending the instruction to avoid the issue, but had to tell it about the python stupidity and ask it 3 times to avoid that happening again. \u003cbr\u003eLong story short, I wasted ten minutes guiding an LLM through it's mistakes after it wrote some perfectly reasonable code and then cocked up the config rendering it all useless.\u003c/p\u003e\n\u003cp\u003eOf course nothings perfect, and I had a few extra ideas that I added, along with correcting some assumptions / code choices I disagreed with. Then testing it on Replit was a success! Finally I chucked some data and desired output at chatGPT and it gave me the \u003ccode\u003ejq\u003c/code\u003e \"filter\" (query) I needed:\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003e{results: (.results | map({title}))}\u003c/code\u003e \u003cbr\u003e(This will create a new results object with the old results mapped to just their title field)\u003c/p\u003e\n      \n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"button","content":"http://jq.gdenu.fi/test","metadata":{"class":"btn btn-large btn-block btn-info","button_type":"info","target":"_blank","zip_folder":"false"}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003eCheap / Fast / Reliable Hosting\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003e\u003cbr\u003eAh the triangle of cheap/fast/good, a famous impossible-to-optimise problem. Sure you can optimise for two out of three, but something always suffers...\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eReplit does hosting, yay for them, and they give you free hosting while you work to test on, but it turns itself off after a short period unless you pay to deploy it in a permanent or auto-scaling solution. \u003cbr\u003eThe cost per month seemed prohibitive (starting at $15/month), considering I have a server with 6gb RAM for $10/month hosting lots of insanity and another for $40/year mostly unused, I thought I probably should just use one of them.\u003c/p\u003e\n\u003cp\u003eSo I did! Quickly setup a different user account (avoid admin/root), test the script starts and server responds to jq requests, add a service file to run the script at boot, test, repeat. Job done?\u003c/p\u003e\n\u003cp\u003eNot quite... I have a couple of domains, wanting to use a subdomain of jq for the api, I setup the DNS on the server for the short domain (gdenu.fi), tested that, and then went to setup DNS for the longer more official domain (good-enough.cloud) in cloudflare and thought that would be enough.\u0026nbsp;\u003cbr\u003eOf course the server wasn't ready to expect requests for jq.good-enough.cloud yet, as cloudflare just proxies you on to my server, so I had to also define jq.good-enough.cloud as an alias of jq.gdenu.fi\u003c/p\u003e\n\u003cp\u003eWhy do both domains / all this?\u003c/p\u003e\n\u003cp\u003eWell firstly the cloudflare proxied one is more likely to remain correctly configured, whereas the gdenu.fi domain has it's DNS hosted on itself (and could therefore be wiped out easily - ignoring backups).\u003c/p\u003e\n\u003cp\u003eSecondly, HTTPS takes an incredible amount of additional memory compared to HTTP.\u003cbr\u003e\u003cbr\u003eUnfortunately the internet world has decided no-one should rely on HTTP anymore and as a result provided a mechanism to automatically upgrade connections from HTTP to HTTPS in most situations (like cloudflare does automatically).\u003cbr\u003e\u003cbr\u003eSo having one domain still support HTTP insecure requests seems like a great idea for low-memory microcontrollers, and that is available at \u003ccode\u003e\u003ca href=\"http://jq.gdenu.fi/\"\u003ehttp://jq.gdenu.fi/\u003c/a\u003e\u003c/code\u003e\u0026nbsp;\u003cstrong\u003e [Yes this allows HTTP to HTTPS proxy]\u003c/strong\u003e\u003cbr\u003e\u003cbr\u003eIt's also true that mega long urls can be problematic for flash space consumption, and don't fit in QRcodes well, so having the shorter one work for HTTP is the best of both worlds.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n  \n  \n        \u003ch1\u003e⏩End Result - \u003ca href=\"https://jq.good-enough.cloud/\" target=\"_blank\"\u003ehttps://jq.good-enough.cloud/\u003c/a\u003e\n\u003c/h1\u003e\n\u003cp\u003e\u003cbr\u003eIt could be pretty, it could support HTTP POST (not just GET) requests, and it could allow custom headers to be added to the request, but for now it's good enough.\u003c/p\u003e\n\u003cp\u003eThe interface takes a URL, then a simple comma separated list of fields to remove/ignore.\u003c/p\u003e\n\u003cp\u003eAlternatively the jq query / filter can be specified, (use an LLM to work out the filter) with a linked tutorial, and then the final option of whether to sort the fields/keys or not.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUrl Syntax is:\u003c/strong\u003e /filter?apiKey=DEMO\u0026amp;url=\u003ccode\u003eURL\u003c/code\u003e\u0026amp;jqQuery=\u003ccode\u003efilter like . or use ignoredFields\u003c/code\u003e\u0026amp;ignoredFields=\u003ccode\u003ething1,thing2\u003c/code\u003e\u0026nbsp; \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eI use the form at the main root page \u003ca href=\"https://jq.gdenu.fi/\" target=\"_blank\"\u003ehttps://jq.gdenu.fi/\u003c/a\u003e, or get a url encoded version some other way\u003c/p\u003e\n\u003cp\u003ee.g.\u003cbr\u003e\u003cem\u003eOriginal Url: \u003ca href=\"https://api.github.com/repos/adafruit/circuitpython/commits?per_page=10\" target=\"_blank\"\u003ehttps://api.github.com/repos/adafruit/circuitpython/commits?per_page=10\u003c/a\u003e\u003c/em\u003e\u003cbr\u003e\u003cem\u003eCut down using this filter:\u0026nbsp;\u003ccode\u003e[.[] | {name: .commit.author.name, message: .commit.message}]\u003c/code\u003e\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eBecomes:\u003cbr\u003e\u003ca href=\"http://jq.gdenu.fi/filter?apiKey=DEMO\u0026amp;url=https%3A%2F%2Fapi.github.com%2Frepos%2Fadafruit%2Fcircuitpython%2Fcommits%3Fper_page%3D10\u0026amp;jqQuery=[.[]%20|%20{name:%20.commit.author.name,%20message:%20.commit.message}]\" target=\"_blank\"\u003ehttp://jq.gdenu.fi/filter?apiKey=DEMO\u0026amp;url=https%3A%2F%2Fapi.github.com%2Frepos%2Fadafruit%2Fcircuitpython%2Fcommits%3Fper_page%3D10\u0026amp;jqQuery=[.[]%20|%20{name:%20.commit.author.name,%20message:%20.commit.message}]\u003c/a\u003e\u0026nbsp;[Add \u0026amp;sort=1 if desired]\u003c/p\u003e\n      \n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n        \u003cp\u003eHere's a screenshot of the form (remember I said it could be prettier):\u003c/p\u003e\n      \n\n","metadata":{}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/643/original/image.png?1740750881","metadata":{"caption":""}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch1\u003eAssets:\u003c/h1\u003e\n\u003ch2\u003eCode available at:\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://replit.com/@TyethGundry/SecureJsonFilter\"\u003ehttps://replit.com/@TyethGundry/SecureJsonFilter\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/tyeth/jq-service\"\u003ehttps://github.com/tyeth/jq-service\u003c/a\u003e [needs updating from server copy to include sort]\u003c/p\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"text","content":"\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n        \u003ch2\u003eMore Images I generated for a Hero Image:\u003c/h2\u003e\n      \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/647/original/DALL%C2%B7E_2025-02-28_14.02.35_-_A_concept_illustration_of_jq_%28JSON_processor%29_integrating_with_Express.js_%28Node.js_web_framework%29._The_image_shows_a_JSON_document_or_console_with_jq_.jpg?1740752225","metadata":{"caption":""}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/645/original/DALL%C2%B7E_2025-02-28_14.01.27_-_A_concept_illustration_combining_jq_%28JSON_processor%29_and_Express.js_%28Node.js_web_framework%29._The_image_features_a_JSON_document_with_structured_key-va.jpg?1740752145","metadata":{"caption":"Image generated by Dall-E, representing 'jq' filtering the internet into a microcontroller"}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/644/original/DALL%C2%B7E_2025-02-28_14.06.50_-_A_futuristic_concept_illustration_of_jq_%28JSON_processor%29_acting_as_a_funnel_filter__extracting_and_refining_JSON_data_streams_from_the_internet._The_p.jpg?1740751691","metadata":{"caption":"Image generated by Dall-E, representing 'jq' filtering the internet into a microcontroller"}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/646/original/DALL%C2%B7E_2025-02-28_14.04.20_-_A_concept_illustration_of_jq_%28JSON_processor%29_as_a_funnel_filter__pulling_in_raw_JSON_data_streams_from_the_internet._The_data_flows_through_the_jq_fi.jpg?1740752205","metadata":{"caption":""}},{"element_type":"user_image","content":"https://cdn-learn.adafruit.com/user_assets/assets/000/001/649/original/DALL%C2%B7E_2025-02-28_14.05.51_-_A_concept_illustration_of_jq_%28JSON_processor%29_as_a_funnel_filter__pulling_in_raw_JSON_data_streams_from_the_internet._The_data_flows_through_the_jq_fi.jpg?1740752286","metadata":{"caption":""}}]