A merchant asks Sidekick which products are almost out of stock. The frontier model that talks to the merchant hands that job to a GraphQL subagent: read the Admin API schema, search the docs, write the query, run it, and return the rows.
Shopify Engineering describes that subagent as context-heavy. It reads about 40 tokens for every one it writes, and a frontier-model API charges for every token in that window. For that one job they post-trained NVIDIA Nemotron 3.5 Lightning into a task expert, then served the weights on their own GPUs.
On this GraphQL task the specialist scores slightly higher than the frontier model, responds 7× faster, and costs about 90% less to serve. This article walks the October 2026 loop that produced those numbers, then names what the frontier-plus-specialist split means for merchant teams.
Introduction
This article teaches the model optimisation flywheel Shopify published on 21 October 2026 in How Shopify turns open models into task experts. Andrew McNamara and Logan Vegna walk the setup steps, the nightly self-healing pass, Blackwell post-training, and the GraphQL scoreboard after six loops.
Merchant decisions about Sidekick as a daily admin layer belong in Shopify Sidekick's Continual Learning Flywheel: What Merchants Should Know. Reviewing a Canvas or Sidekick storefront edit before publish belongs in Shopify Canvas: How to Review Sidekick Theme Changes Before They Go Live. External AI that writes through a connector is a different permission model, covered in Shopify Claude Connector: Setup, Benefits, and Risks.
The job here is narrower. Walk the October architecture, keep every figure on Shopify’s page, and state the merchant implication of a specialist that now owns token-heavy Admin GraphQL.
What Shopify built: a frontier planner plus task-expert subagents
Sidekick still runs on a frontier model that talks to the merchant, plans the work, and calls tools. Focused jobs go to subagents. Several of those subagents now run open models Shopify post-trains for that one task.
The GraphQL subagent is the worked example. When a merchant asks which products are almost out of stock, it reads Shopify’s API schema, searches the docs, writes and runs a query, and returns the results. That read-heavy window is expensive on a frontier API and slow if every request pays frontier latency.
Shopify’s response is a task expert: a post-trained open-model checkpoint they own, serve on their own GPUs, and update when the flywheel produces a better score. Because they control the weights, they also decide when the specialist changes.
| Layer | Frontier planner | GraphQL task expert |
|---|---|---|
| Job in Sidekick | Talk to the merchant, plan the work, choose the tool | Read schema and docs, write and run Admin GraphQL, return results |
| Model in the October write-up | Frontier model API | Post-trained NVIDIA Nemotron 3.5 Lightning |
| GraphQL judge score | 73 | 74.5 after six loops and 61,000 healed examples |
| Median full response (AIPerf replay) | 6.50 seconds | 0.87 seconds |
| Serving cost on that traffic | Frontier API pricing | About 90% lower; 90–98% lower across Shopify’s post-trained models |
| Who owns the weights | The frontier vendor | Shopify, served on Shopify GPUs |
Those rows are Shopify’s measurements for their GraphQL subagent, reported in the Engineering post. They describe how Sidekick is built, not a discount on your plan or a change to the Admin GraphQL contract on your shop.
How they define quality and raise the frontier bar
Every task expert starts with a quality rubric written by product experts. For GraphQL the rubric covers completeness, execution, and response quality. How closely two experts agree on the same conversations sets the bar.
Shopify then tunes an LLM judge with the GEPA and ACE optimisers until the judge agrees with held-out human labels at the same level as those experts. From that point the judge scores every agent change, every candidate model, and every training run.
An open model does not enter until the frontier-model version of the same agent has been pushed as far as it will go. An autoresearch loop proposes one small change at a time to prompts, tool descriptions, or control flow, and keeps the change only when the judge’s score improves. The improved frontier agent is the bar the open model has to beat.
That order matters for anyone reading the scoreboard later. A 74.5 on the GraphQL judge is a win against a raised frontier baseline of 73, after the harness has already been improved. The open model is competing with a tuned agent, not with a raw API demo.
How they choose an open model and measure serving
Shopify skips pretraining and tests open models such as NVIDIA Nemotron inside the real subagent, with the same tools and the same judge, then weighs starting quality, serving speed, and support in NVIDIA Megatron Bridge, the library they train with.
For GraphQL they chose Nemotron 3.5 Lightning, a mixture-of-experts model with three billion of its 30 billion parameters active per token. Out of the box that checkpoint scored 37.7 on the GraphQL judge. The gap to 73 is the work the flywheel has to close.
Serving is measured before post-training, because a model that is too slow for production is not worth the training pass. Shopify benchmarks every workload with NVIDIA AIPerf, an open-source load tester in NVIDIA Dynamo, which replays production traces with synthetic text instead of customer data. Shopify contributed live-response replay, so multi-turn tests hit the server’s prefix cache the way production does.
Those measurements pick the serving stack. For Buyer Profile, a batch job, AIPerf compared vLLM, SGLang, and NVIDIA TensorRT-LLM. TensorRT-LLM did the most work per GPU. In NVIDIA’s measurements, SimGym, Shopify’s shopping simulator, ran 4.5× faster on Blackwell B200 than on H200, at 1.48× lower cost.
Sidekick task-expert flywheel
How self-healing turns failed GraphQL conversations into training data
Self-healing turns a failed production conversation into a training example. Every night the judge scores a sample of production GraphQL conversations, and anything short of a perfect score goes into the loop.
A frontier model acting as critic writes a hint, never the answer. A simulated merchant then reruns the request with that hint against the same store. If the judge passes the rerun, Shopify removes the hint, anonymises the conversation, and adds it to the training set. If the rerun still fails, the critic tries again, up to three times.
Over 86 nightly runs, self-healing repaired 59,535 GraphQL conversations, 69% of the failures it tried. Human annotators corrected the rest, and those corrections join the ground truth that calibrates the judge.
The training set therefore grows from production traffic Shopify already sees, plus a critic pass that refuses to leak the answer into the example. Merchant data in the published write-up is described as anonymised before it lands in training. The loop still depends on conversations that look like real admin GraphQL work: stock questions, schema reads, and queries that either execute or fail in a way the judge can score.
How they post-train on Blackwell and what the GraphQL scores show
Post-training uses NVIDIA Megatron Bridge, which brings standard Hugging Face checkpoints onto NVIDIA’s Megatron Core training library without rewriting the model, and scales across nodes on Nebius by changing settings rather than code.
Blackwell makes each loop cheaper. On the same FP8 post-training job for Shopify’s Analytics model, eight B300 GPUs trained 1.86× faster than eight H200s, using 46% fewer GPU-hours. The judge then scores the new checkpoint and the flywheel starts again.
On GraphQL, the scoreboard after six loops is the claim that matters for Sidekick:
- Out of the box, Nemotron 3.5 Lightning scored 37.7; the frontier model scored 73.
- The first loop took the open model to 61.5.
- After six loops and 61,000 healed examples, the post-trained model scored 74.5.
Speed moved with quality. In an AIPerf replay of production GraphQL traffic, the post-trained Nemotron returned a full response in 0.87 seconds at the median, against 6.50 seconds for the frontier model’s API. The slowest 1% of requests finished 8× sooner. The same checkpoint also beat another post-trained open model of comparable size, which was close at the median and more than twice as slow at p99.
Serving cost on that GraphQL traffic is about 90% lower than the frontier model. Across Shopify’s post-trained models, self-hosting cuts serving cost by 90–98% compared with frontier-model pricing.
McNamara and Vegna close the architecture in one line the rest of the flywheel supports: the frontier model keeps the work it is best at, talking to the merchant and deciding what to call, while token-heavy specialist work moves to open models Shopify owns.
What the specialist split means for merchant teams
Merchant teams feel the product as Sidekick using a specialist for schema-heavy Admin GraphQL, while the frontier model still holds the conversation and the training stack stays inside Shopify.
Three implications follow from the published loop.
Native GraphQL questions get a specialist that can keep improving. Production traffic grows the training set, human corrections sharpen the judge, and each hardware generation makes the next loop cheaper to run. Inventory, product, and order questions that already sit on clean Admin GraphQL fields are the workload this flywheel is built to serve. A lean beauty and skincare catalogue such as Filter by Molly-Mae, built for a focused product set a small team can merchandise, is the shape of data that kind of question can read.
Custom stack work stays outside the flywheel until it appears on a path Sidekick can query. RTR Vehicles is an auto and moto dealer-inventory build: a Shopify-native finder on metaobjects, VIN deduplication, a bi-weekly CSV upload, and GraphQL updates the team owns. Sidekick’s GraphQL specialist can ask native product and inventory questions. Dealer stock that lives in that metaobject model is only as visible as the schema Sidekick is allowed to read. The CSV workflow remains the freshness owner.
Admin GraphQL fluency is now part of how Shopify trains its own assistant. The October post is explicit that the specialist reads the API schema and the docs, then writes the query. Stores that already strain under slow admin paths, the pattern in How to Fix Shopify Operational Friction When Simple Tasks Still Take Too Long, will see Sidekick inherit that shape. A metafield the team cannot find after the admin redesign is the same field a GraphQL specialist will miss or misread.
Shopper-tab agents are a different surface. Sequencing WebMCP against a staff Claude connector belongs in When Shopify WebMCP Matters vs the Claude Admin Connector. The task-expert loop is Shopify’s internal serving architecture for Sidekick subagents, not a merchant-facing model picker.
Conclusion
You can now walk Shopify’s October 2026 task-expert loop: rubric and judge, a raised frontier bar, an open-model choice inside the live subagent, AIPerf serving measurements, self-healing that turned 59,535 GraphQL failures into training rows, and Blackwell post-training that took Nemotron from 37.7 to 74.5 on the GraphQL judge while cutting median response time from 6.50 seconds to 0.87.
The merchant implication is architectural. Sidekick’s conversation still sits on a frontier planner. Token-heavy Admin GraphQL now sits on a specialist Shopify owns and retrains from production traffic. Clean native fields help that specialist. Private apps, CSV-owned inventory, and checkout logic your team maintains still need an owner outside the chat.
If those ops questions already live in a custom app or an integration Sidekick cannot see, Shopify app development and a stack assessment map which answers belong on native GraphQL and which still need a maintainable build.
