Close-up of water droplets on a textured surface, standing in for Shopify Sidekick task experts as isolated specialist models around a frontier planner

Sidekick's stock answers now come from a specialist Shopify owns

Sidekick's stock answers now come from a specialist Shopify owns

Oct 26, 2026

Octavian Contis 9 minutes

Share

A merchant types: which products are almost out of stock?

The frontier model still talking to them hands that job to a GraphQL subagent: read the Admin API schema, search the docs, write the query, run it, and return the rows.

That handoff used to be expensive. Shopify Engineering describes the GraphQL subagent as context-heavy. It reads about 40 tokens for every one it writes, and a frontier-model API charges for every token in that window.

For that one job they post-trained NVIDIA Nemotron 3.5 Lightning into a task expert, then served the weights on their own GPUs.

On this GraphQL task the specialist scores slightly higher than the frontier model, responds 7× faster, and costs about 90% less to serve.

Call it the specialist split.

Introduction

The October write-up is the architecture for that split: rubric and judge, a raised frontier bar, the open-model choice, self-healing, Blackwell post-training, and the GraphQL scoreboard after six loops. Merchant decisions about Sidekick as a daily admin layer live in Shopify Sidekick's Continual Learning Flywheel: What Merchants Should Know. Reviewing a Canvas or Sidekick storefront edit before publish lives in Shopify Canvas: How to Review Sidekick Theme Changes Before They Go Live. External AI that writes through a connector is a different permission model, covered in Shopify Claude Connector: Setup, Benefits, and Risks.

Shopify published the loop on 21 October 2026 in How Shopify turns open models into task experts. Andrew McNamara and Logan Vegna adapted it from their NVIDIA GTC Berlin 2026 talk. GraphQL is their worked example.

What is the specialist split?

A frontier model still talks to the merchant, plans the work, and chooses the tool. Focused jobs go to subagents. Several of those subagents now run open models Shopify post-trains for one task.

Picture a floor manager who still greets the customer. Beside them sits a picking desk that only ever walks the warehouse. The manager plans. The desk fetches stock. That is the specialist split in one shop-floor picture.

A task expert is a post-trained open-model checkpoint they own, serve on their own GPUs, and update when the flywheel produces a better score. Because they control the weights, they also decide when the specialist changes.

LayerFrontier plannerGraphQL task expert
Job in SidekickTalk to the merchant, plan the work, choose the toolRead schema and docs, write and run Admin GraphQL, return results
Model in the October write-upFrontier model APIPost-trained NVIDIA Nemotron 3.5 Lightning
GraphQL judge score7374.5 after six loops and 61,000 healed examples
Median full response (AIPerf replay)6.50 seconds0.87 seconds
Serving cost on that trafficFrontier API pricingAbout 90% lower; 90 to 98 percent lower across Shopify’s post-trained models
Who owns the weightsThe frontier vendorShopify, served on Shopify GPUs

Those rows are Shopify’s measurements for their GraphQL subagent. They describe how Sidekick is built. Your shop’s Admin GraphQL contract stays the same.

How do they raise the bar before the open model walks in?

Every task expert starts with a quality rubric written by product experts. For GraphQL the rubric covers completeness, execution, and response quality. How closely two experts agree on the same conversations sets the bar.

Shopify then tunes an LLM judge with the GEPA and ACE optimisers until the judge agrees with held-out human labels at the same level as those experts. From that point the judge scores every agent change, every candidate model, and every training run.

An open model waits. First an autoresearch loop raises the frontier-model version of the same agent. It proposes one small change at a time to prompts, tool descriptions, or control flow. It keeps the change only when the judge’s score improves.

The open model has to beat a raised bar.

That is why 74.5 against 73 reads as a win after the harness has already been improved. The open model is competing with a tuned agent rather than a raw API demo.

Which open model, and is it fast enough to bother training?

Shopify skips pretraining. They test open models such as NVIDIA Nemotron inside the real subagent, with the same tools and the same judge. They weigh starting quality, serving speed, and support in NVIDIA Megatron Bridge, the library they train with.

For GraphQL they chose Nemotron 3.5 Lightning. It is a mixture-of-experts model with three billion of its 30 billion parameters active per token.

Out of the box that checkpoint scored 37.7 on the GraphQL judge. The frontier model sat at 73. The gap is the work the flywheel has to close.

Serving is measured before post-training. A model that is too slow for production is not worth the training pass. Shopify benchmarks every workload with NVIDIA AIPerf, an open-source load tester in NVIDIA Dynamo. It replays production traces with synthetic text instead of customer data. Shopify contributed live-response replay, so multi-turn tests hit the server’s prefix cache the way production does.

Those measurements pick the serving stack. For Buyer Profile, a batch job, AIPerf compared vLLM, SGLang, and NVIDIA TensorRT-LLM. TensorRT-LLM did the most work per GPU.

In NVIDIA’s measurements, SimGym, Shopify’s shopping simulator, ran 4.5× faster on Blackwell B200 than on H200. Cost sat 1.48× lower.

Sidekick task-expert flywheel

Quality rubricand LLM judgeRaise the frontieragent firstTest an open modelin the live subagentMeasure servingwith AIPerfSelf-heal failedGraphQL trafficPost-train onBlackwellSidekick task-expert flywheel
Sidekick task-expert flywheel

Will Sidekick learn my store?

Only in a narrow sense.

Self-healing turns a failed production conversation into a training example. Every night the judge scores a sample of production GraphQL conversations. Anything short of a perfect score goes into the loop.

A frontier model acting as critic writes a hint. Never the answer. A simulated merchant then reruns the request with that hint against the same store. If the judge passes the rerun, Shopify removes the hint, anonymises the conversation, and adds it to the training set. If the rerun still fails, the critic tries again, up to three times.

Self-healing ran for 86 nights and repaired 59,535 GraphQL conversations, 69% of the failures it tried. Human annotators corrected the rest. Those corrections join the ground truth that calibrates the judge.

So the specialist improves on conversations that look like native Admin GraphQL. Stock questions. Schema reads. Queries that succeed or fail in a way the judge can score.

A dealer-inventory metaobject, a VIN-deduped CSV, or a private app webhook sits outside that curriculum until Sidekick can query it. Merchant data in the published write-up is described as anonymised before it lands in training.

The specialist inherits your schema. Broken variant titles still return a confident wrong row.

What do the GraphQL scores look like after six loops?

Post-training uses NVIDIA Megatron Bridge. It brings standard Hugging Face checkpoints onto NVIDIA’s Megatron Core training library without rewriting the model. It scales across nodes on Nebius by changing settings rather than code.

Blackwell makes each loop cheaper. On the same FP8 post-training job for Shopify’s Analytics model, eight B300 GPUs trained 1.86× faster than eight H200s. They used 46% fewer GPU-hours. The judge then scores the new checkpoint and the flywheel starts again.

On GraphQL, the scoreboard after six loops is the claim that matters for Sidekick. Out of the box, Nemotron 3.5 Lightning scored 37.7. The frontier model scored 73. The first loop took the open model to 61.5. Six loops later, the healed set held 61,000 examples, and the post-trained model scored 74.5.

Speed moved with quality. In an AIPerf replay of production GraphQL traffic, the post-trained Nemotron returned a full response in 0.87 seconds at the median, while the frontier model’s API sat at 6.50 seconds. The slowest 1% of requests finished 8× sooner.

The same checkpoint also beat another post-trained open model of comparable size. That peer was close at the median and more than twice as slow at p99.

Median GraphQL full-response time in Shopify’s AIPerf replay of production traffic, as reported in the October 2026 Engineering post.

Serving cost on that GraphQL traffic is about 90% lower than the frontier model.

Across Shopify’s post-trained models, self-hosting cuts serving cost by 90 to 98 percent compared with frontier-model pricing.

McNamara and Vegna close the architecture in one line: the frontier model keeps talking to the merchant and deciding what to call, while token-heavy specialist work moves to open models Shopify owns.

What should a merchant team do with the specialist split?

Native GraphQL questions get a specialist that can keep improving, because production traffic grows the training set, human corrections sharpen the judge, and each hardware generation makes the next loop cheaper to run.

Inventory, product, and order questions that already sit on clean Admin GraphQL fields are the workload this flywheel is built to serve. A lean beauty and skincare catalogue such as Filter by Molly-Mae, built for a focused product set a small team can merchandise, is the shape of data that kind of question can read.

Custom stack work stays outside the flywheel until it appears on a path Sidekick can query. RTR Vehicles is an auto and moto dealer-inventory build: a Shopify-native finder on metaobjects, VIN deduplication, a bi-weekly CSV upload, and GraphQL updates the team owns. Sidekick’s GraphQL specialist can ask native product and inventory questions. Dealer stock that lives in that metaobject model is only as visible as the schema Sidekick is allowed to read. The CSV workflow remains the freshness owner.

Admin GraphQL fluency is now part of how Shopify trains its own assistant. The October post is explicit that the specialist reads the API schema and the docs, then writes the query. Slow admin paths, the pattern in How to Fix Shopify Operational Friction When Simple Tasks Still Take Too Long, still shape what the picking desk can fetch. A metafield the team cannot find after the admin redesign is the same field a GraphQL specialist will miss or misread.

Shopper-tab agents are a different surface. Sequencing WebMCP against a staff Claude connector belongs in When Shopify WebMCP Matters vs the Claude Admin Connector. The task-expert loop is Shopify’s internal serving architecture for Sidekick subagents. It is not a merchant-facing model picker.

Clean native fields help the picking desk. Custom stack work still needs an owner on the floor.

If the ops question already lives on product and inventory fields, keep those fields tidy so the task expert can read them. If the truth still lives in a private app or a CSV the team uploads on its own cadence, name an owner for that path. Sidekick chat leaves that work on the floor.

Conclusion

The specialist split on the merchant side is now practical. Native GraphQL questions now sit on a specialist Shopify owns and retrains from production traffic. Custom apps, CSV-owned inventory, and checkout logic your team maintains still need an owner outside the chat.

When those answers still live in a custom app Sidekick cannot see, Shopify app development and a stack assessment map which questions belong on native GraphQL and which still need a maintainable build.

Frequently Asked Questions

A task expert is an open model Shopify post-trains for one Sidekick subagent job, then serves on its own GPUs. The frontier model still talks to the merchant, plans the work, and decides which tool to call. Token-heavy jobs such as Admin GraphQL then move to the specialist. Shopify Engineering’s October 2026 write-up uses NVIDIA Nemotron 3.5 Lightning as the GraphQL example.

Shopify skips pretraining. Product experts write a scoring rubric, an LLM judge is tuned until it matches held-out human labels, and an autoresearch loop first raises the frontier-model version of the agent. An open model is then tested inside the same subagent with the same tools and judge. Failed production conversations go through self-healing, anonymised examples join the training set, and NVIDIA Megatron Bridge post-trains the checkpoint on Blackwell GPUs.

Shopify tested open models inside the real GraphQL subagent and weighed starting quality, serving speed, and support in NVIDIA Megatron Bridge. For that job they chose Nemotron 3.5 Lightning, a mixture-of-experts model with three billion of its 30 billion parameters active per token. After six flywheel loops the post-trained checkpoint scored 74.5 on their GraphQL judge, slightly above the frontier model’s 73, while returning a median response in 0.87 seconds against 6.50 seconds for the frontier API.

Each night a calibrated judge scores a sample of production GraphQL conversations. Anything short of a perfect score goes to self-healing. A frontier model acting as critic writes a hint, never the answer, and a simulated merchant reruns the request against the same store. If the judge passes, Shopify removes the hint, anonymises the conversation, and adds it to the training set. The critic may try up to three times. Over 86 nightly runs, self-healing repaired 59,535 GraphQL conversations, 69 percent of the failures it tried. Human annotators corrected the rest.

The serving numbers describe Shopify’s model stack, not a new Admin GraphQL contract on your shop. The specialist still reads the store schema, writes a query, and returns results through the same API surface merchants already use. Catalogue quality, metafield shape, and inventory locations still decide whether the answer matches the warehouse. Custom apps and private admin tools stay outside that loop unless they appear on paths Sidekick can query.

Shopify is compressing time and cost on token-heavy native jobs such as Admin GraphQL. A dealer-finder metaobject model, a CSV inventory upload, or a checkout Function still lives in the stack your team maintains. Sidekick can answer a stock question from native product and inventory fields when those fields are clean. Workflows that write through a private app, an ERP, or a custom admin tool still need Shopify app development and a documented owner.

Andrew McNamara and Logan Vegna published How Shopify turns open models into task experts on the Shopify Engineering blog on 21 October 2026. The piece is adapted from their NVIDIA GTC Berlin 2026 talk. Figures in this article come from that post: GraphQL judge scores, AIPerf latency, self-healing counts, and Blackwell training comparisons.

Related Articles