The full menu sync hits the limit before it finishes
In short
A menu sync that times out or applies half a menu is almost never failing on data volume — it is failing on request count, because the job makes one call per item where a batch endpoint exists and runs every location against a shared quota at the same minute. Count the requests per run before changing anything: the number is usually 10 to 40 times what it needs to be.
Key takeaways
- Count requests per sync run per location. That single number diagnoses this faster than any log search.
- A 400-item menu with 1,200 modifier options is 1,600 writes done naively and 16 done in batches of 100.
- Honour the Retry-After header, which carries either a delay in seconds or an HTTP date, and stop guessing your backoff.
- Retrying a whole failed run recreates the load that caused the failure. Retry the failed batch, not the job.
- Change detection is the largest single win: most nightly syncs push a menu that did not change.
The menu is not too large. A restaurant catalogue is small data — a few hundred items, a few thousand modifier options, comfortably under a megabyte as JSON. What fails is the shape of the job: one HTTP request per item, then one per modifier, then the whole tree again for the next location, all of it starting at the same minute against a quota the platform applies across your entire application.
This matters because the obvious fixes are the wrong ones. A longer timeout makes the failure later and larger. More parallelism makes it faster and worse. The number to establish first is how many requests one sync run actually makes, and for most builds the honest answer is that nobody has ever counted.
Count the requests, not the items
- Instrument the sync to emit 1 line per outbound call: location, HTTP method, endpoint, status, and elapsed time. Nothing clever — a counter and a log line.
- Run it once against your largest location and total the calls. Split reads from writes, because platforms frequently meter them differently and a read-heavy job has a different fix from a write-heavy one.
- Compare that total with the platform's published limits for your application tier, read from the platform's own documentation on the day you look. These change, and the limit that applies to a partner application is often not the one on the general developer page.
- Multiply by the number of locations you sync in the same window. If every site starts at 02:00, that product is your real peak, not the per-location figure.
- Count the retries separately. A run that makes 1,600 calls and retries a third of them is making over 2,100, and the retries arrive exactly when the quota is already exhausted.
One call per item, where the platform offers a batch endpoint
Most POS catalogue APIs expose some form of bulk or batch write — a single call carrying many objects, often with an upsert semantic. Integrations written against the reference documentation item by item never find it, because the per-item endpoint is the one that appears first and works immediately in a test with 3 products.
The arithmetic is stark. A 400-item menu with 1,200 modifier options is 1,600 write calls done one at a time. Batched at 100 objects per call it is 16. That is not an optimisation, it is a different job — and it usually turns a sync that could not finish into one that finishes in under a minute. Confirm the batch size ceiling and the object types supported in the platform's own reference, because both vary and both are enforced.
Pushing a menu that did not change
The second largest win, and the cheapest to implement. A nightly full push sends the identical catalogue every night whether or not anything moved, which means the busiest night of your quota year is a Tuesday when nobody edited anything.
- Hash each item. Serialise the item and its modifiers canonically — sorted keys, fixed number formatting — and store the hash against the mapping row. Push only rows whose hash changed.
- Hold the hash per location, not per item. The same product can carry a different price or availability at 2 restaurants, so a single global hash will suppress a change that mattered at 1 site.
- Keep a full push available, and schedule it rarely — weekly, or on demand after a mapping repair. Incremental syncs drift; a periodic full reconciliation is how you find out.
- Log what changed rather than that a sync ran. "18 items updated, 2 created, 0 deleted" is an operational record; "sync complete" is not.
Walking the modifier tree again for every item that shares it
Restaurant menus are dense with shared structure. One modifier group — sizes, milk choices, spice level — is attached to dozens of items, and a naive traversal reads or writes that group once per parent. On a coffee menu this is the single biggest multiplier in the job, and it is invisible in the item count because the explosion happens below the item.
Resolve the graph once per run: build the set of distinct modifier groups and options, sync those, then sync items referencing them by id. If the platform requires the tree to be written with the item, at least deduplicate reads. And expect surprises in how variants and half-and-half constructions map, because a menu structure that looks obvious to a chef often has no equivalent in the POS model — half-and-half orders arriving as whole ones is the same modelling gap seen from the order side.
Every location syncing at the same minute against one quota
Rate limits are usually applied per application, sometimes per merchant, occasionally both. A cron entry that fires all 40 locations at 02:00 turns a comfortable per-location budget into a wall. The fix is scheduling, not engineering.
| What you observe | Underlying cause | The change |
|---|---|---|
| Small menus fine, large menus time out | Per-item writes | Batch endpoint, with the platform's documented batch ceiling |
| Every run costs the same regardless of edits | Full push with no change detection | Per-location content hash on each mapping row |
| Request count far exceeds item count | Shared modifier groups traversed per parent | Resolve distinct groups once per run |
| Failures cluster at one time of night | All locations sharing a quota window | Stagger start times and add jitter per location |
| The run gets slower each time it fails | Whole-job retries stacking on the next attempt | Retry the failed batch with backoff, never the job |
Stagger start times across the service window a restaurant can tolerate — for most sites that is roughly 02:00 to 05:00 local, after close and before prep — and add a random offset per location so that a redeploy does not resynchronise everything onto the same second. Give each location a request budget for the run and have the job stop and report when it exceeds it, rather than pushing on into a throttle.
Retries that recreate the load that caused the failure
The worst version of this problem is self-inflicted. A run fails at 70 per cent, the scheduler retries the whole job, and the retry starts from item 1 — spending the entire quota again to redo work that already succeeded. Three retries and a partially applied menu becomes a location locked out for the rest of the window.
A sync that retries the job instead of the batch is not resilient. It is an amplifier pointed at the limit that just stopped it.
Make the unit of retry the batch, make batches idempotent so a repeat is harmless, and use exponential backoff with jitter so 40 locations do not retry in lockstep. A run that borrows against someone else's window has the same shape as a payment authorisation borrowing against its own expiry — authorise, release, capture — and in both cases the design question is what you do when the allowance runs out, not how to avoid ever running out.
When the limit is genuinely too low for the estate
Sometimes the arithmetic does not close. Once the job is batched, change-detected and staggered, and a full weekly reconciliation across every location still does not fit, the answer is commercial: ask the platform for a higher tier, which is usually available to approved partners and usually requires you to show the traffic profile you have just measured. Bring the request counts, not an assertion.
Two structural questions sit behind this and are worth settling before the next platform is added. Whether a cloud POS and a terminal on the counter should be treated as one integration at all is answered in a cloud POS and a terminal are two different jobs. And missed status changes during service — a different failure from a slow sync — belong to the POS says it sent the event and your system never moved. Reconciling what the POS ended up with is a separate discipline again, shared with your total and the POS total never quite agree.
Building the sync as a scheduled, budgeted, self-reporting job is AI agents and automation work in the unglamorous sense of the term, and "what does your sync do when it gets a 429" is exactly the sort of question that separates teams who have run this in production from teams who have demoed it — one of the tests in choosing an AI development partner. The rest of this silo sits under integrating with the POS on the counter, inside our restaurants and food service practice.
Frequently asked questions
Short answers to the follow-ups this page tends to raise.
Why does a POS menu sync fail only on larger menus?
Because the job scales with request count rather than data size. A per-item sync makes one call per product and one per modifier option, so a 400-item menu with 1,200 options is around 1,600 calls, and somewhere on the way up that crosses the platform's rate limit. The same menu sent through a batch endpoint at 100 objects per call is about 16 requests and finishes easily.
Should we run a full menu sync or an incremental one?
Incremental for the routine, full for reconciliation. Hash each item and its modifiers per location and push only what changed, which removes almost all of the traffic on a typical night. Then schedule a full push weekly or on demand, because incremental pipelines drift and a periodic full run is the only thing that reveals it.
How should a sync handle a 429 from the POS API?
Stop, read the Retry-After header, and wait for what it says. That header carries either a number of seconds or an HTTP date, and honouring it is better than any locally chosen delay. Then retry the failed batch with exponential backoff and jitter — never the whole job, because restarting from the first item spends the quota again on work that already succeeded.
How do we sync 40 restaurants without exhausting the quota?
Stagger and budget. Spread start times across the overnight window with a random per-location offset so a redeploy does not put every site on the same second, and give each run a request budget it must report against rather than silently exceed. Rate limits are commonly applied per application, so 40 locations at 02:00 is one job from the platform's point of view.
- pos
- menu sync
- rate limits
- diagnostics
The work behind this page
Builds from our portfolio that this page draws on.
Read next
- External IDs: the mapping table nobody designs until it breaksThe identity map between your catalogue and the till is the contract that makes every order possible, and 4 routine events break it silently. Design it as a versioned artefact.definition
- The POS says it sent the event and your system never movedA status event that arrives after service is worth nothing. Five places it dies before your handler, and why the durable fix is a polling sweep over open orders rather than better webhook code.diagnostic
- It works at the first location and returns an auth error at the other fourIdentical code, one store working and four rejecting, means authorisation was granted per merchant location and only completed once. The fix is an inventory, not a retry.diagnostic
- Order source and dining option: two fields your reporting rests onTwo small POS fields decide whether an injected order is taxed, routed and attributed correctly — and a wrong value is invisible until the first report nobody can answer.definition
- The order lands in the POS and nothing prints at the stationThe POS accepted your order and the line never saw it. Routing is configuration — dining option, revenue centre, station map — and no field in your payload can override it.diagnostic
- What a POS partner programme gates, and what it does notA partner programme is a commercial gate wearing technical clothing. It controls scopes, production credentials and listing — and the wait is somebody else's decision, not engineering.definition
Working on something in this space?
Tell us where you are in a sentence or two. We'll tell you honestly whether we're the right team, and what a sensible first slice of the work looks like.
Start the conversation