Your bot protection is blocking the AI surfaces you sell through

If AI surfaces are not quoting your prices, check your edge before you check your data. The rule blocking them is almost never a misconfiguration — it is a blanket automation rule doing precisely what someone wrote it to do, at a time when all automated traffic was a cost and none of it was a buyer. The awkward part is that a buying agent and a price scraper arrive with the same broad characteristics, and nobody wants to widen the door without knowing which one walks through.

So the work splits in 2. First establish that a block is actually happening and what is being blocked. Then decide what an identified agent is allowed to reach, which is a policy question with a real security cost attached, not a checkbox.

The origin log will not show you this

A challenged or rate-limited request never reaches your application, so your application logs are silent about the exact traffic you are investigating. Everything below has to be read from the CDN or WAF event log, filtered to non-200 responses on catalogue paths, over at least 7 days.

1. Filter the edge log to 403, 429 and challenge-issued events on product, collection, search and any manifest or feed path. Group by declared user agent and by ASN.
2. Separate managed-rule blocks from your own custom rules. Vendor bot categories update on their own schedule, so a block can appear on a day nobody touched your configuration.
3. Fetch your own robots.txt and manifest paths from outside your network and check the status code, not the body. A challenge page returns 200 with HTML that is not the file, which passes every naive monitor you have.
4. Plot blocked requests per identity over time. A steady low-rate line is a crawler behaving itself; a burst of thousands in a minute from one ASN is not.
5. Check the geography rules separately. Blocking by country blocks the datacentre region an assistant runs in, which has no relationship to where its users are.

The 4 rules that catch buying agents, and how each one reads in the log

Rate limiting deserves a note of its own because it is the one tuned against the wrong model of a visitor. A human browsing generates a burst of requests for one page — document, styles, scripts, a dozen images — then pauses to read. An agent reads structured data, fetches no images, and issues a steady sequence of API-shaped requests with no pause at all. A limit calibrated on human burstiness reads that pattern as an attack, which is why an agent frequently gets 20 products in before the 429 arrives.

Telling a buyer from a scraper without believing the header

The declared user agent is a claim, not a credential — anyone can send any string. But identity is only one of 3 signals, and the other 2 are behavioural and much harder to fake convincingly at scale.

• Declared identity. The product token in the user agent, plus whatever verification the operator publishes: a documented IP range, a reverse-DNS record that resolves forward again, or a request signature. HTTP Message Signatures, standardised as RFC 9421, is the mechanism that makes this checkable rather than declarative — a Signature-Input field naming the covered components and a key identifier, and a Signature field carrying the value.
• Request shape. A buying agent goes depth-first: a search, then a handful of specific products, then availability and policy for one of them. A scraper goes breadth-first, walking every collection page in order, and it does not stop at 40 products because a shopper only asked about one.
• Endpoints touched. An assistant working a purchase reads price, stock, policy and delivery for named items. A harvester reads listings, never policy, never a single deep item twice, and never anything that would only matter to someone about to buy.

What robots.txt settles, and what it has no power over

Two properties of the file catch teams out. Product-token matching is case-insensitive and tokens may only contain letters, underscores and hyphens, so a rule written to match one capitalisation of an agent's name matches all of them — and a token typed with a slash or a version number in it matches nothing. And when several rules could apply, the most specific match wins, measured by the number of octets in the path; where 2 rules are equally specific, allow beats disallow.

Crawlers are also told not to use a cached copy for more than 24 hours, and to parse at least 500 KiB of the file. A change therefore takes up to a day to take effect, and a robots.txt padded past that parse limit by a generator has rules nobody is obliged to read. None of this constrains your WAF, which is the more common blocker: robots.txt is a request, and a firewall rule is an outcome. Fix the firewall first.

Allow reads, refuse writes, and give agents their own limit

An allowlist that simply exempts an identity from bot management is the version of this that goes wrong, because it grants the identity everything your site can do. Scope it instead.

1. Define the surface first. Product detail, search, availability, published policy and your structured feeds — the read boundary argued in designing the tool surface a shopping agent is allowed to call.
2. Exclude every write path explicitly. Cart mutation, account creation, coupon application, review submission and anything that sends an email stay behind the full ruleset. An agent that needs to transact does it through your checkout endpoints, under their own authentication, not through a relaxed WAF rule.
3. Give each identified agent its own rate-limit bucket, sized for sequential structured reads rather than a browsing human, with 429 and a Retry-After header rather than a silent drop. RFC 6585 defines 429 for exactly this, and those responses must not be stored by a cache.
4. Exempt machine-read paths from interactive challenges everywhere. A JavaScript challenge on a feed, a manifest or robots.txt is a permanent block dressed as a 200.
5. Log allowed agent traffic to its own stream and review it monthly. An allowlist nobody looks at becomes an allowlist nobody can defend.

Widening access is only worth doing if what the agent then reads is good enough to quote. An allowlist in front of thin attributes buys you nothing, which is why the readiness gates in what has to be true before an assistant may quote price and stock and the attribute decisions in designing the attribute schema before the import come before this work, not after it. The same applies to terms: an agent that can reach your pages and not your returns policy will paraphrase one, and publishing policy an agent can quote safely is how you stop it.

The part of this nobody can settle for you

Be plain about the exposure. A user-agent string is forgeable, so any allowlist keyed on identity alone can be worn as a costume by a competitor's price monitor. Signature verification and verified reverse DNS raise the cost of that considerably, but not every operator publishes either, and coverage across the agent ecosystem is uneven and moving.

You are choosing between two costs: some scraped prices, or absence from the surfaces where a growing share of buying decisions now gets made.

That is a commercial judgement, and it belongs to whoever owns the channel rather than to whoever owns the firewall. What engineering can do is make the choice reversible and observable — an allowlist expressed as configuration, per-identity traffic visible on a dashboard, and a single switch that closes it again. That is internal tools and ops work, and it is worth more than a permanently open rule nobody feels able to touch.

Unblocking traffic also does not make you legible. An agent that can reach your store still has to find your capability declaration, read your catalogue and complete a purchase — the rest of agentic commerce, inside our retail and e-commerce practice. If agents are fetching your files successfully and still passing you over, the fault is in what those files say, not in whether they arrive.

• agentic commerce
• bot management
• edge configuration
• rate limiting