A handful of sellers are taking everything: is it the ranker or the market?

There are two very different situations behind the same complaint, and they need opposite responses. Either the ranking function built the concentration — it weights past volume, past volume came from past ranking, and the loop has been running for a year — or the market really does have five sellers who are better, faster and more available than everyone else, and the distribution is honest. A counterfactual re-rank distinguishes them in an afternoon. Opinions do not, and the argument recurs every quarter until somebody measures it.

Whether to cap a dominant seller's share is a separate question with its own answer, argued in capping how much one seller can win. Decide what caused the concentration before deciding whether to intervene in it.

Measure the shape before arguing about its cause

Most concentration arguments run on an anecdote: a supply manager notices the same names in every result. Replace it with two numbers per query class — a query class being a category-and-area pair such as deep cleaning in one postcode district. Take the share of accepted leads held by the top 5 sellers, and the share of eligible supply those 5 represent. A top-5 share of 62% is unremarkable if only 7 sellers are eligible and alarming if 180 are.

The counterfactual: yesterday's queries, without the memory

1. Replay a fixed window of real requests — a week is usually enough, and use the eligibility filter as it was at the time, not as it is now.
2. Score each request twice. Once with the live scoring function. Once with every history-derived signal set to a neutral constant: completed jobs, acceptance rate, review count, review score, response-time averages, badges earned from volume.
3. Keep the non-history signals in both runs. Distance, availability, price, verification state, category fit and language are properties of the seller today, not echoes of what the ranker did last year.
4. Compare the top-5 share of the two result sets over the same requests, and the rank correlation between them. A concentrated market with a neutral ranker still concentrates, because distance and availability alone favour whoever covers the most ground.
5. Read the split. Concentration that survives the neutral run is structural. Concentration that collapses to near-uniform was manufactured by the signals you just switched off.

This is an offline job over logged requests, so it is cheap to run and safe to repeat, and it belongs in a scheduled harness rather than a notebook somebody has to rediscover. Building that harness is ordinary automation work — the sort of thing we ship under AI agents and automation — but note the volume: if the replay takes a scoring call per candidate per request, its cost grows with your candidate sets, which is the same arithmetic that governs which stage of matching stopped being linear.

Four mechanisms that manufacture a winners' circle

The first two are ranking defects and are fixable inside the scoring function. The last two are not ranking problems at all, and every hour spent tuning weights against them is wasted. Before concluding that the mid-tier is being starved, confirm the mid-tier was actually reached: leads recorded as sent are not leads anyone saw, and the difference is the subject of what sent, delivered and read actually prove. Sellers who never received the offer look identical in the acceptance data to sellers who lost on rank — the four-way split traced in requests that reach sellers and nobody takes.

Click data cannot audit the ranker that produced it

A seller in position 1 is clicked more than an equally good seller in position 6 for reasons that have nothing to do with quality: buyers read from the top and stop early. So any signal derived from clicks or bookings encodes the previous ranking, and feeding it back trains the model to agree with itself. This is why a counterfactual with history removed can still under-report the problem — the remaining signals may themselves be contaminated.

• Estimate the bias rather than assuming a curve. Randomly swap adjacent positions on a small share of traffic and measure the click difference between a seller shown at 2 versus 3. That difference is your position effect, measured on your buyers rather than borrowed from someone else's.
• Weight logged interactions by the inverse of the chance that result was shown at all. Interactions from rarely-shown sellers count for more, which is what stops the log from being a record of your own past decisions.
• Give new supply a bounded exploration budget so evidence exists to weight — the mechanism argued in giving a listing with no history a chance.
• Keep seller-level performance data inside your own boundary when you can. It is commercially sensitive in a way listing text is not, and shipping it to a third-party ranking service is a decision that deserves the scrutiny set out in private and on-premises model deployment.

A ranker trained on its own clicks is not learning what buyers prefer. It is learning what it showed them.

When the honest answer is that you need more sellers

If both runs concentrate and the eligible-seller count per request is small, the market is thin and the concentration is a description of your supply, not a bug in your code. In a thin, time-constrained market the binding constraint is usually coverage and travel time rather than match quality at all, which is the whole argument in dispatching forty cleaners across one city. Redistributing demand across a pool that cannot serve it produces late arrivals and cancellations, and the buyer side pays for the fairness gesture.

The lever that does work here is the allocation rule rather than the score: rotation with a quality gate spreads demand deliberately, an auction concentrates it on whoever can pay, and the two behave very differently when supply is scarce — compared directly in auction or round-robin for handing out demand. Change the rule knowingly, or recruit. Do not do it by re-weighting review counts and hoping.

Signals, logging, or recruitment

1. Does concentration survive the neutral re-rank? If it collapses, the fix is in the signals: cap the influence of cumulative counts, prefer rates over totals, decay history so a year-old streak stops deciding today's result.
2. Does it survive the neutral run but collapse under randomised positions? Your training data is biased. Fix the logging and the weighting before touching the model, because retraining on the same log reproduces the same winners.
3. Is the median eligible-seller count 3 or fewer? Stop. This is a filter and coverage problem, and ranking is reordering a list that barely exists.
4. Do both runs concentrate with a wide eligible pool? The distribution is real. Decide whether you want it — sometimes concentrating demand on the best sellers is exactly right — and if you do not, change the allocation rule rather than the score.
5. Whatever you change, re-run the counterfactual monthly. Concentration returns quietly, because every ranking function that uses history is a loop with a slow time constant.

The rest of this silo sits under matching, ranking and dispatch, inside our marketplace and two-sided platform practice.

• ranking
• supply side
• position bias
• measurement