Triaging a feature request backlog with a coding agent is not the agent deciding what matters. It is the agent doing the sorting, fast and consistently, against categories a person defined before it touched a single request. The method holds whether the backlog is a few dozen requests or a few hundred.
This is the triage stage of a larger loop. For the whole thing end to end, see running support with a coding agent.
Why define the categories before you start, not while you go?
An agent sorting requests as it goes, inventing a category the moment a request does not fit the existing ones, ends up with a taxonomy nobody chose and nobody can explain later. Writing the categories down first, with a one-line description of what belongs in each, turns triage into a lookup problem for the agent rather than a judgement call it is making fresh on every request. A short, fixed list beats a long, discovered one: five or six categories a person can hold in their head are more useful than twenty an agent generated to avoid ever saying "none of these quite fit." Writing that fixed list somewhere the agent reads before every pass is the same idea an AGENTS.md for your support centre applies to tone and boundaries: decided once, not reinvented per request.
A starting list rarely needs to be longer than this:
| Category | What belongs in it |
|---|---|
| Bug | Something that used to work, or should, and does not |
| Feature request | A capability that does not exist yet |
| Duplicate | The same underlying request as one already open |
| Already answered | Covered by existing documentation or a shipped feature |
| Question | Not a request at all, it just needs a reply |
Why sort before you judge?
Sorting and prioritising are two different jobs, and running them together is where triage usually goes wrong. Sorting asks what kind of request this is: a bug, a feature, a duplicate, a question that already has an answer. Prioritising asks whether it matters enough to build. An agent is genuinely good at the first job, reading text and matching it against a written definition is exactly what it is built to do well. The second job is a business call about what to spend engineering time on, and running it at the same time as sorting means every categorisation decision is quietly tangled up with a priority opinion nobody asked for yet.
How does batching actually work in practice?
Run the agent over the backlog in batches rather than one request at a time, and review a batch before moving to the next one. A batch of twenty or so is small enough to spot-check properly and large enough that a person is not waiting after every single item. Each batch produces a proposed category for every request in it, sitting as a written list, not applied to anything yet. Nothing in a batch is final until the batch has been checked.
What do you actually check by hand?
Not every request. Checking the whole backlog by hand, whether it holds a few dozen requests or a few hundred, would remove the point of using an agent at all. What is worth checking on every batch:
- The edge cases. Anything the agent flagged as uncertain, or where it stated two possible categories, is exactly the request a person should look at first.
- A random sample of the confident ones. Confidence in the agent's output does not track accuracy, so a handful of requests it filed without hesitation still need spot-checking, not because they are more likely to be wrong, but because nothing in the tone tells you either way.
- Anything filed into a category that changes what happens next, such as a request routed straight to "duplicate" or "already answered." Those are the categorisations with the most consequence if they are wrong, so they earn the closest look regardless of how confident the agent sounded. Deduplicating requests covers this specific category in full, because "duplicate" is the single most consequential label triage can apply.
One payment, no subscription, unlimited products.
How do you verify the agent didn't quietly mis-file things?
The dangerous mistake in triage is not the request that gets flagged for review and turns out fine. It is the request that gets filed confidently, correctly-looking, and wrong, and never comes back up again because nothing about it looked like it needed a second look. Two habits catch most of this. First, after a batch is reviewed and applied, periodically pull a random sample from the whole backlog, not just the newest batch, and re-check it against the category definitions cold. Second, watch the distribution: if one category is unexpectedly swelling or another has stayed empty the entire run, that is worth reading through by hand before assuming the category itself was simply rare.
LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence.
Xiong et al., Can LLMs Express Their Uncertainty?
What does a backlog usually get wrong about its own categories?
A category list written before triage starts is a guess, and a guess drawn up without real requests in front of it is usually wrong in a predictable way: too coarse in the place with the most volume, and inventing a distinction nobody actually needed elsewhere. The signal is in the distribution, not in how the list reads on paper. A category swallowing half the backlog is really two categories that have not been separated yet. A category sitting empty after the first batch either never applied, or was defined too narrowly to match how customers actually write.
Fixing this mid-run is a deliberate, visible step, not a quiet edit. Split or merge the category, note when it happened, and re-run only the batches affected by the change rather than assuming the earlier ones still hold. A category definition an agent is triaging against needs to be exactly as stable as any other fact it is told not to invent past.
Frequently asked questions
Why define categories before triage rather than let them emerge?
Categories chosen mid-run reflect whatever the first few odd requests happened to look like, not a considered structure. Defining them first, with a one-line description of what belongs in each, turns sorting into a lookup task rather than an improvised judgement call made fresh on every request.
Should sorting and prioritising happen in the same pass?
No. Sorting asks what kind of request something is; prioritising asks whether it is worth building. An agent is strong at the first and should not be making the second decision at all, so keeping them as separate passes stops a categorisation choice from quietly carrying a priority opinion nobody asked for.
What is the actual risk in letting an agent triage at volume?
Not the requests it flags as uncertain, those get looked at. The risk is a request filed confidently and wrongly, which looks identical to one filed confidently and correctly, and therefore never gets a second look unless someone deliberately re-samples the backlog.
How large should a review batch be?
Small enough to check properly, large enough that a person is not reviewing after every single request. Roughly twenty is a reasonable starting point: big enough to move at a real pace, small enough that spot-checking a batch does not become a chore nobody actually does.
What should always get a human look, regardless of how confident the agent sounded?
Anything the agent itself flagged as uncertain, a random sample of the confident ones, and anything routed into a category with real consequence, such as being marked a duplicate or already answered. Confidence in the output is not evidence of accuracy, so sampling matters more than trusting the tone.