Coding 5,000 open-text responses: building a codebook that survives review

You have 5,000 free-text responses from an exit survey, a support backlog, or an NPS field nobody has opened in eight months. A stakeholder asks what people are saying. You have two days.

The instinct is to reach for topic modelling. This is a guide to why that usually disappoints, and to the boring alternative that holds up in a review: building a codebook on a sample, applying it at scale, and being able to defend both.

Why topic modelling underdelivers on survey text

LDA, NMF and the embedding-plus-clustering family are good at what they optimise for: grouping documents by word co-occurrence. That is not the same as grouping them by the distinction a decision-maker needs.

The output is not actionable. A topic that surfaces as price, expensive, cost, cheap, value tells you people mention price. It does not tell you whether they think the product is overpriced, whether a competitor undercut you, or whether the pricing page confused them. Those are three different decisions and the model cannot separate them, because the words are the same.

Short responses break the assumptions. Classic topic models assume documents long enough to carry a distribution over topics. A seven-word answer carries almost no signal. Half of a typical survey export is under fifteen words.

k is a guess, and the result moves. Change the number of topics, the random seed, or the stopword list, and the topics change. The first time a colleague re-runs your notebook and gets different topics, the finding loses its authority — and rightly so.

Naming is the analysis, and it is still manual. Even with modern embeddings and HDBSCAN, you get clusters. Someone has to read them and decide what each one means. That step is the expensive part, and no clustering method removes it — it just hides it until later.

Topic modelling earns its place in exploration: it is a fast way to see the rough shape of a corpus you have never read. It is a poor basis for a number you are going to put in front of an executive.

How much do you actually have to read?

Less than you fear. Codebook development saturates quickly, and you can measure the point rather than guess it.

Sample randomly, not from the top. Export order usually correlates with time, and time correlates with releases, incidents and campaigns. The first 200 rows are not 200 random rows.

Read in batches of 25 and track new codes. Batch one will produce ten or twelve codes. Batch two, four or five. By batch five or six you are usually adding zero or one. Stop when a batch of 25 adds at most one new code, then read one more batch to confirm. In practice that lands between 100 and 150 responses for most survey data.

Plotting cumulative distinct codes against responses read takes five minutes and turns "I read enough" into a defensible curve. It is also the single most convincing exhibit you can show a sceptical reviewer.

Stratify if you will report by segment. If the write-up will compare enterprise against self-serve, or detractors against promoters, sample proportionally from each. Otherwise the codebook over-fits whoever writes the longest answers — reliably the unhappy ones — and you will be missing codes for the groups you most want to compare.

Write codes that survive contact with a second coder

A code is a label with a boundary, not a summary. Each one needs three things written down, and the third is the one people skip:

• A definition — one sentence, stating what counts.
• An anchor example — a verbatim response that clearly belongs.
• A boundary rule — what does not count, and which neighbouring code it goes to instead. "Slow support" versus "unhelpful support" will be conflated by every coder until you write the line that separates them.

Two failure modes to watch for. A code that catches 40% of responses is not a code, it is a topic heading and needs splitting. A code that catches three responses across 5,000 is an anecdote and should be merged or dropped — unless it is severe, in which case say so explicitly rather than letting the count speak.

Multi-label, always

The average open-text response contains more than one idea. "Support took four days to reply and then told me to read the docs" is a wait-time complaint and a quality complaint. Forcing one label per response throws away half your data and biases whichever theme happens to be listed first in your prompt or your dropdown.

This has a reporting consequence people get wrong constantly: your percentages will not sum to 100, and they should not. Say so in a footnote before someone points at it.

Measuring agreement without fooling yourself

Double-code a random 50 to 100 responses — either two people, or you against the automated pass — and compute agreement per code, not overall. Overall agreement is dominated by the codes that almost never fire and hides the disagreement that matters.

Cohen's kappa is the standard, and it has a trap worth knowing: when a code is rare, kappa can be low even when the two coders agree on 95% of responses. With skewed prevalence there is little room to beat chance. This is the kappa paradox, and the fix is not to drop the code — it is to report prevalence next to kappa so the number can be read in context. A code that fires on 2% of responses with 96% raw agreement and kappa of 0.45 is usually fine; the same kappa on a code that fires on 30% is a real problem with your definition.

As rough anchors for applied work: above 0.8 is strong, 0.6 to 0.8 is usable, below 0.6 means the definition is ambiguous and needs a boundary rule rather than more training.

Reporting numbers that survive scrutiny

• Report the share of responses, not the share of codes. "18% of responses mentioned onboarding" is interpretable. "Onboarding was 12% of all codes" is not.
• Always show the base n, including how many people skipped the question. A free-text field with a 6% response rate tells you about the 6%, and they are not a random 6%.
• Separate frequent from important. The most mentioned theme is often the most visible one, not the most costly. Cross the counts with something that carries weight — churn, ticket cost, deal size — before ranking.
• Never rank without exposure. A complaint that only affects a feature 5% of users touch is a very different finding at the same raw count.
• Keep verbatims attached. Every number in the deck should be one click from the actual sentences behind it. The first question in the meeting is always "what did they actually say?".

Where automation genuinely helps

The mechanical part — applying a fixed codebook to 5,000 responses, consistently, in an afternoon — is exactly what a language model does well, and it does not get bored at row 3,000 the way a human does.

Two conditions make it trustworthy rather than convenient. First, the codebook stays human: definitions and boundary rules come from your reading, not from asking a model to invent categories, because invented categories drift between runs for the same reason topic models do. Second, every assignment must carry the span it came from. If a code cannot point at the words that triggered it, you cannot audit it, and an unauditable classification is not evidence.

The honest division of labour: you decide what the categories are and adjudicate the disagreements; the machine does the repetition and never forgets a boundary rule at four in the afternoon.

The whole thing in one page

1. Random sample, stratified if you will compare segments.
2. Read in batches of 25, track new codes, stop when the curve flattens.
3. Write definition, anchor and boundary rule for every code.
4. Apply to the full set, multi-label.
5. Double-code 50 to 100, report per-code agreement with prevalence.
6. Report share of responses with base n, and keep the verbatims one click away.

None of this is novel — it is standard content-analysis practice that the survey world quietly stopped doing when the corpora got large. The corpora are the reason to keep doing it, not the reason to stop.