The Questions Behind the Questions
A survey draft comes back for review, and one question stops you. It's well-written and clear, and you can't work out why it's there. So you ask the program area: what will you do differently depending on how people answer this? There's a pause. Then: "We just thought it would be interesting to know."
Interesting to know is not nothing. But it's not a reason to put a question in front of the public, take up their time, and then carry the answer forward into a report that shapes a decision. If you can't say what the answer would change, you can't tell, when the results come in, whether the question did its job. You've collected data you have no standard for evaluating.
This is a companion piece to my "Confident and Wrong" series, which is about a gap in how government adopts AI: we measure whether tools get used, but not whether anyone can tell when they're working. This post explores one part of it.
The practice has an unglamorous name. We call them Questions to be Answered, or QBAs, and they're one of the most useful habits we’ve developed — for survey work, and, it turns out, for working with AI.
What a QBA actually is
A Question to be Answered is not a question respondents see. It's a question the organization needs answered: the real-world decision or knowledge gap the research exists to address. "Should we expand the reduced-fare program to more routes?" is a QBA. The survey questions about ridership, income, and travel patterns are how you get there. The QBA is the destination; the survey questions are the route.
The distinction sounds obvious written down. In practice it almost never is, because QBAs usually aren't written down at all. They live in people's heads, spread unevenly across the program area, the ministry partner, and the analyst. Each person carries a slightly different version, and everyone assumes the others share theirs. Nobody has examined their own version closely. The survey gets built on top of this quiet disagreement, and the disagreement only surfaces at the end, when the findings are in and three people want to characterize them three different ways.
Writing the QBAs down before any analysis begins is what prevents that. It forces the alignment to happen at the start, when it's cheap, rather than at the end, when it's a fight over a briefing note.
Two jobs a written QBA does
A QBA earns its keep at two different moments.
During design, it works as a filter. Every question in the survey should connect to at least one QBA. If a question doesn't map to anything the organization actually needs to decide, that's a strong signal it shouldn't be in the survey. This is where the "interesting to know" question gets caught — not because curiosity is bad, but because every extra question costs respondent attention and raises the dropout rate for the questions that matter.
During analysis, it works as a focus. Open-ended data especially is full of interesting patterns, and most of them are beside the point. The QBAs tell you which patterns the report is actually responsible for addressing, so you're not chasing whatever happens to catch your eye. They keep the analysis pointed at the decision it's meant to inform.
A survey with no written QBAs isn't a survey without goals. It's a survey whose goals are unspoken, which means they get filled in after the fact, by whoever writes the report, in whatever direction is most convenient. The absence of a stated standard doesn't remove the standard. It just hands the standard to chance.
Where this meets AI
This is the connection to the "Confident and Wrong" series. When you use an AI tool to do a first-pass analysis of open-ended responses, it hands you back something clean and finished-looking, with no sign of the judgment calls baked into it. The danger isn't that it's wrong. It's that you have no independent standard to check it against, so "does this look reasonable?" becomes the only test the output can be measured by — and a fluent, well-organized output passes that test whether or not it's any good.
The QBAs are that independent standard. They exist before the AI runs, which means they aren't contaminated by what it produces. When the output comes back, you're not asking whether the AI's themes look plausible. You're asking whether they answer the questions you committed to in advance. That's a question the clean surface of the output can't talk you out of, because you wrote it down first.
That's the general principle underneath all of this: a tool earns its place when you can say, ahead of time, what decision its output is supposed to inform. A QBA is just that principle written down and agreed on before the work starts.
How to actually write one
The hardest part of QBAs is starting, and there's something uncomfortable about writing down the things you don't know. That discomfort is normal. A few ways in, when the blank page is the problem:
Start with the decision, not the gap. Instead of "what do I want to know?", ask "what decision does this research need to support?" Name the decision — expanding a program, changing a service model, setting a direction — and the questions you need answered tend to follow on their own.
Ask what would change your mind. Once you have a sense of where you're leaning, ask: what result would make me reconsider? Whatever comes to mind is almost certainly a QBA. And if nothing comes to mind — if you can't picture a result that would change anything — that's worth sitting with, because it may mean the survey is being used to confirm a decision already made rather than to inform one still open.
Describe the report you want. Work backwards. Imagine the findings are in and the report is good. What's in it that you'd put in a briefing note, that would tell you the engagement did its job? Describe that, even roughly, and the QBAs are usually sitting inside the description.
React, don't generate. A blank page is harder than a red pen. If you can't write QBAs from nothing, write down what you know about the file — the context, the policy question, the tensions you're already aware of — and turn that into a rough draft you can correct. Pushing back on a draft is far easier than producing one cold.
Before you write a single survey question — or run a single AI analysis — write down the three to five decisions the work is supposed to inform. Then, for each one, name the result that would change your mind. If you can't, you've learned something important before spending a dollar of respondent time: the question you're really asking isn't settled yet.
One last thing, because it stops people: QBAs are a working document, not a final product. They're meant to be revised as the project develops and your thinking sharpens. Getting something rough written down beats waiting until you're certain. The goal at the start is alignment, not perfection — and alignment you can write down is the thing that lets you tell, later, whether any of it worked.
The three "Confident and Wrong" posts are (1) When the Output Looks Clean, (2) A Seat at the Table, Not a Verdict, and (3) Why the Checklist Isn’t Enough. They make the broader case; this post is one worked example of what closing the gap looks like in practice.