Open Text Survey Responses: Using AI in UX Research

Every survey I have run has had the same shape: a scale question people answer in two seconds, and an open-text box that holds everything worth knowing. Open text survey responses are where customers tell you why, in their own words, and they are also the part most teams never read past the first page. At ShootProof, one survey program brought in 1,411 open-text responses. Nobody was going to read those one at a time.

This post is the 2026 version of how I handle that problem. The tools changed a lot since I first wrote it. The job did not: get from a pile of comments to three things the product team should do, without losing the customer’s voice on the way.

Why open text survey responses matter

Closed questions are efficient. They give you a number you can trend, and that is the whole point of them. They also confine people to the options you thought of in advance, which means they cannot surprise you. The 1 to 5 rating tells you a feature is a 2.8. The comment box tells you it is a 2.8 because the export button is hidden under a menu nobody opens.

Open text is also the best interview recruiting list you will ever get. The people who write three sentences in a survey box are the people who will say yes to a 30-minute call, and they already told you what they want to talk about.

The challenge of analyzing open text responses

The challenge is volume and bias. Reading 1,400 comments by hand takes days, and by comment 300 you are pattern-matching to whatever you noticed in the first 50. Keyword counts are faster and worse: “export” shows up in praise and in complaints, and a count does not know the difference.

What you need is a first pass that groups responses into themes and counts them, done the same way every time, followed by a human pass on the outliers. That order matters. The machine does the sorting. You do the judgment.

Two ways to do the first pass in 2026

If the responses came in through Pendo NPS, start inside Pendo. Pendo’s NPS themes feature groups free-text answers into themes, lets you create, rename, and assign themes yourself, and, for accounts with NPS insights, has Pendo AI generate the themes and a summary for each one. Two limits worth knowing before you plan around it, both from Pendo’s NPS themes and insights article: automatic theme generation needs at least 600 English responses, and theme summaries need at least five responses per theme. Below 600 you assign themes by hand, which for a quarterly NPS run is a morning’s work. Also check your contract. Standalone NPS is on Pendo’s free tier; paid subscribers get NPS through the Sentiment add-on, and not every contract includes it.

For everything else, polls, CSAT follow-ups, a Typeform export, support survey comments, I use Claude. The method is the same one I used on the 1,411 ShootProof responses, just faster. One survey question at a time, I paste the responses in with a short instruction: group these into 6 to 10 themes, give each theme a one-line description and a count, and quote two verbatim responses per theme. Then I ask for the responses that fit no theme. Those outliers are usually the most interesting part of the survey, and they are the ones a keyword count would have dropped on the floor.

Two rules I hold to. First, the themes get checked against a random 50 responses by a person before anyone sees them. Second, the quotes in the report are real verbatims, never paraphrases, because the customer’s phrasing is the evidence.

A worked example: the Print Store surveys

Here is a real one from the ShootProof survey program in 2021. We launched a Print Store and ran two short surveys with an open-text follow-up: one at the announcement, aimed at photographers who said they were not interested, and one after first use. The ratings alone told two different stories: 1.9 out of 5 “likely to use in the future” from the announcement group, 4.2 out of 5 “satisfied with this feature” from the first-use group. The comments explained the gap.

The report was three slides: the two ratings, the theme table with verbatim quotes, and what product would do about it. That is the whole deliverable. If your findings deck is 40 slides, nobody will reach the slide with the decision on it.

Using open text to find interview candidates

Sort the responses by length. The longest 10% are your interview list. Email each one directly, thank them for the specific thing they wrote, and ask for 30 minutes. In my experience the yes rate on that email is far higher than any cold recruiting panel, because the person has already decided they care.

Then feed what you learn back into the themes. An interview often reveals that two themes you kept separate are one problem seen from two angles, and that one “feature request” theme is really a workaround for a bug.

Best practices for open text survey responses

• Ask one open question per survey, placed right after the rating it explains. Two open boxes halves the response rate on both.
• Keep the prompt specific. “What is the main reason for your score?” gets better material than “Any other feedback?”
• Theme the responses the same way each cycle so the counts can be compared quarter to quarter.
• Read a random sample by hand every time, no matter how good the automated pass looks.
• Put the themes next to the usage data. A theme that says “the export is hidden” next to a chart showing export usage dropping is a decision, not an opinion. That pairing is the point of the Pendo dashboards I build for clients.
• Separate bug reports from feature requests at intake, which is the argument behind a two-queue Pendo product feedback workflow.

Once the themes exist, they belong in a regular Voice of Customer report so the same five people see them every month. Theming once and never again is how most survey programs quietly die. If you want help setting up the survey, the theming, and the report as one system, that is a large part of our Pendo analytics and reporting work.

Questions people ask

Why collect open text survey responses at all?

Closed questions tell you what happened and give you a number to trend. Open text tells you why, in the customer’s own words, which is what you need to change the product. It is also the best source of interview candidates you have.

How do you analyze a large volume of open text survey responses?

Run a first pass that groups responses into themes with counts, either Pendo’s NPS themes for NPS data or an AI assistant such as Claude for everything else, then read a random sample and the outliers yourself. Report real verbatim quotes, never paraphrases.

How many responses do you need before AI theming is useful?

Pendo’s automatic NPS theme generation needs at least 600 English responses and five responses per theme for summaries. For a general AI pass there is no hard floor, but under about 100 responses you can read them by hand faster than you can check the machine’s work.

If your survey responses are piling up unread, or your NPS comments live in Pendo and nobody has themed them, bring them to a free Pendo strategy call and we will work through the first pass together on the call.