Audit bias in your "How did you hear about us?" survey

If 12 of 20 survey respondents name LinkedIn, LinkedIn accounts for 60% of their answers. Before using that figure to shift marketing budget, check who never saw the question, who skipped it, and what respondents understood “heard about us” to mean.

Self-reported attribution records how customers remember discovering your product. Those answers can suggest channels to investigate, but they do not establish what caused sales. Audit one recent signup cohort before assigning budget credit: measure coverage, inspect the question, and compare the people answering with those missing from the chart.

Define what the answer should measure

“How did you hear about us?” leaves room for several interpretations: the first encounter, the most memorable encounter, or the event that prompted today’s signup.

Choose the decision you need to inform. To learn where people first encountered your product, ask about remembered discovery. To understand the signup trigger, ask a separate question.

For discovery, start with:

Where do you remember first hearing about [product]? Please answer in your own words. It’s fine if you don’t remember.

An optional follow-up can ask:

What prompted you to try [product] today?

Pretest these questions with a few people resembling your intended users. Ask what they think each question means. Someone invited by a coworker may know who sent the invitation but know nothing about the company’s earlier evaluation.

AAPOR recommends short, specific questions that address one concept and avoid steering respondents toward a preferred answer. It also recommends pretesting with people similar to the intended respondents. AAPOR’s survey best practices

Count the people outside the answer chart

Start with a defined population, such as all new workspace owners who completed signup during a calendar month. Choose a counting unit: person or account. Five teammates joining one workspace should not silently become five independent acquisition stories.

For that population, count eligible people or accounts, those shown the question, and those who answered. Include “don’t remember” in the answer count. Separately record how many answers identify a source clearly enough to classify.

Keep “not shown,” “shown but blank,” and “don’t remember” separate. If your software cannot measure whether the question appeared, record exposure as unknown. You can still count answers, but you cannot reliably split missing answers into people who skipped the question and people who never saw it.

Use the same counting unit in each numerator and denominator:

If exposure is unknown, leave exposure coverage and completion among those exposed uncalculated. If nobody was shown the question, completion is undefined. These are operational audit measures; they do not establish that the survey represents your customer population.

AAPOR explains that a response rate alone cannot reveal whether nonresponse error exists or how large it is. AAPOR’s Standard Definitions

Compare coverage across a few relevant characteristics you already hold, such as signup path, workspace role, or plan. Choose splits that could expose an operational problem. Assisted signups might receive the question reliably while self-service signups rarely reach it.

Check whether the form supplies the answer

Read the survey exactly as a customer encounters it, including nearby copy and default selections. Look for a preselected channel, overlapping choices, missing options, or examples that prominently name the channel your team hopes to validate.

Pew Research Center documents how question format, wording, and answer order can change responses. It also explains why consistent wording and context matter when measuring change over time. Pew’s guide to writing survey questions

A short text field can expose distinctions your categories miss. Consider this hypothetical answer: “A colleague sent me your comparison article.” It names both a person and an asset. A dropdown may force the respondent to choose between “word of mouth” and “content.”

Text responses require interpretation and may be vague. A short list is easier to complete and analyze, but its categories constrain the answer. If you use a list, include an appropriate “Other” path and “Don’t remember,” avoid defaults, and consider rotating unordered channel choices.

Record the question’s wording, choices, placement, and required or optional setting under a version number. Adding an “AI assistant” option changes the measurement. A later increase in that category could reflect the new choice as well as a change in discovery.

Preserve customer wording before assigning categories

Store the raw response beside your interpretation. A coding guide should explain each category and how to handle ambiguity. An answer of “Google,” for example, supports “Google, unspecified.” It does not establish paid search, organic search, or a particular search feature.

For answers naming several encounters, retain multiple mentions if your question allows them. Label the chart “respondents mentioning each source” and use the number of respondents as its denominator; percentages may exceed 100%. If you require one primary category, document the selection rule and retain other mentions separately.

Have a second teammate independently classify a small batch. Review disagreements before coding the rest. If you work alone, revisit ambiguous answers later without looking at your first labels. Use disagreements to revise unclear category definitions.

FindVex’s pain-point evidence sheet guide explains how to separate customer language from interpretation. Apply that separation here, including an uncertainty field for answers you cannot confidently classify.

Worked example: LinkedIn leads the responses

This example is hypothetical. An AI meeting-notes startup has 100 new workspace owners in one month. All saw an optional survey, and 40 answered. Each answer received one category:

LinkedIn accounts for 60% of answers and is named by 24% of the full cohort. Neither percentage establishes LinkedIn’s share of acquired customers.

Splitting the cohort by signup path reveals a gap:

Assisted signups supply 80% of the answers despite representing 40% of the cohort. Their responses heavily influence the overall result. That is a reason to investigate the missing answers before reallocating budget; it does not tell you what nonrespondents would say.

Try a sensitivity exercise. Suppose the 60 people who did not answer had instead supplied 20 search answers and 40 “don’t remember” answers. Search would have 30 mentions to LinkedIn’s 24. This invented scenario is not an estimate of the missing responses. It shows how a possible set of answers could reverse the observed ranking.

For this startup, the next task is to inspect why self-service users skip the question and seek discovery accounts from that group. Weighting the eight self-service answers upward would still assume those eight represent all 60 self-service signups.

Compare survey answers with analytics without forcing agreement

Place the reported source beside the available acquisition data for the same cohort. Keep both fields intact and name the analytics dimension you use.

Google Analytics distinguishes user-scoped dimensions, such as First user source, from session-scoped dimensions, such as Session source. The former describe new-user acquisition; the latter describe acquisition for new sessions. Google’s explanation of traffic-source scopes

A hypothetical customer might report a colleague’s recommendation and later reach your website through search. The records can describe different encounters. Disagreement alone does not identify which record is wrong.

Inspect a few mismatches for question ambiguity, coding errors, or measurement differences. Keep the survey answer even when analytics records another source.

Likewise, an answer naming an AI assistant records a customer’s recollection. It does not verify a specific citation, recommendation, or prompt. Preserve any volunteered detail and leave missing details unknown.

Copy this worksheet into your acquisition review

Use one sheet per cohort. Keep raw answers and coding decisions in a separate tab.

Decision this audit will inform:
Cohort dates and eligibility rules:
Counting unit (person or account):

Question version:
Exact wording and answer choices:
Placement:
Required or optional:

Eligible count:
Shown the question (count or unknown):
Answered, including "don't remember":
  Source identified:
  Don't remember:
  Other unclassifiable answers:
Shown but blank (count or unknown):
Not shown (count or unknown):

Exposure coverage (or unknown):
Answer coverage:
Completion among those exposed (or unknown/undefined):

Groups compared and their answer coverage:
Raw answers retained at:
Ambiguous answers and coding disagreements:
Rule for multiple mentions:
Channel-chart denominator:

Analytics dimension, population, and period:
Mismatches worth inspecting:
What missing answers could change:
Next measurement fix or bounded channel test:
Owner and review date:

The three indented answer categories should add up to the answered count. Keep unclassifiable answers visible instead of silently dropping them from the denominator.

Use recurring mentions to choose what to investigate. For a spending test, define a budget cap and a useful outcome in advance, such as activated workspaces. Evaluate that outcome separately from survey mentions. A small before-and-after change can still reflect other changes in the business.

Your next task: audit one signup cohort

Choose one completed month and fill in the worksheet. Compare answer coverage for your main signup paths, then inspect the raw answers behind the leading channel. Bring the channel counts, their denominator, and the largest unresolved coverage gap to your next acquisition review.

Assign one measurement fix or bounded channel test an owner and review date. If missing responses could change the channel ranking, record that uncertainty before deciding how much budget to move.