Surveys and bias: sampling frames, non-response and wording
What this lesson covers
- A sample can only speak for the population its sampling frame reaches.
- Good design reduces bias; it never eliminates it.
Errors a margin of error misses
- Who can be reached, who answers, how it is asked.
- A bigger sample fixes none of them.
- Design reduces them; it never eliminates.
Target population and sampling frame
- Target: all adult residents.
- Frame: the list actually sampled.
- A landline list misses many people.
| Rates survey | |
|---|---|
| Target | all adult residents |
| Frame | landline directory |
| Missed | no landline |
| Often | younger, renting |
Why a mismatch matters
- A sample speaks only for its frame.
- Missed people may think differently.
- Name the mismatch and explain the limit.
Survey methods: the trade-offs
- Phone, online, face to face, mail.
- Each has an upside and a cost.
| Method | Plus | Minus |
|---|---|---|
| Phone | a real person asks | many do not answer |
| Online | cheap and quick | misses offline people |
| Face to face | high response | slow and costly |
| time to think | few send it back |
In context, not a learned list
- Tie each point to the report.
- Online could miss older rural residents.
- They may feel most strongly about rates.
Non-response
- Selected people who do not take part.
- Those who reply may differ.
- Follow-ups reduce this bias.
| Often | |
|---|---|
| Who replies | strong views, keen |
| Who does not | busy, less interested |
| Reduce it | reminders, callbacks |
Incentives: what and why
- More replies: less non-response.
- But it can change who replies, and how.
- Give the effect and the reason.
| Effect | |
|---|---|
| Incentive | $50 voucher draw |
| Upside | more people reply |
| Risk | prize seekers reply |
| Risk | answers to please |
Question wording
- Leading and loaded words push answers.
- One question, one idea.
- Neutral: do you support or oppose?
| Problem | Example |
|---|---|
| Leading | "Surely you agree..." |
| Loaded | "wasteful spending" |
| Double-barrelled | rates and roads? |
| Neutral | support or oppose? |
Ways to choose a sample
- Random methods reduce selection bias.
- Volunteers are the most likely to be biased.
| Method | How |
|---|---|
| Simple random | all equally likely |
| Stratified | random within groups |
| Cluster | random whole groups |
| Systematic | every 10th on a list |
| Self-selected | volunteers: biased |
Stratified by area
- Split the region into areas.
- Sample each in proportion.
- At random within each area.
| Area | Share | Sample |
|---|---|---|
| City | 50% | 500 |
| Towns | 30% | 300 |
| Rural | 20% | 200 |
| Total | 100% | 1000 |
The 2025 error: "eliminates bias"
- Bias is reduced or minimised.
- A bigger sample does nothing for bias.
- Never write eliminates.
| Avoid | Write | |
|---|---|---|
| Random sample | eliminates bias | reduces bias |
| Stratifying | makes it perfect | more representative |
| Bigger sample | removes bias | smaller MOE only |
Where the marks are
- Achieved: name a source of bias.
- Merit: its effect in this survey.
- Excellence: what and why, precisely.
What to take away
- Check who could be reached, who replied, and how they were asked.
| Frame | who can be reached |
| Mismatch | limits generalising |
| Non-response | responders may differ |
| Wording | neutral, one idea |
| Stratify | random within groups |
| Bias | reduced, not eliminated |
Your turn: four questions
- Pause here and answer each one.
- Say what the effect is, and why.
- Tie it to the survey.
| Survey | Find | |
|---|---|---|
| 1 | online poll of a town | who's missed |
| 2 | cycle lane question | fix wording |
| 3 | café prize draw | its effect |
| 4 | 60% city, 40% rural | split 500 |
Show answer
1. People rarely online are outside the frame; if their views differ, the estimate is biased. 2. Leading and loaded; ask "Do you support or oppose the cycle lanes?" 3. More customers reply, reducing non-response, but it may attract regulars or prize seekers, or answers meant to please. 4. Stratify: 300 city and 200 rural, chosen at random within each area.
The answers
- People rarely online: outside the frame.
- Do you support or oppose the cycle lanes?
- More replies, but who replies changes.
- 300 city, 200 rural, random within each.
Is your child on track?
Take the free placement test