Reading data critically, and claiming only what you can
What this lesson covers
- Elements of chance affect the certainty of results from observational studies and experiments.
- Uncertainty should be taken into account when making claims.
- Data visualisations need to be critically assessed to see if they support or misrepresent the data.
The hard part is not the calculating
- You can find a median and draw a graph.
- The hard question is what those actually entitle you to say.
- Usually less than you would like. Sometimes plenty.
Two honest samples will disagree
- Twenty students, mean height 162. Another twenty, mean 165.
- Same school, both done properly, nobody made a mistake.
- That 3 cm gap is chance. It is not a finding.
| Sample | Result | |
|---|---|---|
| Sample A | 20 students | mean height 162 cm |
| Sample B | 20 others | mean height 165 cm |
| Same school | both done properly | |
| The 3 cm gap | is chance, not a real difference |
Which is why the number matters
- In ten people, one unusual person moves the whole result.
- In two hundred, that same person barely registers.
- So the first question about any finding is: how many?
| Sample size | Effect of one oddity | Evidence |
|---|---|---|
| 10 students | one oddity shifts it all | weak |
| 200 students | one oddity barely shows | stronger |
| So ask | how many was this based on? | always |
And the second question: how were they chosen?
- Everybody knows not to survey the basketball team about sport.
- The dangerous ones are the methods that sound fine.
- Here is one.
Surveying in the library at lunchtime
- Perfectly reasonable place to find thirty people. No bias intended.
- But who is in a library at lunchtime? People who like reading.
- So 80% read daily is true of them, and not of the school.
| Step | What happened | |
|---|---|---|
| Surveyed in the library | at lunchtime | sounds neutral |
| But who is in there? | people who like reading | |
| Finding: 80% read daily | true of library users | |
| Claimed about the school | not supported |
Nobody lied there either
- The survey was real, the answers were honest, the maths was right.
- The finding is true. It is just true about a different group.
- Most bad statistics looks exactly like this.
Now graphs, from the reader’s side
- Last lesson you were drawing these. Now you are being shown one.
- Where does the axis start? Are the bars the same width?
- Is there a scale at all, and how many people was it?
| Ask | Because | |
|---|---|---|
| Where does the axis start? | not at zero means exaggerated | |
| Are the bars the same width? | wider looks like more | |
| Is there a scale at all? | no numbers, no claim | |
| What is the sample size? | often left off entirely |
Same trick, opposite chair
- A hundred against a hundred and five, with the axis starting at 95.
- A five percent change, drawn as a doubling.
- Every number on it is correct. That is what makes it work.
A real pattern, and three explanations
- Ice cream sales and drownings rise and fall together. Genuinely.
- Does ice cream cause drowning? Does drowning sell ice cream?
- Or does something else cause both? Hot weather.
| Claim | Detail | Verdict |
|---|---|---|
| Finding | ice cream sales and drownings | rise together |
| Explanation 1 | ice cream causes drowning | unlikely |
| Explanation 2 | drowning causes ice cream sales | absurd |
| Explanation 3 | hot weather causes both | that is the one |
The third option is the one people miss
- A pattern between two things is a real finding. Keep it.
- But it does not say which caused which, or that either did.
- Always ask what else could be driving both.
| Point | ||
|---|---|---|
| A pattern between two things | is a real finding | |
| It does not say | which caused which | |
| Or whether either did | a third thing may drive both | |
| So always ask | what else could explain this? |
Compare what you found to what you expected
- We thought Year 9s would sleep about eight hours.
- Our median came out at six and a half.
- So either the expectation was wrong, or our method was.
| Stage | Detail | |
|---|---|---|
| We expected | Year 9s to sleep about 8 hours | |
| We found | a median of 6.5 | |
| So either | the expectation was wrong | |
| Or | our method was |
And say which you think it is
- A surprising result is not automatically a discovery.
- It is also not automatically a mistake.
- Your job is to offer an explanation, not to ignore the gap.
How to write it down without overclaiming
- "Students read more" claims something about all students.
- "Students in our sample read more" claims what you measured.
- Name the group, and say what went with what, not what caused what.
| Version | The sentence | |
|---|---|---|
| Overclaimed | Students read more | means all students |
| Honest | Students in our sample read more | names the group |
| Overclaimed | Studying more raises scores | claims a cause |
| Honest | More study went with higher scores | reports what you saw |
But do not swing too far the other way
- All of this could leave you concluding nothing is knowable.
- That is just as wrong as overclaiming, and less useful.
- Aim for a claim exactly as strong as your evidence.
| Failure | What it is | |
|---|---|---|
| Too strong | claiming more than the data shows | |
| Too weak | refusing to conclude at all | |
| The goal | as strong as the evidence | and no stronger |
| Which sometimes | is a confident one |
What to take away
- Six things, and the last one is the skill the rest serve.
| Two honest samples | will disagree, and that is chance |
| Always ask | how many, and how were they chosen? |
| Check every graph’s axis | before you believe the picture |
| A pattern | has at least three possible explanations |
| Compare your finding | to what you expected, and explain the gap |
| Word the claim | so it is exactly as strong as your evidence |
Your turn
- Pause here and try all four.
- Answers are in the description.
| Question | |
|---|---|
| 1 | 12 people show a 5% gap. Strong evidence? |
| 2 | Ice cream and drownings rise together. Why? |
| 3 | Bars of 100 and 105, axis from 95. Problem? |
| 4 | Rewrite "students read more" so it does not overclaim. |
Show answer
1. No. Twelve people is a small sample, so a five percent gap could easily be chance. You would want far more before claiming a real difference. 2. Hot weather. It sends more people to buy ice cream and more people into the water, so it drives both without either causing the other. 3. The difference is five percent, but the second bar looks about twice the height. Every number on the chart is true and the picture still misleads. 4. Something like: students in our sample read more than the students in last year’s sample. It names the group and does not claim it holds everywhere.
The answers
- Twelve people is far too few for a 5% gap.
- Hot weather drives both, so neither causes the other.
- Say students in our sample, not students.
| Answer | |
|---|---|
| 1 | No, too few people |
| 2 | Hot weather |
| 3 | The axis starts at 95 |
| 4 | Name the group |
Is your child on track?
Take the free placement test