Card 09

🎯 👥 ⚠️

A gym surveys its members and finds 95% exercise regularly. Can we conclude that people who join gyms exercise regularly?

💭 How to Think About This

95% sounds impressive! But wait—who's being surveyed? Current members. What about all the people who joined, stopped exercising, and quietly cancelled? They're not in the sample! This is SELECTION BIAS: when your sample isn't representative because of how people got selected into it.

Based on this survey, can we conclude gym members exercise regularly?

🔒 Write your thinking above to unlock hints

The survey reached CURRENT members only. But what about people who joined the gym, didn't exercise, and then quit? They're not counted! You're not measuring "gym joiners"—you're measuring "gym stayers." These are very different groups with very different behavior.

Imagine 100 people join: 50 quit (stopped exercising), 50 remain (still exercising). Survey the 50 remaining → 100% exercise! But of ORIGINAL joiners, only 50% exercise. The non-exercisers selected themselves out of your sample. Dropout creates a massively biased picture.

This trap appears constantly: • Restaurant reviews (only passionate people write reviews) • Customer satisfaction (unhappy customers already left) • "All our graduates found jobs!" (those who didn't graduate aren't counted) • Course reviews (only finishers review). Who's MISSING tells the real story.

To spot selection bias: (1) Ask "Who COULD be in this sample but ISN'T?" (2) Ask "What makes someone LEAVE the sample?" (3) Ask "Does success/failure itself affect who we observe?" The gym should track EVERYONE who ever joined, not just current members.

No! The 95% is misleading because it only counts people who stayed—those who quit aren't in the sample.

Your intuition was drawn to the impressive number, but this reveals the selection bias trap.

What's happening:

• People who exercise → stay as members → get surveyed

• People who don't exercise → quit → don't get surveyed

• Result: The sample is filtered to show only success stories

The key questions to ask:

• "Who COULD be in this sample but isn't?"

• "What would make someone leave the sample?"

• "Is the outcome affecting who we observe?"

The protection: Before trusting any statistic, trace the selection process. Who got included? Who got excluded?

You're right to question WHO was surveyed! The key issue: CURRENT members were surveyed. But what about people who joined, stopped exercising, and quit? They're invisible in this data. You're seeing "gym stayers," not "gym joiners"—a crucial difference.

SELECTION BIAS occurs when the way you choose your sample systematically excludes certain outcomes. Here: exercising → stay → counted. Not exercising → quit → not counted. The outcome (exercising or not) affects whether you're IN the sample. That's the bias!

If 100 people join and 50 quit within a year (stopped exercising), surveying remaining members shows 100% success. But true success rate is 50%! The worse the dropout rate, the more biased the survey. You'd need to track EVERYONE who ever joined to know the real rate.

Your skepticism is valuable! To detect selection bias: (1) Ask "Who's missing?" (2) Ask "Does the outcome affect selection?" (3) Ask "Would failures/dropouts be counted?" If failures systematically disappear from your data, you'll overestimate success rates dramatically.

Exactly right—it depends on who was surveyed, and this sample is biased!

You correctly sensed that the sample composition matters. Here's the full picture:

The selection bias:

• Only CURRENT members were surveyed

• People who quit (non-exercisers) aren't counted

• The sample is filtered to show only successes

What an unbiased study would need:

• Track EVERYONE who ever joined

• Include those who quit in the analysis

• Report: "Of people who join, X% are still exercising after 1 year"

Your habit: Always ask "Who's missing from this data?" Selection bias is everywhere once you learn to see it.

You spotted the SELECTION BIAS! The survey only reached current members—but people who joined and stopped exercising have already quit and aren't surveyed. You're seeing "gym stayers," not "gym joiners." The 95% is massively inflated by survivor bias.

If 100 people join and 80 quit within a year (stopped exercising), surveying the 20 remaining might show 95% exercising. But true rate? Maybe 20%! The 95% tells you about SURVIVORS, not about what happens to people who join gyms. Huge difference!

Your bias-detection skill applies everywhere: • "95% of our graduates got jobs!" (dropouts not counted) • "Our customers love us!" (unhappy ones already left) • "Reviews are 4.5 stars!" (only motivated people review) • "Successful people do X!" (failures doing X aren't interviewed)

To get unbiased data: Track EVERYONE from the start (intention-to-treat analysis). Follow up even with dropouts. Report: "Of 1000 people who joined, 200 were still exercising after 1 year (20%)." This honest reporting reveals the true picture—and is much less impressive!

Exactly right! The sample is biased—people who quit aren't counted, inflating the success rate.

You showed excellent probabilistic thinking by recognizing the selection problem.

The bias mechanism:

• Exercisers → stay → get surveyed → counted as success

• Non-exercisers → quit → not surveyed → invisible

• Result: 95% of SURVIVORS, not 95% of JOINERS

Your detection questions:

• "Who's missing from this data?"

• "Does the outcome affect who gets counted?"

• "What happened to the failures/dropouts?"

The protection: Before trusting impressive statistics, trace who got selected into the sample. Selection bias makes failures invisible, inflating apparent success rates.

🤔 Which thinking lens(es) did you use?

Select all the lenses you used:

Learn more about the 7 Thinking Lenses →

👨‍👩‍👧 For Parents & Teachers

🌱 A Small Everyday Story

"Our coaching students scored 95%+ in boards!"
boasted the advertisement.
But they only admitted students
who already scored 90%+ in mock tests.
And they quietly removed strugglers mid-year.
The "result" was baked into the selection process.

See more guidance →

🧠 Thinking habits this builds:

Automatically asking "who's NOT in this sample?"
Tracing the selection process before trusting statistics
Recognizing that dropout/attrition creates bias
Understanding that success/failure can affect who we observe

🌿 Behaviors you may notice (and reinforce):

"But who dropped out before this survey?" questions
Skepticism about statistics from self-selected groups
Looking for who's missing when hearing impressive numbers
Understanding why reviews skew positive or negative (not neutral)

How to reinforce: When you see impressive statistics, play detective together: "How did people get INTO this sample? Who would have gotten REMOVED? Does that affect what we're seeing?"

🔄 When ideas are still forming:

Some learners may think ALL statistics are useless because of selection bias. Help them see that bias can be minimized through careful study design—random sampling, intention-to-treat analysis, tracking dropouts.

Helpful response: "Selection bias is a problem to manage, not a reason to ignore all data. What would a BETTER study design look like?"

🔬 If you want to go deeper:

Explore "intention-to-treat analysis" in medical trials
Discuss how review platforms try to fight selection bias
Look up "Berkeley admission paradox" (Simpson's Paradox)

Key concepts (for adults): Selection bias, sampling bias, attrition bias, self-selection, non-response bias, convenience sampling, intention-to-treat analysis, representative samples.

A gym surveys its members and finds 95% exercise regularly. Can we conclude that people who join gyms exercise regularly?

🔄 Compare All Perspectives

✅ "95% is strong evidence"

🤔 "It depends who was surveyed"

❌ "The sample is biased"

🤔 Which thinking lens(es) did you use?