Unit 6 · Analysis and Communication
6-1 Statistical Inference
You can never survey everyone, so you study a sample and reason back to the whole population — that is statistical inference. It answers two kinds of question: “about how big?” (estimation) and “is this real, or just chance?” (hypothesis testing). But because a sample is only part of the whole, every answer carries uncertainty. The skill is not to hide that uncertainty but to measure it — and to remember that “statistically significant” is not the same as “important”.
Guiding question
How can we reason from a sample to a whole population, and how do we measure the uncertainty in that reasoning?
Learning outcome. Explain the difference between point estimation and interval estimation including the meaning of a confidence level, and describe the procedure of hypothesis testing and the cautions needed when interpreting its results.
1 · What it is
A sample is drawn from the population and surveyed, and the properties of the population are then inferred from the results of the sample. That is statistical inference.
2 · Its two halves
| Approach | Description | Example |
|---|---|---|
| Estimation | Infer the properties of the population in numerical terms | Infer the population mean height from the sample mean height |
| Hypothesis testing | Determine whether a hypothesis about the population is correct | Determine whether a new learning method is more effective than the conventional one |
3 · And it is never certain
In either case, since the sample is only part of the population, inference always involves uncertainty. Judgment is made while considering the degree of that uncertainty.
1 · A single value, or a range
Point estimation
Estimating the population as a single value calculated from the sample. If the sample mean height is 168 cm, the population mean height is estimated to be 168 cm.
Interval estimation
Estimating that the population falls within a certain range. The population mean height is estimated to be in the range 166 cm to 170 cm.
2 · Confidence interval and confidence level
The interval used in interval estimation is the confidence interval, and how often such intervals contain the population value over repeated surveys is the confidence level. Levels of 95% or 99% are common.
Suppose an interval of “166 cm to 170 cm” is obtained at a 95% confidence level. The “95%” means that if the same survey method is repeated 100 times, about 95 of the resulting confidence intervals will contain the true population mean height.
So let us repeat it 100 times and count
The real mean height of every student is 168 cm.
A survey never knows that. It measures 25 students, takes their mean, and reports an interval around it. We are about to run that exact same survey 100 times — and, unlike the surveys, we can see the answer.
Before you press: if each survey is done perfectly, how many of the 100 intervals do you expect to miss 168 cm?
3 · What each one hides and shows
Point estimation is easy to understand because it is a single value, but the uncertainty of the estimate is invisible. Interval estimation has width, so care is needed in interpreting it — but it has the advantage of also showing the degree of uncertainty.
1 · The two hypotheses
Hypothesis testing is a procedure for determining whether a hypothesis about the population is acceptable, based on the data at hand. Two hypotheses are set up.
| Hypothesis | Description |
|---|---|
| Null hypothesis (H0) | The hypothesis serving as the starting point of the test, such as “no difference” or “no relationship” |
| Alternative hypothesis (H1) | The hypothesis adopted when the null hypothesis is rejected, such as “there is a difference” or “there is a relationship” |
2 · Why you begin by assuming what you want to deny
The probability that data collection could result in the data at hand is evaluated under the assumption that there is no difference or relationship. The reason for first assuming the hypothesis one wants to deny is that, rather than directly showing “there is a difference”, it is logically easier to judge this by showing that, under “there is no difference”, the data at hand would be unlikely to be obtained.
It is difficult to show directly that “the new learning method is effective”. Instead one assumes “there is no effect”, examines how low the probability is of obtaining a difference like the experimental result at hand, and if that probability is sufficiently small, judges that “no effect” is unlikely.
3 · And the verdict
Based on the evaluation, the null hypothesis is judged to be either rejected (denied) or not rejected.
1 · Four steps, in this order
Set the null hypothesis H0 and the alternative hypothesis H1
Decide the significance level (5% = 0.05 and 1% = 0.01 are commonly used)
Compute the p-value from the data at hand
Compare the p-value with the significance level and make a judgment
Notice that the significance level is decided at step ② — before the p-value is computed. Choosing it afterwards, once you have seen the p-value, is choosing the answer you wanted.
2 · The p-value
The p-value is, assuming the null hypothesis is correct, the probability of obtaining a result as extreme as, or more extreme than, the data at hand. The smaller the p-value, the less likely it is that the data would be obtained under the null hypothesis.
3 · The significance level, and the criteria
The significance level is the threshold value used to decide whether to reject the null hypothesis.
| Condition | Judgment |
|---|---|
| p-value < significance level | Reject the null hypothesis and adopt the alternative hypothesis (statistically significant) |
| p-value ≥ significance level | Do not reject the null hypothesis (cannot be said to be statistically significant) |
Make the judgment yourself
The whole judgment is one comparison. Set the significance level, choose a p-value, and read off the decision.
Significance level, decided first
p-value, computed from the data
0.03 < 0.05
Reject the null hypothesis and adopt the alternative. The result can be called statistically significant.
Try p = 0.03 at both significance levels. The data never changed. The same p-value is significant at 0.05 and not at 0.01 — which is exactly why the book says the significance level is a guide chosen in advance, not an absolute criterion of correctness.
A court does not begin by asking “how can we prove this person is guilty?”. It begins by assuming they are innocent, and then asks: if they really were innocent, how unlikely would this evidence be? Fingerprints on the weapon would be very unlikely — so the assumption of innocence becomes hard to hold, and it is rejected.
That is exactly the structure of a hypothesis test, and it maps across one line at a time:
- • “presumed innocent” = the null hypothesis H₀: no difference.
- • the evidence = your data.
- • “how unlikely is this evidence if they are innocent?” = the p-value.
- • “beyond reasonable doubt” = the significance level, chosen before you look.
And the piece almost everybody gets wrong: a verdict of not guilty does not mean “proved innocent”. It means the evidence was not strong enough. This is why the book is careful to say the null hypothesis is “not rejected” — never “accepted”, and never “proved true”.
Where the comparison breaks down: A court eventually has to decide something and the accused goes free or does not. A test can end in a permanent shrug — “not enough evidence, come back with more data” — and often should. Also, a court weighs evidence a person judges; a p-value is arithmetic, and it cannot notice that the study was badly designed.
Note carefully what it is not. It is not the probability that the null hypothesis is true, and it is not the probability that you are wrong. It is a statement about how surprising your data would be in a world where nothing is going on.
The book’s wording, which is what an exam wants: p-value … assuming the null hypothesis is correct, the probability of obtaining a result as extreme as, or more extreme than, the data at hand. (p.32)
🧮 Computing a p-value from actual data — the book’s “new learning method” example
- 1
State the two hypotheses. H₀: the new method makes no difference, so the average stays 70. H₁: there is a difference.
Why this step: The book’s procedure begins here, and the order matters: both hypotheses are fixed BEFORE the data is examined. Deciding what would impress you after seeing the result is how people fool themselves.
- 2
Choose the significance level first: 5%, so α = 0.05.
Why this step: This is the “beyond reasonable doubt” line, and choosing it before you look is the whole safeguard. Pick it afterwards and you can always pick one that gives you the answer you wanted.
- 3
Collect the data: 25 students used the new method and averaged 74 marks. The spread of marks is known to be about 10.
Why this step: Three numbers are needed to judge a gap — how big it is (74 vs 70), how many people it came from (25), and how much marks vary anyway (10). A 4-mark gap means nothing until you know the other two.
- 4
Work out how much the sample average bounces around by chance: 10 ÷ √25 = 10 ÷ 5 = 2 marks.
Why this step: This is the key idea in the whole lesson. Averages of groups vary less than individuals do — and the bigger the group, the less they vary. Dividing by the square root of the sample size is what makes a large survey more trustworthy than a small one.
Most people slip here: Divide by √n, not by n. Using 25 here would give 0.4 and make almost any difference look overwhelming. Quadrupling your sample only halves the wobble, which is why big surveys are expensive.
- 5
Measure the gap in those units: (74 − 70) ÷ 2 = 2.0
Why this step: The gap is now expressed as “two chance-sized steps away from what H₀ predicted”. This is what makes different studies comparable — a 4-mark gap and a 4-centimetre gap mean nothing to each other, but “2 steps out” means the same thing in both.
- 6
Look up how often chance alone goes at least 2.0 steps out, in either direction: p = 0.046.
Why this step: This is the p-value, and now it is not magic — it is the answer to “how often would luck do this?”. Roughly once in 22 attempts. The book’s definition says “as extreme as, or more extreme than”, and “either direction” is what makes it match the H₁ we wrote, which said different rather than better.
- 7
Compare and judge: 0.046 < 0.05, so reject H₀.
Why this step: This is the book’s criterion exactly. The result is statistically significant at the 5% level — and notice how close it is to the line.
Most people slip here: At a 1% significance level, 0.046 is NOT below 0.01 and the very same data would not be significant. The verdict changed without a single number in the experiment changing. That is why the level has to be chosen in advance and stated.
The true population mean is a fixed number. It is either inside your interval or it is not — there is no chance involved once the survey is done. What the 95% describes is the method, not this one interval: as the book puts it on p.31, if the same survey were repeated 100 times, about 95 of the intervals produced would contain the true value. Your interval is one draw from that process, and you cannot know which kind you got. That is exactly what the simulation earlier in this lesson shows when you run it: some of the intervals miss, and nothing about a missing one looks any different.
The trouble comes from what gets published. Interesting positive results get written up; the nineteen that found nothing often do not. So the published record fills up with the lucky ones, and when researchers later went back and repeated large numbers of well-known experiments, a substantial share did not reproduce.
The lesson for your own work is the one the book keeps making: a p-value below 0.05 is evidence, not proof, and a single study is a single study. It is also why the significance level must be chosen before looking at the data — the step most people are tempted to skip.
The reproducibility work published across psychology and medicine over the last decade.
1 · Two tests, two jobs
| Test | Main use | Example |
|---|---|---|
| t-test | Examine whether there is a difference in the means of two groups | Whether students’ average test scores differ between a certain learning method and the conventional method |
| Chi-squared test | Examine whether there is a relationship between two categorical variables (data expressed by classification, such as gender, occupation or preference) | Whether there is a relationship between gender and subject preference |
1 · Two ways a judgment can be wrong
Type I error
Rejecting the null hypothesis when it is actually correct. A false alarm.
Type II error
Not rejecting the null hypothesis when it is actually incorrect. A miss.
2 · And you cannot reduce both at once
Lowering the significance level reduces the Type I error, but tends to increase the Type II error. The two have a trade-off relationship.
3 · Three things a p-value is not
The p-value is NOT “the probability that the null hypothesis is correct.”
A small p-value does NOT mean “the effect is large.”
Comparing the p-value with the significance level is NOT an absolute criterion of correctness. The significance level is chosen according to the purpose of the analysis or the conventions of the field; falling below it does not mean the result is certainly correct.
4 · Significant is not the same as important
Suppose an experiment compares two designs of a website, with click-rate data from 100,000 people split evenly. The old design is clicked by 0.4% of visitors, the new one by 0.5% — only 0.1 percentage points higher. Because the data set is large, this difference is likely to be statistically significant. However, whether that 0.1-point difference leads to actual profit or a better user experience needs to be judged from a separate perspective.
Keep the difference fixed and move only the sample size
Two designs of a page. The old one is clicked by 0.4% of visitors, the new one by 0.5%.
That gap is 0.1 percentage points, and it never changes below. The only thing you can move is how many people were shown each design.
A statistics package run on this data reports p = 0.5973. Against a significance level of 0.05, the result is not statistically significant.
With this much data the difference cannot be told apart from chance. Push the slider right and watch what happens — remembering that the two designs stay exactly as good as they are now.
The book does not ask you to compute a p-value — only to interpret one against a significance level. The number above is reported the way a statistics package would report it.
5 · What statistical inference is for
Statistical inference is a powerful means of drawing conclusions from data, but it is important to understand that it is a tool for judgment that includes uncertainty, and the results should be interpreted appropriately.
How the real paper is built
The ministry sets half multiple choice and half written answer in every subject. The two halves are not the same skill. In the multiple-choice half you have to spot the wrong options quickly — and the wrong options are built out of the mistakes students really make. In the written half nobody gives you the answer to recognise; you have to produce it, in your own words, with a reason.
Practise both. Being good at one half and weak at the other costs you exactly half the paper.
- statistical inference
- — reasoning from a sample to the population.
- estimation
- — inferring the population in numbers.
- hypothesis testing
- — judging a claim about the population.
- point estimation
- — a single estimated value.
- interval estimation
- — an estimated range.
- confidence interval
- — the interval used in interval estimation.
- confidence level
- — how often such intervals contain the true value.
- null hypothesis (H0)
- — the “no difference” starting point.
- alternative hypothesis (H1)
- — adopted if H0 is rejected.
- p-value
- — chance of data at least this extreme if H0 holds.
- significance level
- — the threshold for rejecting H0.
- t-test
- — compares the means of two groups.
- chi-squared test
- — checks a relationship between two categories.
- Type I error
- — rejecting a true null hypothesis.
- Type II error
- — keeping a false null hypothesis.
Worked Example
(1) Mark each statement with ○ if it is true or × if it is false.
A Inferring the properties of the population from the sample is called statistical inference.
B Point estimation is a method that estimates the properties of the population within a certain range.
C In hypothesis testing, the probability that data collection could result in the data at hand is evaluated under the assumption that the null hypothesis is correct.
D If a result is statistically significant, the difference is also necessarily important in practice.
(2) Match each description with a term: A p-value, B Significance level, C Null hypothesis.
a A hypothesis serving as the starting point of the test, such as “no difference” or “no relationship”
b The probability of obtaining a result as extreme as, or more extreme than, the data at hand, assuming the null hypothesis is correct
c The threshold value for deciding whether to reject the null hypothesis
1 · Answer the following questions.
2(1) · Which most appropriately describes the procedure of hypothesis testing?
Mark ○ for the correct order, and × for the ones that have the steps out of order.
Set the null and alternative hypotheses → decide the significance level → compute the p-value → make a judgment
Compute the p-value → set the null hypothesis → decide the significance level → make a judgment
Decide the significance level → adopt the alternative hypothesis → compute the p-value → set the null hypothesis
Make a judgment → set the null hypothesis → decide the significance level → compute the p-value
2(2) · With the significance level set at 0.05 (5%), the p-value is 0.03. Which describes the judgment?
Mark ○ for the correct judgment, and × for the others.
Reject the null hypothesis and adopt the alternative hypothesis.
Do not reject the null hypothesis.
Reject the alternative hypothesis.
Change the significance level to 0.01 and recalculate.
2(3) · Choose the most appropriate test method for each purpose.
Match each description with the choice that fits it.
You want to examine whether students’ average test scores differ between two teaching methods.
You want to examine whether there is a relationship between gender and the category of products purchased.
2(4) · Which most appropriately describes the p-value?
Mark ○ for the correct description, and × for the three misunderstandings the book warns about.
The p-value is the probability that the null hypothesis is correct.
The smaller the p-value, the larger the effect is.
The p-value is the probability of obtaining a result as extreme as, or more extreme than, the data at hand, assuming the null hypothesis is correct.
p-value < 0.05 is an absolute criterion of correctness.
Think as an Engineer — Investigate & Decide
A school club claims its new online quiz tool raises students’ average score on weekly tests. You are asked to judge the claim fairly using a hypothesis test.
1 · Frame the hypotheses.
The club reports its tool group averaged 74 on the last quiz versus 70 for a control group of the same size. Write the null and alternative hypotheses for this exact claim in your own words, state what data you would still need to actually run the test (beyond these two means), and say which significance level you would set (0.05 or 0.01) and why.2 · Analyze the risks.
Describe, in this context, what a Type I error and a Type II error would each mean for the club, and say which one you would most want to avoid here — a judgment the lesson’s tables do not make for you.3 · Decide and interpret.
Suppose the test returns a p-value of 0.04. State your decision against the significance level you chose in step 1 — noting that it would differ at 0.05 versus 0.01. Then, if the result is significant, explain why even a significant result might still not justify switching every class to the tool. In pairs, compare interpretations and agree on what evidence beyond the p-value you would still want.
In a New Context
A shop tests two checkout-button colours with data from 200,000 visitors and finds the new colour’s click rate is 0.05 percentage points higher, with a p-value of 0.01 at the 5% significance level. State whether the result is statistically significant, and explain why the shop might still decide not to change the button.
Lesson Question — Answered
Because a sample is only part of the population, we use statistical inference — either estimation, which reports the population as a single value (point estimation) or as a range (interval estimation) whose confidence level says how often such intervals would contain the true value, or hypothesis testing, which sets a null hypothesis (“no difference”) against an alternative and compares a p-value with a chosen significance level to decide whether to reject the null.
The uncertainty is never removed: it is measured and reported. Judgment can fail as a Type I error (rejecting a true null) or a Type II error (keeping a false null), and lowering one raises the other.
And a result that is statistically significant — especially with very large samples — is not automatically important in practice. Results must be interpreted, not just computed.
2 · Read the passage and fill in the blanks (a)–(g).
In statistical inference, the properties of the population are inferred from the sample. Estimating the properties of the population as a single value is called ( a ), and estimating within a certain range is called ( b ). Also, the procedure of judging a hypothesis about the population is called ( c ). In hypothesis testing, the ( d ) — such as “no difference” or “no relationship” — and the opposite ( e ) are set up, and a judgment is made by comparing the ( f ) computed from the data at hand with the predetermined ( g ).
3(1) · Which most appropriately describes confidence intervals and confidence levels?
Mark ○ for the correct description, and × for the ones that misread what the confidence level belongs to.
A confidence interval with a 95% confidence level means that the true population value is contained in that interval with a probability of 95%.
The higher the confidence level, the narrower the confidence interval.
A confidence interval with a 95% confidence level means that, if the same survey method is repeated, about 95% of the resulting confidence intervals will contain the true population value.
The confidence level is a value used as the criterion for judgment in hypothesis testing.
3(2) · Which is NOT an appropriate description of errors in hypothesis testing?
Mark ○ for the correct descriptions, and × for the one that is not.
A Type I error is rejecting the null hypothesis when it is actually correct.
A Type II error is not rejecting the null hypothesis when it is actually incorrect.
Lowering the significance level can simultaneously reduce both Type I and Type II errors.
Type I and Type II errors have a trade-off relationship.
3(3) · Which most appropriately describes the interpretation of statistical inference?
Mark ○ for the correct statement, and × for the ones that claim more certainty than inference can give.
A statistically significant difference is necessarily an important difference in practice.
With a large amount of data, even a tiny difference can become statistically significant.
If the p-value falls below 0.05, the conclusion is certainly correct.
Statistical inference can completely remove uncertainty.
Key takeaway
Estimate with a point or an interval (the confidence level is how often the method captures the truth). Test by comparing a p-value with a significance level chosen in advance. Watch for Type I and Type II errors, and never confuse statistical significance with practical importance.
One of you press the 100-survey button and read out how many intervals missed. Do it five times. Then say out loud, together, what the 95% is a fact about — because saying it wrongly is worth marks lost, and it is very easy to say wrongly.
Sign In to Keep Going
You've reached the end of the free preview. Sign in — and upgrade to Plus or Pro — to keep going with Programming and Artificial Intelligence.
Full Access
Unlock all 30 lessons