Unit 6 · Analysis and Communication
6-2 Use and Evaluation of Regression Analysis
Data often hides a relationship you can turn into a prediction: the warmer the day, the more ice cream sells. Regression analysis writes that relationship as an equation, so you can put in tomorrow's forecast temperature and predict tomorrow's sales. Yet a prediction equation is only a model — a simplified picture — and a careful analyst asks two more questions: how accurate is it, and does “fits well” really mean “causes”? A line that fits the data is not proof of cause.
Guiding question
How does regression turn data into a prediction, and how do we judge whether that prediction can be trusted?
Learning outcome. Explain the difference between simple and multiple regression and what a regression model represents, and evaluate a model using R², RMSE and MAE while explaining why a good fit does not prove causation.
1 · What it does
Regression analysis expresses the relationship between variables in the data as an equation, and uses it for prediction and simulation. From past data on temperature and sales, one can express how sales change when temperature rises as an equation, and put tomorrow’s forecast temperature into that equation to predict sales.
2 · The two variables have names
The variable that is the result is the response variable (also called the dependent variable). The variable used to explain or predict it is the explanatory variable. In the temperature-and-sales example, sales is the response variable and temperature is the explanatory variable.
3 · And the equation is called a model
The equation obtained from the data is the model. A model is a simplified representation of a real relationship — putting the trend “the higher the temperature, the higher the sales” into a format that can be calculated.
4 · Building it is only half the job
Both creating a model and evaluating whether it can actually be used are important. A model that is only created is not useful; only by confirming its accuracy and limitations does it lead to practical use.
1 · Simple regression — one explanatory variable
Simple regression analysis predicts the response variable using one explanatory variable. Predicting ice cream sales (y) from temperature (x) alone. The prediction equation is a straight line in the format y = ax + b, and this line is the regression line.
a and b are determined by the least-squares method: the method that determines the line so as to minimize the sum of the squares of the discrepancies between the actual values and the regression line. Each of those discrepancies is called a residual.
Try to beat the least-squares line
Eight days of ice-cream sales against temperature. You choose the line. The grey bars are the residuals — the discrepancy between each actual value and your line.
Your line: y = 3.00x + 0. Sum of the squared residuals: 796.0.
The best possible line scores 52.0. Yours scores 796.0 — 744.0 worse. Move the sliders and try to beat it.
Why squared, and not just added up? Because a residual above the line and one below it would cancel, and a line that misses everything by a mile would score zero. Squaring makes every miss count, and makes a big miss count much more than a small one.
2 · Multiple regression — several explanatory variables
Multiple regression analysis predicts the response variable using multiple explanatory variables. In real data, results are rarely determined by a single cause, so multiple regression — which can consider several factors at once — is suited to more realistic prediction. For example, predicting housing prices from a house’s size, age and distance from a transit station.
The prediction equation is the sum of multiple explanatory variables, each multiplied by a coefficient (a numerical value representing the strength of its influence), plus an intercept. With two explanatory variables the format becomes y = a₁x₁ + a₂x₂ + b. When there are three or more, terms are added in the same way.
| Aspect | Simple regression | Multiple regression |
|---|---|---|
| Number of explanatory variables | One | Two or more |
| Characteristics of the model | Simple and easy to interpret | Can express complex relationships |
| Suitable situations | When you want to capture the relationship between two variables concisely | When the response variable is influenced by multiple factors |
1 · Three indicators
To judge how useful a model created by regression analysis is, evaluation indicators are used.
| Indicator | Concept of calculation | Interpretation |
|---|---|---|
| R² (coefficient of determination) | Expresses, on a 0–1 scale, how well the model explains the variability in the data | The closer to 1, the better the variation in the data is explained |
| RMSE (root mean square error) | The square root of the mean of the squared discrepancies between predicted and actual values | The smaller the value, the higher the prediction accuracy; expressed in the same units as the response variable |
| MAE (mean absolute error) | The mean of the absolute values of the discrepancies between predicted and actual values | The smaller the value, the higher the prediction accuracy; less affected by extremely large discrepancies |
2 · Why there are two error measures and not one
Both RMSE and MAE indicate the size of prediction errors, but RMSE tends to weight large discrepancies more heavily. MAE evaluates discrepancies on average, so it is less affected by outliers. They are used differently according to the purpose of the analysis.
One odd day, and three indicators that disagree about it
Add one strange day: 33 °C but only 25 sales — the shop had to close early. It is a real event, not a typo, so you cannot simply delete it. Watch what each indicator does about it.
| Indicator | Now | Without the odd day |
|---|---|---|
| R² (coefficient of determination)how well the model explains the variability in the data, on a 0–1 scale — the closer to 1 the better | 0.988 | 0.988 |
| RMSE (root mean square error)the square root of the mean of the SQUARED discrepancies — weights large misses heavily | 2.55 | 2.55 |
| MAE (mean absolute error)the mean of the ABSOLUTE discrepancies — less affected by extremely large ones | 2.25 | 2.25 |
Eight ordinary days: R² close to 1, and RMSE and MAE both small and close to each other. Now press the button.
Notice one thing that never changes: RMSE is never smaller than MAE. Squaring cannot shrink a gap.
The table above says R² measures “how well the model explains the variability” and that RMSE “weights big misses more”. Both true, and neither tells you what to actually do. So here are all three, worked out by hand on the eight days you have been looking at. The numbers come out whole, which is not an accident — the data was built so you could check every line yourself.
🧮 Residuals, MAE, RMSE and R² — all three indicators from one table
- 1
The least-squares line for these eight days is y = 5x − 55. Use it to predict each day: at 22° it predicts 5(22) − 55 = 55.
Why this step: Every indicator is built from the same starting point — the gap between what actually happened and what the line said would happen. So the first job is always to produce the line’s prediction for each row.
- 2
Subtract to get each residual (actual − predicted): −3, 2, 4, −3, −1, 3, −2, 0
Why this step: A residual is one day’s miss. The sign says which way: negative means the line predicted more than really happened. This is the “discrepancy between the actual value and the regression line” the book names on p.40.
Most people slip here: Never just add the residuals up. They come to 0 here — and they come to almost 0 for any least-squares line, because the overs and unders cancel by design. A total of zero says nothing whatever about accuracy.
- 3
For MAE, drop the signs and average: (3 + 2 + 4 + 3 + 1 + 3 + 2 + 0) ÷ 8 = 18 ÷ 8 = 2.25
Why this step: Taking the absolute value is the simplest way to stop the cancelling. MAE therefore answers the plainest possible question: “on a typical day, how far off is the line?” Answer: about 2.25 ice creams. Every day counts the same amount.
- 4
For RMSE, square them instead: 9 + 4 + 16 + 9 + 1 + 9 + 4 + 0 = 52. Divide by 8 → 6.5. Then take the square root → 2.55
Why this step: Squaring also removes the signs — but it does something else as well, and that something is the whole point of RMSE. A miss of 4 contributes 16 while a miss of 2 contributes only 4, so one big miss weighs more than several small ones. The final square root just brings the answer back into ice creams, so it can be compared with MAE.
Most people slip here: Do the steps in that order: square, then average, then root. Rooting first would simply undo the squaring and hand you something meaningless.
- 5
Compare them: RMSE 2.55 is larger than MAE 2.25.
Why this step: RMSE is never smaller than MAE — that is a mathematical fact, not a property of this data. What is worth reading is the GAP between them: a small gap means the misses are all about the same size, a large gap means a few days are much worse than the rest. Here the gap is small, so the line is evenly wrong rather than occasionally disastrous.
- 6
For R², first ask how badly you would do with no model at all — just always guessing the average, 90. Those errors squared come to 4252.
Why this step: This is the step that makes R² make sense, and the book’s phrase “explains the variability” hides it completely. R² is a COMPARISON: your model against the laziest possible model, which ignores temperature and guesses the same number every day.
- 7
Now compare: R² = 1 − (52 ÷ 4252) = 1 − 0.0122 = 0.988
Why this step: Read it as: the line removed 98.8% of the error that just guessing the average would have left you with. That is what “explains the variability” means, and now it is a number you can act on rather than a phrase.
Most people slip here: R² of 0 does not mean “no relationship exists” — it means this model is no better than guessing the average. A perfect curved relationship fitted with a straight line can score close to 0 while being completely predictable.
The book’s wording, which is what an exam wants: R² … expresses, on a 0–1 scale, how well the model explains the variability in the data. The closer to 1, the better. (p.41)
Where the comparison breaks down: With coats and thermometers you already know which way the causation runs, so the joke is obvious. In real data you usually do not, and that is exactly when regression is most tempting and most dangerous — which is why the book gives correlation-versus-causation its own caution rather than leaving it to common sense.
This is the most examinable idea in the lesson, so state it precisely in an answer: a high R² shows the model predicts well; it says nothing at all about cause. To claim cause you need something regression cannot give you — a controlled experiment, or an argument about the mechanism from outside the data.
Notice too that this is the same shape of problem as the proxy variable in lesson 1-4: a column that predicts well because it stands in for something else entirely.
The standard teaching example in statistics, drawn from real seasonal public-health data.
1 · Four cautions
① Do not confuse correlation with causation.
A prediction equation that fits does not establish causation. Causation needs temporal precedence, exclusion of other factors’ influence, and theoretical support.
② A model is not finished once created.
Confirm accuracy with evaluation indicators, verify validity in light of the purpose, and do not apply it outside the range of the data it was fitted to.
③ Do not judge a model by R² alone.
Make judgments together with other indicators.
④ Adding more explanatory variables does not necessarily improve a model.
Appropriate variables must be chosen according to the purpose.
2 · One perspective, not the truth
The results of regression analysis should not be treated as absolute truth as they are, but as one perspective that the data offers. Interpreting results appropriately and using them in real-world judgment is what matters.
How the real paper is built
The ministry sets half multiple choice and half written answer in every subject. The two halves are not the same skill. In the multiple-choice half you have to spot the wrong options quickly — and the wrong options are built out of the mistakes students really make. In the written half nobody gives you the answer to recognise; you have to produce it, in your own words, with a reason.
Practise both. Being good at one half and weak at the other costs you exactly half the paper.
- regression analysis
- — expressing a relationship as an equation for prediction.
- response variable
- — the result (dependent) variable.
- explanatory variable
- — the input used to make the prediction.
- model
- — the equation obtained from the data.
- simple regression
- — one explanatory variable; y = ax + b.
- regression line
- — the straight line of a simple regression.
- least-squares method
- — minimises the sum of squared residuals.
- residual
- — the discrepancy between an actual value and the line.
- multiple regression
- — two or more explanatory variables.
- coefficient
- — the weight on an explanatory variable.
- R²
- — how well the model explains the variability, 0 to 1.
- RMSE
- — root mean square error; weights big misses more.
- MAE
- — mean absolute error; less swayed by outliers.
- confounder
- — a hidden third cause behind a correlation.
Worked Example
(1) Mark each statement with ○ if it is true or × if it is false.
A Multiple regression analysis is a method of predicting the response variable using one explanatory variable.
B R² (coefficient of determination) is an indicator showing how well the model explains the variability in the data.
C The equation obtained from data in regression analysis is called the model, and is a simplified representation of a real relationship.
D If the prediction equation fits, it can be said that there is a causal relationship between the explanatory variable and the response variable.
(2) Match each description with an indicator: A RMSE, B R², C MAE.
a The mean of the absolute values of the discrepancies between predicted and actual values
b The square root of the mean of the squared discrepancies between predicted and actual values
c How well the model explains the variability in the data
1 · Answer the following questions.
2(1) · Which most appropriately describes the difference between simple and multiple regression?
Mark ○ for the correct description, and × for the ones that have swapped or denied the difference.
Simple regression can express complex relationships, while multiple regression is simple.
Simple regression uses multiple explanatory variables, and multiple regression uses one.
Simple regression uses one explanatory variable, and multiple regression uses multiple explanatory variables.
There is no difference in the number of explanatory variables; only the calculation method differs.
2(2) · Choose the most appropriate regression method for each purpose.
Match each description with the choice that fits it.
You want to predict ice cream sales from temperature (one variable).
You want to predict housing prices from a house’s size, age and distance from a transit station (multiple variables).
2(3) · Which most appropriately describes the interpretation of regression results?
Mark ○ for the correct statement, and × for the ones that read a fit as a cause.
If a prediction equation is obtained from regression analysis, the equation necessarily indicates a causal relationship.
The results of regression analysis should be treated as one perspective that the data offers, and whether causation has been demonstrated must be carefully evaluated.
If R² is high, the model is necessarily a good model.
If the prediction equation fits, one may conclude that the explanatory variable is the cause of the response variable.
2(4) · Which is NOT an appropriate description of evaluation indicators?
Mark ○ for the correct descriptions, and × for the one that is not.
R² takes values between 0 and 1; the closer to 1, the better the variation in the data is explained.
RMSE is the square root of the mean of squared discrepancies, expressed in the same units as the response variable.
MAE is the mean of the absolute values of the discrepancies, and is less affected by extremely large discrepancies.
If the value of R² is close to 1, the model is necessarily a good model without needing to check other indicators.
Think as an Engineer — Investigate & Decide
Your school canteen wants to predict its daily cold-drink sales so it can order the right amount. You plan a regression model.
1 · Set up the model.
Name the response variable, then list the explanatory variables you would try (for example the day’s temperature, the number of students present, and whether it is an exam day). Decide whether this is a simple or a multiple regression, and say why.2 · Judge and question it.
Choose which evaluation indicator you would report (R², RMSE or MAE) and justify it for this purpose. Then name one confounder — a hidden third factor that could make two variables move together without one causing the other — that could mislead you here.3 · Decide and interpret.
Suppose your model gives a high R². State whether that alone would let you claim “hot days cause higher sales”, and what else you would need. In pairs, compare models and agree on the single most useful explanatory variable to keep.
In a New Context
A city finds that months with more ice cream sales also have more swimming accidents, and a regression line fits the two closely. Explain why this does not mean ice cream causes accidents, and name the hidden factor most likely behind both.
Lesson Question — Answered
Regression analysis expresses the relationship between an explanatory variable (the input) and a response variable (the result) as a model: a regression line y = ax + b in simple regression, or a weighted sum such as y = a₁x₁ + a₂x₂ + b in multiple regression, fitted by least squares — minimising the sum of squared residuals.
Trust is judged with the evaluation indicators used together: R² for how much variability is explained, and RMSE and MAE for the size of prediction errors, RMSE weighting large misses more and MAE less swayed by outliers. Never R² alone, and never by adding variables for their own sake.
Above all, a good fit is only a correlation. Causation requires temporal precedence, ruling out other factors, and theory. Regression offers one perspective from the data, to be interpreted — not obeyed.
2 · Read the passage and fill in the blanks (a)–(h).
In regression analysis, the variable that is the result is called the ( a ), and the variable used to predict it is called the ( b ). The method that predicts using one explanatory variable is called ( c ), and the method that predicts using multiple explanatory variables is called ( d ). To judge whether a model is good, evaluation indicators are used. Indicators include ( e ), which shows how well the model explains the variability in the data, and ( f ) and ( g ), which show the size of the discrepancies between predicted and actual values. To claim causation from the results of regression analysis, temporal precedence, exclusion of other factors’ influence, theoretical support and so on are necessary, and one must not conclude that there is ( h ) just because the prediction equation fits.
3(1) · Which most appropriately describes correlation and causation?
Mark ○ for the correct statement, and × for the ones that over-claim or under-claim.
If there is correlation, causation necessarily holds.
To claim causation, temporal precedence, exclusion of other factors’ influence, theoretical support and so on are necessary.
If a prediction equation is obtained from regression analysis, the equation completely demonstrates causation.
If there is no correlation, it can be said that there is no relationship at all between the two variables.
3(2) · Which most appropriately describes the interpretation of evaluation indicators?
Mark ○ for the correct statement, and × for the others.
R² takes values between 0 and 1; the closer to 1, the better the variation in the data is explained.
RMSE and MAE use the same calculation method, and their values always coincide.
A model with a high R² is always a good model in any situation.
MAE is an indicator that weights large discrepancies more heavily.
3(3) · Which is NOT an appropriate point of caution when using regression models?
Mark ○ for the real cautions, and × for the one that is not.
Do not conclude that there is causation just because the prediction equation fits.
Confirm accuracy with evaluation indicators and verify the validity of the model in light of the purpose.
If R² is close to 1, one may judge the goodness of the model without checking other indicators.
Adding more explanatory variables does not necessarily improve accuracy; appropriate variables must be chosen according to the purpose.
Key takeaway
Regression predicts a response variable from one (simple, y = ax + b) or several (multiple) explanatory variables, fitted by least squares. Judge it with R², RMSE and MAE together — and never read a good fit as proof of cause.
One of you set the sliders to your best guess before looking, and write the sum of squared residuals down. Then press Show me the least-squares line. Whoever got closer explains what they were looking at when they guessed — that instinct is worth having.
Sign In to Keep Going
You've reached the end of the free preview. Sign in — and upgrade to Plus or Pro — to keep going with Programming and Artificial Intelligence.
Full Access
Unlock all 30 lessons