Chapter 1 · Information Technology and Society
1-4 Ethical Issues with AI
AI now helps make real decisions about people: selecting candidates for job interviews, identifying individuals through face recognition, and determining how personal data is used. But an AI can be unfair if the data it learned from was biased, and it can be hard to know why it made a decision at all.
Guiding question
What ethical issues arise as AI spreads, and what principles should guide how we use it?
Learning outcome. Explain what algorithmic bias is and describe its main causes, and connect the basic principles of AI ethics — fairness, transparency, privacy protection and accountability — to specific situations.
With a partner. Look at this situation: a face-recognition AI is more likely to misidentify people from some ethnic groups than others. Before you read on, predict who could be harmed by this, and how. Give a reason.
Algorithmic bias
Algorithmic bias … bias in AI judgments caused by bias in the training data.
e.g., a hiring AI unfairly evaluating a particular gender; a face-recognition AI being more likely to misidentify particular ethnic groups
Main causes of bias:
Training data is biased — for example, insufficient data on certain attributes, or past discriminatory tendencies reflected in the data. Bias can also enter through the way the system is built: an inappropriate choice of variables, a proxy variable that indirectly stands in for a protected attribute, or the design of the model itself.
What does NOT cause it.
Algorithmic bias is not caused by the computer being slow, and it is not the AI “choosing” to be unfair. It is a machine faithfully repeating a pattern it was shown. That is why the fix is to check the data and test the results across different groups — not to buy a faster computer.
Nothing about the AI changes between these two runs. Only what it learned from does.
The company mostly hired one group in the past, so that is what the data mostly contains.
What the training data contained
Who the AI then recommended
The AI learned "successful applicants look like the ones we hired before" — and repeated it.
About these numbers. They are made up to show the shape of the problem — the book gives no figures. What is not made up is the mechanism: bias in the training data becomes bias in the AI’s judgments, which is what p.27 says.
Everyone reaches the same idea within about ten seconds of meeting algorithmic bias: if the AI cannot see gender, it cannot discriminate by gender. So delete the column. Problem solved.
It does not work, and understanding why it does not work is the difference between a weak answer and a strong one on this topic.
The book’s wording, which is what an exam wants: Bias can also enter through the way the system is built — for example, an inappropriate choice of variables, a proxy variable that indirectly stands in for a protected attribute, or the design of the model itself. (p.27)
Step 1 of 6 — What the company actually did
Twenty people applied for jobs in the past. Of the ten men, eight were hired. Of the ten women, two were hired. This is the training data — a record of what people DID, not of what they should have done, and the machine has no way of knowing the difference.
The twenty applicants are invented for teaching — the book gives no figures, and presenting made-up statistics as the book’s would be the very thing this lesson warns about. What is real is the mechanism: information that has been deleted from one column can still arrive through another. Keep the book’s own wording for the exam — past discriminatory tendencies are reflected in the data — and keep this picture for knowing why removing the column is not the end of the problem.
🧮 Deleting the column, and watching the bias survive it
- 1
Look at what actually happened in the past: of 10 men who applied, 8 were hired. Of 10 women, 2 were hired.
Why this step: This is the training data, and it records what people DID — not what they should have done. The book’s first cause of bias is exactly this: past discriminatory tendencies are reflected in the data.
Most people slip here: The AI has no way to know this history was unfair. To it, an 80% hire rate and a 20% hire rate are simply two facts about the world it is being asked to learn.
- 2
Apply the obvious fix: delete the gender column entirely, so the model cannot see it.
Why this step: A completely reasonable thing to try, and worth trying. If bias could only travel through the protected column, this would be the end of the problem.
- 3
Now look at what is left. One column reads “played in the men’s football team”: 7 of the 10 men have a 1, and all 10 women have a 0.
Why this step: This column was never intended to be about gender. But it separates men from women almost perfectly, which means it carries the information you just deleted. That is what makes it a proxy.
Most people slip here: Nobody put it there to cheat. It is a normal thing to record about an applicant, and its connection to gender is a fact about the world, not about the database.
- 4
The model learns from the remaining columns. Of the 10 people hired, 7 have a 1 in that column — so it gets a large positive weight.
Why this step: Exactly the process from lesson 1-2: a clue that lines up with the answers gets a big importance score. The model is doing its job correctly. It has found a genuine pattern in the data it was given.
- 5
Check the outcome: the model still favours men at close to the original rate — with no gender column anywhere.
Why this step: The bias travelled through the proxy. Deleting the protected attribute removed the evidence of discrimination without removing the discrimination — which is arguably worse, because now it is harder to prove.
- 6
Conclude: fairness has to be measured in the OUTCOMES, not assumed from the inputs.
Why this step: This is the strong version of the answer, and it is exactly what the book’s Key Fact means by “checking for bias”. You cannot establish that a system is fair by listing what you did not give it. You have to look at what it decides, group by group, and count.
Where the comparison breaks down: A landlord choosing by postcode knows what they are doing and could be challenged on it. A model has no idea it has found a proxy, and neither does anyone else unless they go looking — which is the black-box problem and transparency arriving together, and why the book puts explainable AI (XAI) in the very next section.
The finding is important; so is the method, and the method is the examinable part. A single overall accuracy figure hid the problem completely, and would have gone on hiding it forever. It only appeared when somebody counted the groups separately — which is the practical meaning of the last step in the box above.
Published accuracy audits of commercial face-recognition systems, and the basis of the book’s own example on p.27.
2 · Privacy issues
The development of AI technology has given rise to new issues concerning privacy.
① Surveillance through face recognition
Cameras in public spaces can identify and track individuals.
② Mass collection of personal data
Large amounts of online behaviour data are collected and analysed.
3 · Explainable AI (XAI) and responsibility
Explainable AI (XAI) … technology that makes it possible for humans to understand why an AI made a particular judgment.
When an AI’s decision-making process is opaque (a black box), it is difficult to verify whether the result is correct.
Responsibility … the question of who is responsible when an AI makes an incorrect judgment.
Views on responsibility differ depending on whether one is the developer, the user, the operator, and so on.
Notice what the book does not say. It does not tell you who is responsible — because no clear standard has yet been established. That is an honest answer to an open question, and an exam answer that says “the developer is responsible” as though it were settled has claimed more than is known.
Basic principles of AI ethics
The following basic principles are considered important for using AI appropriately.
Fairness
Not unjustly discriminating against any particular person or group
Transparency
Showing the AI’s decision-making process and inner workings clearly
Privacy protection
Handling personal information appropriately and protecting privacy
Accountability
Being answerable for the AI’s decisions
Each principle answers one of the problems above. Bias → fairness. The black box → transparency. Surveillance and mass data collection → privacy protection. “Who is responsible?” → accountability. Learning them as a matched pair with the problem they solve is far easier than learning four separate definitions.
How the real paper is built
The ministry sets half multiple choice and half written answer in every subject. The two halves are not the same skill. In the multiple-choice half you have to spot the wrong options quickly — and the wrong options are built out of the mistakes students really make. In the written half nobody gives you the answer to recognise; you have to produce it, in your own words, with a reason.
Practise both. Being good at one half and weak at the other costs you exactly half the paper.
- algorithmic bias
- — bias in AI judgments caused by bias in the training data.
- explainable AI (XAI)
- — technology that lets humans understand why an AI judged as it did.
- black-box problem
- — when an AI’s decision process is opaque and cannot be verified.
- responsibility
- — who answers when an AI judgment is wrong.
- fairness
- — no unjust discrimination against a person or group.
- transparency
- — showing the AI’s decision-making process clearly.
- privacy protection
- — handling personal information appropriately.
- accountability
- — being answerable for the AI’s decisions.
Worked Example
(1) Mark each statement with ○ if it is true or × if it is false.
A Algorithmic bias is bias in AI judgments caused by bias in the training data.
B Even if an AI’s decision-making process is opaque, there is no problem as long as the result is correct.
C Explainable AI (XAI) is a technology that makes it possible for humans to understand the AI’s reasoning.
D When an AI makes an incorrect judgment, who is responsible has already been clearly determined.
(2) Match each description (a–d) with the principle it describes: A Fairness, B Transparency, C Privacy protection, D Accountability.
a Not unjustly discriminating against any particular person or group
b Showing the AI’s decision-making process and inner workings clearly
c Handling personal information appropriately and protecting privacy
d Being answerable for the AI’s decisions
1 · Answer the following questions.
2(1) · Which is NOT an appropriate cause of algorithmic bias?
Mark ○ for each one that IS a cause of algorithmic bias, and × for the one that is not.
Training data is biased.
Past discriminatory tendencies are reflected in the data.
The processing speed of the computer is slow.
There is insufficient data on certain attributes.
2(2) · Choose all that are privacy issues associated with AI.
Mark ○ for each one that IS a privacy issue, and × for each one that is not.
Individuals are identified and tracked through face-recognition technology.
Online behaviour data is collected and analysed in large quantities.
AI creates creative works.
AI computation speed improves.
Think as an Engineer — Investigate & Decide
Building on the hiring-AI case above, investigate a real example and decide on a rule.
1 · Investigate.
Find one documented case where an AI system was accused of being unfair or biased, and record the source you used (a news report, study, or official statement). What group did it affect, and what data might have caused it?2 · Consider responsibility.
When an AI makes an unfair decision, who is responsible — the developer, the company using it, or the operator? Note the responsibility each one holds.3 · Decide.
Before a company uses a hiring AI, what one rule should it follow to make the system fairer? Justify your rule using the principles of AI ethics.
Step 1 says “documented” for a reason.
This lesson is about real harm to real people, so an example half-remembered — or worse, one an AI made up when you asked it — would be exactly the failure the chapter has been warning about. Record where you found it. An argument about fairness that rests on an invented case is not an argument at all.
In a New Context
Lesson Question — Answered
The lesson asked what ethical issues arise as AI spreads and what principles should guide how we use it. Because AI now makes decisions that affect people, three concerns stand out: an AI can be unfair when it learns from biased data; it can put people’s privacy at risk; and it can be difficult to understand, or to hold anyone answerable for, its decisions.
Handling these well takes more than accuracy. The lesson sets out four principles to aim for — fairness, transparency, privacy protection and accountability — and keeping AI fair, transparent, privacy-respecting and accountable is what makes its use responsible.
1 · Quiz yourself first.
Cover the Point! sections and name the four principles with what each one means, and say what causes algorithmic bias and what XAI is for.
2 · Read the passage and fill in the blanks.
As AI technology spreads widely in society, attention has been drawn to various ethical issues. For example, there is the issue of ( a ), in which bias arises in AI judgments due to bias in the training data. Also, for the “black-box” problem — where it is unclear why an AI made a particular judgment — a technology called ( b ) is being researched. Furthermore, when an AI makes an incorrect judgment, ( c ) is questioned, but no clear standard has yet been established among developers, users and operators.
3(1) · For each situation, which principle of AI ethics is most closely related?
Match each description with the choice that fits it.
Ensuring that a hiring AI evaluates fairly regardless of gender
Disclosing the AI’s decision-making process to users in an easy-to-understand way
Not using collected personal data for purposes other than as originally intended
Being answerable when an AI’s diagnostic result turns out to be wrong
3(2) · Which most appropriately describes ethical issues with AI?
Mark each statement with ○ if it is true or × if it is false.
There are no ethical issues with AI.
Algorithmic bias is caused by the AI’s processing speed.
Bias in training data can cause discriminatory bias in AI judgments.
Even if an AI’s decision-making process is opaque, there is no problem for users.
Reflect: Has this lesson changed how much you would trust an AI decision about you? Which of the four principles do you think is hardest to guarantee, and why?
Challenge: Choose one place AI is used (hiring, face recognition, or recommendations). Suggest one rule based on the AI ethics principles that would make it more trustworthy.
Keep reading — this lesson marks itself done automatically.