Unit 5 · Data Collection and Cleaning
5-3 Open Data and APIs
Some of the most powerful data in the world is free to use — population figures from the World Bank, global statistics from the UN, competition datasets on Kaggle. But “free” is not the same as “no rules.” To rely on open data you need to know where to get it, how to fetch it, and how you are allowed to use it. Checking the terms of use and citing your source are what turn borrowed data into trustworthy analysis.
Guiding question
Where does reliable open data come from, how do programs fetch it, and what rules govern its use?
Learning outcome. Explain what open data is and identify major reliable sources of it, and describe how data is obtained through an API and why terms of use and source citation matter.
1 · Three things you must know
To use open data in analysis you need to understand where to obtain it, how to obtain it, and how to use it. Observing these points correctly is what makes reliable analysis possible.
2 · Where it comes from
Open data comes from international organizations, government agencies, and datasets published by researchers and companies. It is important to choose data sources with high reliability that match the purpose of the analysis.
1 · The three the book names
| Provider | Description | Main uses |
|---|---|---|
| World Bank Open Data | Statistics on economy, population, education and health for countries around the world | International comparisons; social and economic trends in each country |
| UN Data | International statistics published by the United Nations and its related agencies | Global issues such as population, development and environment |
| Kaggle | A platform that runs data analysis competitions; datasets in various fields are made public | Acquiring data for machine learning study and research |
2 · Two ways to get the data
On these platforms, data can be searched for and downloaded from a browser — and in many cases it can also be obtained through APIs.
1 · What an API is
An API (Application Programming Interface) is a set of rules for how pieces of software interact with each other. In web services, APIs are also widely used as a means of obtaining data. Using them, the data you need can be obtained directly, without going through a browser, which lets large amounts of data be handled efficiently.
Step through the request and the response
Getting data through an API is a conversation with an order. Step through it.
Client
your program
· · ·
Server
stores the data
Nothing has happened yet. Before you press anything: who speaks first?
2 · JSON
The data format most widely used for data returned via APIs today is JSON (JavaScript Object Notation). JSON represents data as combinations of names and values; it is easy for humans to read and easy to handle in various programming languages.
3 · And the rules that come with it
When using an API it is important to follow the terms of use, not send excessive requests in a short time, and confirm the scope of allowed use of the obtained data.
1 · “Free” is not “no rules”
Even with open data, the scope of free use is set for each dataset. Always confirm the terms of use of the data before using it.
2 · Citing the source
When using data, it is required to cite the source. The source generally includes the data provider, the name of the data, the date of acquisition, and the URL.
Take the citation apart and see what each piece was holding up
Here is a complete source citation. Switch a part off and see what stops being possible.
Source
World Bank Open Data, Individuals using the Internet (% of population), obtained 2026-08-27, https://data.worldbank.org/indicator/IT.NET.USER.ZS
Complete. A reader can see who produced it, exactly which data it is, when you took it, and where to go and check. That is what makes the citation do its two jobs: it signals reliability, and it lets others verify your analysis.
3 · Why it matters, in two words
Citing the source is important both as an indication of the reliability of the data and to allow others to verify the analysis.
Think of the difference like this. Downloading a file is like taking a photograph of a noticeboard: you get what it said at that moment, and you have to go back and take another photo when it changes.
An API is like being able to phone the office and ask a question. Your program sends a request to an address — the endpoint — and the answer comes back straight away, up to date, in a form the program can use without a human opening anything.
The book’s wording, which is what an exam wants: Data can be obtained through APIs, which return data directly to a program, often as JSON. (p.23)
What actually comes back — this is JSON
The reply is not a picture and not a page. It is plain text laid out in a strict pattern, so a program can pick out any value without guessing:
{
"country": "Egypt",
"year": 2023,
"population": 112716598,
"source": "World Bank Open Data"
}Read it as pairs: a name, a colon, then its value. So "year": 2023 means the item called year holds 2023. That is all JSON is — labelled values, so a program can ask for “population” and get one number, every time, without a person reading it.
🧮 Deciding whether you are allowed to use a dataset
- 1
Find the terms of use before you use the data — not after your analysis is finished.
Why this step: The rules decide whether your work can be published at all. Discovering afterwards that a dataset forbids the use you made of it means the work has to be redone with different data.
Most people slip here: “Open” and “free to download” do not mean “no rules”. Almost every open dataset carries conditions, and the commonest one is that you must say where it came from.
- 2
Check what the terms actually permit: may you use it commercially? May you republish the data itself? Must you share your results under the same terms?
Why this step: These are the three questions that differ most between datasets. A dataset that is fine for a school project may not be usable in something sold, and one that allows charts may not allow republishing the raw file.
- 3
Judge the source’s reliability against your purpose.
Why this step: The book makes the point that reliability varies. World Bank Open Data and UN Data are published by international organisations. Kaggle is a platform where many people upload datasets — excellent for practice and machine-learning study, but the dataset is only as trustworthy as whoever put it there.
Most people slip here: This is not “Kaggle is bad”. It is that the platform does not vouch for every file on it, so on Kaggle you must check who produced the data; on a World Bank page the organisation itself is the answer.
- 4
Cite the source: the provider, the dataset, and the date you obtained it.
Why this step: Citing lets anybody check your work, and lets you check it yourself later. The date matters because these datasets are updated — figures you downloaded last year may no longer be the figures on that page today.
This is the practical meaning of the book’s warning that reliability varies. It is not that shared datasets are bad; it is that who published it is part of the data, and a dataset with no traceable source cannot support a conclusion no matter how clean it looks. Notice this is the same idea as secondary data in lesson 5-1 — someone else gathered it, for their own purpose, and their choices are now inside your analysis.
A recurring problem with datasets shared on public platforms.
How the real paper is built
The ministry sets half multiple choice and half written answer in every subject. The two halves are not the same skill. In the multiple-choice half you have to spot the wrong options quickly — and the wrong options are built out of the mistakes students really make. In the written half nobody gives you the answer to recognise; you have to produce it, in your own words, with a reason.
Practise both. Being good at one half and weak at the other costs you exactly half the paper.
- open data
- — data made public for anyone to use.
- World Bank Open Data
- — country statistics on economy, population and more.
- UN Data
- — international statistics from the United Nations.
- Kaggle
- — a data analysis competition platform with public datasets.
- API
- — rules for how software interacts, used to fetch data.
- endpoint
- — the URL a request is sent to.
- JSON
- — a common name/value data format.
- terms of use
- — the rules set for each dataset.
- source citation
- — provider, data name, date obtained, URL.
Worked Example
(1) Mark each statement with ○ if it is true or × if it is false.
A World Bank Open Data publishes statistics on economy, population, education and so on for countries around the world.
B Using an API, necessary data can be obtained directly without going through a browser.
C As long as data is open data, it can be used freely without checking the terms of use.
D When using data, it is required to cite the provider, the name of the data, the date of acquisition, the URL and so on as the source.
(2) Match each description with a provider: A UN Data, B World Bank Open Data, C Kaggle.
a Publishes statistics on economy, population, education and so on for countries around the world.
b Provides international statistics published by the United Nations and its related agencies.
c A platform that runs data analysis competitions.
1 · Answer the following questions.
2(1) · Which most appropriately describes the flow of obtaining data via an API?
Mark ○ for the correct flow, and × for the ones that have the conversation wrong.
The client sends a request to the server, and the server returns the data in a format such as JSON.
The server one-sidedly continues to send data to the client.
Clients exchange data directly with each other.
The server only stores data; clients cannot retrieve it.
2(2) · Choose the most appropriate open data provider for each purpose.
Match each description with the choice that fits it.
You want to compare GDP across countries.
You want to find datasets from various fields for machine learning study.
2(3) · Which most appropriately describes the use of open data?
Mark ○ for the correct statement, and × for the ones that treat “free” as “no rules”.
Open data can be used freely, so the terms of use do not need to be checked.
Source citation is required only for data published for a fee.
Even when using open data, it is required to cite the provider, the name of the data, the date of acquisition, the URL and so on as the source.
For data obtained via an API, there is no need to follow the terms of use.
Think as an Engineer — Investigate & Decide
For a school project you want to show how Internet access in Egypt has changed over the past ten years compared with two neighbouring countries. You must choose an open data source and a way to obtain it.
1 · Inspect two sources.
Look up two candidate providers for this exact question (for example World Bank Open Data and UN Data). For each, note whether it actually holds Internet-access figures by country and year, and how current the latest year is.2 · Analyze fit and rules.
Compare the two on a point the lesson table does not list — such as how far back the data goes, or whether its terms of use allow you to republish a chart in a public report. Name one risk of picking the wrong source here.3 · Decide.
Choose one provider and one fetch method (browser download or API) and justify both from what you found. Write the source citation you would include: provider, data name, date obtained, URL. In pairs, compare choices and agree on the more defensible source.
In a New Context
A weather app on a phone shows today’s forecast by fetching it from a public weather service. Describe the request-and-response steps between the app and the server, and name the data format the server most likely returns.
Lesson Question — Answered
Reliable open data comes from trusted providers chosen to fit the purpose — World Bank Open Data for country economic and social statistics, UN Data for United Nations international statistics, and Kaggle for datasets used in machine learning study.
It can be downloaded through a browser, or fetched by a program through an API: the client sends a request to an endpoint, and the server returns the data in a set format, most often JSON, which the client then uses.
Because open data still comes with rules, every dataset carries terms of use that must be followed — including not overloading an API with requests — and every use requires citing the source (provider, data name, date of acquisition, URL), which signals reliability and lets others verify the analysis.
2 · Read the passage and fill in the blanks (a)–(e).
To use open data in analysis, it is necessary to understand where to obtain it, how to obtain it, and how to use it. Statistics on economy, population, education and so on for countries around the world can be obtained from sources such as ( a ), operated by the World Bank. Also, in addition to downloading individually from a browser, by using ( b ), necessary data can be obtained directly in formats such as ( c ) in response to a request. When using open data, the ( d ) determined for each dataset must always be checked, and when data is used, the provider, the name of the data, the date of acquisition, the URL and so on must be cited as the ( e ).
3(1) · Which most appropriately describes Kaggle?
Mark ○ for the description of Kaggle, and × for the others.
A portal operated by the World Bank that publishes statistics for countries around the world.
A site for international statistics operated by the United Nations.
A platform that runs data analysis competitions, on which datasets in various fields are made public.
A system that automatically records browsing histories of websites.
3(2) · Which is NOT an appropriate description of source citation?
Mark ○ for the appropriate descriptions, and × for the one that is not.
Source citation is important as an indication of the reliability of the data.
Source citation is important to allow others to verify the analysis.
Sources include the data provider, the name of the data, the date of acquisition, the URL and so on.
When using open data, it is not necessary to cite the source.
3(3) · Which is NOT an appropriate point of caution when using an API?
Mark ○ for the real cautions, and × for the one that is not.
Use the API in accordance with the terms of use determined by the provider.
Avoid sending excessive requests in a short time.
Confirm the scope of allowed use of the obtained data in advance.
Because of how APIs work, they may be used without checking the terms of use.
Key takeaway
Pick a reliable open data source for your purpose (World Bank Open Data, UN Data, Kaggle), fetch it by browser or API (the client requests an endpoint; the server returns JSON), and always follow the terms of use and cite the source.
Each find one chart online that gives no source at all. Ask each other the same two questions: who made this data, and could either of us go and check it? Notice how quickly a chart stops being evidence.
Sign In to Keep Going
You've reached the end of the free preview. Sign in — and upgrade to Plus or Pro — to keep going with Programming and Artificial Intelligence.
Full Access
Unlock all 30 lessons