Chapter 4 · Web and Media Design
4-3 Methods for Evaluating Websites
How do you know whether a website works well for its users? You can watch real users try it and ask what confuses them — that is qualitative evaluation. Or you can count how many people visit, how long they stay, and how many buy — that is quantitative evaluation. Numbers tell you how much is happening; users' voices tell you why.
Guiding question
How can a website be evaluated, and why is it useful to combine numbers with users’ voices?
Learning outcome. Distinguish qualitative from quantitative evaluation and explain the major indicators used in web analytics, and explain how heuristic evaluation works and why combining both approaches gives a clearer picture.
With a partner. Imagine a shop where 500 people walk in each day and only two buy anything. You know the numbers. What do the numbers not tell you? Write down two questions you would still need to ask a real customer.
Qualitative and quantitative evaluation
Methods for evaluation generally fall into two main categories. These are used in a wide range of applications, and the same concepts apply to website evaluation.
| Evaluation method | Description (purpose) | Specific techniques |
|---|---|---|
| Qualitative evaluation | Explore in depth the reasons behind users’ psychology and behaviour, and evaluate qualitative aspects that cannot be expressed in numbers. | • Usability testing: observe actual operations to identify points that are hard to use • Interviews: hear users’ direct voices to understand their needs and dissatisfactions |
| Quantitative evaluation | Use numerical data such as access counts and time, and evaluate the current state objectively and statistically. | • Web analytics: analyse metrics such as PV count and time spent • A/B testing: compare two options to verify which leads to better results |
Qualitative evaluation is suited to exploring the cause of “why is it hard to use”, while quantitative evaluation is suited to grasping the overall trend of “how much is it being used”. By combining both, more accurate evaluation is possible.
Major indicators in web analytics
Web analytics … a technique for analysing how a website is accessed using numerical values; it is the central method of quantitative evaluation for websites.
| Indicator | Meaning | Description |
|---|---|---|
| (PV (Page View)) | The total number of times a page has been viewed | A basic indicator that measures how widely the site is seen and how much exposure it gets. |
| (Bounce rate) | The proportion of visits that ended after viewing only the first page | An indicator of whether the page content matched the user’s expectations and whether further actions were prompted. |
| (Conversion rate (CVR)) | The proportion of visits that achieved a target action (purchase, registration, etc.) | A very important indicator that measures how much the site contributes to business results (profits). |
Do not swap bounce rate and CVR.
Bounce rate counts people who left after the first page. CVR counts people who reached the target action. One is about leaving early, the other about finishing. The book’s Worked Example swaps them on purpose to see if you notice.
Each case shows the same three indicators with a different problem. Read what the numbers say first — then press the button and see what they cannot say.
PV (Page View)
12,400
total times the page was viewed
Bounce rate
22%
share of visits that ended after only the first page
Conversion rate (CVR)
0.4%
share of visits that reached the target action
What the numbers show (quantitative): Plenty of people arrive and most of them look around — but almost nobody completes the purchase.
Numbers point; they do not explain. Every one of these dashboards tells you exactly where to look and nothing at all about what to change. That is the reason the book says to combine both kinds of evaluation.
🧮 The three numbers, from a month of raw figures
- 1
A shop’s site had 1000 visits last month. Of those, 620 visitors left after seeing only the first page, and 35 visits ended in a purchase.
Why this step: All three of the book’s indicators come from counts like these. Notice each is measured per VISIT, so the same denominator serves all of them.
- 2
Bounce rate = visits that ended on the first page ÷ all visits = 620 ÷ 1000 = 62%.
Why this step: The bounce rate is the share who arrived and left again immediately. It is written as a proportion rather than a count so that months of different sizes can be compared — 620 out of 1000 and 62 out of 100 are the same performance.
Most people slip here: A high bounce rate is not automatically bad. Somebody who searched for your opening hours, found them on the first page and left, got exactly what they came for. This is precisely why the book insists quantitative evaluation is combined with qualitative — the number cannot tell you which kind of leaving it was.
- 3
Conversion rate (CVR) = visits achieving the target action ÷ all visits = 35 ÷ 1000 = 3.5%.
Why this step: CVR measures whether the site does its job, not whether it is busy. It is the one indicator tied to a purpose you chose in advance — and “the target action” has to be defined before you can measure it at all.
- 4
PV (page views) is a different kind of number: the total pages viewed, so one visit can produce many.
Why this step: This is why PV cannot go in the same fraction as the other two. It measures volume of activity, not proportion of visitors — a point the book makes by calling it a “basic indicator that measures how much”.
Most people slip here: Do not divide PV by visits and call it a rate. That gives pages per visit, which is a useful number but is not what any of the three indicators means.
🧮 An A/B test that looks convincing and is not
- 1
Two versions of a page are shown to 1000 visitors each. Version A gets 30 purchases; version B gets 42.
Why this step: This is A/B testing exactly as the book describes it — two versions, compared on real users, one thing changed between them.
- 2
Work out each conversion rate: A = 30 ÷ 1000 = 3.0%. B = 42 ÷ 1000 = 4.2%.
Why this step: Always compare rates, not raw counts, because the two versions rarely get exactly the same number of visitors. Here they happen to match, which makes the arithmetic easy to follow.
- 3
Compare them: B is 1.2 percentage points higher, which is 40% better in relative terms.
Why this step: And this is the moment where teams launch version B. A 40% improvement is a large claim and the numbers look like they support it.
Most people slip here: Notice how differently the same result can be described. “1.2 points better” and “40% better” are the same 12 extra purchases. Whenever a figure sounds dramatic, check which of the two it is.
- 4
Now ask the question from lesson 6-1: how often would luck alone produce a gap this big? Working it through gives about 0.15 — roughly a 1 in 7 chance.
Why this step: Purchases are rare events, so the counts wobble a lot from chance alone. Thirty and forty-two are small numbers; a handful of shoppers changing their minds moves the result by more than the difference being measured.
- 5
Compare with the usual 5% threshold: 0.15 is much bigger than 0.05, so this difference is NOT statistically significant.
Why this step: The honest conclusion is “we do not know yet”, not “B is better” and not “B is the same as A”. The right response is to keep the test running until there is enough data — not to launch, and not to abandon it.
Most people slip here: Stopping a test the moment it looks good is the most common way people fool themselves with A/B testing. If you check every day and launch on the first good-looking day, you will eventually get one by chance — which is the publication problem from lesson 6-1 in miniature.
Where the comparison breaks down: A thermometer is reliable, and analytics numbers often are not measuring quite what you think — a “visit’ ends after a timeout, so somebody reading carefully for twenty minutes can be recorded as having left. Know how a number is defined before trusting it.
Readers found it irritating and many gave up part-way, so the site got worse by every measure that had not been chosen as the target. This is the strongest argument for the book’s insistence on combining indicators and adding qualitative evaluation: any single number can be improved by damaging something it does not measure, and the better a team is at hitting the target, the faster that happens.
A well-known hazard in optimising a single metric.
Heuristic evaluation
Heuristic evaluation … a method in which experts identify problems in a website’s UI based on rules of thumb (heuristics). It is a form of qualitative evaluation, used along with usability testing and other methods to investigate the causes of usability problems.
Major evaluation perspectives:
① Consistency
whether the design and operations are unified across the site
② Error prevention
whether the design makes it hard for users to make mistakes
③ Visibility of system status
whether system status is clearly indicated to users (e.g., processing indicators, input error messages)
Note where it sits. Heuristic evaluation is qualitative, even though experts rather than ordinary users do it. What makes a method qualitative is that it looks for reasons — not who performs it.
How the real paper is built
The ministry sets half multiple choice and half written answer in every subject. The two halves are not the same skill. In the multiple-choice half you have to spot the wrong options quickly — and the wrong options are built out of the mistakes students really make. In the written half nobody gives you the answer to recognise; you have to produce it, in your own words, with a reason.
Practise both. Being good at one half and weak at the other costs you exactly half the paper.
- qualitative evaluation
- — the “why” from users’ voices and behaviour.
- quantitative evaluation
- — the “how much” from numerical data.
- web analytics
- — numbers on how a site is accessed.
- PV
- — total page views.
- bounce rate
- — share of visits that end after only one page.
- CVR
- — share of visits that reach the target action.
- heuristic evaluation
- — experts check the UI by rules of thumb.
Worked Example
(1) Mark each statement with ○ if it is true or × if it is false.
A Qualitative evaluation is a method that uses numerical data to evaluate quantitatively.
B PV (Page View) is an indicator showing the number of times a page has been viewed.
C The bounce rate is an indicator showing the proportion of visits that achieved a target action.
D Heuristic evaluation is a method in which experts identify UI problems based on rules of thumb.
(2) Match each description (a–c) with the indicator: A PV (Page View), B Bounce rate, C Conversion rate.
a The number of times a page has been viewed
b The proportion of visits that ended after viewing only the first page
c The proportion of visits that achieved a target action
1 · Answer the following questions.
2(1) · Choose all that are part of quantitative evaluation methods.
Mark ○ for each one that is QUANTITATIVE, and × for each one that is not.
Usability testing
Web analytics
Interviews
A/B testing
One question sorts them. Does the method produce a number, or does it produce a reason? Web analytics and A/B testing count things. Usability testing and interviews find out why.
2(2) · Which most appropriately describes the conversion rate?
Mark each statement with ○ if it is true or × if it is false.
The proportion of times a page has been viewed
The proportion of visits that ended after viewing only the first page
The proportion of visits that achieved a target action (product purchase, member registration, etc.)
The proportion of viewed pages relative to the total number of pages on the site
Think as an Engineer — Investigate & Decide
A small online bookstore in your city wants to know whether its website really works for customers. You plan how to evaluate it.
1 · Collect data.
Ask five classmates who have used an online bookstore to visit any online shop and, for each, record one number you could measure (for example, how many pages they viewed) and one thing they said was confusing. Put the results in a small table.2 · Analyze the signals.
Suppose the bookstore’s PV is high but its conversion rate is very low. What might that mean — and which method (quantitative or qualitative) would tell you why? Then consider one customer the data may hide — someone shopping only on a small phone screen, or a first-time visitor — and note one difficulty they might face that a raw PV count would never show.3 · Decide.
Recommend which two evaluation methods the bookstore should use first and give two reasons drawn from the table you built in step 1. In pairs, compare your choices and agree together on the single most useful indicator to track.
Notice what step 2 adds. Averages hide people. A high PV can look healthy while a whole group — everyone on a small screen, say — never gets past the first step. Numbers describe the crowd; only watching an individual shows you the obstacle.
In a New Context
Lesson Question — Answered
Evaluation falls into two kinds. Quantitative evaluation uses numerical data — through web analytics indicators such as PV (how many views), bounce rate (how many leave after the first page) and CVR (how many reach a target action) — to show how much a site is used.
Qualitative evaluation explores the reasons behind users’ behaviour through usability testing, interviews and heuristic evaluation, in which experts check the UI against rules of thumb. Numbers reveal what is happening but not why; users’ voices and expert checks reveal the causes. Combining both — the “how much” and the “why” — gives the most accurate evaluation and the clearest path to improvement.
1 · Quiz yourself first.
Cover the Point! sections and say what qualitative and quantitative evaluation each explore, define PV, bounce rate and CVR, and say what heuristic evaluation is and which kind it belongs to.
2 · Read the passage and fill in the blanks.
Methods for evaluating a website include ( a ) evaluation, which observes user voices and behaviour, and ( b ) evaluation, which analyses using numerical data. Typical indicators used in ( b ) evaluation are ( c ), which shows the number of times a page has been viewed; ( d ), which shows the proportion of visits that ended after viewing only the first page; and ( e ), which shows the proportion of visits that achieved a target action.
3(1) · For each evaluation method, which kind does it fall under?
Match each description with the choice that fits it.
Have users actually explore the website and observe its usability
Tally the access counts and page views of the website
Interview users to hear their impressions and improvement requests
3(2) · Which most appropriately describes heuristic evaluation?
Mark each statement with ○ if it is true or × if it is false.
A method in which users explore the site and usability is observed
A method in which numerical data is tallied and site usage is analysed
A method in which experts identify UI problems based on rules of thumb
A method in which two patterns are prepared and compared with data to see which is more effective
Reflect: If you could see only one metric for a site you use, would you choose PV, bounce rate, or CVR — and why?
Challenge: A page has a high bounce rate. Design a short qualitative test that would reveal the cause, and say what you would change first.
Sign In to Keep Going
You've reached the end of the free preview. Sign in — and upgrade to Plus or Pro — to keep going with Programming and Artificial Intelligence.
Full Access
Unlock all 30 lessons