Supervised and Unsupervised Learning

The key difference is whether the training data includes known answers, called labels.

Within supervised learning, predicting a category such as spam or not spam is classification. Predicting a numeric quantity is regression. This topic’s required worked supervised method, k-nearest neighbours, is used here for classification. In unsupervised learning, clustering groups unlabelled examples according to a stated similarity measure; k-means is the required example.

Learning type	Training data	Goal	Syllabus example
supervised learning	labelled examples	predict a label for new data	k-nearest neighbours
unsupervised learning	unlabelled examples	discover groups or structure	k-means

Labels and Features

A feature is an input property used by a model. A label is the known target answer in supervised learning.

A supplied label is not automatically correct or unbiased. It may contain recording errors or reflect subjective human decisions, so label quality and representativeness must be checked before interpreting a model’s predictions.

Example:

Message length	Number of links	Label
20	0	not spam
25	1	not spam
200	5	spam
180	4	spam

Here:

• message length and number of links are features;
• spam and not spam are labels;
• each row is one labelled training example.

Supervised Learning

In supervised learning, every training example includes the expected output.

The system learns a relationship between input features and labels:

labelled training examples -> fitted model
new features -> predicted label

Supervised learning is suitable when:

• past examples are available;
• the desired output categories are already known;
• the goal is to predict a category for new data.

Examples include predicting spam or not spam, recognising a type of object in an image, or predicting whether a simple condition is present.

k-Nearest Neighbours

k-nearest neighbours, or k-NN, is a supervised-learning algorithm.

It stores labelled examples. When a new item arrives, it compares that item with the stored examples and looks at the labels of the nearest ones.

Basic k-NN classification:

For a new item:
1. calculate its distance from each labelled training example
2. order the examples from nearest to furthest
3. select the k nearest examples
4. predict the majority label among those neighbours

One-Dimensional Example

Suppose message length is the only feature:

Length	Label
20	not spam
30	not spam
150	spam
180	spam

For a new message of length 40 and $k=3$:

Training length	Distance from 40	Label
30	10	not spam
20	20	not spam
150	110	spam
180	140	spam

The three nearest labels are not spam, not spam, and spam. The majority is not spam, so this is the prediction.

Two-Dimensional Distance

With two numeric features, a common distance is Euclidean distance. For points $(x_1,y_1)$ and $(x_2,y_2)$:

$$d=\sqrt{(x_2-x_1{)}^2+(y_2-y_1{)}^2}$$

In a small trace question, the distances may be supplied or easy to calculate. The important idea is that smaller distance means greater similarity according to the selected features.

Choosing $k$

Choice	Possible effect
very small $k$	sensitive to noise or one unusual example
very large $k$	may include distant, less relevant examples
even $k$ in a two-class problem	can produce a tied vote

The best value is not chosen merely by preference. Different values should be evaluated using suitable data.

Distance and Feature Scale

Suppose one feature ranges from 0 to 1, while another ranges from 0 to 10000. The larger-scale feature may dominate the distance calculation even if it is not more important.

This is why data preparation can matter before k-NN. At this level, recognise the problem: distances depend on both the selected features and their numerical scales.

If scaling parameters are estimated from data, estimate them from the training set only. Apply those same values unchanged to validation, test and new data. Using evaluation data to decide the scaling leaks information and makes the evaluation less independent.

Unsupervised Learning

In unsupervised learning, the data does not include known labels.

Example:

Customer	Visits per month	Average spend
A	2	15
B	3	18
C	15	90
D	16	95

No customer is already labelled occasional, regular, low value, or high value. The algorithm searches for groups using similarities in the feature values.

Unsupervised learning is suitable when:

• labels are unavailable;
• the aim is to explore structure;
• grouping similar data points would be useful.

k-Means

k-means is an unsupervised-learning algorithm used for clustering.

It groups points into $k$ clusters.

k-means requires numeric feature vectors for which distance and the arithmetic mean are meaningful. Arbitrary category names cannot be averaged without a suitable numerical representation.

Basic k-means process:

1. choose k starting cluster centres
2. assign each point to its nearest centre
3. update each centre to the mean of its assigned points
4. repeat until assignments stop changing or a stopping limit is reached

One Assignment and Update

Suppose the data points are:

2, 3, 10, 11

and the starting centres are 2 and 10.

Assignment step:

cluster 0: 2, 3
cluster 1: 10, 11

Update step:

$$\text{new centre of cluster 0}=\frac{2+3}{2}=2.5$$ $$\text{new centre of cluster 1}=\frac{10+11}{2}=10.5$$

The next iteration uses 2.5 and 10.5 as the centres.

What Cluster Labels Mean

A result such as cluster 0 or cluster 1 is only an identifier. k-means does not know that a cluster means “high-value customers” or “low-value customers.” A human may interpret and name the groups after examining them.

Choosing $k$

In k-means, $k$ is chosen before fitting. Different values create different numbers of clusters. A value that is too small may combine distinct groups; a value that is too large may split one useful group into several smaller ones.

Starting centres can also affect the final grouping and may lead to different local solutions. Practical library implementations may try more than one starting arrangement; this reduces, but does not eliminate, sensitivity to initialisation.

Comparing k-NN and k-Means

Question	k-NN	k-means
Learning type	supervised	unsupervised
Uses labels during training?	yes	no
Main task	classification	clustering
Meaning of $k$	number of neighbours	number of clusters
What is fitted or stored?	labelled training examples	cluster centres
Example output	spam or not spam	cluster identifier
Does output already have real-world meaning?	yes, because labels are known	not necessarily

How to Choose Between Them

Ask these questions in order:

1. Are known target labels available?
2. Do we need to predict one of those labels for a new item?
3. Or do we need to discover groups in unlabelled data?

Example decisions:

Problem	Suitable approach	Reason
predict whether a labelled email is spam	supervised, k-NN	known labels must be predicted
group unlabelled customers by behaviour	unsupervised, k-means	the groups are not supplied beforehand
calculate a fixed delivery fee	neither	a simple programmed rule is sufficient

Common Mistakes

• Saying k-NN and k-means are the same because both use $k$.
• Confusing a feature with a label.
• Saying supervised learning requires a human to supervise every prediction.
• Saying unsupervised learning has no data.
• Saying k-means predicts known class labels.
• Treating cluster identifiers as meaningful category names.
• Forgetting that changing feature scales can change distance-based results.
• Assuming the chosen value of $k$ is automatically correct.

Quick Check

1. A set of flower measurements already has species labels. Which type of learning is suitable for predicting the species of a new flower?
   Supervised learning.

2. A shop has customer records but no customer-group labels. It wants to discover groups with similar behaviour. Which type is suitable?
   Unsupervised learning.

3. What does $k$ mean in k-NN?
   The number of nearest neighbours used for the prediction.

4. What does $k$ mean in k-means?
   The number of clusters to form.

Return to Artificial Intelligence and Machine Learning.