Lecture notes · Probability · Lecture 19

Weak Law of Large Numbers

MIT 6.041 Probabilistic Systems Analysis and Applied Probability, lecture 19, explained in plain English

Want this lecture to stick? In LessonBound you get checked practice questions for every part, can ask about anything you don't get, and can listen with your screen off. Free, no ads.

Learn this lecture free

Watch or read the original lecture: MIT OpenCourseWare.

Part 1 · 0:00–10:00

This part introduces limit theorems, explains why averaging a random sample can estimate an expected value, and builds two inequalities (Markov and Chebyshev) that link averages and variances to probabilities.

Key ideas

The expected value of a height is the average height across the whole penguin population. Measuring every penguin is slow, so you can instead measure a random sample and average those heights. 0:25
The sample mean is the average of your sample. It is a random variable, because a different random sample gives a different average, while the expected value is a single fixed number. 1:29
Limit theorems study what happens when you have very many random variables or take averages of them. Formulas get too hard to compute exactly for many variables, but they simplify in the limit, so useful answers become possible. 3:40
The weak law of large numbers says the sample mean gets close to the expected value as the sample grows. To prove it, the lecture first needs Markov's inequality, which says that for a non-negative random variable, a small expected value means a small chance of a large value. 4:48
Chebyshev's inequality comes from applying Markov's inequality to the squared distance from the mean. It says that if the variance is small, the chance of being far from the mean is also small. 9:45

Words to know

expected value: The long-run average of a random quantity. It is the average over the whole population or the full set of possible outcomes, weighted by how likely each outcome is.
random variable: A number whose value depends on chance, such as the height of a penguin you happen to pick.
variance: A measure of how spread out a random variable is around its mean. A small variance means the values stay close to the mean.
inequality: A statement that one quantity is less than, or greater than, another. Markov and Chebyshev are rules of this kind that bound probabilities.

Example

Imagine you want to know the average height of all penguins at the South Pole, but you can only measure a few. Each time you pick a random group of penguins, their average height will be a little different, which is why the sample mean is a random variable. As you measure more penguins, those averages tend to settle near the true average, which is the idea behind the weak law.

3 checked practice questions for this part are in the app.

Part 2 · 10:00–20:00

This part explains the Chebyshev inequality, which limits how often a random result lands far from its average, and then defines what it means for a sequence of values to converge to a number.

Key ideas

The Chebyshev inequality says the chance of a result being at least c away from the mean is at most the variance divided by c squared. The proof works by ignoring the middle of the distribution and only counting the far-away parts. 10:20
If the variance is small, the distribution is narrow, so results far from the mean are unlikely. A bigger distance c makes that chance even smaller. 12:10
Using k standard deviations instead of c, the chance of being k standard deviations or more from the mean is at most 1 over k squared. For k equal to 3, that is at most 1/9. 13:36
A sequence of numbers converges to a if, for any band around a, the values eventually get inside the band and stay there. 16:10
A sequence of random variables converges to a number if the probability of falling outside a small band around that number shrinks toward 0 as n grows. 18:00

Words to know

variance: A number that measures how spread out a set of random results is. Small variance means the results sit close to the average.
standard deviation: The square root of the variance. It is a typical distance from the average, in the same units as the results.
converges: Gets closer and closer to a target value and stays close. Think of a car slowing down to park at one exact spot.
epsilon: A small positive number that sets how wide the band around a target is. A smaller epsilon means a narrower band.

Example

Think of the scores on a class quiz. Most students score near the class average, and very few are far from it. The Chebyshev inequality tells you the largest share of students who can be 3 standard deviations away from the average, which is 1/9 at most.

3 checked practice questions for this part are in the app.

Part 3 · 20:00–30:00

This part explains what it means for a sample average to settle down near the true average as you collect more data, and why the average's spread shrinks as the sample grows.

Key ideas

Convergence in probability means that as the sample size n grows, the chance of landing outside a narrow band around the true value a shrinks toward zero. 21:27
In the example, the random variable Yn sits at 0 more and more often, and the chance of being outside a band around 0 is 1/n, which goes to zero. So Yn converges in probability to 0. 22:56
Settling down in probability does not guarantee that averages or spreads settle down too. A rare, far-off value can still pull the expected value up, even when it is very unlikely. 24:04
The expected value of the sample mean equals the true mean, so the sample mean is correct on average over many imagined expeditions. 26:00
The variance of the sample mean is sigma squared divided by n, so it gets smaller as the sample grows. Chebyshev's inequality then shows that the chance of being far from the true mean goes to zero. 28:37

Words to know

convergence in probability: A way of saying that a random result is very likely to land close to a certain number once you have enough data, so the chance of being far off keeps shrinking.
expected value: The long-run average you would get if you could repeat the random experiment a huge number of times.
variance: A measure of how spread out a random result is. A small variance means results cluster tightly.
sample mean: The average of the values you actually measured in one set of observations.

Example

Think of guessing the average height of penguins by measuring a few at a time. One small group can give a shaky guess, but as you measure more penguins, your guesses bunch closer and closer to the true average. The bigger your sample, the less your result depends on luck.

3 checked practice questions for this part are in the app.

Part 4 · 30:00–40:00

This part shows how a pollster uses sample averages to estimate a population fraction, and how the Chebyshev inequality helps decide how big the sample must be.

Key ideas

The weak law of large numbers says a sample mean gets closer to the true mean as the sample grows. The closeness is in the 'convergence in probability' sense. 30:21
A poll cannot ask everyone, so it estimates the true fraction f with f_hat, the fraction found in the sample. f_hat is a random variable because a different sample gives a different answer. 31:46
No poll can guarantee an exact answer. Instead, the pollster sets an accuracy target and a confidence level, such as within 1 percentage point with 95% confidence. 32:51
Chebyshev's inequality needs the variance, but the variance depends on the unknown f. So the pollster uses the worst case, which is 1/4 when f is one half. This gives a sample size of 50,000. 36:12
Real polls use about 1,000 people. They get there by loosening the accuracy target to about 3 points and by using a tighter bound than Chebyshev's inequality. 38:49

Words to know

variance: A number that shows how spread out a random result can be. A bigger variance means the results jump around more.
Chebyshev inequality: A rule that gives a safe upper limit on how likely a random result is to land far from its average. It is a rough bound, not an exact answer.
confidence: How sure you are that your answer falls inside your accuracy goal. A 95% confidence level means a 5% chance of being unlucky.
estimate: Your best guess at an unknown number, based on the information you have. Here, f_hat is the guess made from the sample.

Example

Imagine you want to know how salty a big pot of soup is, but you can only taste a spoonful. The spoonful is your sample, and the taste tells you a lot about the whole pot. A bigger spoonful usually gives a better guess, which is the idea behind the weak law of large numbers.

3 checked practice questions for this part are in the app.

Part 5 · 40:00–50:11

This part explains why adding many random values together, then scaling them by the square root of n, leads to a bell-shaped (normal) pattern, which is the idea behind the central limit theorem.

Key ideas

When you add many independent random values together, the mean grows with n and the spread (variance) also grows with n, so the total gets wider and shifts over. 40:41
Dividing the sum by n squeezes the result onto the true mean, which is the weak law of large numbers. Dividing by the square root of n instead keeps the spread steady, so the shape stays interesting. 42:50
To compare sums fairly, you standardize them: subtract the mean and divide by the standard deviation. The result has mean 0 and variance 1, and it tells you how many standard deviations you are from the average. 45:29
The central limit theorem says that the standardized sum's probabilities get close to the standard normal (bell curve) probabilities, which are easy to look up in tables, when n is large. 46:35
In practice you can pretend the sum is normal and do the calculations that way. Strictly speaking, the theorem is about the standardized version, not the raw sum, because the raw sum spreads out too much in the limit. 49:12

Words to know

i.i.d (independent and identically distributed): Random values that are all separate from each other and all follow the same pattern.
variance: A number that shows how spread out a set of values is. Bigger variance means values are more scattered.
standardize: To shift a value so its average is 0, then scale it so its spread is 1. This makes different values easy to compare.
standard normal: The classic bell-shaped curve with average 0 and spread 1. Its probabilities are listed in printed tables.

Example

Think of many people each flipping a coin and counting heads, then adding up all the counts. The total keeps climbing and spreads out more as more people join, so you need to rescale it to see a steady pattern. Once you rescale it the right way, the results start to look like a bell curve, which is what the central limit theorem describes.

3 checked practice questions for this part are in the app.

← Lecture 12: Iterated Expectations; Sum of a Random Number of Random Variables · Lecture 20: Central Limit Theorem →

Notes written by AI (Claude) from the lecture's official transcript, so check anything important against the lecture. MIT OpenCourseWare · CC BY-NC-SA 4.0 · not affiliated with or endorsed by MIT. These notes are shared under the same license.

LessonBound · Free lecture notes · Privacy · Terms · New York