Lecture notes · Probability · Lecture 12

Iterated Expectations; Sum of a Random Number of Random Variables

MIT 6.041 Probabilistic Systems Analysis and Applied Probability, lecture 12, explained in plain English

Want this lecture to stick? In LessonBound you get checked practice questions for every part, can ask about anything you don't get, and can listen with your screen off. Free, no ads.

Learn this lecture free

Watch or read the original lecture: MIT OpenCourseWare.

Part 1 · 0:00–10:00

This part explains conditional expectation, which is the average of a value once you are told something else, and why that average can itself be treated as a random variable.

Key ideas

A conditional expectation is the average of X after someone tells you the value of Y. It is calculated like a normal average, but with the updated probabilities. 1:45
In the stick example, if the first break is at 3.5, the expected position of the second break is 1.75. That is just a number. 3:36
Before the experiment, we do not know Y yet, so the conditional expectation is a random variable. It equals Y divided by 2, whatever Y turns out to be. 5:29
The average of this random variable matches the total expectation theorem. That theorem finds an overall average by taking a weighted average of the averages in each possible scenario. 9:55

Words to know

conditional expectation: The average of one quantity, found after you have been told the value of another quantity that it depends on.
random variable: A quantity whose value depends on chance, so it can come out differently each time you run the experiment.
conditional density function: A formula that gives the likelihood of each value for a continuous quantity, after you have been told some other information.
total expectation theorem: A rule that finds the overall average by splitting into possible scenarios, averaging within each scenario, and then taking a weighted average of those results.

Example

Imagine a stick that you first break at a random spot, then break the leftover piece at a random spot again. If you already know where the first break was, you can predict the average spot for the second break. Before you break the stick, that prediction is not yet a fixed number, because it depends on where the first break will land.

3 checked practice questions for this part are in the app.

Part 2 · 10:00–20:00

This part explains how to find the average of a quantity when you already know a related value, and how the spread (variance) works when the average itself changes at random.

Key ideas

The law of iterated expectations says that averaging the conditional expectations gives you the plain, overall expected value. It is the total expectation idea written in more abstract notation. 10:22
In the stick example, the expected leftover length of the stick is L/4. You get this by averaging the conditional expectation Y/2 over the uniform random variable Y, whose average is L/2. 11:03
A conditional variance is the spread of X in a world where you already know the value of Y. Since Y is not known in advance, the conditional variance is itself a random variable that depends on Y. 13:15
The variance of X is not just the average of the conditional variances. The full formula has two terms instead of one, so you need both parts to get the right answer. 14:21
The proof is a long string of algebra. The lesson says it is mainly useful for making sure you understand each symbol, and that the intuition comes later, when the conditional expectation is treated as an estimate of X. 18:15

Words to know

conditional expectation: The average value of something, worked out after you are told the value of another quantity. Think of it as the average inside a smaller, known situation.
conditional variance: How spread out a result is, worked out after you are told the value of another quantity. It measures spread inside that known situation.
realization: The specific value that a random quantity ends up with once the experiment has happened. Before that, the value is unknown.
variance: A measure of how spread out the values of a random quantity are around its average. A bigger variance means the results vary more.

Example

Imagine you want the average test score for a whole school. You first find the average score in each classroom, then average those class averages, weighted by class size. If you know which classroom a student is in, you can find that class's average, but until you pick a student, the classroom average is unknown and random.

3 checked practice questions for this part are in the app.

Part 3 · 20:00–30:00

This part shows how to find the average and spread of quiz scores for a whole class by first looking at each section on its own.

Key ideas

If you know each section's average and how many students are in it, you can find the overall class average by taking a weighted average of the section averages. A section with more students counts for more. 22:37
A conditional expectation is the average score you would expect once you know which section a student is in. Because which section you get is random, this conditional expectation is itself a random variable. 21:52
The variance of the conditional expectation measures how far apart the section averages are from the overall average. Here it comes out to 200. 23:59
The expected value of the conditional variances measures the spread of scores inside each section. The total spread of the whole class has two parts: spread within sections, and spread between sections. 28:26
The lesson uses three sections as an example, where the sections are very different (undergraduates and post-doctoral students). In that case the spread between sections matters a lot to the class's overall spread. 29:13

Words to know

conditional expectation: The average you would expect for something, once you already know some extra fact, like which section a student is in.
random variable: A number whose value depends on chance, such as the score of a student picked at random.
variance: A number that shows how spread out values are around their average. A bigger variance means the values are more scattered.
weighted average: An average where some values count more than others. Each value is multiplied by how much weight it gets before adding.

Example

Think of two schools' average test scores, where one school has 10 students and the other has 20. To find the average for both schools together, you cannot just average the two numbers; the school with 20 students should count twice as much. The lesson does this same thing with the sections, giving each one a weight based on its share of the students.

3 checked practice questions for this part are in the app.

Part 4 · 30:00–40:00

This part shows how to break a hard probability problem into easier pieces, and how to add up spending when the number of shopping trips is random.

Key ideas

The total spread of a set of values can be split into two parts: the spread inside each group, and how far apart the group averages are from each other. These two parts add up. 30:23
A divide-and-conquer method means you split a hard problem into easier cases, solve each case on its own, and then combine the results. 31:29
The overall average is found by taking each group's average and weighting it by how likely that group is. In the example, the averages 1/2 and 3/2 are weighted by 1/3 and 2/3 to get 7/6. 33:19
The spread inside each piece was 1/12 in both cases, so its average is also 1/12, since it never changes. 35:11
When you visit a random number of stores, you can first pretend the number is fixed. Once you know the count, you can use the rule for adding up a fixed number of values, then average over all the possible counts. 39:15

Words to know

variance: A number that shows how spread out a set of values is, or how far they tend to sit from their average.
conditional expectation: The average value of something, assuming you already know some extra fact, such as which group you are in.
independent: Two things are independent when knowing one tells you nothing about the other, like each store visit having its own unrelated spending.
uniform distribution: A setup where every value in a range is equally likely, like a number picked at random anywhere between 0 and 1.

Example

Imagine you want the average amount of money you spend on books over a weekend. You do not know how many stores you will visit, but you do know roughly how much you spend at each store. You can first figure out the average for each possible number of stores, then combine those averages using how likely each number of stores is.

3 checked practice questions for this part are in the app.

Part 5 · 40:00–47:52

This part shows how to find the average and the spread of total spending when both the number of stores visited and the amount spent in each store are random.

Key ideas

If the number of stores is independent of how much you spend in each store, then knowing the number of stores adds nothing about the spending amounts, so that condition can be dropped. 40:20
If you know you visit exactly n stores, your expected total spending is n times the expected spending in one store, because each store has the same distribution. 40:40
Using the law of iterated expectations, the overall expected spending is the average number of stores times the average spending per store. 42:29
If the number of stores is fixed at n, the variance of the total is the sum of the variances of each store, which is n times one store's variance. 45:45
Multiplying a random number by a fixed number multiplies its variance by that number squared, which is how the spread from the store count is measured. 44:42

Words to know

variance: A number that shows how spread out a set of possible results is. A big variance means the results can vary a lot from the average.
independent: Two things are independent when knowing one tells you nothing new about the other.
expected value: The average result you would get if you could repeat the random process many times.
law of iterated expectations: A rule that says the overall average of something equals the average of its averages taken within each possible case.

Example

Imagine you walk into a random number of shops on a day, and each shop you spend about the same amount on average. If you expect to visit 10 shops and spend about $100 in each, you expect to spend about $1,000 that day, which is the same reasoning the lesson uses.

3 checked practice questions for this part are in the app.

← Lecture 8: Continuous Random Variables · Lecture 19: Weak Law of Large Numbers →

Notes written by AI (Claude) from the lecture's official transcript, so check anything important against the lecture. MIT OpenCourseWare · CC BY-NC-SA 4.0 · not affiliated with or endorsed by MIT. These notes are shared under the same license.

LessonBound · Free lecture notes · Privacy · Terms · New York