Lecture notes · Probability · Lecture 21

Bayesian Statistical Inference I

MIT 6.041 Probabilistic Systems Analysis and Applied Probability, lecture 21, explained in plain English

Want this lecture to stick? In LessonBound you get checked practice questions for every part, can ask about anything you don't get, and can listen with your screen off. Free, no ads.

Learn this lecture free

Watch or read the original lecture: MIT OpenCourseWare.

Part 1 · 0:00–10:00

This part explains what statistical inference is: using data from the real world to build a model, and why that is different from pure probability.

Key ideas

Probability models are built from ideas, but real life gives us data. We observe the real world, and then use those data to build a model of what is going on. 0:41
Once we have a model and its parameters, we can use it to predict things. We can also use it to infer hidden things we cannot observe directly. 1:27
In a probability problem, there is one correct answer. In a statistics problem, reasonable people can pick different methods, and no single method is clearly the best. 5:48
Good statisticians need to know the assumptions behind each method and what guarantees, if any, it gives. Copying a test without understanding it leads to false conclusions. 7:38

Words to know

inference: Making a well-reasoned guess about something you cannot see directly, based on the data you can see.
model: A simplified description of how something in the real world behaves, often written with math.
parameter: A number inside a model that sets its details, such as how fast arrivals come in.
attenuated: Made weaker, or reduced in strength, as a signal gets smaller while travelling through something.

Example

Think of a weather forecaster who has never seen tomorrow. They look at past weather records and today's readings, build a rough model, and use it to guess what tomorrow will be like. Different forecasters might use different methods and still each be reasonable, which is the kind of choice this lesson describes.

3 checked practice questions for this part are in the app.

Part 2 · 10:00–20:00

This part explains the two main kinds of inference problems, the two ways of thinking about unknown quantities, and why Bayesian inference starts with a prior.

Key ideas

Building a model of a system and estimating an unknown signal are mathematically the same kind of problem, so the same tools work for both. 11:43
Unknown quantities come in two types: discrete ones with a few possible values, where you care about making a correct decision, and continuous ones, where you care about how close your guess is. 13:09
There are two ways to think about an unknown. One treats it as a fixed but unknown number. The other, the Bayesian view, treats it as a random variable described by a prior distribution that represents your beliefs before the experiment. 16:22
Even when the unknown is a fixed number, the estimate you compute is a random variable, because it depends on noisy data. That is why estimates are written with hats and capital letters. 17:52
In Bayesian inference, you use Bayes rule to update your prior with the data and get a posterior distribution, which is your updated belief. To start, you need a prior. 19:59

Words to know

prior distribution: Your starting beliefs about an unknown value, written as a range of likely values, before you look at any new data.
posterior distribution: Your updated beliefs about the unknown value after you have seen the data.
attenuation: How much a signal gets weaker as it travels through a medium, such as air or a wire.
Bayes rule: A formula for updating your beliefs about something unknown after you see evidence about it.

Example

Imagine you shout across a valley and listen for the echo to guess how far away the wall is. You know what you shouted, and you hear a fainter or louder version back, so the sound's weakening tells you something about the distance. Before you shout, you might already guess the wall is somewhere between 50 and 200 meters away, and the echo sharpens that guess.

3 checked practice questions for this part are in the app.

Part 3 · 20:00–30:00

This part explains how Bayesian inference works: you start with a belief, collect data, and use Bayes rule to update that belief into a new one about an unknown value.

Key ideas

A prior is your starting belief about the unknown value, held before you run the experiment. A model of your measuring equipment tells you what data to expect for each possible value. 20:00
Once you observe data, Bayes rule combines the prior with the data to give the posterior, which is your updated belief about the unknown value after seeing the data. 21:02
The same method works for discrete data (using probabilities) and continuous data (using densities). Estimation problems, where the unknown is a continuous value, follow the same steps. 21:22
A classical statistician picks one best estimate, such as heads divided by total flips. A Bayesian treats the unknown value as random, starts with a prior, and reports a full range of likely values with their probabilities. 26:30
The usual result of Bayesian work is a plot or list of probabilities for each possible value. When a boss wants one answer, you must still pick a single value, which is a decision made on top of that probability picture. 29:55

Words to know

prior: Your starting belief about an unknown quantity, before you collect any new data.
posterior: Your updated belief about the unknown quantity after you have seen the data.
Bayes rule: A formula that tells you how to combine your prior belief with new data to get your updated belief.
bias: How likely a coin is to land heads. A fair coin has a bias of one half.

Example

Suppose you want to know how often a friend's coin lands heads. Before flipping, you guess it is probably close to fair, which is your prior. After 20 flips with 14 heads, you update that guess, and your new belief is the posterior, which might show that values near 0.6 are most likely.

3 checked practice questions for this part are in the app.

Part 4 · 30:00–40:00

This part explains how to pick a single best guess for an unknown value after seeing data, and why the choice depends on the rules you set.

Key ideas

A posterior distribution shows how likely each possible value of the unknown quantity is, after you have seen the data. In the discrete case, the most likely value (the MAP estimate) is the one that gives the fewest wrong decisions. 30:38
In the continuous case, any exact value has zero probability, so the peak of the density is not really the most likely value. That makes the MAP estimate a weaker choice for continuous problems. 32:27
Another common single answer is the conditional expectation, which is the average of the unknown value over its posterior distribution. A single number never tells the whole story, so showing the full plot can help. 33:11
If you want to minimize the average squared error and have no data, the best guess is the expected value. For a value spread evenly between 4 and 10, that guess is 7. 37:55
Once you have data, you use the same idea with the conditional distribution. The best guess under squared error is then the conditional expectation, the average over the posterior. 39:53

Words to know

posterior distribution: your updated picture of how likely each possible value is, after you have seen the data.
MAP estimate: maximum a posteriori estimate: the single value that has the highest probability (or highest density) after seeing the data.
conditional expectation: the average value of the unknown quantity, worked out using only the data you have, like the center of gravity of the posterior picture.
mean squared error: the average of the squared gap between your guess and the true value; squaring makes big misses count more.

Example

Imagine you are guessing how far away a sound is, and you have a rough sense that it is somewhere between 4 and 10 blocks. If you want your guess to be off by as little as possible on average, you pick the middle, which is 7. Once you hear more clues, you narrow the range, and the middle of the new range becomes your best guess.

3 checked practice questions for this part are in the app.

Part 5 · 40:00–48:48

Once you have data, the best single guess for an unknown quantity is its conditional expectation, given that data.

Key ideas

Once you have your data, the best single number to report is the conditional expectation of the unknown quantity, given that data. This is the guess that minimizes mean squared error, which is the average of the squared size of your mistakes. 40:40
The argument works for each possible data value. For any particular data you observe, this guess does at least as well as any other rule someone else could use. 42:47
Because the conditional guess wins for every possible data value, it also wins when you average over all possible data. So it is the best estimator overall. 43:30
In real problems you have several data points, and choosing a prior is a judgment call. The math can also be hard to compute, so a simpler alternative is useful. The lecture says it will cover one next time. 47:29

Words to know

conditional expectation: The average value you would expect for something, once you already know some facts. It changes when new information arrives.
prior: What you believe about the unknown quantity before you look at any data.
posterior: What you believe about the unknown quantity after you have seen the data. It mixes your prior belief with what the data say.
estimator: A rule or machine that takes in data and outputs a guess about an unknown quantity.

Example

Imagine guessing how many apples are in a box you cannot see. Before anyone looks, your best guess is an average over what is likely. After someone shows you a few apples, your best guess changes to a new average that uses that information. The lesson says the best single guess is always this updated average, given what you have seen.

3 checked practice questions for this part are in the app.

← Lecture 20: Central Limit Theorem

Notes written by AI (Claude) from the lecture's official transcript, so check anything important against the lecture. MIT OpenCourseWare · CC BY-NC-SA 4.0 · not affiliated with or endorsed by MIT. These notes are shared under the same license.

LessonBound · Free lecture notes · Privacy · Terms · New York