Data Wrangling
MIT 6.NULL The Missing Semester of Your CS Education, lecture 4, explained in plain English
Want this lecture to stick? In LessonBound you get checked practice questions for every part, can ask about anything you don't get, and can listen with your screen off. Free, no ads.
Learn this lecture freeWatch or read the original lecture: MIT OpenCourseWare.
Part 1
This part teaches how to reshape messy log text into a short, useful list, using pipes, the sed tool, regular expressions, and sort with uniq.
Key ideas
- Data wrangling means turning data from one format into another format that is more useful to you. Filtering a huge log down to just the lines you care about is a simple example.
- The pipe symbol | sends the output of one command into the next command. You can chain several small steps, each one narrowing or reshaping the data.
- The sed tool changes text using short commands. Its substitute command, s/REGEX/SUBSTITUTION/, finds text that matches a pattern and replaces it.
- Capture groups let you keep part of a match. Text inside parentheses is saved as \1, \2, and so on, so you can keep the username and drop the rest of the line.
- Once the usernames are clean, sort puts identical names next to each other, and uniq -c counts how many times each name appears.
Words to know
- regular expression
- A pattern written with plain letters and special symbols that describes the text you want to find, such as 'any characters followed by the word Disconnected from'.
- pipe (|)
- A symbol that passes the output of one command straight into the next command, like a conveyor belt between steps.
- greedy
- When a pattern grabs as much text as it possibly can, which can make it match more than you intended.
- capture group
- A part of a pattern wrapped in parentheses so that the text it matched can be saved and reused, for example as \2.
Example
Think of sorting a pile of mail to find who sent the most letters. First you pull out only the letters from one sender, then you cut off the return address and keep just the name, and finally you line up the names so the same ones sit together and count them. Each step is a small tool passing its result to the next one.
3 checked practice questions for this part are in the app.
Part 2
This part shows how to count, sort, and reshape lines of text in the shell, and how to use awk and bc to do more with that text.
Key ideas
- uniq -c collapses lines that repeat next to each other into one line and puts a count in front. Because it only merges neighbors, the lines usually need to be sorted first.
- sort -n sorts by number instead of by letters, and -k1,1 tells sort to use only the first column of each line. Combined with tail, this lets you keep only the most common results.
- paste -sd , joins many lines into one line with commas between the items, which is useful for making a list for a config file. On macOS, the paste command in the example needs GNU coreutils, which the lesson covers in another exercise.
- awk reads text one line at a time and splits each line into fields, with whitespace as the default divider. $1 is the first field, $2 is the second, and so on. A pattern can limit which lines the action applies to.
- bc is a calculator that reads numbers from input. You can add a list of numbers by joining them with plus signs and sending them to bc.
Words to know
- field
- One chunk of a line, separated from the next chunk by a divider. By default, awk treats spaces and tabs as dividers, so each word in a line is a field.
- delimiter
- The character that separates one piece of text from the next, such as a comma or a space.
- awk
- A small programming language built for processing text line by line. The lesson uses it to pick out fields and do simple counting.
- regular expression
- A short written pattern that describes a set of text, such as words that start with c and end with e. The lesson uses one inside an awk pattern.
Example
Imagine a stack of paper ballots where people wrote their favorite fruit. You could group the same fruit together, count each group, and then list the most popular ones first. The shell tools in this lesson do the same kind of job on lines of text.
3 checked practice questions for this part are in the app.
← Lecture 3: Editors (Vim) · Lecture 5: Command-line Environment →
Notes written by AI (Claude) from the lecture's official transcript, so check anything important against the lecture. MIT CSAIL · CC BY-NC-SA 4.0 · not affiliated with or endorsed by MIT. These notes are shared under the same license.
LessonBound · Free lecture notes · Privacy · Terms · New York