Explore Real Data
Data Science
- You should already know
- Notebook cells, tables and charts from Notebooks.
- You'll build
- An annotated notebook with a count, two ranked tables and a histogram.
- You'll be able to
- Select rows with a condition, count and rank them, find and explain missing values, and label a chart.
- You need
- A browser. The notebook runs in the page; nothing to install.
In Notebooks, we loaded a year of weather and drew it. That was a year of ordinary days. This chapter looks at the extraordinary ones: every tropical storm and hurricane recorded in the Atlantic since 1851.
A table this size is too long to read, so we'll ask it questions. How many storms became hurricanes? Which were the strongest? What is missing from the record, and why?
Open the Notebook
Open hurricanes.ipynb. It runs in your browser, and the data file hurricanes.csv is already beside it.
The data comes from the Atlantic hurricane database, known as HURDAT2, kept by the National Hurricane Center at NOAA. The original file tracks each storm every six hours. We prepared this table from it, keeping one row for each storm.
Read each cell and predict its result before running it with Shift-Enter. Write your predictions in Markdown cells of your own.
Inspect Before Counting
The first cell reads the file and shows its first five rows:
import pandas as pd import matplotlib.pyplot as plt storms = pd.read_csv('hurricanes.csv') storms.head()
Each row is one storm, with five columns:
| Column | What it holds |
|---|---|
year |
The year of the storm. |
name |
Its name, if it had one. |
peak_wind_mph |
Its highest sustained wind, in miles per hour. |
lowest_pressure_mb |
The lowest air pressure at its center, in millibars. |
category |
1 to 5 for a hurricane, and 0 for a storm that stayed below hurricane strength. |
The first rows are from 1851, and two of their columns show NaN. That is how pandas displays a missing value: the file has nothing in that position. We'll come back to those.
Practice on Three Rows First
A storm is a hurricane once its sustained wind reaches 74 miles per hour. Before applying that rule to the whole table, the notebook gives you an invented one with three rows:
practice = pd.DataFrame({'peak_wind_mph': [60, 80, None]}) selection = practice['peak_wind_mph'] >= 74 print(selection) print('Selected rows:', selection.sum()) print('Missing:', practice['peak_wind_mph'].isna().sum()) practice.loc[selection]
Predict which row remains. The comparison asks the question of every row at once and produces False, True and False, so only the 80 row is selected. .sum() counts the True values as ones. .loc[selection] keeps the rows where the selection is True.
The third row is missing. None creates a missing entry in this practice table; it isn't a measured zero. A missing wind is not at least 74, so that row isn't selected, and .isna() counts it separately.
Checkpoint: Change the threshold to exactly 80. Should the 80 row remain? Change it to 81 and check again. Restore 74 before moving on.
Apply the Selection to Every Storm
The same two lines work on the real table:
print('Storms:', len(storms)) hurricanes = storms.loc[storms['peak_wind_mph'] >= 74] print('Hurricanes:', len(hurricanes)) print(storms['category'].value_counts().sort_index())
There are 1,988 storms on record, and 981 of them reached hurricane strength. That is very nearly half.
value_counts() is the counting loop from Notebooks, and sort_index() puts the categories in order from 0 to 5. Add up the counts for categories 1 to 5. They should come to 981, because the category was worked out from the same wind.
What Is Missing, and Why
.isna().sum() counts the missing entries in every column at once:
print(storms.isna().sum()) early = storms.loc[storms['year'] < 1950] print('Storms before 1950:', len(early)) print('With no name:', early['name'].isna().sum()) print('With no recorded pressure:', early['lowest_pressure_mb'].isna().sum())
Two columns have gaps. More than half the storms have no name, and about a third have no pressure. The second half of the cell shows where the gaps are: none of the 806 storms before 1950 has a name, and 558 of them have no pressure.
Both gaps have a reason. Storms weren't given names until 1950. The pressure at the center of a storm could be measured only if a ship or a weather station happened to be there, until aircraft and then satellites made it routine.
A missing pressure isn't zero, and those storms were just as real. So we leave their rows in the table and set them aside only for a question that needs the pressure:
known = storms.dropna(subset=['lowest_pressure_mb']) print('Storms with a recorded pressure:', len(known)) known[['year', 'name', 'lowest_pressure_mb']].sort_values('lowest_pressure_mb').head(5)
dropna() makes a new table without the rows that are missing a pressure; storms is unchanged. The double brackets choose three columns to display. sort_values() orders the rows by pressure from smallest to largest, and lower pressure means a stronger storm, so the most intense are at the top.
Checkpoint: len(known) plus the number of missing pressures should equal the number of storms. Check it.
The Strongest Storms
For wind, the strongest storms have the largest values, so we sort the other way:
strongest = storms.sort_values('peak_wind_mph', ascending=False) strongest[['year', 'name', 'peak_wind_mph', 'category']].head(10)
Predict a name you expect to see before running it. .head(10) limits what is displayed; it doesn't remove rows from storms.
One of the ten has NaN for a name. It struck the Florida Keys in 1935, fifteen years before storms were named. The missing value tells us about the record, not about the storm.
Draw and Question a Chart
A histogram groups the peak winds into intervals and counts the storms in each one. Predict its shape first: are most storms weak, strong, or somewhere in between?
fig, ax = plt.subplots(figsize=(7, 4)) ax.hist(storms['peak_wind_mph'], bins=15, edgecolor='white') ax.set(xlabel='Peak wind (mph)', ylabel='Storms', title='Atlantic storms, 1851 to 2025') fig.tight_layout() plt.show()
bins sets the number of intervals, and each bar's height is the number of storms in its interval. The labels say what was measured and in what unit, so nobody has to guess.
Most storms are weak, and very few are extreme. Notice the left edge: the chart stops dead, with no gentle slope down to zero. A storm only enters this record once it is a storm. There is no row for a breezy afternoon, so the table can't tell us anything about winds below its starting point.
That is worth asking of any dataset: what had to be true for a row to be here at all?
Your Turn
A major hurricane has winds of at least 111 miles per hour. Record your prediction of how many storms reach that in a Markdown cell, then change the threshold. Check your count against the category counts: it should match categories 3, 4 and 5 added together.
Try 5 histogram bins and then 30. Which features of the chart stay, and which appear only at one setting? At 30, gaps open up. Winds in this record are measured in steps of 5 knots, so some narrow intervals can't contain any storm. More bins don't mean more data.
Count the storms in each year with storms['year'].value_counts().sort_index() and draw the result with ax.plot(...). The line rises. Before concluding that storms have become more common, remember what the missing pressures showed: in 1851, a storm that no ship met was never recorded. This table can't separate more storms from better watching.
Then choose Restart Kernel and Run All Cells. A result that depends on a deleted or out-of-order cell is a notebook bug.
In Your Own Words
Explain the difference between a missing value and a zero, using a storm's pressure as your example. Then say why dropping the rows with missing values would have been the wrong first step.
Next, we'll stop writing the rule ourselves. In Train Your First Model, the computer works out a rule from examples.