6.4. Transforming#

Data scientists transform dataframe columns when they need to change each value in a feature in the same way. For example, if a feature contains heights of people in feet, a data scientist might want to transform the heights to centimeters. In this section, we’ll introduce apply, an operation that transforms columns of data using a user-defined function:

baby = pd.read_csv('babynames.csv')
baby
Name Sex Count Year
0 Liam M 19659 2020
1 Noah M 18252 2020
2 Oliver M 14147 2020
... ... ... ... ...
2020719 Verona F 5 1880
2020720 Vertie F 5 1880
2020721 Wilma F 5 1880

2020722 rows × 4 columns

In the baby names New York Times article, Pamela mentions that names starting with the letter L or K became popular after 2000. On the other hand, names starting with the letter J peaked in popularity in the 1970s and 1980s and dropped off in popularity since. We can verify these claims using the baby dataset.

We approach this problem using the following steps:

  1. Transform the Name column into a new column that contains the first letters of each value in Name.

  2. Group the dataframe by the first letter and year.

  3. Aggregate the name counts by summing.

To complete the first step, we’ll apply a function to the Name column.

6.4.1. Apply#

pd.Series objects contain an .apply() method that takes in a function and applies it to each value in the series. For instance, to find the lengths of each name, we apply the len function:

names = baby['Name']
names.apply(len)
0          4
1          4
2          6
          ..
2020719    6
2020720    6
2020721    5
Name: Name, Length: 2020722, dtype: int64

To extract the first letter of each name, we define a custom function and pass it into .apply():

# The argument to the function is an individual value in the series. 
def first_letter(string):
    return string[0]

names.apply(first_letter)
0          L
1          N
2          O
          ..
2020719    V
2020720    V
2020721    W
Name: Name, Length: 2020722, dtype: object

Using .apply() is similar to using a for loop. The preceding code is roughly equivalent to writing:

result = []
for name in names:
    result.append(first_letter(name))

Now we can assign the first letters to a new column in the dataframe:

letters = baby.assign(Firsts=names.apply(first_letter))
letters
Name Sex Count Year Firsts
0 Liam M 19659 2020 L
1 Noah M 18252 2020 N
2 Oliver M 14147 2020 O
... ... ... ... ... ...
2020719 Verona F 5 1880 V
2020720 Vertie F 5 1880 V
2020721 Wilma F 5 1880 W

2020722 rows × 5 columns

Note

To create a new column in a dataframe, you might also encounter this syntax:

baby['Firsts'] = names.apply(first_letter)

This mutates the baby table by adding a new column called Firsts. In the preceding code, we use .assign(), which doesn’t mutate the baby table itself; it creates a new dataframe instead. Mutating dataframes isn’t wrong but can be a common source of bugs. Because of this, we’ll mostly use .assign() in this book.

6.4.2. Example: Popularity of “L” Names#

Now we can use the letters dataframe to see the popularity of first letters over time:

letter_counts = (letters
 .groupby(['Firsts', 'Year'])
 ['Count']
 .sum()
 .reset_index()
)
letter_counts
Firsts Year Count
0 A 1880 16740
1 A 1881 16257
2 A 1882 18790
... ... ... ...
3638 Z 2018 55996
3639 Z 2019 55293
3640 Z 2020 54011

3641 rows × 3 columns

fig = px.line(letter_counts.loc[letter_counts['Firsts'] == 'L'],
              x='Year', y='Count', title='Popularity of "L" names',
              width=350, height=250)
fig.update_layout(margin=dict(t=30))
../../_images/pandas_transforming_15_0.svg

The plot shows that “L” names were popular in the 1960s, dipped in the decades after, but have indeed resurged in popularity since 2000.

What about “J” names?

fig = px.line(letter_counts.loc[letter_counts['Firsts'] == 'J'],
              x='Year', y='Count', title='Popularity of "J" names',
              width=350, height=250)
fig.update_layout(margin=dict(t=30))
../../_images/pandas_transforming_17_0.svg

The NYT article says that “J” names were popular in the 1970s and ’80s. The plot agrees and shows that they have become less popular since 2000.

6.4.3. The Price of Apply#

The power of .apply() is its flexibility—you can call it with any function that takes in a single data value and outputs a single data value.

Its flexibility has a price, though. Using .apply() can be slow, since pandas can’t optimize arbitrary functions. For example, using .apply() for numeric calculations is much slower than using vectorized operations directly on pd.Series objects:

%%timeit

# Calculate the decade using vectorized operators
baby['Year'] // 10 * 10
9.66 ms ± 755 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit

def decade(yr):
    return yr // 10 * 10

# Calculate the decade using apply
baby['Year'].apply(decade)
658 ms ± 49.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The version using .apply() is 30 times slower! For numeric operations in particular, we recommend operating on pd.Series objects directly.

In this section, we introduced data transformations. To transform values in a dataframe, we commonly use the .apply() and .assign() functions. In the next section, we’ll compare dataframes with other ways to represent and manipulate data tables.