```import warnings
# Ignore numpy dtype warnings. These warnings are caused by an interaction
# between numpy and Cython and can be safely ignored.
# Reference: https://stackoverflow.com/a/40846742
warnings.filterwarnings("ignore", message="numpy.dtype size changed")
warnings.filterwarnings("ignore", message="numpy.ufunc size changed")

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
%matplotlib inline
import ipywidgets as widgets
from ipywidgets import interact, interactive, fixed, interact_manual

sns.set()
sns.set_context('talk')
np.set_printoptions(threshold=20, precision=2, suppress=True)
pd.options.display.max_rows = 7
pd.options.display.max_columns = 8
pd.set_option('precision', 2)
# This option stops scientific notation for pandas
# pd.set_option('display.float_format', '{:.2f}'.format)
```

# 6.3. Apply, Strings, and Plotting¶

In this section, we will answer the question:

Can we use the last letter of a name to predict the sex of the baby?

Here’s the Baby Names dataset once again:

```baby = pd.read_csv('babynames.csv')
baby.head()
# the .head() method outputs the first five rows of the DataFrame
```
Name Sex Count Year
0 Mary F 9217 1884
1 Anna F 3860 1884
2 Emma F 2587 1884
3 Elizabeth F 2549 1884
4 Minnie F 2243 1884

Breaking the Problem Down

Although there are many ways to see whether prediction is possible, we will use plotting in this section. We can decompose this question into two steps:

1. Compute the last letter of each name.

2. Group by the last letter and sex, aggregating on Count.

3. Plot the counts for each sex and letter.

## 6.3.1. Apply¶

`pandas` Series contain an `.apply()` method that takes in a function and applies it to each value in the Series.

```names = baby['Name']
names.apply(len)
```
```0          4
1          4
2          4
..
1891891    5
1891892    6
1891893    8
Name: Name, Length: 1891894, dtype: int64
```

To extract the last letter of each name, we can define our own function to pass into `.apply()`:

```def last_letter(string):
return string[-1]

names.apply(last_letter)
```
```0          y
1          a
2          a
..
1891891    a
1891892    e
1891893    p
Name: Name, Length: 1891894, dtype: object
```

## 6.3.2. String Manipulation¶

Although `.apply()` is flexible, it is often faster to use the built-in string manipulation functions in `pandas` when dealing with text data.

`pandas` provides access to string manipulation functions using the `.str` attribute of Series.

```names = baby['Name']
names.str.len()
```
```0          4
1          4
2          4
..
1891891    5
1891892    6
1891893    8
Name: Name, Length: 1891894, dtype: int64
```

We can directly slice out the last letter of each name in a similar way.

```names.str[-1]
```
```0          y
1          a
2          a
..
1891891    a
1891892    e
1891893    p
Name: Name, Length: 1891894, dtype: object
```

We suggest looking at the docs for the full list of string methods (link).

We can now add this column of last letters to our `baby` DataFrame.

```baby['Last'] = names.str[-1]
baby
```
Name Sex Count Year Last
0 Mary F 9217 1884 y
1 Anna F 3860 1884 a
2 Emma F 2587 1884 a
... ... ... ... ... ...
1891891 Verna M 5 1883 a
1891892 Winnie M 5 1883 e
1891893 Winthrop M 5 1883 p

1891894 rows × 5 columns

## 6.3.3. Grouping¶

To compute the sex distribution for each last letter, we need to group by both Last and Sex.

```# Shorthand for baby.groupby(['Last', 'Sex']).agg(np.sum)
baby.groupby(['Last', 'Sex']).sum()
```
Count Year
Last Sex
a F 58079486 915565667
M 1931630 53566324
b F 17376 1092953
... ... ... ...
y M 18569388 114394474
z F 142023 4268028
M 120123 9649274

52 rows × 2 columns

Notice that `Year` is also summed up since each non-grouped column is passed into the aggregation function. To avoid this, we can select out the desired columns before calling `.groupby()`.

```# When lines get long, you can wrap the entire expression in parentheses
# and insert newlines before each method call
letter_dist = (
baby[['Last', 'Sex', 'Count']]
.groupby(['Last', 'Sex'])
.sum()
)
letter_dist
```
Count
Last Sex
a F 58079486
M 1931630
b F 17376
... ... ...
y M 18569388
z F 142023
M 120123

52 rows × 1 columns

## 6.3.4. Plotting¶

`pandas` provides built-in plotting functionality for most basic plots, including bar charts, histograms, line charts, and scatterplots. To make a plot from a DataFrame, use the `.plot` attribute:

```# We use the figsize option to make the plot larger
letter_dist.plot.barh(figsize=(10, 10))
```
```<matplotlib.axes._subplots.AxesSubplot at 0x1a17af4780>
``` Although this plot shows the distribution of letters and sexes, the male and female bars are difficult to tell apart. By looking at the `pandas` docs on plotting (link) we learn that `pandas` plots one group of bars for row column in the DataFrame, showing one differently colored bar for each column. This means that a pivoted version of the `letter_dist` table will have the right format.

```letter_pivot = pd.pivot_table(
baby, index='Last', columns='Sex', values='Count', aggfunc='sum'
)
letter_pivot
```
Sex F M
Last
a 58079486 1931630
b 17376 1435939
c 30262 1672407
... ... ...
x 37381 644092
y 24877638 18569388
z 142023 120123

26 rows × 2 columns

```letter_pivot.plot.barh(figsize=(10, 10))
```
```<matplotlib.axes._subplots.AxesSubplot at 0x1a17c36978>
``` Notice that `pandas` conveniently generates a legend for us as well. However, this is still difficult to interpret. We plot the counts for each letter and sex which causes some bars to appear very long and others to be almost invisible. We should instead plot the proportion of male and female babies within each last letter.

```total_for_each_letter = letter_pivot['F'] + letter_pivot['M']

letter_pivot['F prop'] = letter_pivot['F'] / total_for_each_letter
letter_pivot['M prop'] = letter_pivot['M'] / total_for_each_letter
letter_pivot
```
Sex F M F prop M prop
Last
a 58079486 1931630 0.967812 0.032188
b 17376 1435939 0.011956 0.988044
c 30262 1672407 0.017773 0.982227
... ... ... ... ...
x 37381 644092 0.054853 0.945147
y 24877638 18569388 0.572597 0.427403
z 142023 120123 0.541771 0.458229

26 rows × 4 columns

```(letter_pivot[['F prop', 'M prop']]
.sort_values('M prop') # Sorting orders the plotted bars
.plot.barh(figsize=(10, 10))
)
```
```<matplotlib.axes._subplots.AxesSubplot at 0x1a18194b70>
``` ## 6.3.5. Summary¶

We can see that almost all first names that end in ‘p’ are male and names that end in ‘a’ are female! In general, the difference between bar lengths for many letters implies that we can often make a good guess to a person’s sex if we just know the last letter of their first name.

We’ve learned to express the following operations in `pandas`:

Operation

`pandas`

Applying a function elementwise

`series.apply(func)`

String manipulation

`series.str.func()`

Plotting

`df.plot.func()`