Scraping, Cleaning, and Importing: an Overview
Posted on Sun 19 June 2016 in posts
The first challenge in this project was retrieving the OCRed directories from their home on Hathitrust and into my main data frame. In text format, a single directory on Hathitrust typically consists of hundreds of individual text files--far too many to download individually, but an easy task for Python. Once downloaded, the text file needs to be "cleaned" of html markup and the resulting text imported into the database. I've already applied this three-stage process of scraping, cleaning, and importing to ninety-two directories. I'll go into more detail about each step in that process at a later time, but for now, here's a general overview of how I went about it.
import os
import pandas as pd
As I've discussed in an earlier blog entry, I used BeautifulSoup to create a 'catalog' data frame that contains information about each of the 111 New York City directories that I've found in HathiTrust. Among other things, the data frame keeps track of which directories have been scraped, cleaned, and imported. The most up-to-date version is stored locally as a pickle. I'll begin by loading that data:
df_cat = pd.read_pickle('../blogdata/catalog_overview.p')
In addition to a large number of nineteenth-century directories, the catalog data frame contains quite a few entries from the late 1700s and the early 1900s. Since those aren't of interest to me at the moment, I'll exclude them from my working data frame. I'll also limit the working data frame to the seven columns relevant for understanding the process of scraping, cleaning, and importing the Hathitrust directories: the directory title, the publisher, the year it was published, the total number of pages in the digital version of that directory, and the number of pages that have already been scraped, cleaned, and imported:
df_cat = df_cat[(df_cat.pub_year > 1799) & (df_cat.pub_year < 1900)]
select_cols = ['publisher', 'pub_year', 'total_pages',
'scraped', 'cleaned', 'imported']
df_cat = df_cat[select_cols]
Next, I'll transform those column names into a more user-friendly version for display here:
display_names = ['Publisher', 'Year', 'Total Pages', 'Scraped', 'Cleaned', 'Imported']
df_cat.columns = display_names
Scraping Hathitrust¶
The scraping process grabs each OCRed page from each directory in the catalog and stores it locally as a text file with html markup. Scraping a single directory takes a fair amount of time: about two seconds per page. This can add up, especially when scraping directories from the second half of the nineteenth centuries, which grew in size along with the population of New York City. Happily, I've downloaded the text files for most of the nineteenth century directories in Hathitrust and only the directories from the 1890s are left:
unscraped = (df_cat.Scraped == 0)
print("\n\nNumber of directories waiting to be scraped: " + str(sum(unscraped)))
df_cat[unscraped].sort_values('Year')
It's worth noting the wide variation in the length of the four directories above. The 1890 and 1895 publications are surprisingly short at less than 500 pages each--a length more typical of directories from the 1820s. By the 1890s, New York City was a bona fide metropolis with a population of approximately 1.5 million. That, combined with the addition of advertisements, meant that most New York City address directories of that decade had 1500 pages or more.
So what's going on here? The shorter publications above are business directories, not residential ones. That's fun data to have, so there's no harm in scraping, cleaning, and importing the business directories too, but I'll exclude them from the main analysis later.
Cleaning¶
As each page is scraped, the page number is recorded in the "Scraped" column. If the process is interrupted, the script picks up where it left off the next time it runs. Once all pages in a directory have been scraped, the directory is ready for cleaning.
The uncleaned directory pages exist locally as text files, full of html markup. Thankfully, Hathitrust was judicious and consistent in their use of "id" and "class" tags, so removing the markup is easy using BeautifulSoup. Currently, four recently-scraped directories, all from the 1880s, are sitting in the cleaning queue:
unclean = (df_cat.Scraped > 0) & (df_cat.Cleaned == 0)
print("\n\nNumber of scraped directories waiting to be cleaned: " + str(sum(unclean)))
df_cat[unclean].sort_values('Year')
The cleaning process produces a collection of text files, each containing html-free, OCRed text from one scanned page. Blank pages--pages that had no OCRed text at all--are omitted during the cleaning process. As each page is processed, the "Cleaned" column is updated with its page number.
Importing¶
Once cleaned, the files will be imported line-by-line into an existing Pandas data frame. Fourteen directories, mostly from the 1870s, have been scraped and cleaned and are waiting to join the seventy-four directories that have already been imported into the project data frame:
ready_for_import = (df_cat.Cleaned > 0) & (df_cat.Imported == 0)
print("\n\nNumber of cleaned directories waiting to be imported: " + str(sum(ready_for_import)))
df_cat[ready_for_import].sort_values('Year')
Currently the main data frame has over six million rows, each representing one line of text from a directory. Once all of the nineteenth-century directories have been imported into the main data frame, the data frame will have about ten million rows and I'll move on to the next phase of this project: identifying which rows represent address entries and parsing those entries into distinct columns ('last name', 'first name', 'occupation', etc.).
But before doing that, I'll need to determine which of the rows in the main data frame represent address entries and which represent other types of text. As this subset of the main data frame shows, many of the rows are in fact addresses:
pd.read_pickle('../blogdata/address_sample.p')
At the same time, many directory pages contain no addresses at all. These include the introduction and table of contents, and supplemental pages, such as weather tables, informational essays, and advertisements. The rows produced from these pages will need to be excluded from the analysis:
pd.set_option('display.max_colwidth',100)
pd.read_pickle('../blogdata/suppl_sample.p')
Fortunately, there are a number of ways to use Python to isolate address entries from other types of text. I'll be talking about those in a future blog entry.