Web scraping

Ranae Dietzel and Andee Kaplan

Don’t abuse your power

If you scrape a website, please read the terms and conditions!!
It’s sometimes more efficient/appropriate to find the API.
If a website public offers an API, USE IT (instead of scraping)!!!

http://www.wired.com/2014/01/how-to-hack-okcupid http://www.vox.com/2016/5/12/11666116/70000-okcupid-users-data-release

Let’s scrape some data!

We’re going to scrape the open access journal “Agronomy” for papers about nitrogen and get some information from them:

Title
Authors
Year
Tables

Your turn

Open up the lab script and get going! Here are some helpful function names from rvest:

read_html
html_nodes
html_text
html_table

SelectorGadget may also be helpful.