Pre-R Data

Experimental design

• Reproducibility and collaboration begin before you head to the field or lab
  
• If you are going to work with a statistician (and you should), talk with them when you are designing your project
  
• Search for data related to your field site to determine expected variability and potential problems
  
• Set up a GitHub repo and introduce the crazy idea of version control to your collaborators
  

Experimental design

Collecting data

• Establish a recorded protocol complete with references
  
• Test if everyone can follow the protocol before they are left on their own
  
• Bring the protocol with you to the field and record deviations
  
• Name everything that is unique – each chamber, depth, or rep

Collecting data

• Take notes on anything that might be slightly relevant, but don’t go overboard – you should have established most of the required information before you got to the field/lab
  
• Take pictures/video in the field/lab until your coworkers are just beyond the point of total annoyance

Do not assume you will remember anything. Remembering indicates there is a piece of data that exists only in your head!

Collected data

• If your data exists only on paper – scan or photograph the paper
  
• There will be an error and you’ll need to refer back
  
• Make data entry as easy as possible for those entering data
  
• If data collection was automated, keep it just as it comes in

Never manually change your raw data

Once you have the data, organize it according to Karl Broman’s spreadsheet data organization

Be consistent

• Use consistent codes for variable names
  
• Not “Soy”, “soy”, “Soybean”
• Use a single fixed code for any missing values
  
• NA, NAN, . . .
  
• Use consistent codes for categorical variables
• Location - “Arm”, “Lewis”, “Lew”
  *Use common data layout if you have multiple files

Write dates as YYYY-MM-DD

xckd.com

Excel is a date monster

Be very, very careful.

Fill in all of the cells

• If a cell is blank, no one knows if it is intentional or if the data are missing
  
• Don’t depend on one cell to cover the information for several cells, you may understand, but a computer won’t

Put just one thing in a cell

• Separate the information as well as you can
  
• For example, don’t combine block “A” and plot “23” as “A23”, or you will need to separate that information later

Make it a rectangle

• Rows should correspond to identifiers (blocks, treatments, reps)
  
• Columns should correspond to variables
  
• If you treat your spreadsheet like a virtual desk with data and calculations and graphs spread all over the place, even other humans are going to have a tough time

More on tidy data later

Create a data dictionary

• Keep a separate file that explains what everything is (also in a rectangle)
  
• Exact variable name
  
• Version of name used for visualization
  
• Long explanation of the variable
  
• Measurement units

No calculations in raw data files

• If you just can’t wait, make a copy and create your messes there
  
• Otherwise known as “relapse”

Don’t use font color or highlighting as data

Choose good names for things

• So hard
  
• Don’t use spaces
  
• Avoid special characters($, @, %, #, &, *, (, ), !, etc.)
  
• Short, but meaningful

Make backups

• Best policy is three locations
  
• One local (hard drive/server)
• One off-site (GitHub)
  
• Another one off-site, preferably hosted in a different region (Dropbox, Cybox)

Save data in plain text files

• This is a non-proprietary format, so anyone can open it at any point in the future
  
• .csv preferred
  
• Keep as much data in one tab as possible, otherwise save each tab as a separate file

If you have a lot of tabs or files, maybe you should be using a database. We will talk about these next week!

Spreadsheet cautionary tales

Washington Post

Spreadsheet cautionary tales

Genome Biology

Further resources

Jenny Bryan’s Spreadsheet Page
Data Carpentry Workshop
Checklist from rOpenSci
Also very good:
How to share data with a statistician