Data cleaning process

When the raw data arrived, first check whether the data contains information you meant.
The goal of data cleaning is to make normalized data tables. Therefore, once the structure of the raw data is understood, we should first outline the list of normalized data tables and then consider how to clean raw data sets.
We first pre-clean each raw data set. This includes selecting relevant columns, renaming the columns, replacing values, and mutating new variables, if necessary. We should pre-clean each raw data one by one: one pre-cleaning file for one raw data set.
We then combine multiple pre-cleaned data sets to generate the target normalized tables. This step no longer reads the raw data sets. It should only read the pre-cleaned data sets.

Check the raw data

Check the candidate for the primary key variables of each raw data.
- How about the number of unique values of the candidate's primary key variables? Does the size match your expectation?
Check the candidate for the foreign key variables of each raw data.
- How about the matching of the foreign key variables across the raw data? Do they match most of the observations?
Check the variations of the variables you are going to use.
- How about the missing values? Do they have enough non-missing values?
- How about the variation? Do they have enough reasonable variations?

The name of each data set.
The path to each data set
List up columns/information to select
- Rename to: write the new variable if you rename the column.
- Values to: write how you replace values and how you deal with NAs.
- Mutate to: write how you will make new variables based on.
The information in the selected columns should be the superset of the information contained in the variables of the cleaned data tables.