What is data cleaning?

• The goal of data cleaning is to generate a list of normalized tables of available information that satisfies several restrictions that are common across following analyses and store them in cleaned folder.
• A normalized table is a concept that is mathematically defined in the relational database modeling.
• All following analyses should only refer to the output of data cleaning process, the list of normalized tables and are strictly prohibited from referring to the raw information.
• For this reason, we distinguish cleaned folder that only contains the list of normalized tables from input folders that contain raw data and intermediate folder that contains temporary data generated in the data cleaning step. The above requirement means that the following analysis should never read data from iput or intermediate folders but should only refer to files in cleaned folder.
• If an analyst finds that s/he needs raw information not yet contained in the list of normalized tables, s/he should go back to the data cleaning step to update the list of normalized tables so that they contain the relevant information.
• The list of normalized tables are not formatted for specific analysis. In this sense, it should be distinguished from data transformation (or data representation or feature engineering), which reads the list of normalized tables and generates data objects that are convenient for each analysis.

Normalized tables

• Reference in Japanese:　https://www.amazon.co.jp/dp/4781913903/ref=cm_sw_r_tw_dp_BKJJ15S0KQWMJYAA5EXH
• Reference in English: TBA
• A domain is a set of elements. For example, the set of names $D_1 = \{\text{John}, \text{Bob}, \text{Alice}\}$, the set of grades $D_2 = \{A, B, C, D\}$.
• Consider the direct product of the domains given n domains, $D = D_1 \times D_2 \times \cdots \times D_n$.
• An element of the direct product of the domains is called a tuple $r \in D$. In the case of the set of names $D_1 = \{\text{John}, \text{Bob}, \text{Alice}\}$ and the set of grades $D_2 = \{A, B, C, F\}$, we have $3 \times 4 = 12$ tuples in the direct product, such as $(John, A)$. You can imagine that a tuple is a possible realization of a data record.
• A relation $R$ is defined as the subset of the direct product of the domains, $R \subset D = D_1 \times D_2 \times \cdots \times D_n$. You can imagine that a relation is the data.
• The number of tuples in $R$ is called the cardinality of $R$. The number of domains that define the relation is called the degree of $R$. You can imagine that cardinality is the number of observations and degree is the number of the variables in the data.
• We define the attribute name of a domain and the relation name of the relation. You can imagine that the attribution name of a domain is the name of the variable and the relation name is the name of the data.
• Technically, we first define the set of attribute names and define a domain function $dom: A_i \to D_i$ that returns a domain given an attribute name to define the domain.
• Let $R$ is the name of a relation, $A_1, \cdots, A_n$ are the set of attribute names, and $dom$ is the domain function, we define the relation $R(A_1, \cdots, A_n)$ as a subset of $dom(A_1)\times \cdots \times dom(A_n)$.
• The set of attribute names $A_1, \cdots, A_n$ and the domain function $dom$ that generate relations as a subset of $dom(A_1) \times \cdots \times dom(A_n)$ is called the relation scheme and denoted by $\boldsymbol{R}(A_1, \cdots, A_n)$. A specific relation generated from the relation scheme is called an instance of the relation scheme.
• Let $\Omega_{\boldsymbol{R}}$ denote the set of attribute names $\{A_1, \cdots, A_n\}$. For example, if $A_1$ is the name and $A_2$ is the grade, then $\Omega_{\boldsymbol{R}}$ is the name and grade. Let $t(A_i)$ dente the tuple $t$’s value of attribute $A_i$ and $t[A_{i_1}, \cdots, A_{i_J}]$ be the tulple’s values of $A_{i_1}, \cdots, A_{i_J}$. For example, $t(A_1)$ is the name of tuple $t$ and $t(A_1, A_2)$ is the name and grade of tuple $t$.