---
title: "EDA: Exploring your dataset in R"
output: html_notebook
---

---

- EDA (Exploratory Data Analysis) 是資料科學的核心步驟。

- 從特徵工程 feature engineering、建模 modeling 到溝通 communication 都扮演著重要的角色。

- **注意** 結合自己的領域知識與興趣是關鍵，否則容易被自動化的技術取代。

- 以下是一個簡單 EDA 自動化的例子 (`DataExplorer`: https://boxuancui.github.io/DataExplorer/)。參考來源是一個可愛的計畫(https://www.littlemissdata.com/blog/steam-data-art2)

```{r}
# install.packages("DataExplorer")
```


- ![圖片來源](https://images.squarespace-cdn.com/content/v1/58eef8846a4963e429687a4d/1541391131777-YNJ8X44SZ1PB6AUWA9CQ/ke17ZwdGBToddI8pDm48kHfqQC-zsNgVx8BfEetvzTtZw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZUJFbgE-7XRK3dMEBRBhUpzPD3satZ1UO_T9bCxVh7-Rt3xZnBSKhYaXwjIaM0_FUvaJrUbeykrdF7lReoNjj9k/dataExplorerGifLg.gif?format=1500w)


---

# 示範

```{r warning=FALSE}
# load the readr library to bring in the dataset
library(readr)
# Download the data set
d <- read_csv('https://raw.githubusercontent.com/lgellis/STEM/master/DATA-ART-1/Data/FinalData.csv', col_names = TRUE)

```

- 基本觀察數據的指令與概念妳應該都很熟悉了

```{r}
head(d, 10)
dim(d)
```



- 妳也可以使用 `dplyr` 的 `glimpse()`, 可以提供數據的垂直觀 (vertical preview of the dataset): a preview of all columns as a row so that it's very easy to take in. 以及 `summary()`。



```{r}
library(dplyr)
glimpse(d)
summary(d)
```

- 如果妳要更炫一點，可以用 `skimr` 來直接秀出 missing values, inline histogram 等等。 (不一定要為此就安裝套件) 

```{r, warning=FALSE}
#install.packages('skimr')
library(skimr)
skim(d)
```



- 此外，` visdat` 套件也可以較直觀的看圖形與統計結果。(一樣，不一定要為此就安裝套件)

```{r}
#library(devtools)
#devtools::install_github("ropensci/visdat")
library(visdat)
vis_miss(d)
vis_dat(d)
```



# 有趣的來了 QQ

- Automating Data Scientist/EDA: DataExplorer 套件中的 `create_report()`會自動幫妳做基本的 EDA 報告。 

> It pulls a full data profile of your data frame and produces a html file with the basic statistics, structure, missing data, distribution visualizations, correlation matrix and principal component analysis for your data frame! 

- 試試先 

```{r message=FALSE, warning=FALSE}
library(DataExplorer)
library(ggplot2)
create_report(d)
# 自動儲存並開啟瀏覽器分頁顯示產生的報告文件
```


- 也可以改圖形主題的參數 (`theme_config()` to pass on the theme paramaters). 以及其他的分別作圖函數 (`plot_box()` or `plot_histgram()`) 


```{r}
plot_intro(d, ggtheme = theme_minimal(),
             title = "Automated EDA with Data Explorer",
             )
```


- 其實自己也可以做看看。


## 進階使用

```{r}
#devtools::install_github("alastairrushworth/inspectdf")
library(inspectdf)
```


- 先做 data transformation, 建立三個 data frame 

  - `allGrades` is the full data frame with the complete set of survey results

  - `oldGrades` includes a subset of the survey results for all grades greater than 5.  This includes grades 6-8.

  - `youngGrades` includes a subset of the survey results for all grades less than 6.  This includes grades 3-5.

We will use allGrades for the single data frame analysis and oldGrades and youngGrades for the data frame comparisons.  


```{r}
allGrades <- d
oldGrades <- allGrades %>% 
  filter(Grade > 5)
youngGrades <- allGrades %>% 
  filter(Grade < 6)

#View the distribution of grade to ensure it was split properly
library(ggplot2)
ggplot(oldGrades, aes(x=Grade)) + 
  geom_histogram()
ggplot(youngGrades, aes(x=Grade)) + 
  geom_histogram()

```

### 比較 data frame

-  use the `inspect_types()` command to very easily see a breakdown of character vs numeric variables.

```{r}
inspect_types(allGrades, show_plot = TRUE)
inspect_types(youngGrades, oldGrades, show_plot = TRUE)
```


- `inspect_mem()` function will tell us some basic sizing information, including data frame columns, rows, total size and the sizes of each variable.  

```{r}
all<- inspect_mem(allGrades)
show_plot(all)
m <- inspect_mem(youngGrades, oldGrades)
show_plot(m)
```



- The `inspect_na()` function shows us the percentage of na values for each variable.  The comparison view is quite neat as it highlights variables with unequal na percentages.  



```{r}
all.na <- inspect_na(allGrades)
yo.na<- inspect_na(youngGrades, oldGrades)
show_plot(all.na)
show_plot(yo.na)
```



## 琢磨一下，期中考後會有個練習題




## Reference
- https://www.littlemissdata.com/blog/simple-eda
- https://www.littlemissdata.com/blog/inspectdf





