---
title: "funModeling quick-start"
author: "Pablo Casas"
output:
rmarkdown::html_vignette:
md_extensions: [
"-autolink_bare_uris"
]
date: "`r Sys.Date()`"
vignette: >
%\VignetteIndexEntry{funModeling quick-start}
%\VignetteEngine{knitr::rmarkdown}
%\usepackage[utf8]{inputenc}
---
```{r setup, echo = FALSE}
knitr::opts_chunk$set(
out.width = "700px"
)
```
# funModeling quick-start {#quick_start}
This package contains a set of functions related to exploratory data analysis, data preparation, and model performance. It is used by people coming from business, research, and teaching (professors and students).
`funModeling` is intimately related to the _Data Science Live Book_ -Open Source- (2017) in the sense that most of its functionality is used to explain different topics addressed by the book.
## Blog posts based on `funModeling`:
* [Exploratory Data Analysis in R (introduction)](https://blog.datascienceheroes.com/exploratory-data-analysis-in-r-intro/)
* [Automatic data types checking in predictive models](https://blog.datascienceheroes.com/automatic-data-types-checking-in-predictive-models/)
* [Fast data exploration for predictive modeling](https://blog.datascienceheroes.com/fast-data-exploration-for-predictive-modeling/)
* [New discretization method: Recursive information gain ratio maximization](https://blog.datascienceheroes.com/discretization-recursive-gain-ratio-maximization/)
## Opening the black-box
Some functions have in-line comments so the user can open the black-box and learn how it was developed, or to tune or improve any of them.
All the functions are well documented, explaining all the parameters with the help of many short examples. R documentation can be accessed by: `help("name_of_the_function")`.
### About this quick-start
This quick-start is focused only on the functions. All explanations around them, and the how and when to use them, can be accessed by following the "_**Read more here.**_" links below each section, which redirect you to the book.
Below there are most of the `funModeling` functions divided by category.
## Exploratory data analysis
### `status`: Dataset health status (2nd version)
Similar to `df_status`, but it returns all percentages in the 0 to 1 range (not 1 to 100).
```{r, message=FALSE, warning=FALSE}
library(funModeling)
status(heart_disease)
```
Note: `df_status` will be deprecated, please use `status` instead.
### `data_integrity`: Dataset health status (2nd version)
A handy function to return different vectors of variable names aimed to quickly filter NA, categorical (factor / character), numerical and other types (boolean, date, posix).
It also returns a vector of variables which have high cardinality.
It returns an 'integrity' object, which has: 'status_now' (comes from status function), and 'results' list, following elements can be found: vars_cat, vars_num, vars_num_with_NA, etc. Explore the object for more.
```{r, message=FALSE, warning=FALSE}
library(funModeling)
di=data_integrity(heart_disease)
# returns a summary
summary(di)
# print all the metadata information
print(di)
```
### `plot_num`: Plotting distributions for numerical variables
Plots only numeric variables.
```{r, fig.height=4, fig.width=6}
plot_num(heart_disease)
```
Notes:
* `bins`: Sets the number of bins (10 by default).
* `path_out` indicates the path directory; if it has a value, then the plot is exported in jpeg. To save in current directory path must be dot: "."
[**Read more here.**](https://livebook.datascienceheroes.com/exploratory-data-analysis.html#plotting-numerical-variable)
### `profiling_num`: Calculating several statistics for numerical variables
Retrieves several statistics for numerical variables.
```{r}
profiling_num(heart_disease)
```
Note:
* `plot_num` and `profiling_num` automatically exclude non-numeric variables
[**Read more here.**](https://livebook.datascienceheroes.com/exploratory-data-analysis.html#numerical-profiling-in-r)
### `freq`: Getting frequency distributions for categoric variables
```{r distribution1, message=FALSE, fig.height=4, fig.width=6, warning=FALSE}
library(dplyr)
# Select only two variables for this example
heart_disease_2=heart_disease %>% select(chest_pain, thal)
# Frequency distribution
freq(heart_disease_2)
```
Notes:
* `freq` only processes `factor` and `character`, excluding non-categorical variables.
* It returns the distribution table as a data frame.
* If `input` is empty, then it runs for all categorical variables.
* `path_out` indicates the path directory; if it has a value, then the plot is exported in jpeg. To save in current directory path must be dot: "."
* `na.rm` indicates if `NA` values should be excluded (`FALSE` by default).
[**Read more here.**](https://livebook.datascienceheroes.com/exploratory-data-analysis.html#profiling-categorical-variables)
## Correlation
### `correlation_table`: Calculates R statistic
Retrieves R metric (or Pearson coefficient) for all numeric variables, skipping the categoric ones.
```{r}
correlation_table(heart_disease, "has_heart_disease")
```
Notes:
* Only numeric variables are analyzed. Target variable must be numeric.
* If target is categorical, then it will be converted to numeric.
[**Read more here.**](https://livebook.datascienceheroes.com/exploratory-data-analysis.html#linear-correlation)
### `var_rank_info`: Correlation based on information theory
Calculates correlation based on several information theory metrics between all variables in a data frame and a target variable.
```{r}
var_rank_info(heart_disease, "has_heart_disease")
```
Note: It analyzes numerical and categorical variables. It is also used with the numeric discretization method as before, just as `discretize_df`.
[**Read more here.**](https://livebook.datascienceheroes.com/selecting-best-variables.html#select_features_var_rank_info)
### `cross_plot`: Distribution plot between input and target variable
Retrieves the relative and absolute distribution between an input and target variable.
Useful to explain and report if a variable is important or not.
```{r profiling1, fig.height=4, fig.width=8}
cross_plot(data=heart_disease, input=c("age", "oldpeak"), target="has_heart_disease")
```
Notes:
* `auto_binning`: `TRUE` by default, shows the numerical variable as categorical.
* `path_out` indicates the path directory; if it has a value, then the plot is exported in jpeg.
* `input` can be numeric or categoric, and `target` must be a binary (two-class) variable.
* If `input` is empty, then it runs for all variables.
[**Read more here.**](https://livebook.datascienceheroes.com/selecting-best-variables.html#profiling_target_cross_plot)
### `plotar`: Boxplot and density histogram between input and target variables
Useful to explain and report if a variable is important or not.
**Boxplot:**
```{r boxplot_analysis, fig.height=3, fig.width=5}
plotar(data=heart_disease, input = c("age", "oldpeak"), target="has_heart_disease", plot_type="boxplot")
```
[**Read more here.**](https://livebook.datascienceheroes.com/selecting-best-variables.html#target-profiling-using-boxplots)
**Density histograms:**
```{r density_histogram, fig.height=3, fig.width=5}
plotar(data=mtcars, input = "gear", target="cyl", plot_type="histdens")
```
[**Read more here.**](https://livebook.datascienceheroes.com/selecting-best-variables.html#target-profiling-using-density-histograms)
Notes:
* `path_out` indicates the path directory; if it has a value, then the plot is exported in jpeg.
* If `input` is empty, then it runs for all numeric variables (skipping the categorical ones).
* `input` must be numeric and target must be categoric.
* `target` can be multi-class (not only binary).
### `categ_analysis`: Quantitative analysis for binary outcome
Profile a binary target based on a categorical input variable, the representativeness (`perc_rows`) and the accuracy (`perc_target`) for each value of the input variable; for example, the rate of flu infection per country.
```{r}
df_ca=categ_analysis(data = data_country, input = "country", target = "has_flu")
head(df_ca)
```
Note:
* `input` variable must be categorical.
* `target` variable must be binary (two-value).
This function is used to analyze data when we need to reduce variable cardinality in predictive modeling.
[**Read more here.**](https://livebook.datascienceheroes.com/data-preparation.html#high_cardinality_predictive_modeling)
## Data preparation
### Data discretization
#### `discretize_get_bins` + `discretize_df`: Convert numeric variables to categoric
We need two functions: `discretize_get_bins`, which returns the thresholds for each variable, and then `discretize_df`, which takes the result from the first function and converts the desired variables. The binning criterion is equal frequency.
Example converting only two variables from a dataset.
```{r}
# Step 1: Getting the thresholds for the desired variables: "max_heart_rate" and "oldpeak"
d_bins=discretize_get_bins(data=heart_disease, input=c("max_heart_rate", "oldpeak"), n_bins=5)
# Step 2: Applying the threshold to get the final processed data frame
heart_disease_discretized=discretize_df(data=heart_disease, data_bins=d_bins, stringsAsFactors=T)
```
The following image illustrates the result. Please note that the variable name remains the same.
Notes:
* This two-step procedure is thought to be used in production with new data.
* Min and max values for each bin will be `-Inf` and `Inf`, respectively.
* A fix in the latest `funModeling` release (1.6.7) may change the output in certain scenarios. Please check the results if you were using version 1.6.6. More info about this change [here](https://s3.amazonaws.com/datascienceheroes.com/img/blog/changes_discretize_df.png).
[**Read more here.**](https://livebook.datascienceheroes.com/data-preparation.html#data-discretization)
### `equal_freq`: Convert numeric variable to categoric
Converts numeric vector into a factor using the equal frequency criterion.
```{r}
new_age=equal_freq(heart_disease$age, n_bins = 5)
# checking results
Hmisc::describe(new_age)
```
[**Read more here.**](https://livebook.datascienceheroes.com/data-preparation.html#equal-frequency-binning)
Notes:
* Unlike `discretize_get_bins`, this function doesn't insert `-Inf` and `Inf` as the min and max value respectively.
### `discretize_rgr`: Variable discretization based on gain ratio maximization
This is a new method developed in `funModeling` developed improve the binning based on a binary target variable.
```{r}
input=heart_disease$oldpeak
target=heart_disease$has_heart_disease
input2=discretize_rgr(input, target)
# checking:
summary(input2)
```
Adjust max number of bins with: `max_n_bins`; `5` as default.
Control minimum sample size per bin with `min_perc_bins`; `0.1` (or 10%) as default)
### `range01`: Scales variable into the 0 to 1 range
Convert a numeric vector into a scale from 0 to 1 with 0 as the minimum and 1 as the maximum.
```{r}
age_scaled=range01(heart_disease$oldpeak)
# checking results
summary(age_scaled)
```
## Outliers data preparation
### `hampel_outlier` and `tukey_outlier`: Gets outliers threshold
Both functions retrieve a two-value vector that indicates the thresholds for which the values are considered as outliers.
The functions `tukey_outlier` and `hampel_outlier` are used internally in `prep_outliers`.
**Using Tukey's method:**
```{r}
tukey_outlier(heart_disease$resting_blood_pressure)
```
[**Read more here.**](https://livebook.datascienceheroes.com/data-preparation.html#detecting-outliers-using-tukey-method)
**Using Hampel's method:**
```{r}
hampel_outlier(heart_disease$resting_blood_pressure)
```
[**Read more here.**](https://livebook.datascienceheroes.com/data-preparation.html#detecting-outliers-using-hampel-method)
### `prep_outliers`: Prepare outliers in a data frame
Takes a data frame and returns the same data frame plus the transformations specified in the `input` parameter. It also works with a single vector.
Example considering two variables as input:
```{r}
# Get threshold according to Hampel's method
hampel_outlier(heart_disease$max_heart_rate)
# Apply function to stop outliers at the threshold values
data_prep=prep_outliers(data = heart_disease, input = c('max_heart_rate','resting_blood_pressure'), method = "hampel", type='stop')
```
Checking the before and after for variable `max_heart_rate`:
```{r, echo=FALSE}
# Checking max and min value for 'max_heart_rate' before the transformation
sprintf("Before transformation -> Min: %s; Max: %s", min(heart_disease$max_heart_rate), max(heart_disease$max_heart_rate))
# Apply function to stop outliers at the threshold values
data_prep=prep_outliers(data = heart_disease, input = c('max_heart_rate','resting_blood_pressure'), method = "hampel", type='stop')
# Checking the results, the maximum value is now 174.5 (the minimum remains the same)
# Checking max and min value for 'max_heart_rate' before the transformation
sprintf("After transformation -> Min: %s; Max: %s", min(data_prep$max_heart_rate), max(data_prep$max_heart_rate))
```
The min value changed from 71 to 86.23, while the max value remained the same at 202.
Notes:
* `method` can be: `bottom_top`, `tukey` or `hampel`.
* `type` can be: `stop` or `set_na`. If `stop` all values flagged as outliers will be set to the threshold. If `set_na`, then the flagged values will set to `NA`.
[**Read more here.**](https://livebook.datascienceheroes.com/data-preparation.html#analyze-binary-outcome-vs-categorical-variable)
## Predictive model performance
### `gain_lift`: Gain and lift performance curve
After computing the scores or probabilities for the class we want to predict, we pass it to the `gain_lift` function, which returns a data frame with performance metrics.
```{r performance, fig.height=3, fig.width=7}
# Create machine learning model and get its scores for positive case
fit_glm=glm(has_heart_disease ~ age + oldpeak, data=heart_disease, family = binomial)
heart_disease$score=predict(fit_glm, newdata=heart_disease, type='response')
# Calculate performance metrics
gain_lift(data=heart_disease, score='score', target='has_heart_disease')
```
[**Read more here.**](https://livebook.datascienceheroes.com/model-performance.html#gain_and_lift)
### `coord_plot`: Coordinate plot (clustering models)
Useful when we want to profile cluster results in terms of its means.
Imagine `cyl` can be the cluster number.
```{r cluster_performance, fig.height=3, fig.width=6}
coord_plot(data=mtcars, group_var="cyl", group_func=median, print_table=TRUE)
```
---
* Github repository (report bugs or improvements): [https://github.com/pablo14/funModeling](https://github.com/pablo14/funModeling)
* Creator and maintainer: Pablo Casas [https://twitter.com/pabloc_ds](https://twitter.com/pabloc_ds)