Scorecard Building in R – Part V – Rejected Sample Inference, Grade Analysis and Scoring Techniques

In the previous section, part IV of the scorecard building process, I trained, validated and tested a logistic regression model serving as the heart of the scorecard. In this section, I address the obvious sample selection problem where loans are accepted based on merit and personal credit information, and also rejected because of lack of credentials. I also look into analyzing model assumptions where the predicted scores for the training set is used to built a grading scheme. As an extra exercise, I scale the log-odds score into a more understandable scoring function.

To fully account for the sample selection bias in our model, performance inference methods are utilized to predict the performance of rejected clients if they were actually given a loan. The first step is creating a function that maps some sort of domain knowledge of features to the log-odds of accepted customers. This function will then be used to predict the probabilities of rejected customers. Here, the rejected data set does not include which applicants applied for a 36-month loan term so I assume that all of these applicants were considered.

I will utilize the following packages:


I will require the dataset of rejected applicants and the data available from them. This can be downloaded here.

rejected_data <- read.csv("C:/Users/artemior/Desktop/Lending Club Model/RejectStatsD.csv")

To start off, I create a new logistic regression model with the same Bad_Binary variables and only the features that are common between both accepted and rejected applicants. In this case, both contain zip code, state and employment length. The data that I will use comes from section 2, more specifically WOE_matrix and Bad_Binary. It is assumed that building this inference model takes into account all variable WOE transformations. Bad_Binary_Inference calls upon the original Bad variable from the features_36 vector.

Bad_Binary_Original <- features_36$Bad
sample_inference_features <- WOE_matrix_final[c("zip_code", "addr_state", "emp_length")]
sample_inference_features["Bad_Binary"] <- Bad_Binary_Original

Run a simple generalized linear model on the accepted applicants data set.

sample_inference_model <- glm(Bad_Binary ~ ., data = sample_inference_features)

Here, I calculate the WOEs for the rejected applicants by applying the WOE tables from the accepted applicants onto the rejected applicant data set. The code and methodology of transformation is exactly that in part III.

features_36_inference % select(-Amount.Requested, -Application.Date,
-Loan.Title, -Risk_Score,
-Debt.To.Income.Ratio, -Policy.Code)

features_36_inference_names <- colnames(features_36_inference)

Initiate cluster for parallel processing.


number_cores <- detectCores() – 1
cluster <- makeCluster(number_cores)
clusterExport(cluster, c("IV", "min_function", "max_function",
"only_features_36", "recode", "WOE_tables"))

Create the WOE matrix table for the rejected data applicants.

WOE_matrix_table_inference <- parSapply(cluster, as.matrix(features_36_inference_names),
FUN = WOE_tables_function)

WOE_matrix_inference is the converted WOE matrix for the rejected applicant data set. This is the dataset we will be using to predict their scores for model performance inference.

WOE_matrix_inference <- parSapply(cluster, features_36_inference_names,
FUN = create_WOE_matrix)

Using WOE_matrix_inference to come up with predicted probabilities using the sample_inference_model, which was built on the accepted applicant data set.

rejected_inference_prob <- predict(sample_inference_model,
data = WOE_matrix_inference,
type = "response")
rejected_inference_prob_matrix <- as.matrix(rejected_inference_prob)
rejected_inference_prob_dataframe <-
colnames(rejected_inference_prob_dataframe) <- c("Probabilities")


Now I obtain the predicted probabilities using the cross-validated elastic-net logistic regression model from section 4 for all accepted applicants

number_cores <- detectCores()
cluster <- makeCluster(number_cores)

accepted_prob <- predict(, LC_WOE_Dataset, type = "prob")
accepted_prob_matrix <- as.matrix(accepted_prob[,2])
accepted_prob_dataframe <-
colnames(accepted_prob_dataframe) <- c("Probabilities")

I combine the probabilities from both the rejected and accepted applicants and generate a graph that depicts the distribution of the probabilities.

probability_matrix <- rbind(accepted_prob_matrix, rejected_inference_prob_matrix)
probability_matrix <-
colnames(probability_matrix) <- c("Probabilities")

I use ggplot to plot the distribution of probabilities of default. Here we see that the distribution is left skewed. This is a representation of both accepted and rejected applications. It is also useful to analyze the distribution of the accepted and rejected applicants separately.

probability_distribution <- ggplot(data = probability_matrix, aes(Probabilities))
probability_distribution <- probability_distribution + geom_histogram(bins = 50)

accepted_probability_distribution <- ggplot(data = accepted_prob_dataframe, aes(Probabilities))
accepted_probability_distribution <- accepted_probability_distribution + geom_histogram(bins = 50)


rejected_probability_distribution <- ggplot(data = rejected_inference_prob_dataframe, aes(Probabilities))
rejected_probability_distribution <- rejected_probability_distribution + geom_histogram(bins = 50)


The accepted probability distribution is left-skewed while the distribution of the rejected applicants is normal. This could be interpreted as rejected applicants exhibiting a normally distributed score if they were to be funded. Therefore, rejected sample selection bias is minimal if applicants are being rejected randomly through a normal distribution. If this was not the case, I would suspect some bias in the acceptance and rejection of applicants.

After gaining some insight on how our model will perform on a population it has never seen before. We look to formalizing the scorecard by creating a grading scheme that defines several levels of risk.

Here, I am going to organize the accepted applicant probability data set into bins to initiate a lift analysis. We use lift analysis to help us determine which bins of scores are going to be described by particular letter grades. I create 25 different bins to mimic the sub-grade system that LC has from their given public data set. I then append the “Bad” column from features_36 to this vector and summarize the information so that I may be able to calculate the proportions of bads within each bin.

bins = 25
Bad_Binary_Values <- features_36$Bad
prob_bad_matrix <-, Bad_Binary_Values))
colnames(prob_bad_matrix) <- c("Probabilities", "Bad_Binary_Values")

I sort the probabilities and binary values in an increasing order based on the probabilities column. By ordering the first column, the corresponding values in the second column are also sorted.

Probabilities <- prob_bad_matrix[,1]
Bad_Binary_Values <- prob_bad_matrix[,2]
order_accepted_prob <- prob_bad_matrix[order(Probabilities, Bad_Binary_Values, decreasing = FALSE),]

I create the bins based on the sorted probabilities and create a new data frame consisting of only the bins and bad binary values. This will be the data frame I use to conduct a lift analysis.

bin_prob <- cut(order_accepted_prob$Probabilities, breaks = bins, labels = 1:bins)
order_bin <-, order_accepted_prob[,2]))
colnames(order_bin) <- c("Bin", "Bad")

I summarize the information where I calculate the proportion of bads within each bin.

bin_table <- table(order_bin$Bin, order_bin$Bad)

Bin_Summary <- group_by(order_bin, Bin)

Bad_Summary <- summarize(Bin_Summary, Total = n(), Good = sum(Bad), Bad = 1 - Good/Total)

Using Bad_Summary, I plot a bar plot that represents the lift analysis.

lift_plot <- ggplot(Bad_Summary, aes(x = Bin, y = Bad))
lift_plot <- lift_plot + geom_bar(stat = "identity", colour = "skyblue", fill = "skyblue")
lift_plot <- lift_plot + xlab("Bin")
lift_plot <- lift_plot + ylab("Proportion of Bad")


Here, the graph shows 25 bins and the proportion of bad customers within each bin. As expected, the bins have a decreasing trend of proportion of bads which shows the effectiveness of our classifer.By separating them into 25 bins, I mimic LC’s subgrading system a nd could apply the exact same logic to this scorecard.

To finalize the scorecard, I generate a linear function of log-odds and apply a three-digit score mapping system that will assist upper management in understanding the risk score obtained from the scorecard. First I convert the probability scores into log-odds form and figure out what type of linear transformation I would like to apply.

Accepted_Probabilities <- Probabilities
LogOdds <- log(Accepted_Probabilities)

The score is up to the analyst and how they feel is the best way to present it to upper management for easy interpretation. Here, I will apply a three-digit score transformation to the log-odds using a simple linear function. To calculate the slope of this linear line, I use the minimum and maximum log-odds and use that as my range for a score range from 100 – 1000.

max_score <- 1000
min_score <- 100
max_LogOdds <- max(LogOdds)
min_LogOdds <- min(LogOdds)

linear_slope <- (max_score - min_score)/(max_LogOdds - min_LogOdds)
linear_intercept <- max_score - linear_slope * max_LogOdds

Here, the linear slope is 82.6 and the intercept is 1000. This means that the average applicant will have a risk score that is maxed out, guaranteed funding . As the Log-Odds decreases based on the features that are inputted into the model, the score will decrease significantly. The way to interpret the score here is that for every 1 unit of log-odds, the score will decrease by 82.6 units. That means someone that is twice as risky as someone with 1 unit of log-odds will have their score decreased by 165.2. Therefore, every 82.6 units in a score indicates levels of riskiness that is easily understood by non-technical audiences.

Concluding Remarks

There you have it! After 5 parts into the scorecard building process, the scorecard is ready for presentation and production. Now, I could go even further into explaining some of the things that could be changed for the scorecard, such as changing scoring thresholds that meet business requirements or accepted levels of risk. I could even go further into explaining how you can link the scorecard, grades and expected losses and margins of profit for the company. This stems beyond what I have demonstrated here but is definitely possible for utmost consideration especially for a financial company.

Source Code

The R code for all 5 parts of the scorecard building process can be found at my Github page.

Scorecard Building in R – Part IV – Training, Testing and Validating the Logistic Regression Model

Continuing from part III where the Weight-of-Evidence matrix and information values were determined to give us an idea of how the consumer credit information could lead to predict the performance of 36-month loans. In this part, we train, test and validate an elastic-net Logistic Regression model. This statistical model is one of the most widely used machine learning techniques that maps a bernoulli distributed variable to a continuous log-odds value. Here we will also use parallel processing to speed up the high amounts of calculations and algorithms done by the caret package on the data set.


Set the seed so that we may receive reproducible results when we train our model.


Redefine the WOE matrix obtained from part III as our main dataset for this section. Please see part III to see how WOE_matrix_final was obtained.

LC_WOE_Dataset <- WOE_matrix_final

Use createDataPartition to divide the random sample into a training and test set where 75% of the data goes to training and 25% goes to testing

partition <- createDataPartition(LC_WOE_Dataset$Bad_Binary, p = 0.75, list = FALSE)
training <- LC_WOE_Dataset[partition,]
testing <- LC_WOE_Dataset[-partition,]

Define the type of resampling that will be used. Here, I am interested in using repeated k-fold cross-validation. More specifically, I apply 3-fold cross-validation by setting the number of folds to 3. Later on, I want to select a model that maximizes the statistic AUC for this specific classification model. I set savePredictions to TRUE to save predictions for each hold-out in each step of the cross-validation. I set classProbs to TRUE to compute class probabilities and predicted values for each resample. summaryFunction is set to twoClassSummary which allows us to compute true-positive rates and false-positive rates later on.

fitControl <- trainControl(method = "cv",
number = 3,
savePredictions = TRUE,
classProbs = TRUE,
summaryFunction = twoClassSummary)

Now we train the model. First we set the seed to ensure that the algorithm is being run on the exact same data in each fold.


Here, parallel processing is initiated to speed up algorithms thereafter.

number_cores <- detectCores()

cluster <- makeCluster(number_cores)


Set up the lambda and alpha grids in which the train function will used to generate an elastic-net logistic regression model. The alpha term acts as a weight between L1 and L2 regularizations, where in such extremes, alpha = 1 gives the LASSO regression and alpha = 0 gives the RIDGE regression. Penalized linear regression models aims to balance the bias-variance trade-off who exhibits a relationship of increasing bias to decrease variance. The lambda parameter further penalizes coefficient estimate to 0 which indirectly serves as variable reduction. It is also the result of an elastic-net regression to handle collinearities very well.

Caution: the following code takes a long time to run. Here, I let the algorithm use its default grid of alpha and lambda parameters to obtain a solution. I could have more control over this if I set up my own sequences of alpha and lambda values and include a ‘tuneGrid’ entry into the train function. <- train(Bad_Binary ~., data = training,
method = "glmnet",
family = "binomial",
metric = "ROC",
trControl = fitControl,
tuneLength = 5)

When completed, we take a look at the summary of tuning parameters that were used in the cross-validation process. Here, we use the ROC to optimize alpha and lambda. Lambda = 1 is used and therefore, the model converges to a LASSO regression model. We also plot the graph of changing the tuning parameter, lambda.


There are some terminology to address. Specificity as presented in the summary is the fraction of loans that were good and predicted good by the model. Sensitivity is the fraction of loans that were bad and predicted bad by the model. In the summary, all sensitivities and specificities are presented for every alpha parameter and lambda hyperparameters. All corresponding ROC’s which actually represent the AUC value are presented.

Given the code above, 3-fold cross-validation splits the data set into 3 parts, labelled Set 1, 2 and 3. The algorithm selects 1 of the sets to be used as a test set and trains the model on the other two sets. It then calculates the AUC and stores it in memory. The algorithm repeats this procedure for every combination of training and test sets, ie. Training = ((1,2), (1,3), (2,3)), Testing = ((3), (2), (1)). After all ROCAUC’s are calculated, it averages over them and presents this for a particular initiated alpha parameter and lambda hyperparameter. In this case, the train function runs 3-fold cross-validation given the grid of starting parameters. THe ROCAUC given out by the algorithm represents the average over the 3 models that were trained in each iteration.

Now that we obtained an ROCAUC of 0.8754938 from the cross-valiation step. We now need to test how well the model performs on a set the model has never seen, our initial test set. If a similar ROCAUC is shown from testing the model on the test set, then we can conclude that overfitting has been appropriately addressed. It is up to the discretion of the analyst to decide the threshold similarity between cross-validation ROCAUC and test set ROCAUC. calculates the predicted probabilities used to generate the ROCAUC. <- predict(, testing, type = "prob")
auc.condition <- ifelse(testing$Bad_Binary == "Good", 1, 0) <- roc(auc.condition,[[2]]) provides an ROCAUC of 0.8792 which comes really close to our cross-validated ROCAUC. Here, I am happy with the results and proceed to use this as the final model for the purpose of LC’s risk scorecard.

I proceed to set up a visualization of the ROCAUC through a roc plot using the package pROC.

plot(, col = "red", grid = TRUE)


Since the primary focus of this project is to set up a logistic regression scorecard for Lending Club, the model obtained here is sufficient enough. I could go further and test out several different classification machine learning models such as random forests, binary trees, etc. By having the elastic-net logistic regression, I produce the coefficient estimates corresponding to the regularization parameter used. The heart of the model lies within the coefficient estimates. As previously mentioned, the algorithm is a form of variable selection in that it pins down features to 0 if overfitting is suspected or collinearities are present within the WOE dataset.

final.model <-$finalModel <- as.matrix(coef(final.model,$bestTune$lambda))

In the next section, Scorecard Building – Part V – Rejected Sample Inference, Grade Analysis and Scoring Techniques, I discuss how the model is evaluated and analyzed for further business implications.

Scorecard Building in R – Part III – Data Transformation

In part II of the scorecard building process, I had prepared the Lending Club data in order to create a Logistic Regression model that would enact as a scorecard in predicting good customers from bad ones.

In this section, I transform the data set by applying weight-of-evidence (WOE) value conversions where in the previous section, a weight-of-evidence was created for specific binning groups. Here, for each feature column, I take each data point and assign the weight-of-evidence value from its corresponding binning group. For example, for the home ownership variable, all customers who are paying mortgage on their homes have a weight-of-evidence of 0.29. Every entry in the home ownership column in the data set with a value of ‘Mortgage’ would then be replaced with 0.29. This transformation takes place for all features within the data set so that the new matrix contains only weight-of-evidence valued transformations.

The following code begins this process. Here, I continue to use the code from part II. I use the package ‘parallel’ to apply some basic parallel processing that will help make the code run faster.


First, here is a function to obtain the minimum value of range in string form and maximum value of range in string form.

min_function <- function(x) {
remove_brackets <- gsub("\\[|\\]", "", x = x)
take_min <- gsub(",.*", "", remove_brackets)
min_value <- as.numeric(take_min)

max_function <- function(x) {
remove_brackets <- gsub("\\[|\\]", "", x = x)
take_max <- gsub(".*,", "", remove_brackets)
max_value <- as.numeric(take_max)

The following presents a function that tabulates all WOE with their respective categories. This will help group all variables and allow for easier lookups when NA’s are transformed into -1.

features_36_names_WOE <- colnames(features_36)[-ncol(features_36)]
features_36_names_WOE_vector_length <- length(features_36_names_WOE)
only_features_36 <- features_36[-ncol(features_36)]

WOE_tables_function <- function(x) {
table_text <- sprintf("IV$Tables$%s", x)
create_table <- eval(parse(text = table_text))
MIN <- sapply(create_table, min_function, USE.NAMES = FALSE)[,1]
MAX <- sapply(create_table, max_function, USE.NAMES = FALSE)[,1]
MIN_equal_NA <-
count_MIN_equal_NA <- length(MIN[MIN_equal_NA])

if (count_MIN_equal_NA == 1) {
MIN[] <- -1
MAX[] <- -1
WOE <- create_table$WOE
categories <- create_table[,1]
table <-, WOE)

} else {
WOE <- create_table$WOE
table <- cbind(MIN, MAX, WOE)

To obtain the results of this WOE_tables function quickly, we assign the function to three cores attributed to the laptop I am using. This is known as Parallel Processing. This allows the slow task of applying a function over each row of data to be sped up.

First, calculate the number of cores that are located within the laptop. The memory used to apply the WOE_tables_function will be distributed among the number of cores minus 1. We need to save the last core for any other sort of activity we want to do that may or may not be programming related.

number_cores <- detectCores() - 1

Initiate Cluster, where the cluster is just the defined group of cores that are designated to process the memory.

cluster <- makeCluster(number_cores)

I assign the main functions that the cluster will be handling and processing. These include the functions that are called within the main function WOE_tables_function.

clusterExport(cluster, c("IV", "min_function", "max_function"))

WOE_tables is the resulting matrix which is created off the cluster. We use the function parSapply which is very similar to the sapply function except is run with parallel processing.

WOE_tables <- parSapply(cluster, as.matrix(features_36_names_WOE), FUN = WOE_tables_function)

Usually at this point we would close the cluster so that the computer may resume using memory for other computer functions. Since we will still require its work, we do not close it and resume coding our way to obtaining the final aggregated WOE matrix.

recode is a helper function that takes in the feature column name and searches for it in the WOE_tables vector. It then replaces all raw data inputs in the feature vector with their respective WOE values.

recode <- function(x, y) {
r_WOE_table_text <- sprintf("WOE_tables$%s", y)
create_r_WOE_table <- eval(parse(text = r_WOE_table_text))
data_type_indicator <- create_r_WOE_table[1,1]

if (is.factor(data_type_indicator)) {
category_Table <- as.numeric(create_r_WOE_table[,1])
corresponding_WOE_Table <- as.character(create_r_WOE_table[,2])
category_Table_length <- length(category_Table)
raw_variable <- as.numeric(factor(x))

for (i in 1:category_Table_length) {
condition_1 <- raw_variable == category_Table[i]
raw_variable[condition_1] <- corresponding_WOE_Table[i]


} else if (data_type_indicator == -1) {
min_r_Table <- create_r_WOE_table[,1]
max_r_Table <- create_r_WOE_table[,2]
corresponding_WOE_Table <- as.character(create_r_WOE_table[,3])
min_r_Table_length <- length(min_r_Table)
raw_variable <- x

for (i in 2:min_r_Table_length) {
condition_1 = min_r_Table[i]
condition_2 <- raw_variable <= max_r_Table[i]
raw_variable[condition_1 & condition_2] <- as.numeric(corresponding_WOE_Table[i])

condition_3 <-
raw_variable[condition_3] <- corresponding_WOE_Table[1]


} else {
min_r_Table <- create_r_WOE_table[,1]
max_r_Table <- create_r_WOE_table[,2]
corresponding_WOE_Table <- create_r_WOE_table[,3]
min_r_Table_length <- length(min_r_Table)
raw_variable <- x

for (i in 1:min_r_Table_length) {
condition_1 = min_r_Table[i]
condition_2 <- raw_variable <= max_r_Table[i]
raw_variable[condition_1 & condition_2] <- corresponding_WOE_Table[i]



WOE_matrix_final applies the recode function over the entire vector of feature names. Another helper function create_WOE_matrix allows the matrix to be created.

create_WOE_matrix <- function(x) {
variable_text <- sprintf("only_features_36$%s", x)
create_variable <- eval(parse(text = variable_text))
variable <- create_variable
variable_name <- x
WOE_vector <- recode(variable, variable_name)

Finally, create WOE_matrix through parallel processing. Again, we export the set of subfunctions that will be called by the main function create_WOE_matrix to the cluster. After creating WOE_matrix, it is important to include the Binary vector of whether a loan has resulted to be “Good” or “Bad”. Finally, we obtain the goal of this section of the project, WOE_matrix_final. Notice that the last line of code stops the cluster.

clusterExport(cluster, c("only_features_36", "create_WOE_matrix", "recode", "WOE_tables"))

WOE_matrix <- parSapply(cluster, features_36_names_WOE, FUN = create_WOE_matrix)
WOE_matrix <-
Bad_Binary <- features_36$Bad
Bad_Condition_1 <- Bad_Binary == 1
Bad_Condition_0 <- Bad_Binary == 0
Bad_Binary[Bad_Condition_1] <- "Good"
Bad_Binary[Bad_Condition_0] <- "Bad"
Bad_Binary <- as.factor(Bad_Binary)
WOE_matrix["Bad_Binary"] <- Bad_Binary
WOE_matrix_final <- WOE_matrix


In the next section, Scorecard Building – Part IV – Training, Testing and Validating the Logistic Regression Model I will take the transformed data set and apply various machine learning techniques to get a preliminary scorecard.

Scorecard Building in R – Part II – Data Preparation and Analysis

I used the dataframe manipulation package ‘dplyr’, some basic parallel processing to get the code running faster with the package ‘parallel’, and the ‘Information’ package which allows me to analyze the features within the data set using weight-of-evidence and information value.


First, I read in the Lending Club csv file downloaded from Lending Club website. The file is saved on my local desktop which is easily accessed by the read.csv function.

data <- read.csv("C:/Users/artemior/Desktop/Lending Club model/LoanStats3d.csv")

Next, I create a column that indicates whether I will keep an observation (row) or not. This will be based on the loan statuses because for a predictive logistic regression model, I would like all the statuses that will be strictly defined as a ‘Good’ loan or a ‘Bad’ loan.

data <- mutate(data,
Keep = ifelse(loan_status == "Charged Off" |
loan_status == "Default" |
loan_status == "Fully Paid" |
loan_status == "Late (16-30 days)" |
loan_status == "Late (31-120 days)",
"Keep", "Remove"))

After creating the ‘Keep’ column I filter the data depending on whether the observation had “Keep” or “Remove”.

sample <- filter(data, Keep == "Keep")

I further filter the data set to create two new samples. The Lending Club offers two exclusive types of loan products. To improve predictability of the riskiness of its loans, we can create two sub-risk models, one for all 36-month term loans and 60-month term loans.

sample_36 <- filter(sample, term == " 36 months")
sample_60 <- filter(sample, term == " 60 months")

For the purposes of this scorecard building demonstration I will create a model using the 36-month term loans. Using the mutate function, I create a new column called ‘Bad’ which will be my binary independent variable used
in the logistic regression.

sample_36 <- mutate(sample_36, Bad = ifelse(loan_status == "Fully Paid", 1, 0))

The next step is to clean up the table to remove any data points I do not want to include in the prediction model. Variables such as employment title would take more time to analyze so for the purposes of this analysis I remove them.

features_36 % select(-id, -member_id, -loan_amnt,
-funded_amnt, -funded_amnt_inv, -term, -int_rate, -installment,
-grade, -sub_grade, -pymnt_plan, -purpose, -loan_status,
-emp_title, -out_prncp, -out_prncp_inv, -total_pymnt, -total_pymnt_inv,
-total_rec_int, -total_rec_late_fee, -recoveries, -last_pymnt_d, -last_pymnt_amnt,
-next_pymnt_d, -policy_code, -total_rec_prncp, -Keep)

To further understand the data, I want to take a look at the number of observations per category under each variable. This will weed out any data points that could be problematic in future algorithms.

Once the features table is complete, I use the methodology of information value to transform the raw feature data. In theory, transforming the raw data into a proportional log-odds value as seend in the Weight-of-Evidence maps better onto a logistic regression fitted curve.

IV <- create_infotables(data = features_36, y = "Bad")

We can generate a summary of the IV’s for each feature. The IV for a particular feature represents the sum of individual bin IV’s.


We can even check the IV tables for individual features and see how each feature was binned, the percentage of observations that the bin represents out of the total number of observations, the WOE attributed to the bin and as well as the IV. The following code is an example of presenting the feature summary for the last credit pull date.


I analyze the behaviors of continuous and ordered-discrete variables by plotting their weight-of-evidences. In theory, the best possible transformation occurs when weight-of-evidences exhibit a monotonic relationship. First, I define features_36_names as the vector of column names. This will serve as the vector which I will use a function that plots every WOE graph for each feature in the features_36_names matrix. I remove features from the list that are categorical and would generate way too many bins to plot later on. For example, I removed the feature zip_code as there would be over 500 different kinds.

features_36_names_plot <- colnames(features_36)[c(-7, -11, -ncol(features_36))]

Here is the code for the ploeWOE function as I previously mentioned. This function generates a WOE plot for input x, where x is a string that represents the column name of a specific feature. Recall that I generated a list of strings in features_36_names.

plotWOE <- function(x) {
p <- plot_infotables(IV, variable = x, show_values = TRUE)
return(p) }

To make my for loop code clean and faster, I define a number as the length of the features name vector.

feature_name_vector_length_plot <- length(features_36_names_plot)

Now for the fun part, to generate a graph for each feature, I use a for loop which will go over every string object in the features_names_36 list, and plot a WOE graph for each string name that corresponds to a feature in the features_36 matrix. To be safe, I created an error-handling portion of code because somewhere in this huge matrix of features, I may have missed a feature or two in which a WOE plot cannot be created. This would occur if a particular feature only contained 1 category or value for every observed loan.

for (i in 1:feature_name_vector_length_plot) {
p <- tryCatch(plotWOE(features_36_names_plot[i]),
error = function(e)
{print(paste("Removed variable: ",
features_36_names_plot[i])); NaN})
print(p) }

About 90 graphs are generated using for loop. Below I present and discuss two examples of what kinds of graphs are presented and what they mean.


The home ownership weight-of-evidence plot displays how a greater proportion of good consumer loan customers own their homes and a greater proportion of bad consumer loans pay rent where they live. Those who still pay mortgage are slightly better customers.


The months since delinquency (or time since you failed to pay off some form of credit) weight-of-evidence plot presents another intuitive relationship. The more months that pass since a customer’s most recent delinquency will make them more likely to be a good customer in paying off their loan. The lower the amount of months since a customer’s most recent delinquency means that they have just recently failed to pay off other forms of credit. This goes to show that even if you had a delinquency in your lifetime, you can improve your credit management and behaviors over time.

In the plot, something weird happens when customers had their delinquency between 19 – 31 months before they received another consumer loan. This could suggest a lagging effect where it takes some time to fully chase down a customer. It could be the case that sometimes months and months of notification is given before the customer is actually classified as delinquent.

In the next post, Scorecard Building – Part III – Data Transformation, I am going to describe how the data we prepared and analyzed using Information Theory will be transformed to better suit a logistic regression model.

Scorecard Building in R – Part I – Introduction

Part of my job as a Data Scientist is to create, update and maintain a small-to-medium business scorecard. This machine learning generated application allows its users to identify applicants that are more likely to pay back their loan or not. Here, I take the opportunity to showcase the steps I take in building a reliable scorecard, and the analysis associated with evaluating it by using R. I will accomplish this with the use of public data provided by the consumer and commercial lending company, Lending Club (downloaded here).

Here is an overview of the essential steps to take when building this scorecard:

  1. Data Collection, Cleaning and Manipulation
  2. Data Transformations: Weight-of-Evidence and Information Value
  3. Training, Validating and Testing a Model: Logistic Regression
  4. Scorecard Evaluation and Analysis
  5. Finalizing Scorecard with other Techniques

See the next post, Scorecard Building – Part II – Data Preparation and Analysis to see how the data is prepared for further scorecard building.

Time-Series Model Building for TSX Stock Prices Using R

Time-series modelling and forecasting was definitely a core concept to learn and one of the more important technical skills to pick up in a masters program in economics. Having primarily focused on economic applications of time-series such as the estimation and prediction of future Canadian Real GDP and Real Interest Rates, I was able to get a sense of the power of time-series modelling.

One of the things I wished these applied econometric courses taught me was how to at least follow some sort of standard procedure in building a good time-series model! It can be misleading to have a time-series model project that you worked really hard on, knowing that you followed every step in the textbook, but realize in practice that it is a horrible model!

Luckily, I was able to pick up on one of the fundamental statistical model building procedures: splitting your data into a training and testing set with time-series data.

In this mini-project, I use a data set of 141 observations from Yahoo Finance Canada where each observation represents the S&P/TSX stock price for a particular month. Here, I wanted to demonstrate the importance of building a forecasting model on a training set, using this model to forecast future values, and comparing the forecasted values with actual observed values to see how well the model performed. Take note on the Charizard and Venusaur colour palettes of graphs during the read!

The following describes the procedures taken to create this model:

  1. Perform decomposition of time-series using LOESS on full data set of stock prices.
  2. Obtain a vector of seasonally adjusted stock prices (remainder) after the seasonal and trend components are removed.
  3. Separate this remainder into a training and test set where the training set consists of all seasonally adjusted stock prices between January 2006 to December 2015.
  4. Build an ARIMA model on the training set and obtain predicted values into the present month (October 2016).
  5. Compare predicted values to actual values observed and assess model performance.

Using the above procedures, I was able to obtain the following graph using an ARIMA(5,0,0) model or more simply an MA(5) model.


When we see the estimated model (Fitted line) compared to the actual behaviour of the stocks, the two are seemingly close. It is evident though that the model predictions into 2016 come close to what was actually observed but still over-predict the seasonally adjusted stock prices. About 6 months into the forecast, the predictions start to decrease and under-predict actual stock performance. Now, forecasts such as these are not meant to go out for too long as they become unreliable. For demonstration purposes, it is nice to see that the model can reliably predict behaviour for a few months into the future.

Since I do not want to put complete faith into this one model, I also ran a Simple Exponential Smoothing time-series model using HoltWinters.

The following describes the procedures taken to create this model:

  1. Separate the raw time-series data into a training and test set where the training set consists of all seasonally adjusted stock prices between January 2006 to December 2015.
  2. Build an Simple Exponential Smoothing (SES) model using Holt-Winters on the training set and obtain predicted values into the present month (October 2016).
  3. Compare predicted values to actual values observed and assess model performance.

Using the above procedures, I was able to obtain the following graph:stockpriceforecastHW

I am happy with the results of this graph. First, one needs to note that this model takes in raw stock prices and as such should be interpreted as their raw prices. The SES model fits closely with the actual values and the forecasting behaviour is similar to that obtained from the ARIMA(5,0,0) model. The confidence intervals shaded in blue show some boundaries to these values which is also very nice.

Further Work

In this post, I addressed the idea of being able to split a time-series data set into a training and testing set and model accordingly. In the first model, I used LOESS, a decomposition method to make the stock price data stationary by removing the seasonal and trending components. This allowed me to reliably apply an ARIMA model and make subsequent predictions. In the second model, I directly applied the Holt-Winters method and obtained similar results.

Although one could see that the time-series models were trained well and to a certain degree able to predict well, there is much more left to be said here. It is much better practice to compare the performance between the two models through calculations of their prediction errors. To take it even further, we could apply more advanced time-series modelling via neural networks.

Source Code

This project has been done in R. The source code can be found at my github here.

A Friendly BC Hydro Electricity Consumption Analysis using Tableau

If there is something to appreciate about the Canadian West Coast, it is definitely in the way that it leads by example through environmentally friendly practices. One of the ways that British Columbia takes on this initiative is through administering electrical energy in the cleanest and most cost-efficient way.  This is all made possible through BC Hydro, a Canadian-controlled crown corporation responsible for providing British Columbia residences reliable and affordable electricity. British Columbia prides itself in delivering electrical energy and is known to have one of the lowest electricity consumer prices in Canada. One way to show appreciation for natural resource consumption is to, of course, take a look at our very own personal electricity consumption. Before diving into any analysis, first it might be useful to provide some context into how BC Hydro prices electricity consumption.

Electricity Pricing

BC Hydro uses a two-stage pricing algorithm where consumers are required to pay $0.0858 per kWh up to a max consumption threshold of 1350 kWh within a two-month period. This rate increases to the second stage at $0.1287 per kWh if the consumer uses over 1350 kWh within the two months. In addition to the two-stage pricing algorithm, consumers are required to pay a base rate of $0.1899 times the number of days in their billing period, and pay a rider-rate which is a buffer cost to consumers to cover unpredictable economic circumstances such as abnormal market prices or inaccurate water level forecasts. The entire cost becomes subject to GST and the total is the payable amount during every billing period.  If you want to read for knowledge’s sake, BC Hydro thoroughly explains the pricing of electricity consumption on their website.


The motivation behind this post is to analyze a very special electricity consumption data set (special because I was given permission by my great friend Skye to analyze his electricity consumption data!) I will be analyzing Skye’s personal electricity consumption in the full calendar years of 2015 and 2016.

Data, Set, Go!

It is amazing how accessible BC Hydro makes personal electricity consumption data to its paying customers. Skye has revealed that to obtain your personalized data set, you would simply login to your MyHydro account, and request an exported .csv file. Within 24 hours of submitting a request, you will receive an e-mail with a personalized .csv file.

The file contains two data columns, the first column containing the Start Interval Time/Date representing the time stamp at the beginning of each hour of every day of the year, and the second column containing Net Consumption (kWh) representing the amount of electricity used up until the end of the hour measured in kilowatts per hour!

One thing I noticed within Skye’s data set is that there were some missing Net Consumption values.  Missing values were indicated with N/A attached to some time stamps. Further looking into this data and without any prior knowledge to Skye’s electricity consumption behavior, there is no way to really know for certain why the data was missing. To rectify this situation, I simply replaced missing values with the most recent level of net consumption. For example, if there was a missing value on September 13, 2016 at 10:00am, I would assume a forward-looking value such that this missing value would be replaced with the net consumption value from 9:00am. If there was trailing missing values, or consecutive missing values, I would replace them all with the most recent available net consumption value.

Without further ado, I shall begin reporting what I found from Skye’s electricity consumption data using Tableau Public. Pokemon Fans, please take notice in the Venusaur colour palette!

There was an increase of about 3% in net consumption expenditure from 2015 to 2016. Skye exhibits typical seasonal trends in electricity consumption.

Electricity Consumption Seasonal

Net Expenditure is what Skye is charged during the two-stage pricing algorithm on a per month basis. He does not actually ever step into stage-two of pricing as he is well below the 1350 kWh threshold every two months (which is amazing, save energy and save money!)

Visually, it seems that Skye exhibits a typical consumption behavior, where his expenditure in the first three quarters of each year has a decreasing trend, he hits a minimum in September, then scales back up in the Fall and Winter months. Is it possible that what we see visually is not verified statistically? We can validate this theory that Skye is behaving typically or the way he should be. Consider the following R code:

table2015 <- c(42.87, 39.01, 33.9, 29.25, 22.5, 26.68, 33.06, 24.45, 15.57,
 27.11, 49.37, 49.39)
table2016 <- c(44.46, 31.57, 32.68, 29, 25.53, 25.43, 24.53, 29.42, 20.43,
 32.03, 43.96, 67.62)
chisq.test(table2015, 2016)

Behavior Distribution

Here, I perform a simple chi-square test to validate that these data-points are in fact Skye’s typical behavior. The null hypothesis is that the expenditures in 2015 and 2016 are independent, or in simpler terms, we hypothesize the possibility that Skye’s behavior has changed and therefore has significantly changed his electrical consumption. Since the result presents a p-value of 0.2329 (much greater than 0.05,  a benchmark value to consider this null hypothesis), we reject the null hypothesis and conclude that there is evidence to suggest that Skye is behaving how he should be!

Although we have statistical evidence regarding his typical behavior, one still needs to question what happened in December 2016, where Skye’s expenditure increased by almost 37% from 2015! Could this probably be a result of one of the coldest winters that Lower Mainland has experienced in years?

Skye has more expenditure control in 2016. His net daily expenditures are more sporadic in 2015 with monthly averages between $0.80 and $1.40 per day, whereas 2016 was less sporadic with monthly averages between $0.70 and $1.20 (exception, December 2016 at average of $2.20).


Daily Expenditure 2015


Daily Expenditure 2016

Here, I use the term sporadic to describe the range in distributions of net daily expenditures per month.  For example, the box-plot ranges in the first quarter of 2015 are much wider than the box-plots in the first quarter of 2016. This is especially evident in the summer months. To put it simply, Skye has more consistency and better control of his electricity consumption and expenditures in 2016.

We have seen that Skye’s expenditure has increased by 3.42% from 2015 to 2016. One would think that with more controlled electricity consumption in 2016, his expenditure would be lower. By taking a look at December 2016, we can see that his expenditure is abnormally higher (by almost an additional $0.70 per day!) It is interesting to see that electricity consumption behavior was definitely more different in 2016 and kept at all-time low until the winter season.

Another thing to point out is that outlying value of about $4.00 in December 2016. Anecdotally, Skye says it is most likely because he forgot to turn off the stove that day!

Skye’s favourite days of electricity consumption are Tuesdays, Wednesdays, Saturdays and Sundays. Net hourly expenditure is seasonally and consistently higher (more expenditures are greater than $0.04 per hour) during these days.

Net Hourly Expenditure 2015 ($)

Monthly Daily Expenditure 2015

Net Hourly Expenditure 2016 ($)

Monthly Daily Expenditure 2016

Consistent with what we have been seeing, the winter months continue to observe the highest expenditure per hour. In addition, it seems that Skye has more of a liking to use higher levels of electricity on Tuesdays, Wednesdays and on the weekends with expenditures ranging between $0.03 to $0.09 per hour.

To account for a smaller difference, we could see that in 2015, Skye used more electricity throughout November everyday in 2015, but has decreased his hourly expenditure over November weekends in 2016.

Skye exhibits behavioral change in 2016 mornings and mid-evenings with higher net hourly expenditures compared to 2015.

Net Hourly Expenditure 2015 ($)

Daily Hour Charge 2015

Net Hourly Expenditure 2016 ($)

Daily Hour Charge 2016

It seems that Skye is consuming more electricity in the morning in 2016, evidently with an additional $0.01 to $0.02 per hour from 2015 rates. A behavioral change is signaled by the 7AM to 8AM mark in the morning and by the 6PM to 7PM mark in the evening. Even times like Mondays in 2016 at 11PM in the late night exhibit increase in electricity consumption.

Skye spends about 57% less than the average British Columbian household.

Compare Rates

According to BC Hydro, the average BC household consumes an average of about 900 kWh of electricity per month (not including seasonality). If we apply the exact same logic by taking the average consumption of Skye’s past 2 years, we see that he actually consumes way less than the average household.

Watt to Consider for the Future

This analysis was a great way to continue displaying fun data in Tableau. With just a two-column personalized electricity consumption data set, I was able to dig a little deeper on the spending behavior of Skye. Some things came to mind as I was conducting this analysis which could be used as motivation for further analysis posts.

This analysis serves as a perfect transition into utilizing machine learning methods to forecast future expenditures. We can definitely come to understand what exactly our forecasting machine learning models intend to capture and how these behaviors will help us predict future behavior. In this analysis, 2015 and 2016 data was used but in reality, data up to the current date and data before 2015 can be obtained. This gives more opportunity to build and test forecasting models accordingly.

In addition to the amazing visualizations produced by Tableau, a perfect consideration for future time-series modelling is to plot the data using ggplot2 in R. Interchangeably using these two visualization tools can serve as good practice and could provide more insights when used in conjunction with one another.

A special thanks to Skye for letting me use his data, it was fun!


My Personal Vancouver Transit Usage: Analysis using Tableau

One of the things I love about Vancouver is its public transportation system, Translink. I grew up loving trains, and so it only seemed natural that riding the Skytrain be one of the funnest things I have experienced when I first came to Vancouver. It has been around two years since I have moved here and I still use it to go everywhere in and out of the Vancouver area. A cool feature of the Translink system is the Compass Card, a re-loadable fare pass in which frequent riders will use to tap themselves on and off the transit system through fare gates. Part of the reason why I love the idea of tapping on and off the fare gates or on bus rides is because of how the system records data of where and when you have tapped.

The thought of Translink’s ability to easily conduct commuter analysis using the millions of data recorded everyday for strategic pricing and vehicle allocation is intriguing. As such, this is what motivates riders like me to analyze my personal rider behavior. Conveniently, the Compass Card website allows you to download your own personal .csv file. The file contains lines of transactions representing every single time you have tapped on and off the system.

The motivation behind this post is to showcase some data analysis. I would love to present what I have learned about my transit behavior between September 2016 and August 2017 using Tableau Public. For any Pokemon fans out there, visualizations take on a Charizard colour palette.

On average, I began my travels with the bus 2.7 times more than the train each month. Equivalently, the bus began 73% of my trips.

Rider Usage Growth

73 Percent Bus Usage

This makes sense as the bus begins my commute to almost everywhere I go when I begin at home. It is interesting to see that my ridership has consistently increased up until the second quarter in 2017. The slight kink in the graph is due to the fact that I spent most of the month of May 2017 travelling (I went to Japan for the first time!)

I used the transit system in 289 out of 365 days and most days, I took 2-3 trips.

Trip Data

Here, I defined a trip as one where I would be required to make a new full fare payment. It is possible that multiple forms of transit may be used within an hour and a half time interval before having to pay again. These potential ways of transferring between types of transit (ie. bus to a train) are not considered as trips.

I tried avoiding the morning transit rush. I am more likely to use transit during evening rush hour. Weekend usage often starts in the late morning.

Daily Trip Schedule

This huge morning spread in my transit usage behavior reflects my choice to go to the gym before I go to work, especially during the spring and summer seasons when it gets brighter outside earlier in the day. Therefore, I can begin using transit as early as 5:00am! Also, being given a flexible work schedule, sometimes I choose to head to work as late as anywhere between 8:00am and 9:00am.

I often go out Friday and Saturday evenings and as much as I love taking transit, there is no clear increase in transit usage behavior during this time because depending on the activity, I may already be in walking distance of what I want to do, or transit may not be my ideal form of transportation.

I saved money getting a Zone 1 Monthly Pass at $91.00 with my behavior! I would have spent on average, $103.00 a month on individual fares.

Fare Usage

Often times, I don’t think about how many times I tap on and off the system and overlook what I would be paying if I did not have a monthly pass. This is full proof that getting a zone 1 monthly pass is worth it as a frequent transit user and I do not have to worry about other financial alternatives.

If I was more nit-picky, I could definitely save more money by not getting the monthly pass during the months where it would not be worth it. For example, every December,  I fly out to Toronto for two weeks to visit family for the holidays. One might think that this could signal a behavioral change to pay closer attention to my budget allocation towards public transit. In reality, I actually prefer not having to worry about loading my compass card every month. Hence, I have it set to auto-load where the system automatically charges my credit card and loads a monthly pass to my compass card.

Further Considerations

This analysis was a great introductory way for me to explore Tableau as an analytical tool. I will definitely be using it more often to create vibrant visualizations and hone in on insights from interesting data. Some future considerations I have for these kinds of analysis is to utilize maps and locations to enhance the visualization and stories behind transit data. In this particular case, almost all of my trips began in Vancouver and rarely in any other surrounding city so geographic visuals may have not been much use.

Another future consideration is to augment the existing transit data with other data sources such as the distances traveled using transit possibly obtained from Google Maps for example. Some analysis on how much it would cost per kilometer traveled or personal summary statistics on distances traveled also sounds interesting.

Amidst the world of available data in everyday life, one last future consideration is that the next time you tap off the transit system, think about how that is one more data point for your next analysis!

Implementing a Predictive Model Pipeline using R and Microsoft Azure Machine Learning

In this post, I aim to demonstrate the process of building a simple machine learning model in R and implementing it as predictive web-service in Microsoft Azure Machine Learning (Azure ML). One is not limited to the built-in machine learning capabilities of Azure ML since the Azure ML environment enables the use of R scripts, and the ability to upload and utilize R packages.

In practice, this gives the Data Scientist the flexibility they need to use their own carefully R-crafted machine learning models within the Azure ML environment. Once an R-built machine learning model is fully implemented within the Azure ML environment, the web-service provides an API in which calls can be made by external applications. This provides large value to any organization that seeks to automate decision processes using predictive modelling.

This demonstration is subdivided into three sections:

  1. Building a Machine Learning Model in R
  2. Preparing for Model Implementation in Microsoft Azure Machine Learning
  3. Creating a Predictive Web Service in Microsoft Azure Machine Learning

Before we begin, this demonstration assumes all of the following are satisfied:

  1. We have a verified and registered Microsoft Azure Machine Learning account. Microsoft allows you to try the product for free if you do not have a subscription. Click the “Sign-up here” on the far right using the link above and follow the steps to gain access to Azure ML.
  2. We have R and an associated IDE installed. I will be using the free version of R-Studio throughout this process.
  3. We have the following R packages installed: dplyr, dummies, caretcaretEnsemble
  4. We have the Human Resources Analytics data set downloaded as this will be the main source of data for building our predictive model.

Building a Machine Learning Model in R

To keep things simple, I will be building a simple logistic regression model.

Data Pre-processing

First, I pre-process the Human Resources Analytics data set to hot-encode the categorical features sales and salary.


dataset <- read.csv("HRdata.csv")
dataset <-, names = c("sales", "salary"), sep = "_")
dataset <- dataset[-c(15,19)] 

# Columns 15 and 19 represent sales_RandD and salary_high which are removed to prevent the dummy variable trap

Training the Logistic Regression Model

Next, I train a Logistic Regression Model and check that it can successfully generate predictions for new data.


# Create logistic regression model
glm_model - glm(left ~ ., data = dataset)

# Generate predictions for new data
newdata <- data.frame(satisfaction_level = 0.5, last_evaluation = 0.5, number_project = 1, average_montly_hours = 160, time_spend_company = 2, Work_accident = 0, promotion_last_5years = 1, sales_accounting = 0, sales_hr = 0, sales_IT = 0, sales_management = 0, sales_marketing = 0, sales_product_mng = 0, sales_sales = 1, sales_support = 0, sales_technical = 0, salary_low = 0, salary_medium = 1)
prediction <- predict(object = stack.rf, newdata = newdata) 


Executing the above gives us a probability of 0.193 indicating that this employee has low risk of leaving.

Saving the R-Built Model

Since the goal is to use our very own R-built model in Microsoft Azure Machine Learning, we need to be able to utilize our model without having to generate the above code over again. We run the following code to save our model and all of its parameters:

saveRDS(glm_model, file = "glm_model.rds")

Running this code will save the glm_model.rds file in the active working directory within R-Studio. In this demonstration, the glm_model.rds file is saved to my desktop.


Creating Package Project Environment

The next couple of steps are crucial in ensuring that we end up with a package file that can be uploaded to Azure ML. The reason for creating this package is to ensure that Azure ML can call on our logistic regression model to generate predictions.

First, we must initialize the package creation process by starting a new project. In R-Studio this achieved by the following:

  • Click “File” in the top left corner
  • Click “New Project…” and a pop-up screen will appear


  • Click “New Directory”


  • Click “R Package”


  • Type in a package name. Here I used “azuremlglm” as my package name. Make sure to create the project folder by setting a project subdirectory. Here, I used my desktop as the location for this project folder.
  • After clicking “Create Project”, a new R-Studio working environment will open with the default R file being “hello.R”. Since I saved my project to my desktop, I also noticed that a new folder was created.


  • Now we are set to build our package. Within our package environment in R-Studio, we can close the “hello.R” file and create three new R scripts by hitting ctrl + shift + N twice. These three scripts will be needed in the following sections.

Filling the Package with Necessary Items

By successfully setting up the package creation environment, we are now free to fill this package with anything that we may find useful in our predictive modelling pipeline. For the purposes of this demonstration, this package will only include the logistic regression model built from the first section, and a function that Azure ML can use to generate predictions.

Before writing anything in the new R script, we write the following R code in the first script to add the glm_model.rds file to our package. To better accomplish this, we can drag the .rds file to the azuremlglm folder since the R-Studio working directory is that project folder.

# Read the .rds file into the package environment
glm_model_rds <- readRDS("stack_randomforest_model.rds")


By reading in the .rds file that contained our logistic regression model into the package environment, we are now free to utilize the model in any way we wish. It is important that we save this script within the project folder. Here, I saved it as glm_model_rds.R as seen on the tab.


The Prediction Function

Since the primary use of this package is to utilize the logistic regression model to produce predictions, we need to create a function that takes in a data frame containing new data and outputs a prediction. This is very similar to the prediction verification procedure we did in the first section after building the model and using the predict function on new data.

In the new R script that we created, we write the following R code:

# Create function that allows Azure ML to generate predictions using logistic regression model

prediction_function <- function(newdata) {
 prediction <- predict(glm_model_rds, newdata = newdata)

Here, I saved this function as prediction_function.rds


The Decision Function

When Azure ML receives new data and passes its arguments to this function, we expect the resulting predictive web-service to produce the predicted probability. What if our decision process required more than just the predicted probability?

The added benefit of being able to create your own models and packages to use in Azure Machine Learning are tenfold. In many cases, you may want the Azure ML API call to output decision processes as a result of the predictions created by your machine learning model. Consider the following example:

# Create function with decision policy

decision_policy <- function(probability) {
 if (probability < 0.2) {return("Employee is low risk, occassional check-up where necessary.")}
 else if (probability >= 0.2 & probability < 0.6) {return("Employee is medium risk, take action in employee retention where necessary.")}
 else (return("Employee is high risk, notify upper management to ensure risk is mitigated in work environment."))

This decision function takes the logistic regression model’s predicted probability of a new observation and applies a Human Resource policy that meets the organization’s needs. As you can see, instead of a predicted probability, this function is recommending some form of action to be taken one the predictive model is used. It is possible that the result of a predictive model can trigger many different company-wide policies, no matter what the industry-specific application.

Here’s another example in the alternative business-financing industry. A predicted probability of risk to a specific business owner can trigger different loan-product pricing policies, and trigger different employee actions to be taken. If the Azure ML API call can output a series of policies and rules, there is huge value in being able to automate decision processes in order to get that loan out faster or rejected faster.

Creating your own models in R and including decision policies within your R packages could be the solution to an automated decision process within any organization.

Now, back to package creation. Given the newly created decision_function, we need to be sure to update our prediction_function to be able to implement these new policies.

# Create function that allows Azure ML to generate predictions using stacked model

prediction_function <- function(newdata) {
 prediction <- predict(glm_model_rds, newdata = newdata)

With the prediction function and decision function ready to go, it is important that we run these functions so that it is saved within the package environment.

It is also important that we save this R script within the package folder. Here, I saved the decision_policy.R function and re-saved the prediction_function.R as shown in the tabs.



Once these three separate R scripts are saved, we are ready to build and save our package. To build and save the package, we do the following:

  • Click the “Build” tab in the top right corner


  • Click “Build & Reload”


  • Verify that the package was built and saved by going to the R library folder, “R/win-library/3.3”. Here, my library is saved in my Documents folder.


  • With the package folder from above, you want to create a .zip file of it. You can do this by right-clicking the file, going to “send to”, then selecting “Compressed (zipped) folder”. After doing so, it will create a .zip file of your package. Do not rename this .zip file. I also proceeded to drag this .zip file to my desktop.


  • This next step is extremely important. With the newly saved .zip file, you want to create ANOTHER .zip file of it. The reason for this is because of the weird way that Azure ML reads in package files. This time I renamed the new .zip file as “2_azuremlglm”.  You should now have a .zip file that contains a .zip file that contains the actual azuremlglm folder package. You can delete the first .zip file created from the previous step as it is no longer needed.


  • This is the resulting package file that will be uploaded to Azure ML.

Creating a Predictive Web Service in Microsoft Azure Machine Learning

We are in the final stretch of the implementation process! This last section will describe how to configure Microsoft Azure Machine Learning to utilize our logistic regression model and decision rules.

Uploading the Package File and Creating a New Experiment

Once we have logged in, we want to do the following steps:

  • Click the “NEW” button in the bottom left corner, click “DATASET”, and then click “FROM LOCAL FILE” as shown


  • Upload the .zip file created from the previous section


  • When the upload is successful, you should receive the following message at the bottom of the screen


  • Next, we create a new blank experiment. We do this by clicking the “NEW” button at the bottom left corner again, click “EXPERIMENT”, and then click the first option “Blank Experiment”


  • Now we are ready to configure our Azure Machine Learning experiment


Setting up the Experiment Platform

In order for Microsoft Azure Machine Learning to utilize our logistic regression model, we need to set up the platform in such a way that it knows to take in new data inputs and produce prediction outputs. We accomplish this with the following layout.


  • The Execute R Script on the left defines the schema of the inputs. This module will connect to the first input node of the second Execute R Script. The code inputted in this module is as follows


  • A module was placed for the package so that it can be installed within the Azure ML environment. This module is inputted into the third node of the second Execute R Script module.
  • The Execute R Script in the center is where we utilize the logistic regression model package. This module contains the following code


  • Once all of the above are satisfied, we are ready to deploy the predictive web service.

Deploying the Predictive Web service

  • At the bottom of the screen, we will deploy the web service by clicking on DEPLOY WEB SERVICE”, then clicking “Deploy Web service (classic)”.


  • Azure ML will then automatically add the Web service input and Web service output modules to the appropriate nodes as follows


  • The Web service output automatically connected to the second output node of the Execute R Script module. We actually want this to connect to the first output node of the Execute R Script as shown


  • Click the “RUN” button at the bottom of the screen to verify the web service
  • Click the “DEPLOY WEB SERVICE” button once again, and select “Deploy Web service (Classic). The following page will show up


  • Finally, we are able to test that our predictive model works by clicking the blue “Test” button in the “REQUEST/RESPONSE” row.


  • After confirming the test, we should get the following result


  • This confirms that our predictive model works and all decision policies have been correctly implemented. The API is ready to go and can be consumed by external applications.

Further Considerations

Throughout this post, I showcased the process of implementing a simple predictive model using R and Microsoft Azure Machine Learning model. Of course, there are much more efficient ways of utilizing predictive models such as directly using the platform of Azure ML  to train, validate and test machine learning models, or directly using the Execute R Script module and doing all the R hard-coding there.

I want to emphasize that the process outlined here may seem less efficient to build and carry out, but I think it offers a good way to organize and automate decision pipelines. By going through the process of building and creating R packages that can then be uploaded to Azure ML, we are able to implement many decision rules within the R package. For example, an organization may choose to implement several product pricing rules  or internal decision policies as a result of what the predictive model outputs. There is plenty of room to automate these decisions for faster turnaround of work. Creating packages also gives us the ability to train, validate, and test more complex machine learning models and saving their results accordingly. I am sure there are plenty of other reasons and uses than the ones I stated here in which building your own machine learning R packages and then uploading it to Azure ML is highly beneficial.

In the future, I look to implement this process by using more complex machine learning models rather than the simple logistic regression. I also look to learn some more software application development as this is clearly not the end of the data science pipeline. With Azure ML producing an API, it would be nice to be able to see the full extent of this pipeline by utilizing the API through my own created applications. Finally, some important takeaways from this post are the abilities to organize and automate an operational data science pipeline and the thought-process behind automating company-related decisions.

Overcoming the First Hurdle: From Knowing a Little to Learning a Lot

Growth as a data scientist will take on many forms and scale up several different paths depending on the function that you serve within your work environment. The learning curve as an early-stage Data Scientist will vary on several things such as your background education and knowledge, prior experiences within the field and industry, and whether you work within a team of data scientists, or as a standalone data scientist.

For myself, the learning curve was and continues to be steep and challenging. I began my career as a standalone data scientist for a start-up company, coming straight out of school and having very limited knowledge of the financial industry. All I had under my knowledge-base at the time was an in-depth understanding of the Logistic Regression, some economic analytical projects involving time-series, and a toolkit consisting of R and Microsoft Excel. Out of uplifting encouragement, I could of done more to add to my skill set before I started my job, but with what I knew, with an eagerness to learn, and with an immense curiosity, I had exactly what I needed to begin my career.

My role as a data scientist is to build and maintain proprietary credit scoring models, and provide adhoc analysis and reports upon request. There was already a whole list of challenges that I faced when I first started: a lack of appropriate credit scorecard building knowledge, a lack of knowledge on advanced data analytic techniques, verifying that my work met industry standards, and lacking the knowledge to closely monitor model effects.

These challenges pushed me to figure out the best practices and processes in the best way I thought possible. Here are some of the ways I went about addressing the challenges I faced during the start of my career.

Conducting Independent Research

My first gut instinct to approach a problem where you virtually have almost no background experience and no one to turn to for answers is to research! Having obtained a Master’s degree from a program that infused independent research heavily within its curriculum, this only came natural to me. For example, the most important thing in tackling a scorecard building project was first understanding its entirety and breaking it down into manageable and understandable pieces. It was extremely important to know why it is used, how it is used, and how it will benefit my company’s operations.

What often happened throughout my research was that I would find complex solutions that were difficult to implement without advanced enterprise software or advanced programming knowledge, or I would find solutions that seemed too easy and not convincing enough to use. This process of researching and attempting to reproduce certain projects on the internet definitely increased my technical understanding and in many ways helped me boost my proficiency in R. Along the way, I even picked up some Python and I also learned to how to write queries in Microsoft SQL Server and MySQL to better streamline my data and model building processes.


Another challenge was ensuring that the credit scoring models were built following best practices within the financial industry. This was a little more difficult for two reasons. The first one being that a scorecard for an alternative business-lending company would differ immensely from the more common scorecards developed in the industry such as that for personal loans. Secondly, the modelling practices for alternative subprime business-lending is still relatively new with the emergence of these industries stemming back since the 2008 Financial Crisis. Therefore, research is limited and most ideas behind these driving forces are mostly proprietary.

To overcome this challenge, I engaged in some more internet research, but more importantly, I networked with industry professionals and took what I could from my discussions with them. Most of our discussions involved understanding what techniques were used widely in the industry. During this time, LinkedIN, and my personal connections contributed to my learning of overcoming this challenge. I learned to set up interactions with professionals online as well learned to generate and connect ideas between professionals within my own work.

Engaging in Trial and Error

At first, there is high pressure when you first start as a data scientist with expectations of completing your projects within specified deadlines. The scorecard was my very first project and with the limited knowledge that I had, I was almost forced into a situation of trial and error. Initially, my practices involved researching and building in an endless cycle, where I often updated the scorecard to meet new standards and practices I learned along the way. At the time, there was very little internal user feedback on the scorecard because it was assumed that it was performing exactly the way it should be. It was essential that through this trial and error process that there was constant communication and understanding among the company in order to continue building a robust scorecard. Here, I learned a lot about not only the technical side of model building, but also found that my role as a standalone data scientist has a unique place within the operational team.

Being Prepared and Building Confidence

No data science problems at high levels of technicality and knowledge can be solved so easily. As a standalone data scientist where you are mostly doing things on your own accord and expected to make educated executive decisions, you are bound to run into personal hurdles such as worries and frustrations. When something goes wrong with your models, you become the first person accountable which in many ways can be offsetting. I came to realize that all of these feelings were natural and it was perfectly fine!

In order to overcome this challenge, it was always in my best interest to be prepared to provide thorough answers to questions that the company asked me, and be able to address concerns. Whenever there was a problem or concern raised with the models I built, or the data analysis methodologies, I was always forward with a positive answer or came up with a solution. It was in my best interest to be accountable and honest with my abilities. This stemmed from the realization that I do not know everything, but I do want to learn to make sure I do my best work in order to help the company grow. With the appropriate communication among upper management and their moral support, these personal challenges slowly faded and I actually began to expedite my learning of more applied business data science.

Moving Forward

What I appreciate the most about the early stages of my career is the amounts of learning that I have done and the huge amounts of growth I experienced as a person. With that said, the learning never ends as new modelling needs occur, data repositories grow with new data to be analyzed, and new modelling techniques and solutions are introduced with new technologies.

I know that as I continue along this career path, I am bound to learn some more programming, apply other predictive models, and conduct interesting kinds of analysis. With these ongoing changes within a fast-growing company, there is bound to be one problem solved with ten more problems arising. The best part of being in the early-stage of my career is that I know I still have a lot to learn, and as I move forward, I will anticipate the challenges ahead, and be more than happy to tackle them one step at a time.