# BrainlessLabs Diary

## Finding Donors

Published on March 02, 2019

# Introduction

## Project: Finding Donors for CharityML

Welcome to the second project of the Machine Learning Engineer Nanodegree! In this notebook, some template code has already been provided for you, and it will be your job to implement the additional functionality necessary to successfully complete this project. Sections that begin with 'Implementation' in the header indicate that the following block of code will require additional functionality which you must provide. Instructions will be provided for each section and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!

Note: Please specify WHICH VERSION OF PYTHON you are using when submitting this notebook. Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.

In this project, you will employ several supervised algorithms of your choice to accurately model individuals' income using data collected from the 1994 U.S. Census. You will then choose the best candidate algorithm from preliminary results and further optimize this algorithm to best model the data. Your goal with this implementation is to construct a model that accurately predicts whether an individual makes more than $50,000. This sort of task can arise in a non-profit setting, where organizations survive on donations. Understanding an individual's income can help a non-profit better understand how large of a donation to request, or whether or not they should reach out to begin with. While it can be difficult to determine an individual's general income bracket directly from public sources, we can (as we will see) infer this value from other publically available features. The dataset for this project originates from the UCI Machine Learning Repository. The datset was donated by Ron Kohavi and Barry Becker, after being published in the article "Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid". You can find the article by Ron Kohavi online. The data we investigate here consists of small changes to the original dataset, such as removing the 'fnlwgt' feature and records with missing or ill-formatted entries. ## Exploring the Data Run the code cell below to load necessary Python libraries and load the census data. Note that the last column from this dataset, 'income', will be our target label (whether an individual makes more than, or at most,$50,000 annually). All other columns are features about each individual in the census database.

# Import libraries necessary for this project
import numpy as np
import pandas as pd
from time import time
from IPython.display import display, HTML # Allows the use of display() for DataFrames
import seaborn as sns
# Import supplementary visualization code visuals.py
import visuals as vs
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)
np.warnings.filterwarnings('ignore')

# Pretty display for notebooks
%matplotlib inline
%lsmagic
# %pylab inline
sns.set(color_codes=True)
# Load the Census dataset
print('Data upload done')
Data upload done
# Section for helper functions
def display_html(display_string: str, color: str='blue', heading_value: str = 'h4'):
"""
Display some text in HTML form inside the display sections
"""
display(HTML(' <span style="color:{0}"><{2}>{1}</{2}> </span>  '.format(color, display_string, heading_value)))

def print_general_information(title:str, data):
"""
Display general information related to data
"""
display_html('Display basic information about the records "{}":'.format(title))
# Success - Display the first record
display(data.info())
print("Get an overview of the data before we proceed:")
display(data.describe())
display_html('Checking the shape of the data:')
display(data.shape)
# Print unique values in the income column. This is the column that will be used for training and prediction.
display_html('Check the unique values for income:')
display(data['income'].unique())

def display_pair_plot(title:str, data):
"""
Display pairplot
"""
display_html('Pairplot "{}":'.format(title))
sns.pairplot(data)
plt.show()
print_general_information('Data Variable', data)

#### Display basic information about the records "Data Variable":

age workclass education_level education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country income
0 39 State-gov Bachelors 13.0 Never-married Adm-clerical Not-in-family White Male 2174.0 0.0 40.0 United-States <=50K
1 50 Self-emp-not-inc Bachelors 13.0 Married-civ-spouse Exec-managerial Husband White Male 0.0 0.0 13.0 United-States <=50K
2 38 Private HS-grad 9.0 Divorced Handlers-cleaners Not-in-family White Male 0.0 0.0 40.0 United-States <=50K
3 53 Private 11th 7.0 Married-civ-spouse Handlers-cleaners Husband Black Male 0.0 0.0 40.0 United-States <=50K
4 28 Private Bachelors 13.0 Married-civ-spouse Prof-specialty Wife Black Female 0.0 0.0 40.0 Cuba <=50K
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 45222 entries, 0 to 45221
Data columns (total 14 columns):
age                45222 non-null int64
workclass          45222 non-null object
education_level    45222 non-null object
education-num      45222 non-null float64
marital-status     45222 non-null object
occupation         45222 non-null object
relationship       45222 non-null object
race               45222 non-null object
sex                45222 non-null object
capital-gain       45222 non-null float64
capital-loss       45222 non-null float64
hours-per-week     45222 non-null float64
native-country     45222 non-null object
income             45222 non-null object
dtypes: float64(4), int64(1), object(9)
memory usage: 4.8+ MB

None

Get an overview of the data before we proceed:
age education-num capital-gain capital-loss hours-per-week
count 45222.000000 45222.000000 45222.000000 45222.000000 45222.000000
mean 38.547941 10.118460 1101.430344 88.595418 40.938017
std 13.217870 2.552881 7506.430084 404.956092 12.007508
min 17.000000 1.000000 0.000000 0.000000 1.000000
25% 28.000000 9.000000 0.000000 0.000000 40.000000
50% 37.000000 10.000000 0.000000 0.000000 40.000000
75% 47.000000 13.000000 0.000000 0.000000 45.000000
max 90.000000 16.000000 99999.000000 4356.000000 99.000000

#### Checking the shape of the data:

(45222, 14)

#### Check the unique values for income:

array(['<=50K', '>50K'], dtype=object)

Checking the pairplot

display_pair_plot('Data Variable', data)

### Implementation: Data Exploration

A cursory investigation of the dataset will determine how many individuals fit into either group, and will tell us about the percentage of these individuals making more than $50,000. In the code cell below, you will need to compute the following: • The total number of records, 'n_records' • The number of individuals making more than$50,000 annually, 'n_greater_50k'.
• The number of individuals making at most $50,000 annually, 'n_at_most_50k'. • The percentage of individuals making more than$50,000 annually, 'greater_percent'.

HINT: You may need to look at the table above to understand how the 'income' entries are formatted.

# TODO: Total number of records
n_records = np.alen(data)

# TODO: Number of records where individual's income is more than $50,000 n_greater_50k = data[data.income == '>50K'].shape[0] # TODO: Number of records where individual's income is at most$50,000
n_at_most_50k = np.alen(data[data.income == '<=50K'])

# TODO: Percentage of individuals whose income is more than $50,000 greater_percent = np.multiply(np.divide(np.float64(n_greater_50k), n_records), 100) # Print the results print("Total number of records: {}".format(n_records)) print("Individuals making more than$50,000: {}".format(n_greater_50k))
print("Individuals making at most $50,000: {}".format(n_at_most_50k)) print("Percentage of individuals making more than$50,000: {}%".format(greater_percent))
Total number of records: 45222
Individuals making more than $50,000: 11208 Individuals making at most$50,000: 34014

#### Note: Recap of accuracy, precision, recall

Accuracy measures how often the classifier makes the correct prediction. It’s the ratio of the number of correct predictions to the total number of predictions (the number of test data points).

Precision tells us what proportion of messages we classified as spam, actually were spam. It is a ratio of true positives(words classified as spam, and which are actually spam) to all positives(all words classified as spam, irrespective of whether that was the correct classificatio), in other words it is the ratio of

[True Positives/(True Positives + False Positives)]

Recall(sensitivity) tells us what proportion of messages that actually were spam were classified by us as spam. It is a ratio of true positives(words classified as spam, and which are actually spam) to all the words that were actually spam, in other words it is the ratio of

[True Positives/(True Positives + False Negatives)]

For classification problems that are skewed in their classification distributions like in our case, for example if we had a 100 text messages and only 2 were spam and the rest 98 weren't, accuracy by itself is not a very good metric. We could classify 90 messages as not spam(including the 2 that were spam but we classify them as not spam, hence they would be false negatives) and 10 as spam(all 10 false positives) and still get a reasonably good accuracy score. For such cases, precision and recall come in very handy. These two metrics can be combined to get the F1 score, which is weighted average(harmonic mean) of the precision and recall scores. This score can range from 0 to 1, with 1 being the best possible F1 score(we take the harmonic mean as we are dealing with ratios).

### Question 1 - Naive Predictor Performace

• If we chose a model that always predicted an individual made more than $50,000, what would that model's accuracy and F-score be on this dataset? You must use the code cell below and assign your results to 'accuracy' and 'fscore' to be used later. Please note that the the purpose of generating a naive predictor is simply to show what a base model without any intelligence would look like. In the real world, ideally your base model would be either the results of a previous model or could be based on a research paper upon which you are looking to improve. When there is no benchmark model set, getting a result better than random choice is a place you could start from. HINT: • When we have a model that always predicts '1' (i.e. the individual makes more than 50k) then our model will have no True Negatives(TN) or False Negatives(FN) as we are not making any negative('0' value) predictions. Therefore our Accuracy in this case becomes the same as our Precision(True Positives/(True Positives + False Positives)) as every prediction that we have made with value '1' that should have '0' becomes a False Positive; therefore our denominator in this case is the total number of records we have in total. • Our Recall score(True Positives/(True Positives + False Negatives)) in this setting becomes 1 as we have no False Negatives. ''' TP = np.sum(income) # Counting the ones as this is the naive case. Note that 'income' is the 'income_raw' data encoded to numerical values done in the data preprocessing step. FP = income.count() - TP # Specific to the naive case TN = 0 # No predicted negatives in the naive case FN = 0 # No predicted negatives in the naive case ''' from sklearn.metrics import accuracy_score, precision_score, recall_score, fbeta_score # Calculating the performance without sklearn display_html("Calculating the performance without sklearn:") TP = np.float64(np.sum(income)) FP = np.float64(np.subtract(income.count(), TP)) TN = np.float64(0.0) FN = np.float64(0.0) # TODO: Calculate accuracy, precision and recall accuracy = np.divide(np.float64(TP), income.count()) precision = np.divide(TP, np.add(TP, FP)) recall = np.divide(TP, np.add(TP, FN)) print('accuracy = {}, recall = {}, precision = {}'.format(accuracy, recall, precision)) # TODO: Calculate F-score using the formula above for beta = 0.5 and correct values for precision and recall. def calculate_fscore(precision, recall, b): b2 = np.float64(np.square(b)) numerator = np.multiply(precision, recall) numerator = np.multiply(numerator, (b2 + 1)) denominator = np.add(np.multiply(b2, precision), recall) fscore_ret = np.divide(numerator, denominator) return fscore_ret beta = 0.5 fscore = calculate_fscore(precision=precision, recall=recall, b=beta) # Print the results print("Naive Predictor: [Accuracy score: {:.4f}, F-score: {:.4f}]".format(accuracy, fscore)) # Calculating the performance with sklearn display_html("Calculating the performance with sklearn:") all_one_pred = [np.float64(1) for i in range(income.size)] accuracy = accuracy_score(income, all_one_pred) precision = precision_score(y_true=income, y_pred=all_one_pred) recall = recall_score(y_true=income, y_pred=all_one_pred) fscore = calculate_fscore(precision=precision, recall=recall, b=beta) print('accuracy = {}, recall = {}, precision = {}'.format(accuracy, recall, precision)) print("Naive Predictor: [Accuracy score: {:.4f}, F-score: {:.4f}]".format(accuracy, fscore)) #### Calculating the performance without sklearn: accuracy = 0.2478439697492371, recall = 1.0, precision = 0.2478439697492371 Naive Predictor: [Accuracy score: 0.2478, F-score: 0.2917] #### Calculating the performance with sklearn: accuracy = 0.2478439697492371, recall = 1.0, precision = 0.2478439697492371 Naive Predictor: [Accuracy score: 0.2478, F-score: 0.2917] ### Supervised Learning Models The following are some of the supervised learning models that are currently available in scikit-learn that you may choose from: • Gaussian Naive Bayes (GaussianNB) • Decision Trees • Ensemble Methods (Bagging, AdaBoost, Random Forest, Gradient Boosting) • K-Nearest Neighbors (KNeighbors) • Stochastic Gradient Descent Classifier (SGDC) • Support Vector Machines (SVM) • Logistic Regression ### Question 2 - Model Application List three of the supervised learning models above that are appropriate for this problem that you will test on the census data. For each model chosen • Describe one real-world application in industry where the model can be applied. • What are the strengths of the model; when does it perform well? • What are the weaknesses of the model; when does it perform poorly? • What makes this model a good candidate for the problem, given what you know about the data? HINT: Structure your answer in the same format as above^, with 4 parts for each of the three models you pick. Please include references with your answer. Answer: This is a classification problem and the output is either 0 or 1 so following are the methods I will be using to test on the census data (The links for various articles and books are provided in the reference section at the end): Logistic Regression • Real World Application: Investigation of risk factors associated with injuries to horses undertaking jump racing in Great Britain • Strengths of the model: • Easy to underatand and interprete, so a good baseline to start with and quickly get some answer. • Easy on computing resources and effecient to train • It has low variance and so is less prone to over-fitting • Can provide good results in case of less features • Weakness of the model: • Cannot solve non-linear problem with this, the descision surface is linear for logistic regression. • Sensitive to outliers. • Logistic regression requires that the variables are independent, so in the training data care has to be given to include only indepenedent variables. • Needs large sample size to provide stable results. • Why use: • This is a binary clasification problem of detecting if the income is over 50K or not. So logistic regression can be employed here. The benefit of having logistic regression is that its simple to implement and then the model can be improved( like using stochastic gradient descent). I believe this will act as a better baseline than the naive method we used earlier. Support Vector Machines (SVM) • Real World Application: Using SVM for natural language processing to find out the study region in environmental science • Strengths of the model: • Effective in high dimensional spaces so can model non linear decision boundaries. • Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient. • Different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels. • Weakness of the model: • SVN works by creating hyper planes on n-dimensional feature space, so for larger feature set training SVM can be time consuming if the dataset is big. • Overfit problem can occur when the data is too noisy • If the number of features is much greater than the number of samples it may suffer from overfitting if the kernel is not choosen properly. So choosing the kernel is bit essential. • Why use: • SVM works effectively in the binary classification that we are having here. Apart from that from the pair plots plotted. I do not see a very clear 2-dimensional distinction descision surface. This will be helpful as it operates on hyperplanes to work with data. Here the other task is to find a best possible boundary, SVM can be useful as its built on large margin classification. This will effectively help classify the margins in a better way. Gradient Boosting • Real World Application: Predict survival for cancer patient • Strengths of the model: • Good on large data sets. Good choice to reduce bias and variance. • Good for both linear and non linear data set. • Good for both regression and classification tasks. • New predictors learn from mistakes committed by previous predictors, so it takes less time/iterations to reach close to actual predictions. • Weakness of the model: • Very sensitive to feature set and training set. • Predictions are not easy to understand. This may affect the chance that it will be tuned right in the first shot. May need better understanding. • If stopping crieteria is not choosen properly can lead to overfitting. • Why use: • The data we have may not be necessarily linear so gradient boosting can be applied here as it works with both kind of data. The data set looks to have some class imbalance and a ensemble method like Gradient boosting can help achieve a better prediction model. ### Implementation - Creating a Training and Predicting Pipeline To properly evaluate the performance of each model you've chosen, it's important that you create a training and predicting pipeline that allows you to quickly and effectively train models using various sizes of training data and perform predictions on the testing data. Your implementation here will be used in the following section. In the code block below, you will need to implement the following: • Import fbeta_score and accuracy_score from sklearn.metrics. • Fit the learner to the sampled training data and record the training time. • Perform predictions on the test data X_test, and also on the first 300 training points X_train[:300]. • Record the total prediction time. • Calculate the accuracy score for both the training subset and testing set. • Calculate the F-score for both the training subset and testing set. • Make sure that you set the beta parameter! # TODO: Import two metrics from sklearn - fbeta_score and accuracy_score def train_predict(learner, sample_size, X_train, y_train, X_test, y_test): ''' inputs: - learner: the learning algorithm to be trained and predicted on - sample_size: the size of samples (number) to be drawn from training set - X_train: features training set - y_train: income training set - X_test: features testing set - y_test: income testing set ''' results = {} # TODO: Fit the learner to the training data using slicing with 'sample_size' using .fit(training_features[:], training_labels[:]) start = time() # Get start time learner = learner.fit(X_train[0:sample_size], y_train[0:sample_size]) end = time() # Get end time # TODO: Calculate the training time results['train_time'] = end - start # TODO: Get the predictions on the test set(X_test), # then get predictions on the first 300 training samples(X_train) using .predict() start = time() # Get start time predictions_test = learner.predict(X_test) predictions_train = learner.predict(X_train[:300]) end = time() # Get end time # TODO: Calculate the total prediction time results['pred_time'] = end - start # TODO: Compute accuracy on the first 300 training samples which is y_train[:300] results['acc_train'] = accuracy_score(y_train[:300], predictions_train) # TODO: Compute accuracy on test set using accuracy_score() results['acc_test'] = accuracy_score(y_test, predictions_test) # TODO: Compute F-score on the the first 300 training samples using fbeta_score() beta = 0.5 results['f_train'] = fbeta_score(y_train[:300], predictions_train, beta = beta) # TODO: Compute F-score on the test set which is y_test results['f_test'] = fbeta_score(y_test, predictions_test, beta = beta) # Success print("{} trained on {} samples.".format(learner.__class__.__name__, sample_size)) # Return the results return results ### Implementation: Initial Model Evaluation In the code cell, you will need to implement the following: • Import the three supervised learning models you've discussed in the previous section. • Initialize the three models and store them in 'clf_A', 'clf_B', and 'clf_C'. • Use a 'random_state' for each model you use, if provided. • Note: Use the default settings for each model — you will tune one specific model in a later section. • Calculate the number of records equal to 1%, 10%, and 100% of the training data. • Store those values in 'samples_1', 'samples_10', and 'samples_100' respectively. Note: Depending on which algorithms you chose, the following implementation may take some time to run! # TODO: Import the three supervised learning models from sklearn np.warnings.filterwarnings('ignore') from sklearn.linear_model import LogisticRegression from sklearn.ensemble import GradientBoostingClassifier from sklearn.svm import SVC # TODO: Initialize the three models random_state = 43 # Logistic Regression clf_A = LogisticRegression(random_state=random_state) # Support Vector Machines (SVM) clf_B = SVC(random_state=random_state) # Gradient Boosting clf_C = GradientBoostingClassifier(random_state=random_state) # TODO: Calculate the number of samples for 1%, 10%, and 100% of the training data # HINT: samples_100 is the entire training set i.e. len(y_train) # HINT: samples_10 is 10% of samples_100 (ensure to set the count of the values to be int and not float) # HINT: samples_1 is 1% of samples_100 (ensure to set the count of the values to be int and not float) samples_100 = y_train.size samples_10 = np.int64(np.multiply(samples_100, 0.1)) samples_1 = np.int64(np.multiply(samples_100, 0.01)) # Collect results on the learners # results = {} # model_set = [clf_A, clf_B, clf_C] # model_set = [clf_A, clf_C] def calc_result_of_learner(model_set): results = {} for clf in model_set: clf_name = clf.__class__.__name__ results[clf_name] = {} for i, samples in enumerate([samples_1, samples_10, samples_100]): results[clf_name][i] = \ train_predict(clf, samples, X_train, y_train, X_test, y_test) return results def print_learner_results_in_table(results, accuracy, fscore): for res in results.items(): display_html(display_string=res[0], color='blue', heading_value='h5') display(pd.DataFrame(res[1]).rename(columns={0:'1%', 1:'10%', 2:'100%'})) vs.evaluate(results, accuracy, fscore) # Run metrics visualization for the three supervised learning models chosen display_html("Run metrics visualization for the three supervised learning models chosen (With SVM):") model_set = [clf_A, clf_B, clf_C] results = calc_result_of_learner(model_set=model_set) print_learner_results_in_table(results=results, accuracy=accuracy, fscore=fscore) # vs.evaluate(results, accuracy, fscore) display_html("Run metrics visualization for the three supervised learning models chosen (Without SVM):") model_set = [clf_A, clf_C] results = calc_result_of_learner(model_set=model_set) # vs.evaluate(results, accuracy, fscore) print_learner_results_in_table(results=results, accuracy=accuracy, fscore=fscore) #### Run metrics visualization for the three supervised learning models chosen (With SVM): LogisticRegression trained on 361 samples. LogisticRegression trained on 3617 samples. LogisticRegression trained on 36177 samples. SVC trained on 361 samples. SVC trained on 3617 samples. SVC trained on 36177 samples. GradientBoostingClassifier trained on 361 samples. GradientBoostingClassifier trained on 3617 samples. GradientBoostingClassifier trained on 36177 samples. ##### LogisticRegression 1% 10% 100% acc_test 0.823991 0.841570 0.843118 acc_train 0.860000 0.850000 0.830000 f_test 0.638170 0.680017 0.682812 f_train 0.727273 0.691318 0.642202 pred_time 0.007558 0.004691 0.002988 train_time 0.003045 0.019732 0.246644 ##### SVC 1% 10% 100% acc_test 0.758872 0.834052 0.841791 acc_train 0.763333 0.850000 0.840000 f_test 0.000000 0.668927 0.682872 f_train 0.000000 0.703422 0.670103 pred_time 0.207695 1.739599 14.293886 train_time 0.009839 0.833704 89.733363 ##### GradientBoostingClassifier 1% 10% 100% acc_test 0.830735 0.863460 0.864787 acc_train 0.966667 0.890000 0.876667 f_test 0.652447 0.735864 0.738580 f_train 0.929577 0.787781 0.759076 pred_time 0.018142 0.016104 0.019581 train_time 0.077291 0.644665 7.494051 #### Run metrics visualization for the three supervised learning models chosen (Without SVM): LogisticRegression trained on 361 samples. LogisticRegression trained on 3617 samples. LogisticRegression trained on 36177 samples. GradientBoostingClassifier trained on 361 samples. GradientBoostingClassifier trained on 3617 samples. GradientBoostingClassifier trained on 36177 samples. ##### LogisticRegression 1% 10% 100% acc_test 0.823991 0.841570 0.843118 acc_train 0.860000 0.850000 0.830000 f_test 0.638170 0.680017 0.682812 f_train 0.727273 0.691318 0.642202 pred_time 0.004381 0.004233 0.003947 train_time 0.002694 0.017182 0.212742 ##### GradientBoostingClassifier 1% 10% 100% acc_test 0.830735 0.863460 0.864787 acc_train 0.966667 0.890000 0.876667 f_test 0.652447 0.735864 0.738580 f_train 0.929577 0.787781 0.759076 pred_time 0.021167 0.016140 0.023832 train_time 0.082876 0.605710 7.868132 ## Improving Results In this final section, you will choose from the three supervised learning models the best model to use on the student data. You will then perform a grid search optimization for the model over the entire training set (X_train and y_train) by tuning at least one parameter to improve upon the untuned model's F-score. ### Question 3 - Choosing the Best Model • Based on the evaluation you performed earlier, in one to two paragraphs, explain to CharityML which of the three models you believe to be most appropriate for the task of identifying individuals that make more than$50,000.

HINT: Look at the graph at the bottom left from the cell above(the visualization created by vs.evaluate(results, accuracy, fscore)) and check the F score for the testing set when 100% of the training set is used. Which model has the highest score? Your answer should include discussion of the:

• metrics - F score on the testing when 100% of the training data is used,
• prediction/training time
• the algorithm's suitability for the data.

• Of the 3 models tested the Gradient Boosting Classifier performed the best. It has scored better in both the testing and training sets. Though its F-Score and accuracy score is nearly same as the other 3 its still on the higher side for both the testing and training data. This means it is a proper balance of prescision (The number of relevent items) and recall (How many relevent items are selected)
• The time taken by the SVM is quite high than the other 3. We can obsereve it by the second set of graph where we are discarding it. Though it takes a significant amout of time more, the accuracy and F-Score on both test and train data are less. So we will be discarding it. Now in Logistic Regression Vs Gradient Boosting; the accuracy and F-Score of the Gradient Boosting is bit more for both test and training data. So we can choose it. Apart from that if we see the timing, we can see that with the % increase in samples the time taken to training time for the gradient boosting is quite more, but for the testing set, though with % increase the time to predict for the Logistic regression increases, the time for Gradient Boosting is nearly same. This may be true if the data size increase further.
• We will be using Gradient Boosting for further analysis. The algorithm scored better with the default parameters. So improving the hyper parameter tuning may possible give a better predction. Apart from that it is fast to work on.

### Question 4 - Describing the Model in Layman's Terms

• In one to two paragraphs, explain to CharityML, in layman's terms, how the final model chosen is supposed to work. Be sure that you are describing the major qualities of the model, such as how the model is trained and how the model makes a prediction. Avoid using advanced mathematical jargon, such as describing equations.

HINT:

When explaining your model, if using external resources please include all citations.

Answer: The algorith we will be using here is summarised:

1. It first models the data with a simpler model (weak model). Because the model is simple it will not be a good fit to generate errors. Now it focuses on that error.
2. Now it uses a different model (predictor) to fix this hard to fit data (error data) and get them right.
3. The above 2 steps are repeated for some time with different predictors so that we have have better results gradually.
4. At the end we combine the predictors in some way to get better results.

For a simple example lets think we have the task to grade a students paper (lets say math combination of calculus, stats and algebra) and instead of getting expert in the field we stick with some people who are not that exprt but can look at a gradient rubic we are giving them and grade. So the assumption is that individually they will not be doing justice in grading as they will work mostly mechanically. So we follow this to train them:

1. Ask one person to grade with the rubic and we verify the result, point out the errors in the grading process. These errors were due to the fact that this person is good in understanding certain question (lets say calculus) and grading well and not good in grading other questions (algebra and stats).
2. Now we assign the 2nd person with grading the questions where the 1st persion failed and so on repeat the process till we are satisfied.
3. After that we average the results using some method.

This is the gist of the ensemble method like gradient boosting.

Applying the similar example to the problem at hand we have the following solution using gradient boosting: Here the first simple solution can be lets say all of them have more than 50K salary as we did in the naive method eariler. This may not be accurate as its not a very good prediction but a weak one. The algorithm will check how good it has done, what is the residual (How much it is deviating from the actual results). For next step it will figure out which variable is causing more trouble and then it will use a descision tree to get a better catagorization on that variable. This process is repeated till the satisfied results are reached.

Gradient boosting has benefits for our approach as we have observed that the time taken to train and predict by this mode is really good and also the end results that is given by the F-Score is also good.

### Implementation: Model Tuning

Fine tune the chosen model. Use grid search (GridSearchCV) with at least one important parameter tuned with at least 3 different values. You will need to use the entire training set for this. In the code cell below, you will need to implement the following:

• Import sklearn.grid_search.GridSearchCV and sklearn.metrics.make_scorer.
• Initialize the classifier you've chosen and store it in clf.
• Set a random_state if one is available to the same state you set before.
• Create a dictionary of parameters you wish to tune for the chosen model.
• Example: parameters = {'parameter' : [list of values]}.
• Note: Avoid tuning the max_features parameter of your learner if that parameter is available!
• Use make_scorer to create an fbeta_score scoring object (with $\beta = 0.5$).
• Perform grid search on the classifier clf using the 'scorer', and store it in grid_obj.
• Fit the grid search object to the training data (X_train, y_train), and store it in grid_fit.

Note: Depending on the algorithm chosen and the parameter list, the following implementation may take some time to run!

# TODO: Import 'GridSearchCV', 'make_scorer', and any other necessary libraries
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer

# TODO: Initialize the classifier
clf = GradientBoostingClassifier(random_state=random_state, verbose=0)

# TODO: Create the parameters list you wish to tune, using a dictionary if needed.
# HINT: parameters = {'parameter_1': [value1, value2], 'parameter_2': [value1, value2]}
# 3 is the default one. So I am starting from 3
# n_estimators starts with 100
# parameters = {'max_depth':[3,4,5,6,7],
#              'loss':['deviance', 'exponential'],
#              'n_estimators':[100, 150, 300, 600],
#              'learning_rate': [0.1, 0.5, 1.0],
#              'warm_start': [True, False]}

# parameters = {'max_depth':[3,4,5,6,7],
#              'loss':['deviance', 'exponential'],
#              'n_estimators':[100, 150, 300, 600]}

# parameters = {'max_depth':[3,4,5,6,7],
#              'loss':['deviance', 'exponential'],
#              'n_estimators':[100, 150, 300, 600],
#              'learning_rate': [0.1, 0.5, 1.0]}

parameters = {'max_depth':[3,4,5,6,7],
'loss':['deviance', 'exponential']}

# TODO: Make an fbeta_score scoring object using make_scorer()
scorer = make_scorer(fbeta_score, beta = 0.5)

# TODO: Perform grid search on the classifier using 'scorer' as the scoring method using GridSearchCV()
start = time()
grid_obj = GridSearchCV(estimator = clf, param_grid = parameters, scoring = scorer)

# TODO: Fit the grid search object to the training data and find the optimal parameters using fit()
grid_fit = grid_obj.fit(X_train, y_train)
end = time()
# Get the estimator
best_clf = grid_fit.best_estimator_

# Make predictions using the unoptimized and model
predictions = (clf.fit(X_train, y_train)).predict(X_test)
# best_predictions = best_clf.predict(X_test)
start = time()
best_predictions = best_clf.predict(X_test)
end = time()
print("Time taken to predict: {}".format(end - start))
Time taken to predict: 0.03802013397216797
# Report the before-and-afterscores
print("Unoptimized model\n------")
print("Accuracy score on testing data: {:.4f}".format(accuracy_score(y_test, predictions)))
print("F-score on testing data: {:.4f}".format(fbeta_score(y_test, predictions, beta = 0.5)))
print("\nOptimized Model\n------")
print("Final accuracy score on the testing data: {:.4f}".format(accuracy_score(y_test, best_predictions)))
print("Final F-score on the testing data: {:.4f}".format(fbeta_score(y_test, best_predictions, beta = 0.5)))
print('Time taken for grid search: {}s'.format(end-start))
display(best_clf)
Unoptimized model
------
Accuracy score on testing data: 0.8648
F-score on testing data: 0.7386

Optimized Model
------
Final accuracy score on the testing data: 0.8724
Final F-score on the testing data: 0.7527
Time taken for grid search: 0.03802013397216797s

learning_rate=0.1, loss='deviance', max_depth=5,
max_features=None, max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=100,
n_iter_no_change=None, presort='auto', random_state=43,
subsample=1.0, tol=0.0001, validation_fraction=0.1,
verbose=0, warm_start=False)
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
conf_mat = confusion_matrix(y_test, best_predictions)
# normalize the data
conf_mat = conf_mat.astype('float') / conf_mat.sum(axis=1)[:, np.newaxis]
display(conf_mat)
sns.heatmap(conf_mat, annot=True, annot_kws={"size":50}, cmap='plasma_r', square=False)
plt.title('Confusion matrix for:\n{}'.format(best_clf.__class__.__name__));
plt.ylabel('True')
plt.xlabel('Predicted')
array([[0.94274476, 0.05725524],
[0.34892251, 0.65107749]])

Text(0.5, 12.5, 'Predicted')

### Question 5 - Final Model Evaluation

• What is your optimized model's accuracy and F-score on the testing data?
• Are these scores better or worse than the unoptimized model?
• How do the results from your optimized model compare to the naive predictor benchmarks you found earlier in Question 1?_

Note: Fill in the table below with your results, and then provide discussion in the Answer box.

#### Results:

Metric Unoptimized Model Optimized Model
Accuracy Score 0.8648 0.8724
F-score 0.7386 0.7527

• Accuracy of optimized model is 0.8724 and F-Score is 0.7527
• Thse score are nearly same as the un-optimized score. Just a little bit better.
• These scores are way above the score of the naive predictor in Question 1. There the Accuracy score: 0.2478, F-score: 0.2917.

## Feature Importance

An important task when performing supervised learning on a dataset like the census data we study here is determining which features provide the most predictive power. By focusing on the relationship between only a few crucial features and the target label we simplify our understanding of the phenomenon, which is most always a useful thing to do. In the case of this project, that means we wish to identify a small number of features that most strongly predict whether an individual makes at most or more than $50,000. Choose a scikit-learn classifier (e.g., adaboost, random forests) that has a feature_importance_ attribute, which is a function that ranks the importance of features according to the chosen classifier. In the next python cell fit this classifier to training set and use this attribute to determine the top 5 most important features for the census dataset. ### Question 6 - Feature Relevance Observation When Exploring the Data, it was shown there are thirteen available features for each individual on record in the census data. Of these thirteen records, which five features do you believe to be most important for prediction, and in what order would you rank them and why? # Just to reiterate the features display(data.columns) Index(['age', 'workclass', 'education_level', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'income'], dtype='object') Answer: Following are the 5 features I believe are important from personal experience: • education_level: Gets the level of education. This is important for the earning. • occupation : The occupation also matter. Some occupations have better pay and some do not. Regardless of education this can be an variable that influences. • capital-gain: Regardless of the above 2 the captail gain will determine if the person can pay for charity. If this is not there we cant expect much. • captail-loss: Same as above. This also influence the capability to pay. • age: I guess age plays a role to tell in what kind of position the person is financially. ### Implementation - Extracting Feature Importance Choose a scikit-learn supervised learning algorithm that has a feature_importance_ attribute availble for it. This attribute is a function that ranks the importance of each feature when making predictions based on the chosen algorithm. In the code cell below, you will need to implement the following: • Import a supervised learning model from sklearn if it is different from the three used earlier. • Train the supervised model on the entire training set. • Extract the feature importances using '.feature_importances_'. # TODO: Import a supervised learning model that has 'feature_importances_' # Fortunately the GraientBoosting has feature_importances_ # TODO: Train the supervised model on the training set using .fit(X_train, y_train) model = best_clf # TODO: Extract the feature importances using .feature_importances_ importances = best_clf.feature_importances_ # Plot vs.feature_plot(importances, X_train, y_train) ### Question 7 - Extracting Feature Importance Observe the visualization created above which displays the five most relevant features for predicting if an individual makes at most or above$50,000.

• How do these five features compare to the five features you discussed in Question 6?
• If you were close to the same answer, how does this visualization confirm your thoughts?
• If you were not close, why do you think these features are more relevant?

I was not expecting the martial-status to impact the individuals ability to earn. But from data it looks like it makes difference. The rest of the features are as I expected.

### Feature Selection

How does a model perform if we only use a subset of all the available features in the data? With less features required to train, the expectation is that training and prediction time is much lower — at the cost of performance metrics. From the visualization above, we see that the top five most important features contribute more than half of the importance of all features present in the data. This hints that we can attempt to reduce the feature space and simplify the information required for the model to learn. The code cell below will use the same optimized model you found earlier, and train it on the same training set with only the top five important features.

# Import functionality for cloning a model
from sklearn.base import clone

# Reduce the feature space
X_train_reduced = X_train[X_train.columns.values[(np.argsort(importances)[::-1])[:5]]]
X_test_reduced = X_test[X_test.columns.values[(np.argsort(importances)[::-1])[:5]]]

# Train on the "best" model found from grid search earlier
start = time()
clf = (clone(best_clf)).fit(X_train_reduced, y_train)
end = time()
print("Time taken for training: {}".format(end-start))
# Make new predictions
start = time()
reduced_predictions = clf.predict(X_test_reduced)
end = time()
print("Time taken for predicting: {}".format(end-start))

# Report scores from the final model using both versions of data
print("Final Model trained on full data\n------")
print("Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, best_predictions)))
print("F-score on testing data: {:.4f}".format(fbeta_score(y_test, best_predictions, beta = 0.5)))
print("\nFinal Model trained on reduced data\n------")
print("Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, reduced_predictions)))
print("F-score on testing data: {:.4f}".format(fbeta_score(y_test, reduced_predictions, beta = 0.5)))
print("Time taken for ")
Time taken for training: 1.6157686710357666
Time taken for predicting: 0.01609182357788086
Final Model trained on full data
------
Accuracy on testing data: 0.8724
F-score on testing data: 0.7527

Final Model trained on reduced data
------
Accuracy on testing data: 0.8627
F-score on testing data: 0.7347
Time taken for 

### Question 8 - Effects of Feature Selection

• How does the final model's F-score and accuracy score on the reduced data using only five features compare to those same scores when all features are used?
• If training time was a factor, would you consider using the reduced data as your training set?

Metric Optimized Model Feature Reduced Model
Accuracy Score 0.8724 0.8627
F-score 0.7527 0.7347
• Checking the table we can see that the accuracy is not much changing with reduced feature model.
• The time taken for prediction changed from 0.03 to 0.015. This will be very useful when we have a lot more data and more feature to choose from.
• From doing the gridsearch and prediction in this exercise which almost melted my laptop, I believe only keeping the most essential features makes more sense when we are computing in constrained environment.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to
File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

Reference

If you like it, share it!