GBDT Note

Introduction

Gradient Boosting explained by Alex Rogozhnikov

Gradient boosting (GB) is a machine learning algorithm developed in the late '90s that is still very popular. It produces state-of-the-art results for many commercial (and academic) applications. This page explains how the gradient boosting algorithm works using several interactive visualizations.

http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html

Packages

XGBoost

XGBoost Python Package - xgboost 1.6.1 documentation

https://xgboost.readthedocs.io/en/stable/python/index.html

LGBM

Parameter Tuning

GBM Parameters

1. Tree-Specific Parameters: These affect each individual tree in the model.

2. Boosting Parameters: These affect the boosting operation in the model.

3. Miscellaneous Parameters: Other parameters for overall functioning.

1. min_samples_split
• Defines the minimum number of samples (or observations) which are required in a node to be considered for splitting.
• Used to control over-fitting. Higher values prevent a model from learning relations which might be highly specific to the particular sample selected for a tree.
• Too high values can lead to under-fitting hence, it should be tuned using CV.

2. min_samples_leaf
• Defines the minimum samples (or observations) required in a terminal node or leaf.
• Used to control over-fitting similar to min_samples_split.
• Generally lower values should be chosen for imbalanced class problems because the regions in which the minority class will be in majority will be very small.

3. min_weight_fraction_leaf
• Similar to min_samples_leaf but defined as a fraction of the total number of observations instead of an integer.
• Only one of #2 and #3 should be defined.

4. max_depth
• The maximum depth of a tree.
• Used to control over-fitting as higher depth will allow model to learn relations very specific to a particular sample.
• Should be tuned using CV.

5. max_leaf_nodes
• The maximum number of terminal nodes or leaves in a tree.
• Can be defined in place of max_depth. Since binary trees are created, a depth of ‘n’ would produce a maximum of 2^n leaves.
• If this is defined, GBM will ignore max_depth.

6. max_features
• The number of features to consider while searching for a best split. These will be randomly selected.
• As a thumb-rule, square root of the total number of features works great but we should check upto 30-40% of the total number of features.
• Higher values can lead to over-fitting but depends on case to case.

1. Initialize the outcome
2. Iterate from 1 to total number of trees
  2.1 Update the weights for targets based on previous run (higher for the ones mis-classified)
  2.2 Fit the model on selected subsample of data
  2.3 Make predictions on the full set of observations
  2.4 Update the output with current results taking into account the learning rate
3. Return the final output.

1. learning_rate
• This determines the impact of each tree on the final outcome (step 2.4). GBM works by starting with an initial estimate which is updated using the output of each tree. The learning parameter controls the magnitude of this change in the estimates.
• Lower values are generally preferred as they make the model robust to the specific characteristics of tree and thus allowing it to generalize well.
• Lower values would require higher number of trees to model all the relations and will be computationally expensive.

2. n_estimators
• The number of sequential trees to be modeled (step 2)
• Though GBM is fairly robust at higher number of trees but it can still overfit at a point. Hence, this should be tuned using CV for a particular learning rate.

3. subsample
• The fraction of observations to be selected for each tree. Selection is done by random sampling.
• Values slightly less than 1 make the model robust by reducing the variance.
• Typical values ~0.8 generally work fine but can be fine-tuned further.

1. loss
• It refers to the loss function to be minimized in each split.
• It can have various values for classification and regression case. Generally the default values work fine. Other values should be chosen only if you understand their impact on the model.

2. init
• This affects initialization of the output.
• This can be used if we have made another model whose outcome is to be used as the initial estimates for GBM.

3. random_state
• The random number seed so that same random numbers are generated every time.
• This is important for parameter tuning. If we don’t fix the random number, then we’ll have different outcomes for subsequent runs on the same parameters and it becomes difficult to compare models.
• It can potentially result in overfitting to a particular random sample selected. We can try running models for different random samples, which is computationally expensive and generally not used.

4. verbose
• The type of output to be printed when the model fits. The different values can be:
• 0: no output generated (default)
• 1: output generated for trees in certain intervals
• >1: output generated for all trees

5. warm_start
• This parameter has an interesting application and can help a lot if used judicially.
• Using this, we can fit additional trees on previous fits of a model. It can save a lot of time and you should explore this option for advanced applications

6. presort
• Select whether to presort data for faster splits.
• It makes the selection automatically by default but it can be changed if needed.

3. Parameter Tuning with Example

1. City variable dropped because of too many categories

2. DOB converted to Age | DOB dropped

3. EMI_Loan_Submitted_Missing created which is 1 if EMI_Loan_Submitted was missing else 0 | Original variable EMI_Loan_Submitted dropped

4. EmployerName dropped because of too many categories

5. Existing_EMI imputed with 0 (median) since only 111 values were missing

6. Interest_Rate_Missing created which is 1 if Interest_Rate was missing else 0 | Original variable Interest_Rate dropped

7. Lead_Creation_Date dropped because made little intuitive impact on outcome

8. Loan_Amount_Applied, Loan_Tenure_Applied imputed with median values

9. Loan_Amount_Submitted_Missing created which is 1 if Loan_Amount_Submitted was missing else 0 | Original variable Loan_Amount_Submitted dropped

10. Loan_Tenure_Submitted_Missing created which is 1 if Loan_Tenure_Submitted was missing else 0 | Original variable Loan_Tenure_Submitted dropped

11. LoggedIn, Salary_Account dropped

12. Processing_Fee_Missing created which is 1 if Processing_Fee was missing else 0 | Original variable Processing_Fee dropped

13. Source – top 2 kept as is and all others combined into different category

14. Numerical and One-Hot-Coding performed

def modelfit(alg, dtrain, predictors, performCV=True, printFeatureImportance=True, cv_folds=5):
    #Fit the algorithm on the data
    alg.fit(dtrain[predictors], dtrain['Disbursed'])

    #Predict training set:
    dtrain_predictions = alg.predict(dtrain[predictors])
    dtrain_predprob = alg.predict_proba(dtrain[predictors])[:,1]

    #Perform cross-validation:
    if performCV:
        cv_score = cross_validation.cross_val_score(alg, dtrain[predictors], dtrain['Disbursed'], cv=cv_folds, scoring='roc_auc')

    #Print model report:
    print "\nModel Report"
    print "Accuracy : %.4g" % metrics.accuracy_score(dtrain['Disbursed'].values, dtrain_predictions)
    print "AUC Score (Train): %f" % metrics.roc_auc_score(dtrain['Disbursed'], dtrain_predprob)

    if performCV:
        print "CV Score : Mean - %.7g | Std - %.7g | Min - %.7g | Max - %.7g" % (np.mean(cv_score),np.std(cv_score),np.min(cv_score),np.max(cv_score))

    #Print Feature Importance:
    if printFeatureImportance:
        feat_imp = pd.Series(alg.feature_importances_, predictors).sort_values(ascending=False)
        feat_imp.plot(kind='bar', title='Feature Importances')
        plt.ylabel('Feature Importance Score')

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm0 = GradientBoostingClassifier(random_state=10)
modelfit(gbm0, train, predictors)

General Approach for Parameter Tuning

1. Choose a relatively high learning rate. Generally the default value of 0.1 works but somewhere between 0.05 to 0.2 should work for different problems

2. Determine the optimum number of trees for this learning rate. This should range around 40-70. Remember to choose a value on which your system can work fairly fast. This is because it will be used for testing various scenarios and determining the tree parameters.

3. Tune tree-specific parameters for decided learning rate and number of trees. Note that we can choose different parameters to define a tree and I’ll take up an example here.

4. Lower the learning rate and increase the estimators proportionally to get more robust models.

Fix learning rate and number of estimators for tuning tree-based parameters

1. min_samples_split = 500 : This should be ~0.5-1% of total values. Since this is imbalanced class problem, we’ll take a small value from the range.

2. min_samples_leaf = 50 : Can be selected based on intuition. This is just used for preventing overfitting and again a small value because of imbalanced classes.

3. max_depth = 8 : Should be chosen (5-8) based on the number of observations and predictors. This has 87K rows and 49 columns so lets take 8 here.

4. max_features = ‘sqrt’ : Its a general thumb-rule to start with square root.

5. subsample = 0.8 : This is a commonly used used start value

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
param_test1 = {'n_estimators':range(20,81,10)}
gsearch1 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, min_samples_split=500,min_samples_leaf=50,max_depth=8,max_features='sqrt',subsample=0.8,random_state=10),
param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch1.fit(train[predictors],train[target])

gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_

1. If the value is around 20, you might want to try lowering the learning rate to 0.05 and re-run grid search

2. If the values are too high ~100, tuning the other parameters will take long time and you can try a higher learning rate

Tuning tree-specific parameters

1. Tune max_depth and num_samples_split

2. Tune min_samples_leaf

3. Tune max_features

param_test2 = {'max_depth':range(5,16,2), 'min_samples_split':range(200,1001,200)}
gsearch2 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, n_estimators=60, max_features='sqrt', subsample=0.8, random_state=10),
param_grid = param_test2, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch2.fit(train[predictors],train[target])
gsearch2.grid_scores_, gsearch2.best_params_, gsearch2.best_score_

param_test3 = {'min_samples_split':range(1000,2100,200), 'min_samples_leaf':range(30,71,10)}
gsearch3 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, n_estimators=60,max_depth=9,max_features='sqrt', subsample=0.8, random_state=10),
param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch3.fit(train[predictors],train[target])
gsearch3.grid_scores_, gsearch3.best_params_, gsearch3.best_score_

modelfit(gsearch3.best_estimator_, train, predictors)

param_test4 = {'max_features':range(7,20,2)}
gsearch4 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, n_estimators=60,max_depth=9, min_samples_split=1200, min_samples_leaf=60, subsample=0.8, random_state=10),
param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch4.fit(train[predictors],train[target])
gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_

• min_samples_split: 1200

• min_samples_leaf: 60

• max_depth: 9

• max_features: 7

Tuning subsample and making models with lower learning rate

param_test5 = {'subsample':[0.6,0.7,0.75,0.8,0.85,0.9]}
gsearch5 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, n_estimators=60,max_depth=9,min_samples_split=1200, min_samples_leaf=60, subsample=0.8, random_state=10,max_features=7),
param_grid = param_test5, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch5.fit(train[predictors],train[target])
gsearch5.grid_scores_, gsearch5.best_params_, gsearch5.best_score_

predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm_tuned_1 = GradientBoostingClassifier(learning_rate=0.05, n_estimators=120,max_depth=9, min_samples_split=1200,min_samples_leaf=60, subsample=0.85, random_state=10, max_features=7)
modelfit(gbm_tuned_1, train, predictors)

predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm_tuned_2 = GradientBoostingClassifier(learning_rate=0.01, n_estimators=600,max_depth=9, min_samples_split=1200,min_samples_leaf=60, subsample=0.85, random_state=10, max_features=7)
modelfit(gbm_tuned_2, train, predictors)

predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm_tuned_3 = GradientBoostingClassifier(learning_rate=0.005, n_estimators=1200,max_depth=9, min_samples_split=1200, min_samples_leaf=60, subsample=0.85, random_state=10, max_features=7,
warm_start=True)
modelfit(gbm_tuned_3, train, predictors, performCV=False)

predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm_tuned_4 = GradientBoostingClassifier(learning_rate=0.005, n_estimators=1500,max_depth=9, min_samples_split=1200, min_samples_leaf=60, subsample=0.85, random_state=10, max_features=7,
warm_start=True)
modelfit(gbm_tuned_4, train, predictors, performCV=False)

Remark