Learning Bagging Ensemble Methods with R: A Step-by-Step Guide

The Instability of Single Decision Trees

When statistical analysts and data scientists embark on building predictive models, a common and often intuitive starting point is the construction of a single decision tree. This methodology offers immense appeal due to its inherent simplicity and remarkable ease of interpretation. A decision tree mirrors human decision-making processes, making it a highly transparent model for initial exploration and understanding of variable relationships within a dataset.

Despite these advantages, relying exclusively on a single decision tree model introduces a critical vulnerability: its high susceptibility to statistical instability, commonly referred to as high variance. This instability manifests because the structure of the tree is extremely sensitive to minor fluctuations or noise present in the training data. If the initial dataset were subjected to a small, random perturbation—such as removing or adding a few observations—the resultant tree structure could change drastically, leading to dramatically different and unreliable predictions.

This inherent fragility means that a single decision tree often performs poorly when attempting to generalize to new, unseen data, resulting in a high test error rate. To successfully overcome this pervasive issue and reduce the model’s variance without sacrificing the desirable low bias often associated with deep trees, we must employ sophisticated ensemble methods. Among the most effective techniques utilized in the machine learning community is Bagging, which is an acronym derived from Bootstrap Aggregating.

Understanding the Core Mechanism of Bagging

Bagging, or Bootstrap Aggregating, stands as a cornerstone technique in ensemble learning, specifically designed to enhance the stability and predictive accuracy of machine learning algorithms, particularly complex models like decision trees. The fundamental power of Bagging lies in its systematic introduction of controlled randomness during the training phase, followed by the aggregation of results from multiple, independently trained models. This process effectively diversifies the models and averages out their individual errors.

The operational efficiency of Bagging is built upon three distinct yet interconnected steps. These steps ensure that a large pool of varied models is created, whose collective wisdom ultimately surpasses the capability of any single component model:

1. The procedure begins by drawing a predefined number, b, of bootstrapped samples from the original training dataset. Bootstrapping involves sampling observations with replacement, which means some observations may appear multiple times in a single sample, while others may not appear at all. This technique is crucial for generating diverse training subsets.
2. An individual decision tree model is meticulously constructed and fitted for each of the b unique bootstrapped samples. Because each sample is slightly different, the resulting individual trees will exhibit structural differences, promoting the necessary diversity within the ensemble.
3. Finally, the predictions generated by all b individual trees are averaged (for regression tasks) or subjected to a majority vote (for classification tasks) to formulate a single, robust, and finalized prediction model.

By constructing a large quantity of these individual decision trees—often ranging from hundreds to thousands—and subsequently averaging their output, the final bagged model successfully smooths the high variance inherent in any single, deep tree. The outcome is a highly stable fitted model that consistently demonstrates a significantly lower test error rate and superior generalization performance on unseen data.

Step 1: Loading Essential R Libraries and Packages

The successful implementation of a sophisticated Bagging model in the R environment requires the installation and activation of several specialized statistical packages. These libraries furnish the core functions necessary for efficient data manipulation, rigorous model fitting, and the subsequent calculation of variable importance metrics crucial for interpretation.

For this comprehensive demonstration, we will rely on five core libraries, each serving a distinct and critical function in the data science pipeline:

• dplyr: Recognized globally for providing a fluent and highly efficient grammar for data wrangling and manipulation tasks, simplifying complex data transformations.
• e1071: A versatile library offering a suite of advanced statistical functionality, including methods essential for calculating critical performance metrics such as variable importance.
• caret: Stands for Classification And REgression Training. This package provides a unified interface for streamlined general model fitting, complexity control, and training workflow management.
• rpart: The foundational package utilized for fitting standard, single decision tree models, which serve as the fundamental base learners that are aggregated within the Bagging framework.
• ipred: This library is paramount, as it provides the specific functions required for the fitting and analysis of bagged decision trees, including the central bagging() function.

The following R code snippet executes the necessary commands to load these essential packages into the active session, ensuring all required functions are accessible for the subsequent modeling steps:

library(dplyr)       #for data wrangling
library(e1071)       #for calculating variable importance
library(caret)       #for general model fitting
library(rpart)       #for fitting decision trees
library(ipred)       #for fitting bagged decision trees

Step 2: Data Preparation and Fitting the Bagged Model

To provide a tangible and practical application of Bagging, we will utilize the renowned built-in R dataset known as airquality. This dataset comprises 153 daily observations detailing various ambient air quality measurements—specifically, Ozone concentration, Solar Radiation (Solar.R), Wind speed, and Temperature (Temp)—recorded in New York during a specific period. Our primary objective is to construct a robust bagged regression model capable of accurately predicting the Ozone level based on the influence of all other recorded variables.

Before initiating the model fitting process, it is standard practice to examine the structural integrity of the data to ensure proper variable handling and identify potential missing values. The str() function provides a concise summary of the dataset’s variables, their data types, and the initial few observations:

#view structure of airquality dataset
str(airquality)

'data.frame':	153 obs. of  6 variables:
 $ Ozone  : int  41 36 12 18 NA 28 23 19 8 NA ...
 $ Solar.R: int  190 118 149 313 NA NA 299 99 19 194 ...
 $ Wind   : num  7.4 8 12.6 11.5 14.3 14.9 8.6 13.8 20.1 8.6 ...
 $ Temp   : int  67 72 74 62 56 66 65 59 61 69 ...
 $ Month  : int  5 5 5 5 5 5 5 5 5 5 ...
 $ Day    : int  1 2 3 4 5 6 7 8 9 10 ...

With the data structure understood, we proceed to fit the bagged model using the bagging() function, which is the core utility provided by the ipred library. We specify nbagg = 150, which instructs the function to perform 150 bootstrap replications, meaning 150 unique trees will be built and aggregated. Additionally, we set coob = TRUE, enabling the calculation of performance metrics based on Out-of-Bag samples, providing an immediate, unbiased estimate of generalization error. Crucially, we use rpart.control() arguments to regulate the complexity of the base learners:

#make this example reproducible
set.seed(1)

#fit the bagged model
bag <- bagging(
  formula = Ozone ~ .,
  data = airquality,
  nbagg = 150,   
  coob = TRUE,
  control = rpart.control(minsplit = 2, cp = 0)
)

#display fitted bagged model
bag

Bagging regression trees with 150 bootstrap replications 

Call: bagging.data.frame(formula = Ozone ~ ., data = airquality, nbagg = 150, 
    coob = TRUE, control = rpart.control(minsplit = 2, cp = 0))

Out-of-bag estimate of root mean squared error:  17.4973 

Step 3: Interpreting Model Parameters and Performance Metrics

A key aspect of implementing Bagging involves understanding how the control parameters influence the individual decision trees—the base learners—that constitute the ensemble. In our execution, we deliberately employed aggressive settings within the rpart.control() function to encourage the growth of extremely deep and complex base trees:

• minsplit = 2: This parameter defines the minimum number of observations that must exist in a node before any further split is permitted. By setting this value to a minimum of 2, we permit the individual trees to be highly granular and specialized, allowing them to capture intricate patterns in the training data, even those that might be considered noise.
• cp = 0 (Complexity Parameter): The complexity parameter dictates the threshold for improvement in overall fit required for a split to be accepted. Setting cp to zero ensures that pruning is essentially minimized. Any potential split, regardless of how marginally it improves the model’s accuracy, is considered, thus forcing the trees to grow to their maximum possible depth.

These intentional settings result in base decision trees characterized by inherently low bias (because they fit the training data extremely well) but consequently exhibit high variance. The genius of Bagging is its ability to harness the strength of these low-bias, high-variance base learners. By leveraging the diversity created through bootstrapped samples and the principle of averaging, the ensemble effectively reduces the overall variance, yielding a final model that retains low bias while achieving dramatically improved generalization performance.

The output from the bag object provides a critical measure of performance: the Out-of-Bag (OOB) estimate of the Root Mean Squared Error (RMSE), which is calculated here as 17.4973. The OOB method provides a highly reliable, cross-validated measure of predictive accuracy. It works by using observations that were *not* included in the bootstrap sample used to train a specific tree (the OOB samples) to test that tree. The OOB RMSE is then the average difference between the predicted Ozone value and the actual observed Ozone value across all such out-of-bag observations, serving as an unbiased estimate of the model’s performance on truly unseen data.

Step 4: Quantifying and Visualizing Predictor Importance

While ensemble methods like Bagging deliver superior predictive accuracy, they often come at the cost of reduced interpretability compared to a single, easily visualized decision tree. For robust analytical insight, it remains essential to understand which predictor variables exert the most significant influence on the final predictions, even within the complex structure of an aggregated model.

We can systematically quantify the importance of each predictor by measuring the total reduction in the Residual Sum of Squares (RSS) that is achieved by all splits involving that specific predictor, averaged across all 150 trees in the ensemble. A higher resulting value for a variable indicates that it contributed more substantially to reducing the prediction error throughout the model construction process, thereby signifying it as a more important predictor.

The following R code leverages the powerful varImp() function from the caret library to compute this variable importance metric. Subsequently, the results are sorted in descending order and visualized using a horizontal bar plot, offering a clear, comparative view of predictor influence:

#calculate variable importance
VI <- data.frame(var=names(airquality[,-1]), imp=varImp(bag))

#sort variable importance descending
VI_plot <- VI[order(VI$Overall, decreasing=TRUE),]

#visualize variable importance with horizontal bar plot
barplot(VI_plot$Overall,
        names.arg=rownames(VI_plot),
        horiz=TRUE,
        col='steelblue',
        xlab='Variable Importance')

As clearly depicted in the resulting visualization, the predictor variable Solar.R exhibits the highest importance score, signifying it as the most critical factor in determining Ozone levels within this ensemble model. Conversely, the variable Month appears to be the least influential predictor among those considered, suggesting its impact on Ozone concentration, after accounting for other factors, is minimal.

Step 5: Applying the Model to Generate New Predictions

The final and most practical stage of the modeling process is the deployment of the robust, fitted ensemble model to generate accurate predictions for entirely new, unseen observations. This step validates the entire process by demonstrating the Bagging technique’s ability to generalize beyond the training data effectively.

To illustrate this deployment, we first define a hypothetical new observation, assigning specific, realistic values for each of the predictor variables used in the model (Solar.R, Wind, Temp, Month, and Day):

#define new observation
new <- data.frame(Solar.R=150, Wind=8, Temp=70, Month=5, Day=5)

#use fitted bagged model to predict Ozone value of new observation
predict(bag, newdata=new)

24.4866666666667

By inputting the specified values for this new observation into the fitted bagging model, the resulting prediction indicates that the estimated Ozone value for that particular day is approximately 24.487. This successful utilization of the predict() function concludes the comprehensive step-by-step guide on implementing and analyzing a bagged regression model in R, showcasing its power in generating stable and accurate forecasts.

The complete R code utilized throughout this tutorial, including data setup and visualization scripts, is available for review and execution through the external repository here.