Abstract
Bagging is a commonly used ensemble method, which involves the bootstrap sampling of training observations. Bootstraps are typically needed to decouple the models. However, this is based on the well-understood properties of deterministic algorithms. To perform bagging for stochastic algorithms, there is an implicit assumption that both variance due to the training data and variance due to the algorithm need to be reduced. This assumption may not be correct and only a reduction in variance due to the algorithm may need to be targeted. Bootstrapping may cause an unnecessary increase in error due to bias because it reduces the number of unique training observations. Bootstrapped and non-bootstrapped ensembles are compared across multiple machine learning algorithms and data sets using an extended error decomposition. The results show that bootstrap sampling is often not required as it increases error due to bias and internal variance. The prediction error associated with an algorithm and data set needs to be more fully decomposed in order to distinguish between the different sources of error. This allows the most appropriate ensemble method to be chosen.