MultiBoosting [also known as Boost Bagging] integrates boosting and bagging, combining boosting's superior bias reduction with bagging's superior variance reduction. MultiBoosting is an example of Multi-Strategy Ensemble Learning. We have shown that combining ensemble learning techniques can substantially reduce error.
Publications
Ting, K. M.; Wells, J.; Tan, S.; Teng, S.; Webb, G. I.
Feature-subspace aggregating: Ensembles for stable and unstable learners
Machine Learning, vol. 82, no. 3, pp. 375-397, 2011, ISSN: 0885-6125.
@article{TingEtAl11,
title = {Feature-subspace aggregating: Ensembles for stable and unstable learners},
author = {K. M. Ting and J. Wells and S. Tan and S. Teng and G. I. Webb},
doi = {10.1007/s10994-010-5224-5},
issn = {0885-6125},
year = {2011},
date = {2011-01-01},
journal = {Machine Learning},
volume = {82},
number = {3},
pages = {375-397},
publisher = {Springer},
address = {Netherlands},
abstract = {This paper introduces a new ensemble approach, Feature-Subspace Aggregating (Feating), which builds local models instead of global models. Feating is a generic ensemble approach that can enhance the predictive performance of both stable and unstable learners. In contrast, most existing ensemble approaches can improve the predictive performance of unstable learners only. Our analysis shows that the new approach reduces the execution time to generate a model in an ensemble through an increased level of localisation in Feating. Our empirical evaluation shows that Feating performs significantly better than Boosting, Random Subspace and Bagging in terms of predictive accuracy, when a stable learner SVM is used as the base learner. The speed up achieved by Feating makes feasible SVM ensembles that would otherwise be infeasible for large data sets. When SVM is the preferred base learner, we show that Feating SVM performs better than Boosting decision trees and Random Forests. We further demonstrate that Feating also substantially reduces the error of another stable learner, k-nearest neighbour, and an unstable learner, decision tree.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
This paper introduces a new ensemble approach, Feature-Subspace Aggregating (Feating), which builds local models instead of global models. Feating is a generic ensemble approach that can enhance the predictive performance of both stable and unstable learners. In contrast, most existing ensemble approaches can improve the predictive performance of unstable learners only. Our analysis shows that the new approach reduces the execution time to generate a model in an ensemble through an increased level of localisation in Feating. Our empirical evaluation shows that Feating performs significantly better than Boosting, Random Subspace and Bagging in terms of predictive accuracy, when a stable learner SVM is used as the base learner. The speed up achieved by Feating makes feasible SVM ensembles that would otherwise be infeasible for large data sets. When SVM is the preferred base learner, we show that Feating SVM performs better than Boosting decision trees and Random Forests. We further demonstrate that Feating also substantially reduces the error of another stable learner, k-nearest neighbour, and an unstable learner, decision tree.
Ting, K. M.; Wells, J. R.; Tan, S. C.; Teng, S. W.; Webb, G. I.
FaSS: Ensembles for Stable Learners
Proceedings of the 8th International Workshop on Multiple Classifier Systems, MCS 2009, pp. 364-374, Springer, Reykjavik, Iceland, 2009.
@inproceedings{TingEtAl09,
title = {FaSS: Ensembles for Stable Learners},
author = {K. M. Ting and J. R. Wells and S. C. Tan and S. W. Teng and G. I. Webb},
doi = {10.1007/978-3-642-02326-2_37},
year = {2009},
date = {2009-01-01},
booktitle = {Proceedings of the 8th International Workshop on Multiple Classifier Systems, MCS 2009},
pages = {364-374},
publisher = {Springer},
address = {Reykjavik, Iceland},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Webb, G. I.; Zheng, Z.
Multistrategy Ensemble Learning: Reducing Error by Combining Ensemble Learning Techniques
IEEE Transactions on Knowledge and Data Engineering, vol. 16, no. 8, pp. 980-991, 2004.
@article{WebbZheng04,
title = {Multistrategy Ensemble Learning: Reducing Error by Combining Ensemble Learning Techniques},
author = {G. I. Webb and Z. Zheng},
doi = {10.1109/TKDE.2004.29},
year = {2004},
date = {2004-01-01},
journal = {IEEE Transactions on Knowledge and Data Engineering},
volume = {16},
number = {8},
pages = {980-991},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA},
abstract = {Ensemble learning strategies, especially Boosting and Bagging decision trees, have demonstrated impressive capacities to improve the prediction accuracy of base learning algorithms. Further gains have been demonstrated by strategies that combine simple ensemble formation approaches. In this paper, we investigate the hypothesis that the improvement inaccuracy of multi-strategy approaches to ensemble learning is due to an increase in the diversity of ensemble members that are formed. In addition, guided by this hypothesis, we develop three new multi-strategy ensemble-learning techniques. Experimental results in a wide variety of natural domains suggest that these multi-strategy ensemble-learning techniques are, on average, more accurate than their component ensemble learning techniques},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Ensemble learning strategies, especially Boosting and Bagging decision trees, have demonstrated impressive capacities to improve the prediction accuracy of base learning algorithms. Further gains have been demonstrated by strategies that combine simple ensemble formation approaches. In this paper, we investigate the hypothesis that the improvement inaccuracy of multi-strategy approaches to ensemble learning is due to an increase in the diversity of ensemble members that are formed. In addition, guided by this hypothesis, we develop three new multi-strategy ensemble-learning techniques. Experimental results in a wide variety of natural domains suggest that these multi-strategy ensemble-learning techniques are, on average, more accurate than their component ensemble learning techniques
Webb, G. I.
MultiBoosting: A Technique for Combining Boosting and Wagging
Machine Learning, vol. 40, no. 2, pp. 159-196, 2000.
@article{Webb00a,
title = {MultiBoosting: A Technique for Combining Boosting and Wagging},
author = {G. I. Webb},
doi = {10.1023/A:1007659514849},
year = {2000},
date = {2000-01-01},
journal = {Machine Learning},
volume = {40},
number = {2},
pages = {159-196},
publisher = {Springer},
address = {Netherlands},
abstract = {MultiBoosting is an extension to the highly successful AdaBoost technique for forming decision committees. MultiBoosting can be viewed as combining AdaBoost with wagging. It is able to harness both AdaBoost's high bias and variance reduction with wagging's superior variance reduction. Using C4.5 as the base learning algorithm, Multi-boosting is demonstrated to produce decision committees with lower error than either AdaBoost or wagging significantly more often than the reverse over a large representative cross-section of UCI data sets. It offers the further advantage over AdaBoost of suiting parallel execution.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
MultiBoosting is an extension to the highly successful AdaBoost technique for forming decision committees. MultiBoosting can be viewed as combining AdaBoost with wagging. It is able to harness both AdaBoost's high bias and variance reduction with wagging's superior variance reduction. Using C4.5 as the base learning algorithm, Multi-boosting is demonstrated to produce decision committees with lower error than either AdaBoost or wagging significantly more often than the reverse over a large representative cross-section of UCI data sets. It offers the further advantage over AdaBoost of suiting parallel execution.
Zheng, Z.; Webb, G. I.
Stochastic Attribute Selection Committees with Multiple Boosting: Learning More Accurate and More Stable Classifier Committees
Zhong, N.; Zhou, L. (Ed.): Lecture Notes in Computer Science 1574: Methodologies for Knowledge Discovery and Data Mining - Proceedings of the Third Pacific-Asia Conference (PAKDD'99), pp. 123-132, Springer-Verlag, Beijing, China, 1999.
@inproceedings{ZhengWebb99b,
title = {Stochastic Attribute Selection Committees with Multiple Boosting: Learning More Accurate and More Stable Classifier Committees},
author = {Z. Zheng and G. I. Webb},
editor = {N. Zhong and L. Zhou},
year = {1999},
date = {1999-01-01},
booktitle = {Lecture Notes in Computer Science 1574: Methodologies for Knowledge Discovery and Data Mining - Proceedings of the Third Pacific-Asia Conference (PAKDD'99)},
pages = {123-132},
publisher = {Springer-Verlag},
address = {Beijing, China},
abstract = {Classifier learning is a key technique for KDD. Approaches to learning classifier committees, including Boosting, Bagging, SASC, and SASCB, have demonstrated great success in increasing the prediction accuracy of decision trees. Boosting and Bagging create different classifiers by modifying the distribution of the training set. SASC adopts a different method. It generates committees by stochastic manipulation of the set of attributes considered at each node during tree induction, but keeping the distribution of the training set unchanged. SASCB, a combination of Boosting and SASC, has shown the ability to further increase, on average, the prediction accuracy of decision trees. It has been found that the performance of SASCB and Boosting is more variable than that of SASC, although SASCB is more accurate than the others on average. In this paper, we present a novel method to reduce variability of SASCB and Boosting, and further increase their average accuracy. It generates multiple committees by incorporating Bagging into SASCB. As well as improving stability and average accuracy, the resulting method is amenable to parallel or distributed processing, while Boosting and SascB are not. This is an important characteristic for datamining in large datasets.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Classifier learning is a key technique for KDD. Approaches to learning classifier committees, including Boosting, Bagging, SASC, and SASCB, have demonstrated great success in increasing the prediction accuracy of decision trees. Boosting and Bagging create different classifiers by modifying the distribution of the training set. SASC adopts a different method. It generates committees by stochastic manipulation of the set of attributes considered at each node during tree induction, but keeping the distribution of the training set unchanged. SASCB, a combination of Boosting and SASC, has shown the ability to further increase, on average, the prediction accuracy of decision trees. It has been found that the performance of SASCB and Boosting is more variable than that of SASC, although SASCB is more accurate than the others on average. In this paper, we present a novel method to reduce variability of SASCB and Boosting, and further increase their average accuracy. It generates multiple committees by incorporating Bagging into SASCB. As well as improving stability and average accuracy, the resulting method is amenable to parallel or distributed processing, while Boosting and SascB are not. This is an important characteristic for datamining in large datasets.
Zheng, Z.; Webb, G. I.
Stochastic Attribute Selection Committees
Antoniou, G.; Slaney, J. K. (Ed.): Lecture Notes in Computer Science Vol. 1502: Advanced Topics in Artificial Intelligence, Selected Papers from the Eleventh Australian Joint Conference on Artificial Intelligence (AI '98), pp. 321-332, Springer-Verlag, Brisbane, Australia, 1998.
@inproceedings{ZhengWebb98a,
title = {Stochastic Attribute Selection Committees},
author = {Z. Zheng and G. I. Webb},
editor = {G. Antoniou and J. K. Slaney},
year = {1998},
date = {1998-01-01},
booktitle = {Lecture Notes in Computer Science Vol. 1502: Advanced Topics in Artificial Intelligence, Selected Papers from the Eleventh Australian Joint Conference on Artificial Intelligence (AI '98)},
pages = {321-332},
publisher = {Springer-Verlag},
address = {Brisbane, Australia},
abstract = {Classifier committee learning methods generate multiple classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. Two such methods, Bagging and Boosting, have shown great success with decision tree learning. They create different classifiers by modifying the distribution of the training set. This paper studies a different approach: Stochastic Attribute Selection Committee learning of decision trees. It generates classifier committees by stochastically modifying the set of attributes but keeping the distribution of the training set unchanged. An empirical evaluation of a variant of this method, namely Sasc, in a representative collection of natural domains shows that the SASC method can significantly reduce the error rate of decision tree learning. On average Sasc is more accurate than Bagging and less accurate than Boosting, although a one-tailed significance test fails to show that these differences are significant at a level of 0.05. In addition, it is found that, like Bagging, Sasc is more stable than Boosting in terms of less frequently obtaining significantly higher error rates than C4.5 and, when error is raised, producing lower error rate increases. Moreover, like Bagging, Sasc is amenable to parallel and distributed processing while Boosting is not.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Classifier committee learning methods generate multiple classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. Two such methods, Bagging and Boosting, have shown great success with decision tree learning. They create different classifiers by modifying the distribution of the training set. This paper studies a different approach: Stochastic Attribute Selection Committee learning of decision trees. It generates classifier committees by stochastically modifying the set of attributes but keeping the distribution of the training set unchanged. An empirical evaluation of a variant of this method, namely Sasc, in a representative collection of natural domains shows that the SASC method can significantly reduce the error rate of decision tree learning. On average Sasc is more accurate than Bagging and less accurate than Boosting, although a one-tailed significance test fails to show that these differences are significant at a level of 0.05. In addition, it is found that, like Bagging, Sasc is more stable than Boosting in terms of less frequently obtaining significantly higher error rates than C4.5 and, when error is raised, producing lower error rate increases. Moreover, like Bagging, Sasc is amenable to parallel and distributed processing while Boosting is not.
Zheng, Z.; Webb, G. I.
Multiple Boosting: A Combination of Boosting and Bagging
Proceedings of the 1998 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'98), pp. 1133-1140, CSREA Press, Las Vegas, Nevada, 1998.
@inproceedings{ZhengWebb98b,
title = {Multiple Boosting: A Combination of Boosting and Bagging},
author = {Z. Zheng and G. I. Webb},
year = {1998},
date = {1998-01-01},
booktitle = {Proceedings of the 1998 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'98)},
pages = {1133-1140},
publisher = {CSREA Press},
address = {Las Vegas, Nevada},
abstract = {Classifier committee learning approaches have demonstrated great success in increasing the prediction accuracy of classifier learning, which is a key technique for datamining. These approaches generate several classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. It has been shown that Boosting and Bagging, as two representative methods of this type, can significantly decrease the error rate of decision tree learning. Boosting is generally more accurate than Bagging, but the former is more variable than the latter. In addition, bagging is amenable to parallel or distributed processing, while Boosting is not. In this paper, we study a new committee learning algorithm, namely MB (Multiple Boosting). It creates multiple subcommittees by combining Boosting and Bagging. Experimental results in a representative collection of natural domains show that MB is, on average, more accurate than either Bagging or Boosting alone. It is more stable than Boosting, and is amenable to parallel or distributed processing. These characteristics make MB a good choice for parallel datamining.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Classifier committee learning approaches have demonstrated great success in increasing the prediction accuracy of classifier learning, which is a key technique for datamining. These approaches generate several classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. It has been shown that Boosting and Bagging, as two representative methods of this type, can significantly decrease the error rate of decision tree learning. Boosting is generally more accurate than Bagging, but the former is more variable than the latter. In addition, bagging is amenable to parallel or distributed processing, while Boosting is not. In this paper, we study a new committee learning algorithm, namely MB (Multiple Boosting). It creates multiple subcommittees by combining Boosting and Bagging. Experimental results in a representative collection of natural domains show that MB is, on average, more accurate than either Bagging or Boosting alone. It is more stable than Boosting, and is amenable to parallel or distributed processing. These characteristics make MB a good choice for parallel datamining.
Zheng, Z.; Webb, G. I.; Ting, K. M.
Integrating Boosting and Stochastic Attribute Selection Committees for Further Improving The Performance of Decision Tree Learning
Proceedings of the Tenth IEEE International Conference on Tools with Artificial Intelligence (ICTAI-98), pp. 216-223, IEEE Computer Society Press, Taipei, Taiwan, 1998.
@inproceedings{ZhengWebbTing98,
title = {Integrating Boosting and Stochastic Attribute Selection Committees for Further Improving The Performance of Decision Tree Learning},
author = {Z. Zheng and G. I. Webb and K. M. Ting},
year = {1998},
date = {1998-01-01},
booktitle = {Proceedings of the Tenth IEEE International Conference on Tools with Artificial Intelligence (ICTAI-98)},
pages = {216-223},
publisher = {IEEE Computer Society Press},
address = {Taipei, Taiwan},
abstract = {Techniques for constructing classifier committees including boosting and bagging have demonstrated great success, especially boosting for decision tree learning. This type of technique generates several classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. Boosting and bagging create different classifiers by modifying the distribution of the training set. SASC (Stochastic Attribute Selection Committees) uses an alternative approach to generating classifier committees by stochastic manipulation of the set of attributes considered at each node during tree induction, but keeping the distribution of the training set unchanged. We propose a method for improving the performance of boosting. This technique combines boosting and SASC. It builds classifier committees by manipulating both the distribution of the training set and the set of attributes available during induction. In the synergy SASC effectively increases the model diversity of boosting. Experiments with a representative collection of natural domains show that, on average, the combined technique outperforms either boosting or SASC alone in terms of reducing the error rate of decision tree learning.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Techniques for constructing classifier committees including boosting and bagging have demonstrated great success, especially boosting for decision tree learning. This type of technique generates several classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. Boosting and bagging create different classifiers by modifying the distribution of the training set. SASC (Stochastic Attribute Selection Committees) uses an alternative approach to generating classifier committees by stochastic manipulation of the set of attributes considered at each node during tree induction, but keeping the distribution of the training set unchanged. We propose a method for improving the performance of boosting. This technique combines boosting and SASC. It builds classifier committees by manipulating both the distribution of the training set and the set of attributes available during induction. In the synergy SASC effectively increases the model diversity of boosting. Experiments with a representative collection of natural domains show that, on average, the combined technique outperforms either boosting or SASC alone in terms of reducing the error rate of decision tree learning.