Bias and variance provide a powerful conceptual tool for analyzing machine learning performance. My research revealed that as sample size increases, the ratio of variance relative to bias tends to decrease. As has since become widely accepted, this implies that low variance learning algorithms, such as linear models, should be most effective with small data quantity. For large data quantities, low bias learning algorithms, such as deep learning, should be most effective.
Previous approaches to conducting bias-variance experiments have provided little control over the types of data distribution from which bias and variance are estimated. I have developed new techniques for bias-variance analysis that provide greater control over the data distribution. Experiments show that the type of distribution used for bias-variance experiments can greatly affect the results obtained.
Publications
Webb, Geoffrey I; Conilione, Paul
Estimating bias and variance from data
2004, (Unpublished manuscript).
@unpublished{WebbConilione04,
title = {Estimating bias and variance from data},
author = {Geoffrey I Webb and Paul Conilione},
year = {2004},
date = {2004-01-01},
abstract = {The bias-variance decomposition of error provides useful insights into the error performance of a classifier as it is applied to di#erent types of learning task. Most notably, it has been used to explain the extraordinary e#ectiveness of ensemble learning techniques. It is important that the research community have e#ective tools for assessing such explanations. To this end, techniques have been developed for estimating bias and variance from data. The most widely deployed of these uses repeated sub-sampling with a holdout set. We argue, with empirical support, that this approach has serious limitations. First, it provides very little flexibility in the types of distributions of training sets that may be studied. It requires that the training sets be relatively small and that the degree of variation between training sets be very circumscribed. Second, the approach leads to bias and variance estimates that have high statistical variance and hence low reliability. We develop an alternative method that is based on cross-validation. We show that this method allows far greater flexibility in the types of distribution that are examined and that the estimates derived are much more stable. Finally, we show that changing the distributions of training sets from which bias and variance estimates are drawn can alter substantially the bias and variance estimates that are derived.},
note = {Unpublished manuscript},
keywords = {},
pubstate = {published},
tppubtype = {unpublished}
}
The bias-variance decomposition of error provides useful insights into the error performance of a classifier as it is applied to di#erent types of learning task. Most notably, it has been used to explain the extraordinary e#ectiveness of ensemble learning techniques. It is important that the research community have e#ective tools for assessing such explanations. To this end, techniques have been developed for estimating bias and variance from data. The most widely deployed of these uses repeated sub-sampling with a holdout set. We argue, with empirical support, that this approach has serious limitations. First, it provides very little flexibility in the types of distributions of training sets that may be studied. It requires that the training sets be relatively small and that the degree of variation between training sets be very circumscribed. Second, the approach leads to bias and variance estimates that have high statistical variance and hence low reliability. We develop an alternative method that is based on cross-validation. We show that this method allows far greater flexibility in the types of distribution that are examined and that the estimates derived are much more stable. Finally, we show that changing the distributions of training sets from which bias and variance estimates are drawn can alter substantially the bias and variance estimates that are derived.
Brain, D.; Webb, G. I.
The Need for Low Bias Algorithms in Classification Learning From Large Data Sets
Lecture Notes in Computer Science 2431: Principles of Data Mining and Knowledge Discovery: Proceedings of the Sixth European Conference (PKDD 2002), pp. 62-73, Springer-Verlag, Helsinki, Finland, 2002.
@inproceedings{BrainWebb02,
title = {The Need for Low Bias Algorithms in Classification Learning From Large Data Sets},
author = {D. Brain and G. I. Webb},
year = {2002},
date = {2002-01-01},
booktitle = {Lecture Notes in Computer Science 2431: Principles of Data Mining and Knowledge Discovery: Proceedings of the Sixth European Conference (PKDD 2002)},
pages = {62-73},
publisher = {Springer-Verlag},
address = {Helsinki, Finland},
abstract = {This paper reviews the appropriateness for application to large data sets of standard machine learning algorithms, which were mainly developed in the context of small data sets. Sampling and parallelization have proved useful means for reducing computation time when learning from large data sets. However, such methods assume that algorithms that were designed for use with what are now considered small data sets are also fundamentally suitable for large data sets. It is plausible that optimal learning from large data sets requires a different type of algorithm to optimal learning from small data sets. This paper investigates one respect in which data set size may affect the requirements of a learning algorithm the bias plus variance decomposition of classification error. Experiments show that learning from large data sets may be more effective when using an algorithm that places greater emphasis on bias management, rather than variance management},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper reviews the appropriateness for application to large data sets of standard machine learning algorithms, which were mainly developed in the context of small data sets. Sampling and parallelization have proved useful means for reducing computation time when learning from large data sets. However, such methods assume that algorithms that were designed for use with what are now considered small data sets are also fundamentally suitable for large data sets. It is plausible that optimal learning from large data sets requires a different type of algorithm to optimal learning from small data sets. This paper investigates one respect in which data set size may affect the requirements of a learning algorithm the bias plus variance decomposition of classification error. Experiments show that learning from large data sets may be more effective when using an algorithm that places greater emphasis on bias management, rather than variance management
Webb, G. I.
MultiBoosting: A Technique for Combining Boosting and Wagging
Machine Learning, vol. 40, no. 2, pp. 159-196, 2000.
@article{Webb00a,
title = {MultiBoosting: A Technique for Combining Boosting and Wagging},
author = {G. I. Webb},
doi = {10.1023/A:1007659514849},
year = {2000},
date = {2000-01-01},
journal = {Machine Learning},
volume = {40},
number = {2},
pages = {159-196},
publisher = {Springer},
address = {Netherlands},
abstract = {MultiBoosting is an extension to the highly successful AdaBoost technique for forming decision committees. MultiBoosting can be viewed as combining AdaBoost with wagging. It is able to harness both AdaBoost's high bias and variance reduction with wagging's superior variance reduction. Using C4.5 as the base learning algorithm, Multi-boosting is demonstrated to produce decision committees with lower error than either AdaBoost or wagging significantly more often than the reverse over a large representative cross-section of UCI data sets. It offers the further advantage over AdaBoost of suiting parallel execution.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
MultiBoosting is an extension to the highly successful AdaBoost technique for forming decision committees. MultiBoosting can be viewed as combining AdaBoost with wagging. It is able to harness both AdaBoost's high bias and variance reduction with wagging's superior variance reduction. Using C4.5 as the base learning algorithm, Multi-boosting is demonstrated to produce decision committees with lower error than either AdaBoost or wagging significantly more often than the reverse over a large representative cross-section of UCI data sets. It offers the further advantage over AdaBoost of suiting parallel execution.
Brain, D.; Webb, G. I.
On The Effect of Data Set Size on Bias And Variance in Classification Learning
Richards, D.; Beydoun, G.; Hoffmann, A.; Compton, P. (Ed.): Proceedings of the Fourth Australian Knowledge Acquisition Workshop (AKAW-99), pp. 117-128, The University of New South Wales, Sydney, Australia, 1999.
@inproceedings{BrainWebb99,
title = {On The Effect of Data Set Size on Bias And Variance in Classification Learning},
author = {D. Brain and G. I. Webb},
editor = {D. Richards and G. Beydoun and A. Hoffmann and P. Compton},
year = {1999},
date = {1999-01-01},
booktitle = {Proceedings of the Fourth Australian Knowledge Acquisition Workshop (AKAW-99)},
pages = {117-128},
publisher = {The University of New South Wales},
address = {Sydney, Australia},
abstract = {With the advent of data mining, machine learning has come of age and is now a critical technology in many businesses. However, machine learning evolved in a different research context to that in which it now finds itself employed. A particularly important problem in the data mining world is working effectively with large data sets. However, most machine learning research has been conducted in the context of learning from very small data sets. To date most approaches to scaling up machine learning to large data sets have attempted to modify existing algorithms to deal with large data sets in a more computationally efficient and effective manner. But is this necessarily the best method? This paper explores the possibility of designing algorithms specifically for large data sets. Specifically, the paper looks at how increasing data set size affects bias and variance error decompositions for classification algorithms. Preliminary results of experiments to determine these effects are presented, showing that, as hypothesized variance can be expected to decrease as training set size increases. No clear effect of training set size on bias was observed. These results have profound implications for data mining from large data sets, indicating that developing effective learning algorithms for large data sets is not simply a matter of finding computationally efficient variants of existing learning algorithms.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
With the advent of data mining, machine learning has come of age and is now a critical technology in many businesses. However, machine learning evolved in a different research context to that in which it now finds itself employed. A particularly important problem in the data mining world is working effectively with large data sets. However, most machine learning research has been conducted in the context of learning from very small data sets. To date most approaches to scaling up machine learning to large data sets have attempted to modify existing algorithms to deal with large data sets in a more computationally efficient and effective manner. But is this necessarily the best method? This paper explores the possibility of designing algorithms specifically for large data sets. Specifically, the paper looks at how increasing data set size affects bias and variance error decompositions for classification algorithms. Preliminary results of experiments to determine these effects are presented, showing that, as hypothesized variance can be expected to decrease as training set size increases. No clear effect of training set size on bias was observed. These results have profound implications for data mining from large data sets, indicating that developing effective learning algorithms for large data sets is not simply a matter of finding computationally efficient variants of existing learning algorithms.