Manipulation of generality, through appropriate generalization and specialization, can modify classifier performance in predictable and useful ways.
Publications
Webb, G. I.; Brain, D.
Generality is Predictive of Prediction Accuracy
LNAI State-of-the-Art Survey series, 'Data Mining: Theory, Methodology, Techniques, and Applications', pp. 1-13, Springer, Berlin/Heidelberg, 2006, (An earlier version of this paper was published in the Proceedings of PKAW 2002, pp 117-130).
@inproceedings{WebbBrain05,
title = {Generality is Predictive of Prediction Accuracy},
author = {G. I. Webb and D. Brain},
year = {2006},
date = {2006-01-01},
booktitle = {LNAI State-of-the-Art Survey series, 'Data Mining: Theory, Methodology, Techniques, and Applications'},
pages = {1-13},
publisher = {Springer},
address = {Berlin/Heidelberg},
abstract = {During knowledge acquisition it frequently occurs that multiple alternative potential rules all appear equally credible. This paper addresses the dearth of formal analysis about how to select between such alternatives. It presents two hypotheses about the expected impact of selecting between classification rules of differing levels of generality in the absence of other evidence about their likely relative performance on unseen data. We argue that the accuracy on unseen data of the more general rule will tend to be closer to that of a default rule for the class than will that of the more specific rule. We also argue that in comparison to the more general rule, the accuracy of the more specific rule on unseen cases will tend to be closer to the accuracy obtained on training data. Experimental evidence is provided in support of these hypotheses. These hypotheses can be useful for selecting between rules in order to achieve specific knowledge acquisition objectives.},
note = {An earlier version of this paper was published in the Proceedings of PKAW 2002, pp 117-130},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
During knowledge acquisition it frequently occurs that multiple alternative potential rules all appear equally credible. This paper addresses the dearth of formal analysis about how to select between such alternatives. It presents two hypotheses about the expected impact of selecting between classification rules of differing levels of generality in the absence of other evidence about their likely relative performance on unseen data. We argue that the accuracy on unseen data of the more general rule will tend to be closer to that of a default rule for the class than will that of the more specific rule. We also argue that in comparison to the more general rule, the accuracy of the more specific rule on unseen cases will tend to be closer to the accuracy obtained on training data. Experimental evidence is provided in support of these hypotheses. These hypotheses can be useful for selecting between rules in order to achieve specific knowledge acquisition objectives.
Webb, G. I.; Brain, D.
Generality is Predictive of Prediction Accuracy
Yamaguchi, T.; Hoffmann, A.; Motoda, H.; Compton, P. (Ed.): Proceedings of the 2002 Pacific Rim Knowledge Acquisition Workshop (PKAW'02), pp. 117-130, Japanese Society for Artificial Intelligence, Tokyo, Japan, 2002.
@inproceedings{WebbBrain02,
title = {Generality is Predictive of Prediction Accuracy},
author = {G. I. Webb and D. Brain},
editor = {T. Yamaguchi and A. Hoffmann and H. Motoda and P. Compton},
year = {2002},
date = {2002-01-01},
booktitle = {Proceedings of the 2002 Pacific Rim Knowledge Acquisition Workshop (PKAW'02)},
pages = {117-130},
publisher = {Japanese Society for Artificial Intelligence},
address = {Tokyo, Japan},
abstract = {There has been a dearth of research into the relative impacts of alternative high level learning biases. This paper presents two hypotheses about the expected impact of selecting between classification rules of differing levels of generality in the absence of other evidence about their likely relative performance on unseen data. It is argued that the accuracy on unseen data of the more general rule will tend to be closer to that of a default rule for the class than will that of the more specific rule. It is also argued that the accuracy on unseen cases of the more specific rule will tend to be closer to the accuracy obtained on training data than will the accuracy of the more general rule. Experimental evidence is provided in support of these hypotheses. We argue that these hypotheses can be of use in selecting appropriate learning biases to achieve specific learning objectives.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
There has been a dearth of research into the relative impacts of alternative high level learning biases. This paper presents two hypotheses about the expected impact of selecting between classification rules of differing levels of generality in the absence of other evidence about their likely relative performance on unseen data. It is argued that the accuracy on unseen data of the more general rule will tend to be closer to that of a default rule for the class than will that of the more specific rule. It is also argued that the accuracy on unseen cases of the more specific rule will tend to be closer to the accuracy obtained on training data than will the accuracy of the more general rule. Experimental evidence is provided in support of these hypotheses. We argue that these hypotheses can be of use in selecting appropriate learning biases to achieve specific learning objectives.
Webb, G. I.
Cost Sensitive Specialisation
Foo, N. Y.; Goebel, R. (Ed.): Lecture Notes in Computer Science Vol. 1114. Topics in Artificial Intelligence: Proceedings of the Fourth Pacific Rim International Conference on Artificial Intelligence (PRICAI'96), pp. 23-34, Springer-Verlag, Cairns, Australia, 1996.
@inproceedings{Webb96d,
title = {Cost Sensitive Specialisation},
author = {G. I. Webb},
editor = {N. Y. Foo and R. Goebel},
year = {1996},
date = {1996-01-01},
booktitle = {Lecture Notes in Computer Science Vol. 1114. Topics in Artificial Intelligence: Proceedings of the Fourth Pacific Rim International Conference on Artificial Intelligence (PRICAI'96)},
pages = {23-34},
publisher = {Springer-Verlag},
address = {Cairns, Australia},
abstract = {Cost-sensitive specialization is a generic technique for misclassification cost sensitive induction. This technique involves specializing aspects of a classifier associated with high misclassification costs and generalizing those associated with low misclassification costs. It is widely applicable and simple to implement. It could be used to augment the effect of standard cost-sensitive induction techniques. It should directly extend to test application cost sensitive induction tasks. Experimental evaluation demonstrates consistent positive effects over a range of misclassification cost sensitive learning tasks.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Cost-sensitive specialization is a generic technique for misclassification cost sensitive induction. This technique involves specializing aspects of a classifier associated with high misclassification costs and generalizing those associated with low misclassification costs. It is widely applicable and simple to implement. It could be used to augment the effect of standard cost-sensitive induction techniques. It should directly extend to test application cost sensitive induction tasks. Experimental evaluation demonstrates consistent positive effects over a range of misclassification cost sensitive learning tasks.
Webb, G. I.
Generality Is More Significant Then Complexity: Toward An Alternative To Occams Razor
Zhang, C.; Debenham, J.; Lukose, D. (Ed.): Artificial Intelligence: Sowing the Seeds for the Future, Proceedings of Seventh Australian Joint Conference on Artificial Intelligence (AI'94), pp. 60-67, World Scientific, Armidale,NSW, Australia, 1994.
@inproceedings{Webb94b,
title = {Generality Is More Significant Then Complexity: Toward An Alternative To Occams Razor},
author = {G. I. Webb},
editor = {C. Zhang and J. Debenham and D. Lukose},
year = {1994},
date = {1994-01-01},
booktitle = {Artificial Intelligence: Sowing the Seeds for the Future, Proceedings of Seventh Australian Joint Conference on Artificial Intelligence (AI'94)},
pages = {60-67},
publisher = {World Scientific},
address = {Armidale,NSW, Australia},
abstract = {Occam's Razor is widely employed in machine learning to select between classifiers with equal empirical support. This paper presents the theorem of decreasing inductive power: that, all other things being equal, if two classifiers a and b cover identical cases from the training set and a is a generalisation of b, a has higher probability than b of misclassifying a previously unsighted case. This theorem suggests that, to the contrary of Occam's Razor, generality, not complexity, should be used to select between classifiers with equal empirical support. Two studies are presented. The first study demonstrates that the theorem of decreasing inductive power holds for a number of commonly studied learning problems and for a number of different means of manipulating classifier generality. The second study demonstrates that generality provides a more consistent indicator of predictive accuracy in the context of a default rule than does complexity. These results suggest that the theorem of decreasing predictive power provides a suitable theoretical framework for the development of learning biases for use in selecting between classifiers with identical empirical support},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Occam's Razor is widely employed in machine learning to select between classifiers with equal empirical support. This paper presents the theorem of decreasing inductive power: that, all other things being equal, if two classifiers a and b cover identical cases from the training set and a is a generalisation of b, a has higher probability than b of misclassifying a previously unsighted case. This theorem suggests that, to the contrary of Occam's Razor, generality, not complexity, should be used to select between classifiers with equal empirical support. Two studies are presented. The first study demonstrates that the theorem of decreasing inductive power holds for a number of commonly studied learning problems and for a number of different means of manipulating classifier generality. The second study demonstrates that generality provides a more consistent indicator of predictive accuracy in the context of a default rule than does complexity. These results suggest that the theorem of decreasing predictive power provides a suitable theoretical framework for the development of learning biases for use in selecting between classifiers with identical empirical support