The Knowledge Factory is an interactive machine learning and data analytics environment (also known as human-in-the-loop machine learning, mixed-initiative learning and human-computer collaborative learning) that provides tight integration between machine learning and knowledge acquisition from experts. This pioneering research program included one of the few detailed experimental demonstrations that human-in-the-loop machine learning can outperform both autonomous machine learning and an expert specifying rules without the assistance of machine learning.
Software
The Knowledge Factory is a Macintosh System 4 to 7 application.
The Knowledge Factory is best run under Mac OS 7.6. It runs successfully under the Basilisk II Mac emulator on Windows.
TKF.sea is a self extracting archive. Double click on the TKF.sea icon and follow the instructions.
The archive contains the following files:
- The Knowledge Factory - application
- manual.hdr, manual.dat, manual,evl - data files used within the Manual.pdf
- hypo.hdr, hypo.dat, hypo.evl - data files containing 100 hypothyroid cases
- ReadMe - a text file containing features of the application not documented withing the Manual.pdf
- Manual.pdf - Adobe's Portable document format requires the Adobe Acrobat reader
The Knowledge Factory:
- requires no special installation. Simply double click on the The Knowledge Factory icon and the system becomes operational.
- will run on any Macintosh computer operating under System 4.1 or higher so long as sufficient RAM is available.
- is initially configured to request 1000K when it is run under Multifinder. This should normally be sufficient for projects examining up to 4000 example cases.
If The Knowledge Factory displays a message during operation stating that it is unable to complete an action due to a shortage of memory, you should increase the application memory size by:
- saving the current state of the project;
- quitting from The Knowledge Factory;
- selecting the The Knowledge Factory icon;
- choosing Get Info from the Finder’s File Menu; and
- entering the desired value in the Preferred Size box.
Publications
Webb, G. I.
Integrating Machine Learning with Knowledge Acquisition
Leondes, C. T. (Ed.): Expert Systems, vol. 3, pp. 937-959, Academic Press, San Diego, CA, 2002.
@incollection{Webb02,
title = {Integrating Machine Learning with Knowledge Acquisition},
author = {G. I. Webb},
editor = {C. T. Leondes},
year = {2002},
date = {2002-01-01},
booktitle = {Expert Systems},
volume = {3},
pages = {937-959},
publisher = {Academic Press},
address = {San Diego, CA},
keywords = {},
pubstate = {published},
tppubtype = {incollection}
}
Webb, G. I.; Wells, J.; Zheng, Z.
An Experimental Evaluation of Integrating Machine Learning with Knowledge Acquisition
Machine Learning, vol. 35, no. 1, pp. 5-24, 1999.
@article{WebbWellsZheng99,
title = {An Experimental Evaluation of Integrating Machine Learning with Knowledge Acquisition},
author = {G. I. Webb and J. Wells and Z. Zheng},
year = {1999},
date = {1999-01-01},
journal = {Machine Learning},
volume = {35},
number = {1},
pages = {5-24},
publisher = {Springer},
address = {Netherlands},
abstract = {Machine learning and knowledge acquisition from experts have distinct capabilities that appear to complement one another. We report a study that demonstrates the integration of these approaches can both improve the accuracy of the developed knowledge base and reduce development time. In addition, we found that users expected the expert systems created through the integrated approach to have higher accuracy than those created without machine learning and rated the integrated approach less difficult to use. They also provided favorable evaluations of both the specific integrated software, system called The Knowledge Factory, and of the general value of machine learning for knowledge acquisition.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Machine learning and knowledge acquisition from experts have distinct capabilities that appear to complement one another. We report a study that demonstrates the integration of these approaches can both improve the accuracy of the developed knowledge base and reduce development time. In addition, we found that users expected the expert systems created through the integrated approach to have higher accuracy than those created without machine learning and rated the integrated approach less difficult to use. They also provided favorable evaluations of both the specific integrated software, system called The Knowledge Factory, and of the general value of machine learning for knowledge acquisition.
Webb, G. I.; Wells, J.
An Experimental Evaluation of Integrating Machine Learning with Knowledge Acquisition Through Direct Interaction with Domain Experts
Compton, P.; Mizoguchi, R.; Motada, H.; Menzies, T. (Ed.): Proceedings of the 1996 Pacific Knowledge Acquisition Workshop (PKAW'96), pp. 170-189, UNSW Press, Coogee, Sydney, Australia, 1996.
@inproceedings{WebbWells96,
title = {An Experimental Evaluation of Integrating Machine Learning with Knowledge Acquisition Through Direct Interaction with Domain Experts},
author = {G. I. Webb and J. Wells},
editor = {P. Compton and R. Mizoguchi and H. Motada and T. Menzies},
year = {1996},
date = {1996-01-01},
booktitle = {Proceedings of the 1996 Pacific Knowledge Acquisition Workshop (PKAW'96)},
pages = {170-189},
publisher = {UNSW Press},
address = {Coogee, Sydney, Australia},
abstract = {Machine learning and knowledge acquisition from experts have distinct and apparently complementary knowledge acquisition capabilities. This study demonstrates that the integration of these approaches can both improve the accuracy of the knowledge base that is developed and reduce the time taken to develop it. The system studied, called The Knowledge Factory is distinguished by the manner in which it supports direct interaction with domain experts with little or no knowledge engineering expertise. The benefits reported relate to use by such users. In addition to the improved quality of the knowledge base, in questionnaire responses the users provided favourable evaluations of the integration of machine learning with knowledge acquisition within the system.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Machine learning and knowledge acquisition from experts have distinct and apparently complementary knowledge acquisition capabilities. This study demonstrates that the integration of these approaches can both improve the accuracy of the knowledge base that is developed and reduce the time taken to develop it. The system studied, called The Knowledge Factory is distinguished by the manner in which it supports direct interaction with domain experts with little or no knowledge engineering expertise. The benefits reported relate to use by such users. In addition to the improved quality of the knowledge base, in questionnaire responses the users provided favourable evaluations of the integration of machine learning with knowledge acquisition within the system.
Webb, G. I.
Integrating Machine Learning With Knowledge Acquisition Through Direct Interaction With Domain Experts
Knowledge-Based Systems, vol. 9, pp. 253-266, 1996.
@article{Webb96a,
title = {Integrating Machine Learning With Knowledge Acquisition Through Direct Interaction With Domain Experts},
author = {G. I. Webb},
doi = {10.1016/0950-7051(96)01033-7},
year = {1996},
date = {1996-01-01},
journal = {Knowledge-Based Systems},
volume = {9},
pages = {253-266},
publisher = {Elsevier},
abstract = {Knowledge elicitation from experts and empirical machine learning are two distinct approaches to knowledge acquisition with differing and mutually complementary capabilities. Learning apprentices have provided environments in which a knowledge engineer may collaborate with a machine learning system allowing, for a synergy between the complementary approaches. The Knowledge Factory is a knowledge acquisition environment that allows a domain expert to collaborate directly with a machine learning system without the need for assistance from a knowledge engineer. This requires a different form of environment to the learning apprentice. This paper describes techniques for supporting such interactions and their implementation in a knowledge acquisition environment called The Knowledge Factory.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Knowledge elicitation from experts and empirical machine learning are two distinct approaches to knowledge acquisition with differing and mutually complementary capabilities. Learning apprentices have provided environments in which a knowledge engineer may collaborate with a machine learning system allowing, for a synergy between the complementary approaches. The Knowledge Factory is a knowledge acquisition environment that allows a domain expert to collaborate directly with a machine learning system without the need for assistance from a knowledge engineer. This requires a different form of environment to the learning apprentice. This paper describes techniques for supporting such interactions and their implementation in a knowledge acquisition environment called The Knowledge Factory.
Webb, G. I.; Wells, J.
Recent Progress in Machine-Expert Collaboration for Knowledge Acquisition
Yao, X. (Ed.): Proceedings of the Eighth Australian Joint Conference on Artificial Intelligence (AI'95), pp. 291-298, World Scientific, Canberra, Australia, 1995.
@inproceedings{WebbWells95,
title = {Recent Progress in Machine-Expert Collaboration for Knowledge Acquisition},
author = {G. I. Webb and J. Wells},
editor = {X. Yao},
year = {1995},
date = {1995-01-01},
booktitle = {Proceedings of the Eighth Australian Joint Conference on Artificial Intelligence (AI'95)},
pages = {291-298},
publisher = {World Scientific},
address = {Canberra, Australia},
abstract = {Knowledge acquisition remains one of the primary constraints on the development of expert systems. A number of researchers have explored methods for allowing a machine learning system to assist a knowledge engineer in knowledge acquisition. In contrast, we are exploring methods for enabling an expert to directly interact with a machine learning system to collaborate during knowledge acquisition. We report recent extensions to our methodology encompassing a revised model of the role of machine learning in knowledge acquisition; techniques for communication between a machine learning system and a domain expert and novel forms of assistance that a machine learning system may provide to an expert.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Knowledge acquisition remains one of the primary constraints on the development of expert systems. A number of researchers have explored methods for allowing a machine learning system to assist a knowledge engineer in knowledge acquisition. In contrast, we are exploring methods for enabling an expert to directly interact with a machine learning system to collaborate during knowledge acquisition. We report recent extensions to our methodology encompassing a revised model of the role of machine learning in knowledge acquisition; techniques for communication between a machine learning system and a domain expert and novel forms of assistance that a machine learning system may provide to an expert.
Webb, G. I.
Control, Capabilities and Communication: Three Key Issues for Machine-Expert Collaborative Knowledge Acquisition
Aussenac, N.; Boy, G.; Gaines, B.; Linster, M.; Ganascia, J. G.; Kodratoff, Y. (Ed.): Proceedings (Complement) of the Seventh European Workshop on Knowledge Acquisition for Knowledge-based Systems (EWKA'93), pp. 263-275, Springer-Verlag, Toulouse, France, 1993.
@inproceedings{Webb93e,
title = {Control, Capabilities and Communication: Three Key Issues for Machine-Expert Collaborative Knowledge Acquisition},
author = {G. I. Webb},
editor = {N. Aussenac and G. Boy and B. Gaines and M. Linster and J. G. Ganascia and Y. Kodratoff},
year = {1993},
date = {1993-01-01},
booktitle = {Proceedings (Complement) of the Seventh European Workshop on Knowledge Acquisition for Knowledge-based Systems (EWKA'93)},
pages = {263-275},
publisher = {Springer-Verlag},
address = {Toulouse, France},
abstract = {Machine learning and knowledge elicitation are different but complementary approaches to knowledge acquisition. On the face of it there are large potential gains to be reaped from the integration of these two knowledge acquisition techniques. Machine-expert collaborative knowledge acquisition combines these approaches by placing the machine learning system and the human expert as partners in the knowledge-acquisition task. This paper examines three key issues facing machine-expert collaborative knowledge-acquisition where should control reside, what capabilities should each partner bring to the task and how should the partners communicate?},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Machine learning and knowledge elicitation are different but complementary approaches to knowledge acquisition. On the face of it there are large potential gains to be reaped from the integration of these two knowledge acquisition techniques. Machine-expert collaborative knowledge acquisition combines these approaches by placing the machine learning system and the human expert as partners in the knowledge-acquisition task. This paper examines three key issues facing machine-expert collaborative knowledge-acquisition where should control reside, what capabilities should each partner bring to the task and how should the partners communicate?
Webb, G. I.
Man-Machine Collaboration for Knowledge Acquisition
Adams, A.; L., Sterling. (Ed.): Proceedings of the Fifth Australian Joint Conference on Artificial Intelligence (AI'92), pp. 329-334, World Scientific, Hobart, Tas., Australia, 1992.
@inproceedings{Webb92,
title = {Man-Machine Collaboration for Knowledge Acquisition},
author = {G. I. Webb},
editor = {A. Adams and Sterling. L.},
year = {1992},
date = {1992-01-01},
booktitle = {Proceedings of the Fifth Australian Joint Conference on Artificial Intelligence (AI'92)},
pages = {329-334},
publisher = {World Scientific},
address = {Hobart, Tas., Australia},
abstract = {Both machine learning and knowledge elicitation from human experts have unique strengths and weaknesses. Man-machine collaboration for knowledge acquisition allows both knowledge acquisition techniques to be employed hand- in-hand. The strengths of each can alleviate the other's weaknesses. This has the potential to both reduce the time taken to develop an expert system while increasing the quality of the finished product. This paper discusses techniques for man-machine collaboration for knowledge acquisition and describes Einstein, a computer system that implements those techniques},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Both machine learning and knowledge elicitation from human experts have unique strengths and weaknesses. Man-machine collaboration for knowledge acquisition allows both knowledge acquisition techniques to be employed hand- in-hand. The strengths of each can alleviate the other's weaknesses. This has the potential to both reduce the time taken to develop an expert system while increasing the quality of the finished product. This paper discusses techniques for man-machine collaboration for knowledge acquisition and describes Einstein, a computer system that implements those techniques
Webb, G. I.
Einstein: An Interactive Inductive Knowledge-Acquisition Tool
Proceedings of the Sixth Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, pp. (36)1-16, Banff, Canada, 1991.
@inproceedings{Webb91c,
title = {Einstein: An Interactive Inductive Knowledge-Acquisition Tool},
author = {G. I. Webb},
year = {1991},
date = {1991-01-01},
booktitle = {Proceedings of the Sixth Banff Knowledge Acquisition for Knowledge-Based Systems Workshop},
pages = {(36)1-16},
address = {Banff, Canada},
abstract = {Einstein is a knowledge acquisition system that incorporates data-driven inductive rule development and refinement in a user driven production rule development and evaluation environment. This allows the user and the induction system to interact as a cooperative knowledge acquisition team. Unique features of this system include efficient automated inductive refinement of existing production rules, interactive user management of machine learning facilities, including local and global guidance, interactive specification of key examples and counter-examples and interactive case-based rule assessment.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Einstein is a knowledge acquisition system that incorporates data-driven inductive rule development and refinement in a user driven production rule development and evaluation environment. This allows the user and the induction system to interact as a cooperative knowledge acquisition team. Unique features of this system include efficient automated inductive refinement of existing production rules, interactive user management of machine learning facilities, including local and global guidance, interactive specification of key examples and counter-examples and interactive case-based rule assessment.