We discuss basic prediction theory and it's impact on classification success evaluation, implications for learning algorithm design, and uses in learning algorithm execution. This tutorial is meant to be a comprehensive compilation of results which are both theoretically rigorous and practically useful. There are two important implications...

This paper is about bounds on future error rates. We present a theorem for combining an arbitrary test set based bound with an arbitrary training set based bound. Appropriate use of this theorem results in a combined bound with two properties: 1) the combined bound is never much worse than either the training set based bound or the test set based bound and 2) the combined bound is sometimes better than either bound individually. Empirical validation is presented showing the effectiveness of the combined bound.

This document is primarily about the theory of sample complexity for answering the question "Have we learned?". However, we do not neglect the experimental side. In particular, following the theory we will present results for application of sample complexity bounds to machine learning problems. These results are the 'best known results' in terms of bound tightness and should be considered as a guide and challenge to others working on sample complexity bounds