In practical classification, there is often a mix of learnable and unlearnable classes and only a classifier above a minimum performance threshold can be deployed. This problem is exacerbated if the training set is created by active learning. The bias of actively learned training sets makes it hard to determine whether a class has been learned. We give evidence that there is no general and efficient method for reducing the bias and correctly identifying classes that have been learned. However, we characterize a number of scenarios where active learning can succeed despite these difficulties.

This paper addresses the adjustment of the language model (LM) scaling factor of an automatic speech recognition (ASR) system for a new domain using only un-transcribed speech. The main idea is to replace the (unavailable) reference transcript with an automatic transcript generated by an independent ASR system, and adjust parameters using this sloppy reference. It is shown that despite its fairly high error rate (ca. 35%), choosing the scaling factor to minimize disagreement with the erroneous transcripts is still an effective recipe for model selection. This effectiveness is demonstrated by adjusting an ASR system trained on Broadcast News to transcribe the MIT Lectures corpus. An ASR system for telephone speech produces the sloppy reference, and optimizing towards it yields a nearly optimal LM scaling factor for the MIT Lectures corpus. Index Terms: LVCSR, language modeling, domain adaptation, semi-supervised learning, unsupervised learning

functional atlas for the human brain

An important component for making semi-supervised learning applicable to real world data is the task of model selection. For the case of very limited labeled data, for which semi-supervised learning algorithms have the greatest potential to offer improvement in estimating predictive models, model selection is a significant challenge, a key open problem, and often avoided entirely in previous work. While previous work has demonstrated the benefit of semi-supervised learning in cases of very limited labeled training data, in order for such results to be achievable in practice, some effective method of selecting the hyper-parameters for these methods is necessary. In general, existing approaches rely heavily in some way on the labeled data directly for estimating either performance (e.g., error), some key characteristics of the model, or likelihood, and so can suffer when there is not much labeled data. Instead we propose an alternative, sampling approach in order to estimate model performance. The main idea is to evaluate the models on a large number of generated similar data sets, and to prefer those models that perform well on average across the data sets. New training and unlabeled/test data are generated by sampling from the large amount of unlabeled data and estimated conditional probabilities for the labels. Since these data sets are complete with labels, models can then be evaluated using the generated labels for the much larger set of unlabeled/test data. Using a variety of data sets we demonstrate the effectiveness of our approach, and, for small amounts of labeled data, large improvement over traditional methods like cross-validation, as well as better performance on average than the state-of-the-art for semi-supervised model selection. 1