Results 1  10
of
136
Probabilistic Verification of Discrete Event Systems using Acceptance Sampling
 In Proc. 14th International Conference on Computer Aided Verification, volume 2404 of LNCS
, 2002
"... We propose a model independent procedure for verifying properties of discrete event systems. The dynamics of such systems can be very complex, making them hard to analyze, so we resort to methods based on Monte Carlo simulation and statistical hypothesis testing. The verification is probabilistic in ..."
Abstract

Cited by 78 (10 self)
 Add to MetaCart
We propose a model independent procedure for verifying properties of discrete event systems. The dynamics of such systems can be very complex, making them hard to analyze, so we resort to methods based on Monte Carlo simulation and statistical hypothesis testing. The verification is probabilistic in two senses. First, the properties, expressed as CSL formulas, can be probabilistic. Second, the result of the verification is probabilistic, and the probability of error is bounded by two parameters passed to the verification procedure. The verification of properties can be carried out in an anytime manner by starting off with loose error bounds, and gradually tightening these bounds.
Numerical vs. statistical probabilistic model checking: An empirical study
 IN 10TH INTERNATIONAL CONFERENCE ON TOOLS AND ALGORITHMS FOR THE CONSTRUCTION AND ANALYSIS OF SYSTEMS (TACAS’04
, 2004
"... Numerical analysis based on uniformisation and statistical techniques based on sampling and simulation are two distinct approaches for transient analysis of stochastic systems. We compare the two solution techniques when applied to the verification of timebounded until formulae in the temporal st ..."
Abstract

Cited by 52 (8 self)
 Add to MetaCart
Numerical analysis based on uniformisation and statistical techniques based on sampling and simulation are two distinct approaches for transient analysis of stochastic systems. We compare the two solution techniques when applied to the verification of timebounded until formulae in the temporal stochastic logic CSL. This study differs from most previous comparisons of numerical and statistical approaches in that CSL model checking is a hypothesis testing problem rather than a parameter estimation problem. We can therefore rely on highly efficient sequential acceptance sampling tests, which enables statistical solution techniques to quickly return a result with some uncertainty. This suggests that statistical techniques can be useful as a first resort during system prototyping, rather than as a last resort as often suggested. We also propose a novel combination of the two solution techniques for verifying CSL queries with nested probabilistic operators.
Teaching dimension and the complexity of active learning
 In Proceedings of the 20th Conference on Learning Theory
, 2007
"... Abstract. We study the label complexity of poolbased active learning in the PAC model with noise. Taking inspiration from extant literature on Exact learning with membership queries, we derive upper and lower bounds on the label complexity in terms of generalizations of extended teaching dimension. ..."
Abstract

Cited by 27 (8 self)
 Add to MetaCart
Abstract. We study the label complexity of poolbased active learning in the PAC model with noise. Taking inspiration from extant literature on Exact learning with membership queries, we derive upper and lower bounds on the label complexity in terms of generalizations of extended teaching dimension. Among the contributions of this work is the first nontrivial general upper bound on label complexity in the presence of persistent classification noise. 1 Overview of Main Results In supervised machine learning, it is becoming increasingly apparent that welldesigned interactive learning algorithms can provide valuable improvements over passive algorithms in learning performance while reducing the amount of effort required of a human annotator. In particular, there is presently much interest in the poolbased active learning setting, in which a learner can request the label of any example in a large pool of unlabeled examples. In this case, one crucial quantity is the number of label requests required by a learning algorithm: the label complexity. This quantity is sometimes significantly smaller than the sample complexity of passive learning. A thorough theoretical understanding of these improvements seems essential to fully exploit the potential of active learning. In particular, active learning is formalized in the PAC model as follows. The pool of m unlabeled examples are sampled i.i.d. according to some distribution D. A binary label is assigned to each example by a (possibly randomized) oracle, but is hidden from the learner unless it requests the label. The error rate of a classifier h is defined as the probability of h disagreeing with the oracle on a fresh example X ∼ D. A learning algorithm outputs a classifier ˆ h from a concept space C, and we refer to the infimum error rate over classifiers in C as the noise rate, denoted ν. For ǫ,δ,η ∈ (0,1), we define the label complexity, denoted #LQ(C, D,ǫ,δ,η), as the smallest number q such that there is an algorithm that outputs a classifier ˆ h ∈ C, and for sufficiently large m, for any oracle with ν ≤ η, with probability at least 1 − δ over the sample and internal randomness, the algorithm makes at most q label requests and ˆ h has error rate at most ν + ǫ. 1
Online crowdsourcing: rating annotators and obtaining costeffective labels
 In W. on Advancing Computer Vision with Humans in the Loop
, 2010
"... Labeling large datasets has become faster, cheaper, and easier with the advent of crowdsourcing services like Amazon Mechanical Turk. How can one trust the labels obtained from such services? We propose a model of the labeling process which includes label uncertainty, as well a multidimensional mea ..."
Abstract

Cited by 23 (2 self)
 Add to MetaCart
Labeling large datasets has become faster, cheaper, and easier with the advent of crowdsourcing services like Amazon Mechanical Turk. How can one trust the labels obtained from such services? We propose a model of the labeling process which includes label uncertainty, as well a multidimensional measure of the annotators ’ ability. From the model we derive an online algorithm that estimates the most likely value of the labels and the annotator abilities. It finds and prioritizes experts when requesting labels, and actively excludes unreliable annotators. Based on labels already obtained, it dynamically chooses which images will be labeled next, and how many labels to request in order to achieve a desired level of confidence. Our algorithm is general and can handle binary, multivalued, and continuous annotations (e.g. bounding boxes). Experiments on a dataset containing more than 50,000 labels show that our algorithm reduces the number of labels required, and thus the total cost of labeling, by a large factor while keeping error rates low on a variety of datasets. 1.
RATES OF CONVERGENCE IN ACTIVE LEARNING
 SUBMITTED TO THE ANNALS OF STATISTICS
"... We study the rates of convergence in generalization error achievable by active learning under various types of label noise. Additionally, we study the general problem of model selection for active learning with a nested hierarchy of hypothesis classes, and propose an algorithm whose error rate prova ..."
Abstract

Cited by 23 (4 self)
 Add to MetaCart
We study the rates of convergence in generalization error achievable by active learning under various types of label noise. Additionally, we study the general problem of model selection for active learning with a nested hierarchy of hypothesis classes, and propose an algorithm whose error rate provably converges to the best achievable error among classifiers in the hierarchy at a rate adaptive to both the complexity of the optimal classifier and the noise conditions. In particular, we state sufficient conditions for these rates to be dramatically faster than those achievable by passive learning.
A Bayesian Approach to Model Checking Biological Systems ⋆
"... Abstract. Recently, there has been considerable interest in the use of Model Checking for Systems Biology. Unfortunately, the state space of stochastic biological models is often too large for classical Model Checking techniques. For these models, a statistical approach to Model Checking has been sh ..."
Abstract

Cited by 21 (9 self)
 Add to MetaCart
Abstract. Recently, there has been considerable interest in the use of Model Checking for Systems Biology. Unfortunately, the state space of stochastic biological models is often too large for classical Model Checking techniques. For these models, a statistical approach to Model Checking has been shown to be an effective alternative. Extending our earlier work, we present the first algorithm for performing statistical Model Checking using Bayesian Sequential Hypothesis Testing. We show that our Bayesian approach outperforms current statistical Model Checking techniques, which rely on tests from Classical (aka Frequentist) statistics, by requiring fewer system simulations. Another advantage of our approach is the ability to incorporate prior Biological knowledge about the model being verified. We demonstrate our algorithm on a variety of models from the Systems Biology literature and show that it enables faster verification than stateoftheart techniques, even when no prior knowledge is available. 1
TaskDriven Tactile Exploration
"... Abstract — This paper presents a decisiontheoretic approach to problems that require accurate placement of a robot relative to an object in the world, including grasping and insertion. The decision process is applied to a robot hand with tactile sensors, to localize the object and ultimately achiev ..."
Abstract

Cited by 19 (2 self)
 Add to MetaCart
Abstract — This paper presents a decisiontheoretic approach to problems that require accurate placement of a robot relative to an object in the world, including grasping and insertion. The decision process is applied to a robot hand with tactile sensors, to localize the object and ultimately achieve a target placement by selecting among grasping and informationgathering trajectories. The process is demonstrated in simulation and on a real robot. I.
Statistical Model Checking in BioLab: Applications to the automated analysis of TCell Receptor Signaling Pathway ⋆
"... Abstract. We present an algorithm, called BioLab, for verifying temporal properties of rulebased models of cellular signalling networks. BioLab models are encoded in the BioNetGen language, and properties are expressed as formulae in probabilistic bounded linear temporal logic. Temporal logic is a ..."
Abstract

Cited by 16 (5 self)
 Add to MetaCart
Abstract. We present an algorithm, called BioLab, for verifying temporal properties of rulebased models of cellular signalling networks. BioLab models are encoded in the BioNetGen language, and properties are expressed as formulae in probabilistic bounded linear temporal logic. Temporal logic is a formalism for representing and reasoning about propositions qualified in terms of time. Properties are then verified using sequential hypothesis testing on executions generated using stochastic simulation. BioLab is optimal, in the sense that it generates the minimum number of executions necessary to verify the given property. BioLab also provides guarantees on the probability of it generating TypeI (i.e., falsepositive) and TypeII (i.e., falsenegative) errors. Moreover, these error bounds are prespecified by the user. We demonstrate BioLab by verifying stochastic effects and bistability in the dynamics of the Tcell receptor signaling network. 1
Bayesian Statistical Model Checking with Application to Stateflow/Simulink Verification
, 2010
"... We address the problem of model checking stochastic systems, i.e. checking whether a stochastic system satisfies a certain temporal property with a probability greater (or smaller) than a fixed threshold. In particular, we present a novel Statistical Model Checking (SMC) approach based on Bayesian s ..."
Abstract

Cited by 15 (5 self)
 Add to MetaCart
We address the problem of model checking stochastic systems, i.e. checking whether a stochastic system satisfies a certain temporal property with a probability greater (or smaller) than a fixed threshold. In particular, we present a novel Statistical Model Checking (SMC) approach based on Bayesian statistics. We show that our approach is feasible for hybrid systems with stochastic transitions, a generalization of Simulink/Stateflow models. Standard approaches to stochastic (discrete) systems require numerical solutions for large optimization problems and quickly become infeasible with larger state spaces. Generalizations of these techniques to hybrid systems with stochastic effects are even more challenging. The SMC approach was pioneered by Younes and Simmons in the discrete and nonBayesian case. It solves the verification problem by combining randomized sampling of system traces (which is very efficient for Simulink/Stateflow) with hypothesis testing or estimation. We believe SMC is essential for scaling up to large Stateflow/Simulink models. While the answer to the verification problem is not guaranteed to be correct, we prove that Bayesian SMC can make the probability of giving a wrong answer arbitrarily small. The advantage is that answers can usually be obtained much faster than with standard, exhaustive model checking