this article we describe a Markov method that we developed specifically for process discovery, as well as describe two additional methods that we adopted from other domains and augmented for our purposes. The three methods range from the purely algorithmic to the purely statistical. We compare the methods and discuss their application in an industrial case study.

Empirical evaluation of software systems in actual usage situations is critical in software engineering. Prototyping, beta testing, and usability testing are widely used to refine system requirements, detect anomalous or unexpected system and user behavior, and to evaluate software usefulness and usability. The World Wide Web enables cheap, rapid, and large-scale distribution of software for evaluation purposes. However, current techniques for collecting usage data have not kept pace with the opportunities presented by Web-based deployment. This paper presents an approach and prototype system that makes large-scale collection of usage data over the Internet a practical possibility. A general framework for comparing software monitoring systems is presented and used to compare the proposed approach to existing techniques.

this article.

Process studies and improvement efforts typically call for new instrumentation on the process in order to collect the data they have deemed necessary. This can be intrusive and expensive, and resistance to the extra workload often foils the study before it begins. The result is neither interesting new knowledge nor an improved process. In many organizations, however, extensive historical process and product data already exist. Can these existing data be used to empirically explore what process factors might be affecting the outcome of the process? If they can, organizations would have a cost-effective method for quantitatively, if not causally, understanding their process and its relationship to the product. We present a case study that analyzes an in-place industrial process and takes advantage of existing data sources. In doing this, we also illustrate and propose a methodology for such exploratory empirical studies. The case study makes use of several readily available repositories of process data in the industrial organization. Our results show that readily available data can be used to correlate both simple aggregate metrics and complex process metrics with defects in the product. Through the case study, we give evidence supporting the claim that exploratory empirical studies can provide significant results and benefits while being cost effective in their demands on the organization.

Empirical evaluation of software systems in actual use is critical in software engineering. Prototyping, beta testing, and usability testing are widely employed to refine system requirements, to detect anomalous or unexpected system and user behavior, and to evaluate software usefulness and usability. The World Wide Web enables cheap, rapid, and large-scale distribution of software for evaluation purposes. However, current techniques for collecting usage data have not kept pace with the opportunities presented by Webbased deployment. This paper presents an agent-based approach and prototype system that makes large-scale collection of usage data over the Internet a practical possibility.

Understanding the dynamic behavior of a workflow is crucial for being able to modify, maintain, and improve it. A particularly difficult aspect of some behavior is concurrency. Automated techniques which seek to mine workflow data logs to discover information about the workflows must be able to handle the concurrency that manifests itself in the workflow executions. This paper presents techniques to discover patterns of concurrent behavior from traces of workflow events. The techniques are based on a probabilistic analysis of the event traces. Using metrics for the number, frequency, and regularity of event occurrences, a determination is made of the likely concurrent behavior being manifested by the system. Discovering this behavior can help a workflow designer better understand and improve the work processes they are managing.

Process discovery has been shown to be challenging offering limited results, however, most work has been conducted in closed source systems. This paper describes a new approach to process discovery that examines the Internet information spaces of open source software development projects. In searching for an automated solution to the process discovery problem, we first have simulated it by having a human act as an “intelligent spider” searching the Web space for evidence of process activities and reconstructing process fragments based on the clues discovered. The purpose of such an approach is to help reveal the details of our manual process discovery approach. In turn, such knowledge can then be employed to determine the requirements and design of automated process discovery and modeling mechanisms that can be applied to Web-based software development projects.

In order to investigate practitioners ' opinions of software process and software process improvement, we have collected information from 13 companies, in a variety of ways i.e. the use of Repertory Grid Technique, survey and focus group discussions. Both the Repertory Grid Technique and the focus group discussions (43 discussions occurred, in total) produced a large volume of qualitative data. At the same time, other researchers have reported investigations of practitioners, and we are interested in how their reports may relate to our own. Thus, other research publications can also be treated as a form of qualitative data. In this paper, we review advice on a method, content analysis, that is used to analyse qualitative data. Content analysis is a method for identifying and classifying words and phrases used in ordinary language. We use content analysis to describe and analyse discussions on software process and software process improvement. We report preliminary findings from an analysis of both the focus group evidence and some publications. Our main finding is that there is an apparent contradiction between developers saying that they want evidence for software process improvement, and what developers will accept as evidence. This presents a serious problem for research: even if researchers could demonstrate a strong, reliable relationship between software process improvement and improved organisational performance, there would still be the problem of convincing practitioners that the evidence applies to their particular situation.

Applications built in a strongly decoupled, eventbased interaction style have many commendable characteristics, including ease of dynamic configuration, accommodation of platform heterogeneity, and ease of distribution over a network. It is not always easy to humanly grasp the dynamic behavior of such applications, since many threads are active and events are asynchronously (and profusely) transmitted. We present a set of requirements for an aid to assist in the human understanding and exploration of the behavior of such applications through the incremental refinement of rules for determining causality relationships between messages sent among components. A prototype tool is presented, indicating one viable approach to meeting these requirements. Experience with the tool reinforces some of the requirements and indicates others. 1.

A version of this report under second review for