Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.

Software engineering researchers have long been interested in where and why bugs occur in code, and in predicting where they might turn up next. Historical bug-occurence data has been key to this research. Bug tracking systems, and code version histories, record when, how and by whom bugs were fixed; from these sources, datasets that relate file changes to bug fixes can be extracted. These historical datasets can be used to test hypotheses concerning processes of bug introduction, and also to build statistical bug prediction models. Unfortunately, processes and humans are imperfect, and only a fraction of bug fixes are actually labelled in source code version histories, and thus become available for study in the extracted datasets. The question naturally arises, are the bug fixes recorded in these historical datasets a fair representation of the full population of bug fixes? In this paper, we investigate historical data from several software projects, and find strong evidence of systematic bias. We then investigate the potential effects of “unfair, imbalanced” datasets on the performance of prediction techniques. We draw the lesson that bias is a critical problem that threatens both the effectiveness of processes that rely on biased datasets to build prediction models and the generalizability of hypotheses tested on biased data 1.

In this paper, we present a framework for learning adaptation knowledge with knowledge light approaches for case-based reasoning (CBR) systems. "Knowledge light" means that these approaches use knowledge already acquired and represented inside the CBR system. Therefore, we describe the sources of knowledge inside a CBR system along with the different knowledge containers. Next we present our framework in terms of these knowledge containers. Further, we apply our framework to two very different knowledge light approaches for learning adaptation knowledge. After that we point out some issues which should be addressed during the design or the use of such algorithms for learning adaptation knowledge. From our point of view, many of these issues should be the topic of further research. Finally we close with a short discussion. 1 Introduction One of the major challenges during designing a case-based reasoning (CBR) system is the modeling of appropriate adaptation knowledge. Usually adaptati...

Abstract. Ontology tools performance and scalability are critical to both the growth of the Semantic Web and the establishment of these tools in the industry. In this paper, we present briefly the benchmarking methodology used to improve the performance and the scalability of ontology development tools. We focus on the definition of the infrastructure for evaluating the performance of these tools ’ ontology management APIs in terms of its execution efficiency. We also present the results of applying the methodology for evaluating the API of the WebODE ontology engineering workbench. 1

Process support environments (PSEs) are widely used for modelling, enacting and analyzing human intensive processes. The benefits of a PSE become apparent when processes to be supported are long lived and distributed and contain heterogeneous components. Generally, such processes are subject to dynamic evolution, i.e. they have to be changed during their execution.

However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of

Use case modeling, including use case diagrams and use case specifications, is commonly applied to structure and document requirements. Use case specifications are usually structured, textual documents complying with a certain use case template. However, because use case specifications remain essentially textual, ambiguity is inevitably introduced. In this paper, we propose a use case modeling approach, which is composed of a set of well-defined restriction rules and a use case template. The goal is two-fold: (1) restrict the way users can document use case specifications in order to reduce ambiguity and (2) facilitate automated analysis in order to provide tool support to derive initial analysis models, which in UML are typically composed of class diagrams, interaction diagrams, and possibly other types of diagrams and constraints. Though the proposed restriction rules and template are based on a clear rationale, two main questions need to be investigated. Do users find them too restrictive or impractical in certain situations? Second, do the rules and template have a positive, significant impact on the quality of the resulting analysis models? To investigate these questions, we performed and report on a controlled experiment, which evaluates the restriction rules and use case template in terms of whether they are easy to apply while developing use case models and whether they help obtain higher quality analysis models in terms of correctness, completeness, redundancy, and understandability. Results show that, the restriction rules are overall easy to apply and that our use case modeling approach result 1 Carleton University, TR SCE-09-05 May 2009 in significant improvements regarding the correctness of derived class diagrams and the understandability of use case specifications.

Clarity is underappreciated as a requirements specification quality attribute. We studied the clarity of requirements forms, operationalized as ease of problem detection, least obstructive to understanding, and understandability by stakeholders. A set of use cases for an industrial system was translated into sequence diagrams and ScenarioML; problems identified during each translation were noted, and system stakeholders were interviewed and given a questionnaire on all three forms. The data showed that Scenario-ML best supported requirements clarity, then sequence diagrams but only for stakeholders experienced with them, and finally use cases as the least clear form. Use cases were preferred for non-technical stakeholders to write; sequence diagrams were most effective for details of individual events and for showing interaction with architectural components; with ScenarioML preferred in all other situations. That I require a clearness: and with him to leave no rubs

Software engineers of multi-agent systems (MASs) are faced with different concerns such as autonomy, adaptation, interaction, collaboration, learning, and mobility, which are essentially different from classical concerns addressed in object-oriented software engineering. MAS developers however have relied mostly on object-oriented design techniques and programming languages, such as Java. It often leads to a poor separation of MAS concerns and in turn to the production of MASs that are difficult to maintain and reuse. This paper discusses software engineering approaches for MASs, and presents a new method for integrating agents into object-oriented software engineering from an early stage of design. The proposed approach encourages the separate handling of MAS concerns, and provides a disciplined scheme for their composition. Our proposal explores the benefits of aspect-oriented software development for the incorporation of agents into object-oriented systems. We also illustrate our aspect-oriented approach through the Portalware multi-agent system, a web-based environment for the development of e-commerce portals. Keywords: Multi-agent systems, software agents, software engineering, object-oriented systems, aspect-oriented software development.

Abstract. Companies using domain specific languages in a model-driven development process need to measure their models. However, developing and maintaining a measurement software for each domain specific modeling language is costly. Our contribution is a model-driven measurement approach. This measurement approach is model-driven from two viewpoints: 1) it measures models of a model-driven development process; 2) it uses models as unique and consistent metric specifications, w.r.t a metric specification metamodel. This declarative specification of metrics is then used to generate a fully fledged implementation. The benefit from applying the approach is evaluated by two applications. They indicate that this approach reduces the domain-specific measurement software development cost. 1