... This paper discusses the experience of the authors, based upon a collection of experiments, in terms of a framework for organizing sets of related studies. With such a framework, experiments can be viewed as part of common families of studies, rather than being isolated events. Common families of studies can contribute to important and relevant hypotheses that may not be suggested by individual experiments. A framework also facilitates building knowledge in an incremental manner through the replication of experiments within families of studies. To support the framework, this paper discusses the experiences of the authors in carrying out empirical studies, with specific emphasis on persistent problems encountered in experimental design, threats to validity, criteria for evaluation, and execution of experiments in the domain of software engineering.

This paper presents a controlled experiment focusing on the impact of applying quality design principles such as the ones provided by Coad and Yourdon on the maintainability of object-oriented designs.

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments.

Recently we have witnessed a welcomed increase in the amount of empirical evaluation of Software Engineering methods and concepts. It is hoped that this increase will lead to establishing Software Engineering as a well-defined subject with a sound scientifically proven underpinning rather than a topic based upon unsubstantiated theories and personal belief. For this to happen the empirical work must be of the highest standard. Unfortunately producing meaningful empirical evaluations is a highly hazardous activity, full of uncertainties and often unseen difficulties. Any researcher can overlook or neglect a seemingly innocuous factor, which in fact invalidates all of the work. More serious is that large sections of the communuity can overlook essential experimental design guidelines, which bring into question the validity of much of the work undertaken to date. In this paper, the authors address one such factor - Statistical Power Analysis. It is believed and will be demonstrated that a...

The object-oriented paradigm is becoming increasingly popular, apparently as the result of expert opinion and anecdotal evidence, and not on the basis of sound empirical data. This questionnaire survey was undertaken as part of a programme of research to validate unsupported claims about the paradigm. The questionnaire follows structured interviews of experienced object-oriented users [6] with the intention to confirm the findings on a wider user group. It was posted to relevant electronic newsgroups and to members of an object-oriented (postal) mailing list. The survey received 167 responses to the electronic questionnaire and 119 responses (30% response rate) to the postal version. Results show that respondents are of the view that: (i) the object-oriented paradigm has advantages over other paradigms in terms of ease of analysis and design, programmer productivity, software reuse, and ease of maintenance, (ii) inheritance can introduce difficulties when trying to understand object-or...

There is a general acceptance that software engineering research should be supported by empirical evaluation. To make real progress researchers must address the difficulties caused by the human-intensive nature of software developmentaswell as experimental validity. This paper proposes the use of multi-method empirical research programs, as an alternative to `single-shot' empirical studies, to help address these problems. The multi-method approach is based on the combination of complementary empirical research methods. The intention is that the complementary nature of the research methods compensate for weaknesses inherent in individual methods. It is argued that the multi-method approach potentially provides benefits in terms of more robust conclusions, development and investigation of researchhypotheses in an evolutionary manner, and increased understanding of research results. This paper demonstrates an application of the multi-method approachinanempirical investigation of object-oriented technology. This research program consists of a set of structured interviews with practitioners of object-oriented technology, followed by a wide-scale questionnaire survey, and concludes with a set of three controlled laboratory experiments whichinvestigated one of the key findings from the exploratory interview and questionnaire phases. This application finds evidence that unconstrained inheritance usage in object-oriented software may inhibit software maintenance. The paper concludes that the multi-method approach offers the possibiltyofmore reliable and generalizable results from empirical software engineering research. Key words: empirical, multi-method, object-oriented Preprint submitted to Elsevier Preprint 8January 1998 1

The method of structured interviewing is an appropriate means of conducting a primary investigation when beginning a programme of research involving different empirical techniques because (i) interviewing experienced users from academia and industry should determine if academics are exploring issues that industrialists deem important, (ii) interviewing users from industry helps provide external validity, (iii) users of object-oriented systems have their own views on the benefits and drawbacks of the paradigm. These opinions could then be used as evidence to decide whether an empirically unsupported practice is justified and warrants further, more controlled investigation, and (iv) these views can identify issues that have not been previously considered. Structured interviews, each based on a template of 23 questions, were carried out with 13 experienced object-oriented users to explore problems discussed in the object-oriented literature concerned with high level system understanding, ...

This paper presents a controlled experiment focusing on the impact of applying quality design principles such as the ones provided by Coad and Yourdon on the maintainability of object-oriented design documents. Results, which repeat the findings of a previous study, strongly suggest that such design principles have a beneficial effect on the maintainability of object-oriented design documents. It is argued that object-oriented design documents are sensitive to poor design practices because the cognitive complexity introduced becomes increasingly unmanageable. However, as our ability to generalise these results is limited, they should be considered as preliminary, i.e., it is very likely that they can only be generalised to programmers with little object-oriented training and programming experience. Such programmers can, however, be commonly found on maintenance projects. As well as additional research, external replications of this study are required to confirm the results and achieve ...

Abstract. In software engineering it is widely acknowledged that the usage of metrics at the initial phases of the object oriented software life cycle can help designers to make better decisions and to predict external quality attributes, such as maintainability. Following this idea we have carried out three controlled experiments to ascertain if any correlation exists between the structural complexity and the size of UML class diagrams and their maintainability. We used 8 metrics for measuring the structural complexity of class diagrams due to the usage of UML relationships, and 3 metrics to measure their size. With the aim of determining which of these metrics are really relevant to be used as class diagrams maintainability indicators, we present in this work a study based on Principal Component Analysis. The obtained results show that the metrics related to associations, aggregations, generalizations and dependencies, are the most relevant whilst those related to size seem to be redundant.

Maintainability is an increasingly relevant quality aspect in the development of object oriented software systems (OOSS). It is generally accepted that OOSS maintainability is highly dependent on the decisions made early in the development life cycle. Conceptual modelling is an important task of this early development. So that the maintainability of conceptual models have a great influence on the maintainability of the OOSS which is finally implemented. For assessing the conceptual models maintainability it is useful to have quantitative and objective measurement instruments. Conceptual modelling focus on either static aspects or dynamics aspects of the OOSS. Using the Unified Modelling Language (UML) static aspects at conceptual level are mainly represented in structural diagrams such as class diagrams, whilst dynamic aspects are represented in behavioural diagrams such as statechart diagrams, activity diagrams, sequence diagrams and collaboration diagrams. There exists several works related to metrics for structural diagrams such as class diagrams. However, behavioural diagrams have been little studied This fact leaded us to define measures for UML statechart diagrams. The main goal of this paper is to show how we defined those measures in a methodological way, in order to guarantee their validity. We used the DISTANCE framework, based on measurement theory, to define and theoretically validate the measures. In order to gather empirical evidence that the proposed measures could be early maintainability indicators of UML statechart diagrams, we carried out a controlled experiment. The aim of the experiment was to investigate the relationship between the complexity of UML statechart diagrams and their understandability (one maintainability subcharacteristic).