Results 1 - 10
of
10
On Human Analyst Performance in Assisted Requirements Tracing: Statistical Analysis
"... Abstract—Assisted requirements tracing is a process in which a human analyst validates candidate traces produced by an automated requirements tracing method or tool. The assisted requirements tracing process splits the difference between the commonly applied time-consuming, tedious, and error-prone ..."
Abstract
-
Cited by 8 (2 self)
- Add to MetaCart
Abstract—Assisted requirements tracing is a process in which a human analyst validates candidate traces produced by an automated requirements tracing method or tool. The assisted requirements tracing process splits the difference between the commonly applied time-consuming, tedious, and error-prone manual tracing and the automated requirements tracing procedures that are a focal point of academic studies. In fact, in software assurance scenarios, assisted requirements tracing is the only way in which tracing can be at least partially automated. In this paper, we present the results of an extensive 12 month study of assisted tracing, conducted using three different tracing processes at two different sites. We describe the information collected about each study participant and their work on the tracing task, and apply statistical analysis to study which factors have the largest effect on the quality of the final trace. I.
Process improvement for traceability: A study of human fallibility
- In 20th IEEE International Requirements Engineering Conference (RE
, 2012
"... Abstract—Human analysts working with results from automated traceability tools often make incorrect decisions that lead to lower quality final trace matrices. As the human must vet the results of trace tools for mission- and safety-critical systems, the hopes of developing expedient and accurate tra ..."
Abstract
-
Cited by 5 (2 self)
- Add to MetaCart
(Show Context)
Abstract—Human analysts working with results from automated traceability tools often make incorrect decisions that lead to lower quality final trace matrices. As the human must vet the results of trace tools for mission- and safety-critical systems, the hopes of developing expedient and accurate tracing procedures lies in understanding how analysts work with trace matrices. This paper describes a study to understand when and why humans make correct and incorrect decisions during tracing tasks through logs of analyst actions. In addition to the traditional measures of recall and precision to describe the accuracy of the results, we introduce and study new measures that focus on analyst work quality: potential recall, sensitivity, and effort distribution. We use these measures to visualize analyst progress towards the final trace matrix, identifying factors that may influence their performance and determining how actual tracing strategies, derived from analyst logs, affect results.
Code Patterns for Automatically Validating Requirements-to-Code Traces
"... Traces between requirements and code reveal where requirements are implemented. Such traces are essential for code understandingandchangemanagement. Unfortunately,traces are known to be error prone. This paper introduces a novel approach for validating requirements-to-code traces through calling rel ..."
Abstract
-
Cited by 4 (0 self)
- Add to MetaCart
(Show Context)
Traces between requirements and code reveal where requirements are implemented. Such traces are essential for code understandingandchangemanagement. Unfortunately,traces are known to be error prone. This paper introduces a novel approach for validating requirements-to-code traces through calling relationships withinthecode. Asinput, theapproach requires an executable system, the corresponding requirements, and the requirements-to-code traces that need validating. As output, the approach identifies likely incorrect or missingtracesbyinvestigatingpatternsoftraceswithcalling relationships. The empirical evaluation of four case study systems covering 150 KLOC and 59 requirements demonstrates that the approach detects most errors with 85-95% precision and 82-96 % recall and is able to handle traces of varying levels of correctness and completeness. The approach is fully automated, tool supported, and scalable.
B.: Tracing Requirements and Source Code During Software Development
- In ICSEA 12: Proc. 7th Int. Conf. of Software Engineering Advances, Lissabon
, 2012
"... Abstract—Traceability links between requirements and source code are often created after development. This re-duces the possibilities for developers to use these traceability links during the development process. Additionally, existing approaches applied after development do not consider artifacts f ..."
Abstract
-
Cited by 4 (0 self)
- Add to MetaCart
(Show Context)
Abstract—Traceability links between requirements and source code are often created after development. This re-duces the possibilities for developers to use these traceability links during the development process. Additionally, existing approaches applied after development do not consider artifacts from project management, which are used for planning and organizing a project. These artifacts can serve as a mediator between requirements and source code. In contrast to these existing approaches, we present an approach that creates traceability links between requirements and source code as the development progresses by incorporating artifacts from project management. In this paper, we make two key contributions. First, a Traceability Information Model integrating require-ments, source code and artifacts from project management. Second, an approach for the (semi-) automatic creation of traceability links between artifacts from the Traceability In-formation Model achieving traceability between requirements and source code during the development process. We identified a catalog of information needs of developers from literature regarding requirements, source code that realizes these re-quirements, and work done by co-workers implementing these requirements. The presented approach satisfies the information needs of the developers during the development process, while keeping the traceability links up-to-date. Keywords-traceability; requirements; source code; software development; information needs. I.
Observations on the Connectedness between Requirements-to-Code Traces and Calling Relationships for Trace Validation
"... Abstract — Traces between requirements and code reveal where requirements are implemented. Such traces are essential for code understanding and change management. Unfortunately, the handling of traces is highly error prone, in part due to the informal nature of requirements. This paper discusses obs ..."
Abstract
-
Cited by 2 (2 self)
- Add to MetaCart
(Show Context)
Abstract — Traces between requirements and code reveal where requirements are implemented. Such traces are essential for code understanding and change management. Unfortunately, the handling of traces is highly error prone, in part due to the informal nature of requirements. This paper discusses observations on the connectedness between requirements-to-code traces and calling relationships within the source code. These observations are based on the empirical evaluation of four case study systems covering 150 KLOC and 59 sample requirements. We found that certain patterns of connectedness have high or low likelihoods of occurring. These patterns can thus be used to confirm or reject existing traceability – hence they are useful for validating requirements-to-code traces. Keywords-Requirements, Traceability, and Validation I.
Human Recoverability Index: A TraceLab Experiment
"... Abstract—It has been generally accepted that not all trace links in a given requirements traceability matrix are equal- both human analysts and automated methods are good at spotting some links, but have blind spots for some other. One way to choose automated techniques for inclusion in assisted tra ..."
Abstract
-
Cited by 2 (0 self)
- Add to MetaCart
(Show Context)
Abstract—It has been generally accepted that not all trace links in a given requirements traceability matrix are equal- both human analysts and automated methods are good at spotting some links, but have blind spots for some other. One way to choose automated techniques for inclusion in assisted tracing processes (i.e., the tracing processes that combine the expertise of a human analyst and special-purpose tracing software) is to select the techniques that tend to discover more links that are hard for human analysts to observe and establish on their own. This paper proposes a new measure of performance of a tracing method: human recoverability index-based recall. In the presence of knowledge about the difficulty of link recovery by human analysts, this measure rewards methods that are able to recover such links over methods that tend to recover the same links as the human analysts. We describe a TraceLab experiment we designed to evaluate automated trace recovery methods based on this measure and provide a case study of the use of this experiment to profile and evaluate different automated tracing techniques. I.
Towards a Model of Analyst Effort for Traceability Research: A Position Paper
- in Proceedings of Traceability of Emerging Forms of Software Engineering (TEFSE) 2011, an ICSE workshop
"... ABSTRACT This paper posits that a theoretical model of analyst effort in tracing tasks is necessary to assist with study of the analyst. Specifically, it is clear from prior work by numerous research groups that the important factors in such a model are: the amount of time it takes for an analyst t ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
ABSTRACT This paper posits that a theoretical model of analyst effort in tracing tasks is necessary to assist with study of the analyst. Specifically, it is clear from prior work by numerous research groups that the important factors in such a model are: the amount of time it takes for an analyst to vet a given candidate link and the amount of time it takes an analyst to find a missing link. This paper introduces a theoretical model of analyst effort as well as a simplified model. A number of simulations were undertaken in order to build effort curves to assist in evaluating numerous tracing scenarios, such as determining at what point in time an analyst should switch from vetting candidate links to manually searching for links not in the candidate list.
Departures from Optimality: Understanding Human Analyst’s Information Foraging in Assisted Requirements Tracing
"... Abstract—Studying human analyst’s behavior in automated tracing is a new research thrust. Building on a growing body of work in this area, we offer a novel approach to understanding requirements analyst’s information seeking and gathering. We model analysts as predators in pursuit of prey — the rele ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—Studying human analyst’s behavior in automated tracing is a new research thrust. Building on a growing body of work in this area, we offer a novel approach to understanding requirements analyst’s information seeking and gathering. We model analysts as predators in pursuit of prey — the relevant traceability information, and leverage the optimality models to characterize a rational decision process. The behavior of real analysts with that of the optimal information forager is then compared and contrasted. The results show that the analysts’ information diets are much wider than the theory’s predictions, and their residing in low-profitability information patches is much longer than the optimal residence time. These uncovered discrepancies not only offer concrete insights into the obstacles faced by analysts, but also lead to principled ways to increase practical tool support for overcoming the obstacles. Index Terms—Traceability, requirements engineering, study of human analysts, information foraging. I.
Jane Hayes: Research Statement 1
, 2011
"... Table of Contents Traceability ……………………………………………………………………. 2 Pre-Tenure Work …………………………………………………… … 2 ..."
Abstract
- Add to MetaCart
(Show Context)
Table of Contents Traceability ……………………………………………………………………. 2 Pre-Tenure Work …………………………………………………… … 2
On the Effectiveness of Accuracy of Automated Feature Location Technique
"... Abstract—Automated feature location techniques have been proposed to extract program elements that are likely to be relevant to a given feature. A more accurate result is expected to enable developers to perform more accurate feature location. However, several experiments assessing traceability reco ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—Automated feature location techniques have been proposed to extract program elements that are likely to be relevant to a given feature. A more accurate result is expected to enable developers to perform more accurate feature location. However, several experiments assessing traceability recovery have shown that analysts cannot utilize an accurate traceability matrix for their tasks. Because feature location deals with a certain type of traceability links, it is an important question whether the same phenomena are visible in feature location or not. To answer that question, we have conducted a controlled experiment. We have asked 20 subjects to locate features using lists of methods of which the accuracy is controlled artificially. The result differs from the traceability recovery experiments. Subjects given an accurate list would be able to locate a feature more accurately. However, subjects could not locate the complete implementation of features in 83 % of tasks. Results show that the accuracy of automated feature location techniques is effective, but it might be insufficient for perfect feature location. Index Terms—feature location, impact analysis, program com-prehension, human factor I.
