http://www.computer.org/tse The IEEE Transactions on Software Engineering is an archival journal published monthly. We are interested in well-defined theoretical results and empirical studies that have potential impact on the construction, analysis, or management of software. The scope of this Transactions ranges from the mechanisms through the development of principles to the application of those principles to specific environments. Since the journal is archival, it is assumed that the ideas presented are important, have been well analyzed, and/or empirically validated and are of value to the software engineering research or practitioner community. Specific topic areas include: a) development and maintenance methods and models, e.g., techniques and principles for the specification, design, and implementation of software systems, including notations and process models; b) assessment methods, e.g., software tests and validation, reliability models, test and diagnosis procedures, software redundancy and design for error control, and the measurements and evaluation of various aspects of the process and product; c) software project management, e.g., productivity factors, cost models, schedule and organizational issues, standards; d) tools and environments, e.g., specific tools, integrated tool environments including the associated architectures, databases, and parallel and distributed processing issues; e) system issues, e.g., hardware-software trade-off; and f) state-of-the-art surveys that provide a synthesis and comprehensive review of the historical development of one particular area of interest. en-us Wed, 4 Jan 2012 11:00:01 GMT http://csdl.computer.org/common/images/logos/tse.gif List of recently published journal articles http://www.computer.org/tse http://doi.ieeecomputersociety.org/10.1109/TSE.2011.123 With the wide support for object serialization in object-oriented programming languages, persistent objects have become common place and most large object-oriented software systems rely on extensive amounts of persistent data. Such systems also evolve over time. Retrieving previously persisted objects from classes whose schema has changed is however difficult, and may lead to invalidating the consistency of the application. The ESCHER framework addresses these issues through an IDE-integrated approach that handles class schema evolution by managing versions of the code and generating transformation functions automatically. The infrastructure also enforces class invariants to prevent the introduction of potentially corrupt objects. This article describes a model for class attribute changes, a measure for class evolution robustness, four empirical studies, and the design and implementation of the ESCHER system. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.123 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.104 This article describes GenProg, an automated method for repairing defects in off-the-shelf, legacy programs without formal specifications, program annotations, or special coding practices. GenProg uses an extended form of genetic programming to evolve a program variant that retains required functionality but is not susceptible to a given defect, using existing test suites to encode both the defect and required functionality. Structural differencing algorithms and delta debugging reduce the difference between this variant and the original program to a minimal repair. We describe the algorithm and report experimental results of its success on 16 programs totaling 1.25M lines of C code and 120K lines of module code, spanning 8 classes of defects, in 357 seconds, on average. We analyze the generated repairs qualitatively and quantitatively to demonstrate that the process efficiently produces evolved programs that repair the defect, are not fragile input memorizations, and do not lead to serious degradation in functionality. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.104 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.101 Dynamic software updating (DSU) systems patch programs on the fly without incurring downtime. To avoid failures due to the updating process itself, many DSU systems employ timing restrictions. However, timing restrictions are theoretically imperfect, and their practical effectiveness is an open question. This paper presents the first significant empirical evaluation of three popular timing restrictions: activeness safety (AS), which prevents updates to active functions; con-freeness safety (CFS), which only allows modifications to active functions when doing so is provably type-safe; and manual identification of the event-handling loops during which an update may occur. We evaluated these timing restrictions using a series of DSU patches to three programs: OpenSSH, vsftpd, and ngIRCd. We systematically applied updates at each distinct update point reached during execution of a suite of system tests for these programs to determine which updates pass and which fail. We found that all three timing restrictions prevented most failures, but only manual identification allowed none. Further, although CFS and AS allowed many more update points, manual identification still supported updates with minimal delay. Finally, we found that manual identification required the least developer effort. Overall, we conclude that manual identification is most effective. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.101 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.63 As firms increasingly rely on information systems to perform critical functions the consequences of software defects can be catastrophic. Although the software engineering literature suggests that software process improvement can help to reduce software defects, the actual evidence is equivocal. For example, improved development processes may only remove the “easier” syntactical defects, while the more critical defects remain. Rigorous empirical analyses of these relationships have been very difficult to conduct due to the difficulties in collecting the appropriate data on real systems from industrial organizations. This field study analyzes a detailed data set consisting of 7,545 software defects that were collected on software projects completed at a major software firm. Our analyses reveal that higher levels of software process improvement significantly reduce the likelihood of high severity defects. In addition, we find that higher levels of process improvement are even more beneficial in reducing severe defects when the system developed is large or complex, but are less beneficial when requirements are ambiguous, unclear or incomplete. Our findings reveal the benefits and limitations of software process improvement for the removal of severe defects and suggest where investments in improving development processes may have their greatest effects. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.63 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.59 Propositional bounded model checking has been applied successfully to verify embedded software but remains limited by increasing propositional formula sizes and the loss of high-level information during the translation preventing potential optimizations to reduce the state space to be explored. These limitations can be overcome by encoding high-level information in theories richer than propositional logic and using SMT solvers for the generated verification conditions. Here, we propose the application of different background theories and SMT solvers to the verification of embedded software written in ANSI-C in order to improve scalability and precision in a completely automatic way. We have modified and extended the encodings from previous SMT-based bounded model checkers to provide more accurate support for variables of finite bit width, bit-vector operations, arrays, structures, unions and pointers. We have integrated the CVC3, Boolector, and Z3 solvers with the CBMC front-end and evaluated them using both standard software model checking benchmarks and typical embedded software applications from telecommunications, control systems, and medical devices. The experiments show that our ESBMC model checker can analyze larger problems than existing tools and substantially reduce the verification time. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.59 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.56 This paper proposes a novel approach for selecting and ordering a predetermined number of test cases from an existing test suite. Our approach forms an Integer inear Programming problem using two different coverage-based criteria, and uses constraint relaxation to find many close-to-optimal solution points. These points are then combined to obtain a final solution using a voting mechanism. The selected subset of test cases are then prioritized using a greedy algorithm that maximizes minimum coverage in an iterative manner. The proposed approach has been empirically evaluated and the results show significant improvements over existing approaches for some cases and comparable results for the est. Moreover, our approach provides more consistency compared to existing approaches. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.56 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.37 A major challenge of dynamic reconfiguration is Quality of Service (QoS) assurance, which is meant to reduce appli-cation disruption to the minimum for the system’s transformation. However, this problem has not been well studied. This paper investigates the problem for component-based software systems from three points of view. First, the whole spectrum of QoS characteristics is defined. Second, the logical and physical requirements for QoS characteristics are analyzed and solutions to achieve them are proposed. Third, prior work is classified by QoS characteristics and then realized by abstract reconfiguration strategies. On this basis, quantitative evaluation of the QoS assurance abilities of existing work and our own approach is conducted through three steps. First, a proof-of-concept prototype called the reconfigurable component model is implemented to support the representation and testing of the reconfiguration strategies. Second, a reconfiguration benchmark is proposed to expose the whole spectrum of QoS problems. Third, each reconfiguration strategy is tested against the benchmark and the testing results are evaluated. The most important conclusion from our investigation is that the classified QoS characteristics can be fully achieved under some acceptable constraints. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.37 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.28 AJAX-based Web 2.0 applications rely on stateful asynchronous client/server communication, and client-side run-time manipulation of the DOM tree. This not only makes them fundamentally different from traditional web applications, but also more error-prone and harder to test. We propose a method for testing AJAX applications automatically, based on a crawler to infer a state-flow graph for all (client-side) user interface states. We identify AJAX-specific faults that can occur in such states (related to e.g., DOM validity, error messages, discoverability, back-button compatibility) as well as DOM-tree invariants that can serve as oracles to detect such faults. Our approach, called ATUSA, is implemented in a tool offering generic invariant checking components, a plugin-mechanism to add application-specific state validators, and generation of a test suite covering the paths obtained during crawling. We describe three case studies, consisting of six subjects, evaluating the type of invariants that can be obtained for AJAX applications as well as the fault revealing capabilities, scalability, required manual effort, and level of automation of our testing approach. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.28 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.9 Bad smells are signs of potential problems in code. Detecting and resolving bad smells, however, remain time consuming for software engineers despite proposals on bad smell detection and refactoring tools. Numerous bad smells have been recognized, yet the sequences in which the detection and resolution of different kinds of bad smells are performed are rarely discussed because software engineers do not know how to optimize sequences or determine the benefits of an optimal sequence. To this end, we propose a detection and resolution sequence for different kinds of bad smells to simplify their detection and resolution. We highlight the necessity of managing bad smell resolution sequences with a motivating example, and recommend a suitable sequence for commonly occurring bad smells. We evaluate this recommendation on two non-trivial open source applications, and the evaluation results suggest that a significant reduction in effort ranging from 17.64% to 20% can be achieved when bad smells are detected and resolved using the proposed sequence. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.9 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.5 Formal specifications can help with program testing, optimization, refactoring, documentation, and, most importantly, debugging and repair. However, they are difficult to write manually, and automatic mining techniques suffer from 90-99% false positive rates. To address this problem, we propose to augment a temporal-property miner by incorporating code quality metrics. We measure code quality by extracting additional information from the software engineering process, and using information from code that is more likely to be correct as well as code that is less likely to be correct. When used as a preprocessing step for an existing specification miner, our technique identifies which input is most indicative of correct program behavior, which allows off-the-shelf techniques to learn the same number of specifications using only 45% of their original input. As a novel inference technique, our approach has few false positives in practice (63% when balancing precision and recall, 3% when focused on precision), while still finding useful specifications (e.g., those that find many bugs) on over 1.5 million lines of code. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.5 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.124 The injection of software faults in software components to assess the impact of these faults on other components or on the system as a whole, allowing the evaluation of fault tolerance, is relatively new compared to decades of research on hardware fault injection. This paper presents an extensive experimental study (more than 3.8 millions of individual experiments in three real systems) to evaluate the representativeness of faults injected by a state-of-the-art approach (G-SWFIT). Results show that a significant share (up to 72%) of injected faults cannot be considered representative of residual software faults, as they are consistently detected by regression tests, and that representativeness of injected faults is affected by the fault location within the system, resulting in different distributions of representative/non-representative faults across files and functions. Therefore, we propose a new approach to refine the faultload by removing faults that are not representative of residual software faults. This filtering is essential to assure meaningful results and to reduce the cost (in terms of number of faults) of software fault injection campaigns in complex software. The proposed approach is based on classification algorithms, is fully automatic, and can be used for improving fault representativeness of existing software fault injection approaches. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.124 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.122 Though very important in software engineering, linking artifacts of the same type (clone detection) or different types (traceability recovery) is extremely tedious, error-prone, and effort-intensive. Past research focused on supporting analysts with techniques based on Natural Language Processing (NLP) to identify candidate links. Because many NLP techniques exist and their performance varies according to context, it is crucial to define and use reliable evaluation procedures. The aim of this paper is to propose a set of seven principles for evaluating the performance of NLP techniques in identifying equivalent requirements. In this paper we conjecture, and verify, that NLP techniques perform on a given dataset according to both ability and the odds of identifying equivalent requirements correctly. For instance, when the odds of identifying equivalent requirements are very high, then it is reasonable to expect that NLP techniques will result in good performance. Our key idea is to measure this random factor of the specific dataset(s) in use and then adjust the observed performance accordingly. To support the application of the principles we report their practical application to a case study that evaluates the performance of a large number of NLP techniques for identifying equivalent requirements in the context of an Italian company in the defense and aerospace domain. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.122 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.121 A substantial amount of work has shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications have been reported in the literature. Although it is not going to solve all possible testing problems, random testing appears to be an essential tool in the hands of software testers. In this paper, we review and analyze the debate about random testing. Its benefits and drawbacks are discussed. Novel results addressing general questions about random testing are also presented, such as how long random does testing need on average to achieve testing targets (e.g., coverage), how does it scale and how likely is it to yield similar results if we re-run it on the same testing problem (predictability). Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Results show that there are practical situations in which random testing is a viable option. Our theorems are backed up by simulations and we show how they can be applied to most types of software and testing criteria. In light of these results, we then assess the validity of empirical analyses reported in the literature and derive guidelines for both practitioners and scientists. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.121 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.120 Superimposition is a composition technique that has been applied successfully in many areas of software development. Although superimposition is a general-purpose concept, it has been (re)invented and implemented individually for various kinds of software artifacts. We unify languages and tools that rely on superimposition by using the language-independent model of feature structure trees (FSTs). On the basis of the FST model, we propose a general approach to the composition of software artifacts written in different languages. Furthermore, we offer a supporting framework and tool chain, called FeatureHouse. We use attribute grammars to automate the integration of additional languages. In particular, we have integrated Java, C#, C, Haskell, Alloy, and JavaCC. A substantial number of case studies demonstrate the practicality and scalability of our approach and reveal insights into the properties that a language must have in order to be ready for superimposition. We discuss perspectives of our approach and demonstrate how we extended FeatureHouse with support for XML languages (in particular, XHTML, XMI/UML, and Ant) and alternative composition approaches (in particular, aspect weaving). Rounding off our previous work, we provide here a holistic view of the FeatureHouse approach based on rich experience with numerous languages and case studies and reflections on several years of research. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.120 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.112 Model Management addresses the problem of managing an evolving collection of models, by capturing the relationships between models and providing well-defined operators to manipulate them. In this article, we describe two such operators for manipulating feature specifications described using hierarchical state machine models: Match, for finding correspondences between models, and Merge, for combining models with respect to known or hypothesized correspondences between them. Our Match operator is heuristic, making use of both static and behavioural properties of the models to improve the accuracy of matching. Our Merge operator preserves the hierarchical structure of the input models, and handles differences in behaviour through parameterization. This enables us to automatically construct merges that preserve the semantics of hierarchical state machines. We report on tool support for our Match and Merge operators, and illustrate and evaluate our work by applying these operators to a set of telecommunication features built by AT&T. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.112 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.111 Background: Despite decades of research, there is no consensus on which software effort estimation methods produce the most accurate models. Aim: Prior work has reported that, given M estimation methods, no single method consistently outperforms all others. Perhaps rather than recommending one estimation method as best, it is wiser to generate estimates from ensembles of multiple estimation methods. Method: 9 learners were combined with 10 pre-processing options to generate 9 × 10 = 90 solo-methods. These were applied to 20 data sets and evaluated using 7 error measures. This identified the best n (in our case n = 13) solo-methods that showed stable performance across multiple datasets and error measures. The top 2, 4, 8 and 13 solo-methods were then combined to generate 12 multi-methods, which were then compared to the solo-methods. Results: (i) The top 10 (out of 12) multi-methods significantly out-performed all 90 solo-methods. (ii) The error rates of the multi- methods were significantly less than the solo-methods. (iii) The ranking of the best multi-method was remarkably stable. Conclusion: While there is no best single effort estimation method, there exist best combinations of such effort estimation methods. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.111 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.110 Refactoring tools, common to many integrated development environments, can help programmers to restructure their code. These tools sometimes refuse to restructure the programmer's code, instead giving the programmer a textual error message that she must decode if she wishes to understand the reason for the tool's refusal, and what corrective action to take. This article describes a graphical alternative to textual error messages called Refactoring Annotations. It reports on two experiments, one using an integrated development environment and the other using paper mockups, that show that programmers can use Refactoring Annotations to quickly and accurately understand the cause of refactoring errors. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.110 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.109 Software Product Line Engineering (SPLE) is a prominent paradigm for the assembly of a family of products using product line core assets. The modeling of software assets that together form the actual products is critical for achieving the strategic benefits of software product lines. We propose a feature-based approach to software asset modeling based on abstractions provided by Domain-Specific Kits. This approach involves a software asset metamodel used to derive asset modeling languages that define reusable software assets in domain-specific terms. The approach also prescribes a roadmap for modeling these software assets in conjunction with the product line reference architecture. Asset capabilities can be modeled using feature diagrams as the external views of the software assets. Internal views can be expressed in terms of domain-specific artifacts with variability points, where the domain-specific artifacts are created using Domain-Specific Kits. This approach produces loosely coupled and highly cohesive software assets that are reusable for multiple product lines. The approach is validated by assessing software asset reuse in two different product lines in the finance domain. We also evaluated the productivity gains in large-scale complex projects, and found that the approach yielded a significant reduction in the total project effort. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.109 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.108 Dynamic loading of software components is a widely used mechanism for improved system modularity and flexibility. Correct component resolution is critical for reliable and secure software execution, however, programming mistakes may lead to unintended or even malicious components to be resolved and loaded. In particular, dynamic loading can be hijacked by placing an arbitrary file with the specified name in a directory searched before resolving the target component. In this paper, we present the first automated technique to detect unsafe dynamic component loadings. Our analysis has two phases: 1) apply dynamic binary instrumentation to collect runtime information on component loading; and 2) analyze the collected information to detect vulnerable component loadings. For evaluation, we implemented our technique to detect unsafe component loadings in popular software on Microsoft Windows and Linux. Our evaluation shows that unsafe loading is prevalent in software on both OS platforms, and it is more severe on Microsoft Windows. In particular, our tool detected more than 4,000 unsafe component loadings, and some lead to remote code execution on Microsoft Windows. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.108 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.107 Despite much progress, developing a pervasive computing application remains a challenge because of a lack of conceptual frameworks and supporting tools. This challenge involves coping with heterogeneous devices, overcoming the intricacies of distributed systems technologies, working out an architecture for the application, encoding it in a program, writing specific code to test the application, and finally deploying it. This paper presents a design language and a tool suite covering the development life-cycle of a pervasive computing system. The design language allows to define a taxonomy of area-specific building-blocks, abstracting over their heterogeneity. This language also includes a layer to define the architecture of an application, following an architectural pattern commonly used in the pervasive computing domain. Our underlying methodology assigns roles to the stakeholders, providing separation of concerns. Our tool suite includes a compiler that takes design artifacts written in our language as input, and generates a programming framework that supports the subsequent development stages, namely implementation, testing and deployment. Our methodology has been applied on a wide spectrum of areas. Based on these experiments, we assess our approach through three criteria: expressiveness, usability and productivity. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.107 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.106 Over the past few years, many test case prioritization techniques have been proposed in the literature. Most of them require data on dynamic execution in the form of code coverage information for test cases. However, the collection of dynamic code coverage information has several associated drawbacks including cost increases and reduction in prioritization precision. In this paper, we propose an approach to prioritizing test cases in the absence of coverage information that operates on Java programs tested under the JUnit framework. Our approach, JUPTA analyzes the static call graphs of JUnit test cases and the program under test to estimate the ability of each test case to achieve code coverage, and then schedules the order of these test cases based on those estimates. The experimental results show that the test suites constructed by JUPTA are more effective than those in random and untreated test orders in fault-detection effectiveness. Although the test suites constructed by dynamic coverage-based techniques retain fault-detection effectiveness advantages, the fault-detection effectiveness abilities of test suites constructed by JUPTA are close to those dynamic techniques, and the fault-detection effectiveness abilities of the test suites constructed by some of JUPTA's variants are better than several of those techniques. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.106 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.105 Dynamic specification mining observes program executions to infer models of normal program behavior. What makes us believe that we have seen sufficiently many executions? The TAUTOKO typestate miner generates test cases that cover previously unobserved behavior, systematically extending the execution space and enriching the specification. To our knowledge, this is the first combination of systematic test case generation and typestate mining–a combination with clear benefits: On a sample of 800 defects seeded into six Java subjects, a static typestate verifier fed with enriched models would report significantly more true positives, and significantly fewer false positives than the initial models. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.105 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.103 Background: The accurate prediction of where faults are likely to occur in code can help direct test effort, reduce costs and improve the quality of software. Objective: We investigate how the context of models, the independent variables used and the modelling techniques applied, influence the performance of fault prediction models. Method:We used a systematic literature review to identify 208 fault prediction studies published from January 2000 to December 2010. We synthesise the quantitative and qualitative results of 36 studies which report sufficient contextual and methodological information according to the criteria we develop and apply. Results: The models that perform well tend to be based on simple modelling techniques such as Naïve Bayes or Logistic Regression. Combinations of independent variables have been used by models that perform well. Feature selection has been applied to these combinations when models are performing particularly well. Conclusion: The methodology used to build models seems to be influential to predictive performance. Although there are a set of fault prediction studies in which confidence is possible, more studies are needed that use a reliable methodology and which report their context, methodology and performance comprehensively. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.103 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.102 Software inspection is a static analysis technique that is widely used for defect detection, but which suffers from a lack of rigor. In this paper, we address this problem by taking advantage of formal specification and analysis to support a systematic and rigorous inspection method. The aim of the method is to use inspection to determine whether every functional scenario defined in the specification is implemented correctly by a set of program paths and whether every program path of the program contributes to the implementation of some functional scenario in the specification. The method comprises five steps: deriving functional scenarios from the specification; deriving paths from the program; linking scenarios to paths; analyzing paths against the corresponding scenarios; and producing an inspection report, and allows for a systematic and automatic generation of a checklist for inspection. We present an example to show how the method can be used, and describe an experiment to evaluate its performance by comparing it to perspective-based reading (PBR). The result shows that our method may be more effective in detecting function-related defects than PBR but slightly less effective in detecting implementation-related defects. We also describe a prototype tool to demonstrate the supportability of the method, and draw some conclusions about our work. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.102 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.95 In hard real-time systems, gain time is defined as the difference between the worst-case execution time of a hard task and its actual processor consumption at run time. This paper presents the results of an empirical study about how the presence of a significant amount of gain time in a hard real-time system questions the advantages of using the most representative scheduling algorithms or policies for aperiodic or soft tasks in fixed-priority preemptive systems. The work presented here refines and complements many other studies in this research area, in which such policies have been introduced and compared. This work has been performed by using the authors' testing framework for soft scheduling policies, which produces actual, synthetic, randomly-generated applications, executes them in an instrumented real-time operating system, and finally processes such information to obtain several statistical outcomes. The results show that, in general, the presence of a significant amount of gain time reduces the performance benefit of the scheduling policies under study when compared to serving the soft tasks in background, which is considered the theoretical worst case. In some cases, this performance benefit is so small that the use of a specific scheduling policy for soft tasks is questionable. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.95 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.94 With the increasing importance of reliability in business and industrial software systems, new techniques of architecture-based reliability engineering are becoming an integral part of the development process. These techniques can assist system architects in evaluating the reliability impact of their design decisions. Architecture-based reliability engineering is only effective if the involved reliability models reflect the interaction and usage of software components and their deployment to potentially unreliable hardware. However, existing approaches either neglect individual impact factors on reliability or hard-code them into formal models, which limits their applicability in component-based development processes. This paper introduces a reliability modelling and prediction technique that considers the relevant architectural factors of software systems and explicitly models the component usage profile and execution environment. The technique offers a UML-like modelling notation, whose models are automatically transformed into a formal analytical model. Our work builds upon the Palladio Component Model, employing novel techniques of information propagation and reliability assessment. We validate our technique with sensitivity analyses and simulation in two case studies. The case studies demonstrate effective support of usage profile analysis and architectural configuration ranking, together with the employment of reliability-improving architecture tactics. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.94 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.93 To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a non-detected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these mutations for object-oriented classes. This has two advantages: First, the resulting test suite is optimized towards finding defects rather than covering code. Second, the state change caused by mutations induces oracles that precisely detect the mutants. Evaluated on 10 open source libraries, our MuTest prototype generates test suites that find significantly more seeded defects than the original manually written test suites. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.93 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.92 The Next Release Problem (NRP) aims to optimize customer profits and requirements selection for the software releases. The research on the NRP is restricted by the growing scale of requirements. In this paper, we propose a Backbone based Multilevel Algorithm (BMA) to address the large scale NRP. In contrast to direct solving approaches, BMA employs multilevel reductions to downgrade the problem scale and multilevel refinements to construct the final optimal set of customers. In both reductions and refinements, the backbone is built to fix the common part of the optimal customers. Since it is intractable to extract the backbone in practice, the approximate backbone is employed for the instance reduction while the soft backbone is proposed to augment the backbone application. In the experiments, to cope with the lack of open large requirements databases, we propose a method to extract instances from open bug repositories. Experimental results on 15 classic instances and 24 realistic instances demonstrate that BMA can achieve better solutions on the large scale NRP instances than direct solving approaches. Our work provides a reduction approach for solving large scale problems in search based requirements engineering. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.92 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.91 We investigate the relationship between class dependency and change propagation (co-change) in software written in Java. On the one hand we find a strong correlation between dependency and co-change. Furthermore, we provide empirical evidence for the propagation of change along paths of dependency. These findings support the often alleged role of dependencies as propagators of change. On the other hand we find that approximately half of all dependencies are never involved in co-changes and that the vast majority of co-changes pertain to only a small percentage of dependencies. This means that inferring the co-change characteristics of a software architecture solely from its dependency structure results in a severely distorted approximation of co-change characteristics. Any metric which uses dependencies alone to pass judgment on the evolvability of a piece of Java software is thus implausible. As a consequence we suggest to always take both the change characteristics and the dependency structure into account when evaluating software architecture. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.91 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.90 Recent research results suggest a need for code clone management. In this paper, we introduce JSync, a novel clone management tool for evolving software. JSync provides two main functions to support developers in being aware of the clone relation among code fragments as software systems evolve and in making consistent changes as they create or modify cloned code. JSync represents source code and clones as (sub)trees in Abstract Syntax Trees, measures code similarity based on structural characteristic vectors, and describes code changes as tree editing scripts. The key techniques of JSync include the algorithms to compute tree editing scripts, to detect and update code clones and their groups, to analyze the changes of cloned code to validate their consistency, and to recommend relevant clone synchronization and merging. Our empirical evaluation on several real-world systems shows that JSync is efficient and accurate in clone detection and updating, and provides the correct detection of the defects resulting from inconsistent changes to clones and the correct recommendations for change propagation across cloned code. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.90 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.89 Determining how to cope with existing systems is an important issue for information systems development (ISD). In this paper, we investigate how well different ISD patterns are suited for coping with existing systems. Empirical results, gathered from three software development projects undertaken by a financial institution, suggest propositions regarding how ISD patterns and existing systems affect the characteristics of objective ISD complexity, which in turn determine overall experienced complexity. Existing systems increase complexity due to conflicting interdependencies, but ISD patterns that reduce this complexity, such as those that employ bottom-up or concurrent consideration patterns, are best suited for coping with existing systems. In contrast, top-down and iterative focusing patterns, as classically used in new development, increase the complexity associated with conflicting interdependency, which makes them particularly unsuited for coping with existing systems in ISD. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.89 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.88 This study develops and empirically tests the idea that the impact of structural complexity on perfective maintenance of object-oriented software is significantly determined by the team strategy of programmers (independent or collaborative). We analyzed two key dimensions of software structure, coupling and cohesion, with respect to the maintenance effort and the perceived ease-of-maintenance by pairs of programmers. Hypotheses based on the distributed cognition and task interdependence theoretical frameworks were tested using data collected from a controlled lab experiment employing professional programmers. The results show a significant interaction effect between coupling, cohesion, and programmer team strategy on both maintenance effort and perceived ease-of-maintenance. Highly cohesive and low-coupled programs required lower maintenance effort and were perceived to be easier to maintain than the low cohesive programs and high-coupled programs. Further, our results would predict that managers who allocate maintenance tasks to independent or collaborative programming teams depending on the structural complexity of software could lower their team’s maintenance effort by as much as 70% over managers who use simple uniform resource allocation policies. These results highlight the importance of achieving congruence between team strategies employed by collaborating programmers and the structural complexity of software. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.88 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.86 To estimate Software Code Size early in the development process is important for developing cost-efficient embedded systems. We have applied the COSMIC FSM (Functional Size Measurement) method for Size Estimation of Embedded Software Components in the automotive industry. Experiments were conducted using data from two automotive companies. The experiments show strong correlation between Functional Size and Software Code Size, which is important for obtaining accurate estimation results. This paper presents the characteristics and results of our work, and aims to provide a practical framework for how to use COSMIC FSM for Size Estimation purposes. We investigate the results from our earlier experiments, and conduct further experiments to identify such a framework. Based on these activities we conclude that a clear purpose of the estimation process, a well-defined domain allowing categorization of software, consistent content and quality of requirements, and historical data from implemented software are key factors for Size Estimation of Embedded Software Components. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.86 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.85 Modern systems are becoming highly configurable to satisfy the varying needs of customers and users. However, high levels of configurability entail new challenges. Some faults might be revealed only if a particular combination of features is selected in the delivered products. But testing all combinations is usually not feasible in practice, due to their extremely large numbers. Combinatorial testing is a technique to generate smaller test suites for which all combinations of t features are guaranteed to be tested. In this paper, we formally prove that, when there are no constraints among the features that can be part of a product, then random testing is much more effective than one would intuitively expect at detecting feature interaction faults when compared with combinatorial testing. More importantly, random testing becomes even more effective as the number of features increases and converges towards equal effectiveness with combinatorial testing. Given that combinatorial testing entails significant computational overhead in the presence of large numbers of features, the results suggest that there are realistic scenarios in which random testing may outperform combinatorial testing in large systems. However, when constraints are present among features, then random testing can fare arbitrarily worse than combinatorial testing. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.85 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.84 A fundamental problem of finding software applications that are highly relevant to development tasks is the mismatch between the high-level intent reflected in the descriptions of these tasks and low-level implementation details of applications. To reduce this mismatch we created an approach called Exemplar (EXEcutable exaMPLes ARchive) for finding highly relevant software projects from large archives of applications. After a programmer enters a natural-language query that contains high-level concepts (e.g., MIME, data sets), Exemplar retrieves applications that implement these concepts. Exemplar ranks applications in three ways. First, we consider the descriptions of applications. Second, we examine the Application Programming Interface (API) calls used by applcations. Third, we analyze the dataflow among those API calls.We performed two case studies (with professional and student developers) to evaluate how these three rankings contribute to the quality of the search results from Exemplar. The results of our studies show that the combined ranking of application descriptions and API documents yields the most-relevant search results. We released Exemplar and our case study data to the public. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.84 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.83 Inspired by our past manual aspect mining experience, this paper describes a probabilistic random walk model to approximate the process of discovering crosscutting concerns in the absence of domain knowledge about the application investigated. The random walks are performed on the concept graphs extracted from the program sources to calculate metrics of “utilization” and “aggregation” for each program element. We rank all the program elements based on these metrics and use a threshold to produce a set of candidates that represent crosscutting concerns (CCs). We implemented the algorithm as the Prism CC Miner (PCM) and evaluated PCM on Java applications ranging from a small-scale drawing application to a medium-sized middleware application and a large-scale enterprise application server. Our quantification shows that PCM is able to produce comparable results to manual mining efforts. With a 95% accuracy the top 125 crosscutting candidates are identified automatically. PCM is also significantly more effective as compared to conventional mining approaches. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.83 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.81 Reasoning about the performance of models of software systems typically entails the derivation of metrics such as throughput, utilisation, and response time. If the model is a Markov chain, these are expressed as real functions of the chain, called reward models. The computational complexity of reward-based metrics is of the same order as the solution of the Markov chain, making the analysis infeasible when evaluating large-scale systems. In the context of the stochastic process algebra PEPA, the underlying continuous-time Markov chain has been shown to admit a deterministic (fluid) approximation as a solution of an ordinary differential equation, which effectively circumvents state-space explosion. This paper is concerned with approximating Markovian reward models for PEPA with fluid rewards, i.e., functions of the solution of the differential equation problem. It shows that (a) the Markovian reward models for typical metrics of performance enjoy asymptotic convergence to their fluid analogues, and that (b), via numerical tests, the approximation yields satisfactory accuracy in practice. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.81 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.80 This paper considers the problem of reasoning about the reliability of fault-tolerant systems with two "channels" (i.e., components) of which one, A, supports only a claim of reliability, while the other, B, by virtue of extreme simplicity and extensive analysis, supports a plausible claim of "perfection." We begin with the case where either channel can bring the system to a safe state. We show that, conditional upon knowing pA (the probability that A fails on a randomly selected demand) and pB (the probability that channel B is imperfect), a conservative bound on the probability that the system fails on a randomly selected demand is simply pA.pB. That is, there is conditional independence between the events "A fails" and "B is imperfect." The second step of the reasoning involves epistemic uncertainty about (pA, pB) and we show that under quite plausible assumptions, a conservative bound on system pfd can be constructed from point estimates for just three parameters. We discuss the feasibility of establishing credible estimates for these parameters. We extend our analysis from faults of omission to those of commission, and then combine these to yield an analysis for monitored architectures of a kind proposed for aircraft. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.80 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.79 Context: Many software development practices are based upon codification of expert knowledge, often with little or no underpinning from objective empirical evidence. Objectives: To investigate how extensively the use of software design patterns has been subjected to empirical study, and what evidence is available about how and when their use can provide an effective mechanism for knowledge transfer about design. Method: We conducted a systematic literature review in the form of a mapping study, searching the literature up to the end of 2009 to identify relevant primary studies about the use of the 23 patterns catalogued in the book by the `Gang of Four'. Results: Our searches identified 611 candidate papers. Applying our inclusion/exclusion criteria resulted in a final set of ten papers that described 11 instances of `formal' experimental studies of object-oriented design patterns. We augmented our analysis by including seven `experience' reports that described application of patterns using less rigorous observational forms. Conclusions: There was some support for the usefulness of patterns in providing a framework for maintenance, and some qualitative indication that they do not help novices learn about design. We recommend that researchers use case studies that focus upon some key patterns, and seek to identify the impact that their use can have upon maintenance. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.79 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.77 Object-oriented frameworks are widely used to develop new applications. They provide reusable concepts that are instantiated in application code through potentially complex implementation steps such as subclassing, implementing interfaces, and calling framework operations. Unfortunately, many modern frameworks are difficult to use because of their large and complex APIs and frequently-incomplete user documentation. To cope with these problems, developers often use existing framework applications as a guide. However, locating concept implementations in those sample applications is typically challenging due to code tangling and scattering. To address this challenge, we introduce the notion of concept-implementation templates, which summarize the necessary concept-implementation steps and identify them in the sample application code, and a technique, named FUDA, to automatically extract such templates from dynamic traces of sample applications. This article further presents the results of two experiments conducted to evaluate the quality and usefulness of FUDA templates. The experimental evaluation of FUDA with 14 concepts in five widely-used frameworks suggests that the technique is effective in producing templates with relatively few false positives and false negatives for realistic concepts by using two sample applications. Moreover, we observed in a user study with 28 programmers that the use of templates reduced the concept-implementation time compared to when documentation was used. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.77 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.76 In recent years, there has been significant interest in fault-localization techniques that are based on statistical analysis of program constructs executed by passing and failing executions. This paper shows how the Tarantula, Ochiai, and Jaccard fault-localization algorithms can be enhanced to localize faults effectively in Web applications written in PHP, by using an extended domain for conditional and function-call statements, and by using a source mapping. We also propose several novel test-generation strategies that are geared towards producing test suites that have maximal fault-localization effectiveness. We implemented various fault-localization techniques and test-generation strategies in Apollo, and evaluated them on several open-source PHP applications. Our results indicate that a variant of the Ochiai algorithm that includes all our enhancements localizes 87.8% of all faults to within 1% of all executed statements, compared to only 37.4% for the unenhanced Ochiai algorithm. We also found that all the test-generation strategies that we considered are capable of generating test suites with maximal fault-localization effectiveness, when given an infinite time budget for test generation. However, on average, a directed strategy based on path-constraint similarity achieves this maximal effectiveness after generating only 6.5 tests, compared to 46.8 tests for an undirected test-generation strategy. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.76 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.70 Calling contexts are very important for a wide range of applications such as profiling, debugging, and event logging. Most applications perform expensive stack walking to recover contexts. The resulting contexts are often explicitly represented as a sequence of call sites and hence bulky. We propose a technique to encode the current calling context of any point during an execution. In particular, an acyclic call path is encoded into one number through only integer additions. Recursive call paths are divided into acyclic subsequences and encoded independently. We leverage stack depth in a safe way to optimize encoding: if a calling context can be safely and uniquely identified by its stack depth, we do not perform encoding. We propose an algorithm to seamlessly fuse encoding and stack depth based identification. The algorithm is safe because different contexts are guaranteed to have different IDs. It also ensures contexts can be faithfully decoded. Our experiments show that our technique incurs negligible overhead (0%-6.4% on average). For most medium-sized programs, it can encode all contexts with just one number. For large programs, we are able to encode most calling contexts to a few numbers. We also present our experience of applying context encoding to debugging crash based failures. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.70 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.69 This paper extends the CHOiCe reLATion framEwork, abbreviated as CHOC'LATE, which assists software testers in the application of category/choice methods to testing. CHOC'LATE assumes that the tester is able to construct a single choice relation table from the entire specification; this table then forms the basis for test case generation using the associated algorithms. This assumption, however, may not hold true when the specification is complex and contains many specification components. For such a specification, the tester may construct a preliminary choice relation table from each specification component, and then consolidate all the preliminary tables into a final table to be processed by CHOC'LATE for test case generation. However, it is often difficult to merge these preliminary tables because such merging may give rise to inconsistencies among choice relations or overlaps among choices. To alleviate this problem, we introduce a DividE-and-conquer methodology for identifying categorieS, choiceS, and choicE Relations for Test case generation, abbreviated as DESSERT. The theoretical framework and the associated algorithms are discussed. To demonstrate the viability and effectiveness of our methodology, we describe case studies using the specifications of three real-life commercial software systems. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.69 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.68 Architecting software systems according to the service-oriented paradigm, and designing runtime self-adaptable systems are two relevant research areas in today's software engineering. In this paper we address issues that lie at the intersection of these two important fields. First, we present a characterization of the problem space of self-adaptation for service-oriented systems, thus providing a frame of reference where our and other approaches can be classified. Then, we present MOSES, a methodology and a software tool implementing it to support QoS-driven adaptation of a service-oriented system. It works in a specific region of the identified problem space, corresponding to the scenario where a service-oriented system architected as a composite service needs to sustain a traffic of requests generated by several users. MOSES integrates within a unified framework different adaptation mechanisms. In this way it achieves a greater flexibility in facing various operating environments and the possibly conflicting QoS requirements of several concurrent users. Experimental results obtained with a prototype implementation of MOSES show the effectiveness of the proposed approach. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.68 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.67 Ambient Intelligence technologies have not yet been widely adopted in safety critical scenarios. This principally has been due to fact that acceptable degrees of dependability have not been reached for the applications that rely on such technologies. However, the new critical application domains, like Ambient Assisted Living and Smart Hospitals, which are currently emerging, are increasing the need for methodologies and tools that can improve the reliability of the final systems. This paper presents a middleware architecture for safety critical Ambient Intelligence applications, which provides the developer with services for runtime verification. It is now possible to continuously monitor and check the running system against correctness properties defined at the design time. Moreover, a visual tool, which allows the formal design of several of the characteristics of an Ambient Intelligence application and the automatic generation of setting up parameters and code for the middleware infrastructure, is also presented. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.67 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.66 Persistence is a widely used technique which allows the objects that represent the results of some lengthy computation to outlive the process that creates it, in order to considerably speed-up subsequent program executions. We observe that conventional persistence techniques usually do not consider the application contexts of the persistence operations, where not all of the object states are necessary to be persisted. Leveraging this observation, we have designed and implemented a framework, PERT, which first performs static program analysis to estimate the actual usage of the persisted object, given the context of its usage in the program. The Pert runtime uses the statically computed information to make efficiently tailoring decisions to prune the redundant object states during the persistence operations. Our evaluation result shows that the Pert-based optimization can speedup the conventional persistence operations by 1 to 45 times. The amount of persisted data is also dramatically reduced, as the result of the application-aware tailoring. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.66 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.64 The earlier a conflict is detected, the easier it is to resolve – this is the main precept of workspace awareness. Workspace awareness seeks to provide users with information of relevant ongoing parallel changes occurring in private workspaces, thereby enabling the early detection and resolution of potential conflicts. The key approach is to unobtrusively inform developers of potential conflicts arising because of concurrent changes to the same file and dependency violations in ongoing parallel work. This paper describes our research goals, approach, and implementation of workspace awareness through Palantír and includes a comprehensive evaluation involving two laboratory experiments. We present both quantitative and qualitative results from the experiments, which demonstrate that the use of Palantír, as compared to not using Palantír: (1) leads to both earlier detection and earlier resolution of a larger number of conflicts, (2) leaves fewer conflicts unresolved in the code base that was ultimately checked in, and (3) involves reasonable overhead. Furthermore, we report on interesting changes in users’ behavior, especially how conflict resolution strategies changed among Palantír users. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.64 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.62 Reuse and composition are increasingly advocated and put into practice in modern software engineering. However, the software entities that are to be reused to build an application, e.g., services, have seldom been developed to integrate and to cope with the application requirements. As a consequence, they present mismatch, which directly hampers on their reusability and the possibility to compose them. Software Adaptation has become a hot topic as a non-intrusive solution to work mismatch out using corrective pieces named adaptors. However, adaptation is a complex issue, especially when behavioral interfaces, or conversations, are taken into account. In this article, we present state-of-the-art techniques to generate adaptors given the description of reused entities' conversations and an abstract specification of the way mismatch can be solved. We use a process algebra to encode the adaptation problem, and propose on-the-fly exploration and reduction techniques to compute adaptor protocols. Our approach follows the model-driven engineering paradigm, applied to service-oriented computing as a representative field of composition-based software engineering. We take service description languages as inputs of the adaptation process and we implement adaptors as centralized service compositions, i.e., orchestrations. Our approach is completely tool-supported. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.62 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.61 As software systems are becoming increasingly complex, the likelihood of faults and unexpected behaviors will naturally increase. Many researchers have developed methods to contain faults at run-time, by using software and hardware-based techniques to define protection domains. These approaches tend to impose isolation boundaries on software components that are static, and thus remain intact while the system is running. An unfortunate consequence of statically structured protection domains is that they may impose undue overhead on the communication between separate components. This paper proposes a new run-time technique called Mutable Protection Domains (MPD) that trades communication cost for fault isolation in our COMPOSITE operating system. MPD dynamically adapts hardware isolation between interacting software components, depending on observed communication "hot-paths", with the purpose of maximizing fault isolation where possible. In this sense, MPD naturally tends towards a system of maximal component isolation, while collapsing protection domains where costs are prohibitive. By increasing isolation for low-cost interacting components, MPD limits the scope of impact of future unexpected faults. We demonstrate the utility of MPD using a web server, and identify different hot-paths for different workloads, that dictate adaptations to system structure. Experiments show up to 40% improvement in throughput, compared to a statically-organized system, while maintaining high fault isolation. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.61 http://doi.ieeecomputersociety.org/10.1109/TSE.2011.60 Current and future information systems require a better understanding of the interactions between users and systems in order to improve system use, and ultimately, success. The use of personas as design tools is becoming more widespread as researchers and practitioners discover its benefits. This paper presents an empirical study comparing the performance of existing qualitative and quantitative clustering techniques for the task of identifying personas and grouping system users into those personas. A method based on Factor (Principal Components) Analysis performs better than two other methods which use Latent Semantic Analysis and Cluster Analysis as measured by similarity to expert manually defined clusters. http://doi.ieeecomputersociety.org/10.1109/TSE.2011.60