We report on our experiences developing a performance test-bed generator for industrial usage by extending an open-source UML CASE tool. This tool generates client and server code, database configuration and deployment scripts from a high-level software architecture description. It automates the code generation, compilation, deployment and performance metric result collection processes. We identify a range of problems that arose from our previous research on performance test-bed generation that needed to be addressed to scale this automated software engineering technique. We describe a range of approaches we used to solve these problems in our new tool. We then report on industrial deployment and evaluation of our new tool and discuss the effectiveness of these solutions.

Abstract – The increasing complexity of enterprise and distributed systems accompanying a move to grid and utility computing demands automated design, testing, deployment and monitoring of applications. In this paper, we present the Elba project and Mulini generator. The goal of Elba is creating automated staging and testing of complex enterprise systems before deployment to production. Automating the staging process lowers the cost of testing applications. Feedback from staging, especially when coupled with appropriate resource costs, can be used to ensure correct functionality and provisioning for the application. The Elba project extracts test parameters from production specifications, such as SLAs, and deployment specifications, and via the Mulini generator, creates staging plans for the application. We then demonstrate Mulini on an example application, TPC-W, and show how information from automated staging and monitoring allows us to refine application deployments easily based on performance and cost. I.

Distributed applications typically interact with a number of heterogeneous and autonomous components that evolve independently. Methodical development of such applications can benefit from approaches based on domain-specific languages (DSLs). However, the evolution and customization of heterogeneous components introduces significant challenges to accommodating the syntax and semantics of a DSL in addition to the heterogeneous platforms on which they must run. In this paper, we address the challenge of implementing code generators for two such DSLs that are flexible (resilient to changes in generators or input formats), extensible (able to support multiple output targets and multiple input variants), and modular (generated code can be rewritten). Our approach, Clearwater, leverages XML and XSLT standards: XML supports extensibility and mutability for inprogress specification formats, and XSLT provides flexibility and extensibility for multiple target languages. Modularity arises from using XML meta-tags in the code generator itself, which supports controlled addition, subtraction, or replacement to the generated code via XML-weaving. We discuss the use of our approach and show its advantages in two non-trivial code generators: the Infopipe Stub Generator (ISG) to support distributed flow applications, and the Automated Composable Code Translator to support automated distributed application deployment. As an example, the ISG accepts as input an XML description and generates output for C, C++, or Java using a number of communications platforms such as sockets and publish-subscribe. Categories and Subject Descriptors D.2.11 [Software Engineering]: Software Architectures – languages (e.g., description, interconnection, definition), domainspecific

Date Version Reviewer Comment The ability to effectively manage the different steps of the service lifecycle is fundamental to the success of mobile service platforms. Indeed, while service lifecycle has been traditionally viewed as largely isolated steps, the need to take into account the changing mobile environment and the distribution of resources and service components is leading to stronger intertwining. This white paper presents the work on service lifecycle management carried out within projects of the Mobile Services Platform cluster. Specifically, the problems of service creation, monitoring, validation, deployment, discovery and composition in B3G networks are addressed. While these projects have investigated the same research domains, the variations in the targeted application domains or in the underlying infrastructure have lead to different approaches and solutions. This white paper was produced as a result of work in the Mobile Service Platforms (MSP) cluster of IST FP6 projects. The MSP cluster is part of the Sixth Framework Program of European research activities IST (Information Society Technologies). The cluster covers all

The increasing scale and success of distributed multi-tier applications have created increasingly dynamic workload variations that made system performance less predictable. Consequently, staging has become a significant and useful method to characterize the performance and Quality of Service (QoS) of such applications. Manual staging is an expensive, time consuming and error-prone process. In particular, manually exploring a large configuration parameter space of the applications is a cumbersome task. In this article, we outline the design of Mulini, an automated staging framework for large-scale multi-tier applications that realizes the automation via an extensible and flexible code generator. Mulini adopts XSLT/XPath tools and aspectoriented programming (AOP) techniques to manipulate XMLencoded high-level specifications and weave non-functional specifications (e.g., QoS) into staging implementation. To illustrate the usability of the Mulini code generator in complex staging, we apply Mulini to bottleneck detection and observation-based performance characterization of the RUBiS e-Commerce benchmark. Categories and Subject Descriptors D.2.11 [Software Engineering]: Software Architectures – languages (e.g., description, interconnection, definition), domainspecific

Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, such as distribution topology, communication failure, and timing. Furthermore, they do not provide the engineer with a means to evaluate the relative effectiveness of different criteria nor the relative effectiveness of adequate test suites satisfying a given criterion. This paper makes three contributions to the development and use of test adequacy criteria for distributed systems: 1) a testing method based on discrete-event simulations, 2) a fault-based analysis technique for evaluating test suites and adequacy criteria, and 3) a series of case studies that validate the method and technique. The testing method uses a discrete-event simulation as an operational specification of a system in which the behavioral effects of distribution are explicitly represented. Adequacy criteria and test cases are then defined in terms of this simulation-based specification. The fault-based analysis involves mutation of the simulation-based specification to provide a foil against which test suites and the criteria that formed them can be evaluated. Three distributed systems were used to validate the method and technique, including the Domain Name System (DNS).

Domain-Specific Visual Languages (DSVLs) have captured the attention of the programming language world with their simplicity and high-level abstraction. This has encouraged many to use DSVLs as a way to write programs. With little or no programming knowledge, many end-users can program tasks that would be beyond them with conventional programming. Despite their benefits however, DSVLs need validation, just as conventional programs do. Tests are typically done manually and few DSVLs support testing processes inside the language or tool. Motivated by this, we propose a high-level visual test specification model that resides inside DSVL programs. This specification model enables users to design tests within their domains, providing a way to validate their development models.

Abstract—Cloud computing is an emerging technology paradigm that revolutionizes the computing landscape by providing on-demand delivery of software, platform, and infrastructure over the Internet. Yet, architecting, deploying, and configuring enterprise applications to run well on modern clouds remains a challenge due to associated complexities and non-trivial implications. The natural and presumably unbiased approach to these questions is thorough testing before moving applications to production settings. However, thorough testing of enterprise applications on modern clouds is cumbersome and error-prone due to a large number of relevant scenarios and difficulties in testing process. We address some of these challenges through Expertus—a flexible code generation framework for automated performance testing of distributed applications in Infrastructure as a Service (IaaS) clouds. Expertus uses a multi-pass compiler approach and leverages template-driven code generation to modularly incorporate different software applications on IaaS clouds. Expertus automatically handles complex configuration dependencies of software applications and significantly reduces human errors associated with manual approaches for software configuration and testing. To date, Expertus has been used to study three distributed applications on five IaaS clouds with over 10,000 different hardware, software, and virtualization configurations. The flexibility and extensibility of Expertus and our own experience on using it shows that new clouds, applications, and software packages can easily be incorporated.