We describe the Marama suite of meta-tools. This Eclipse-based toolset permits rapid specification of notational elements, metamodels, view editors and view-model mappings. It has a novel set of behavioural specification tools for both visual and model level behaviours. An integrated mapping tool provides model transformation and code generation support. The toolset has been applied to several significant application development tasks and has undergone a variety of evaluations.

Abstract – The increasing complexity of enterprise and distributed systems accompanying a move to grid and utility computing demands automated design, testing, deployment and monitoring of applications. In this paper, we present the Elba project and Mulini generator. The goal of Elba is creating automated staging and testing of complex enterprise systems before deployment to production. Automating the staging process lowers the cost of testing applications. Feedback from staging, especially when coupled with appropriate resource costs, can be used to ensure correct functionality and provisioning for the application. The Elba project extracts test parameters from production specifications, such as SLAs, and deployment specifications, and via the Mulini generator, creates staging plans for the application. We then demonstrate Mulini on an example application, TPC-W, and show how information from automated staging and monitoring allows us to refine application deployments easily based on performance and cost. I.

We present a new approach for performing load testing of web applications by simulating realistic user behaviour with stochastic form-oriented analysis models. Realism in the simulation of user behaviour is necessary in order to achieve valid testing results. In contrast to many other user models, web site navigation and time delay are modelled stochastically. The models can be constructed from sample data and can take into account effects of session history on user behaviour and the existence of different categories of users. The approach is implemented in an existing architecture modelling and performance evaluation tool and is integrated with existing methods for forward and reverse engineering. 1

Abstract. Predicting the performance of component-based applications is difficult due to the complexity of the underlying component technology. This problem is exacerbated when a messaging protocol is introduced to create a loosely coupled software architecture. Messaging uses asynchronous communication, and must address quality of service issues such as message persistence and flow control. In this paper, we present an approach to predicting the performance of Java 2 Enterprise Edition (J2EE) applications using messaging services. The prediction is done during application design, without access to the application implementation. This is achieved by modeling the interactions among J2EE and messaging components using queuing network models, calibrating the performance model with architecture attributes associated with these components, and populating the model parameters using a lightweight, application-independent benchmark. Benchmarking avoids the need for prototype testing in order to obtain the value of model parameters, and thus reduces the performance prediction effort. A case study is carried out to predict the performance of a J2EE application with asynchronous communication. Analysis of the resulting predictions shows the error is within 15%. 1

Abstract. As the complexity of large-scale enterprise applications increases, providing performance verification through staging becomes an important part of reducing business risks associated with violating sophisticated service-level agreement (SLA). Currently, performance verification during the staging process is accomplished through either an expensive, cumbersome manual approach or ad hoc automation. This paper describes an automation approach as part of the Elba project supporting monitoring and performance analysis of distributed multi-tiered applications that helps in bottleneck detection. We use machinelearning to determine service-level objectives (SLOs) satisfaction and locate bottlenecks in candidate deployment scenarios. We evaluate our tools with TPC-W, an on-line bookstore, and RUBiS, an on-line auction site.

Distributed applications typically interact with a number of heterogeneous and autonomous components that evolve independently. Methodical development of such applications can benefit from approaches based on domain-specific languages (DSLs). However, the evolution and customization of heterogeneous components introduces significant challenges to accommodating the syntax and semantics of a DSL in addition to the heterogeneous platforms on which they must run. In this paper, we address the challenge of implementing code generators for two such DSLs that are flexible (resilient to changes in generators or input formats), extensible (able to support multiple output targets and multiple input variants), and modular (generated code can be rewritten). Our approach, Clearwater, leverages XML and XSLT standards: XML supports extensibility and mutability for inprogress specification formats, and XSLT provides flexibility and extensibility for multiple target languages. Modularity arises from using XML meta-tags in the code generator itself, which supports controlled addition, subtraction, or replacement to the generated code via XML-weaving. We discuss the use of our approach and show its advantages in two non-trivial code generators: the Infopipe Stub Generator (ISG) to support distributed flow applications, and the Automated Composable Code Translator to support automated distributed application deployment. As an example, the ISG accepts as input an XML description and generates output for C, C++, or Java using a number of communications platforms such as sockets and publish-subscribe. Categories and Subject Descriptors D.2.11 [Software Engineering]: Software Architectures – languages (e.g., description, interconnection, definition), domainspecific

The increasing scale and success of distributed multi-tier applications have created increasingly dynamic workload variations that made system performance less predictable. Consequently, staging has become a significant and useful method to characterize the performance and Quality of Service (QoS) of such applications. Manual staging is an expensive, time consuming and error-prone process. In particular, manually exploring a large configuration parameter space of the applications is a cumbersome task. In this article, we outline the design of Mulini, an automated staging framework for large-scale multi-tier applications that realizes the automation via an extensible and flexible code generator. Mulini adopts XSLT/XPath tools and aspectoriented programming (AOP) techniques to manipulate XMLencoded high-level specifications and weave non-functional specifications (e.g., QoS) into staging implementation. To illustrate the usability of the Mulini code generator in complex staging, we apply Mulini to bottleneck detection and observation-based performance characterization of the RUBiS e-Commerce benchmark. Categories and Subject Descriptors D.2.11 [Software Engineering]: Software Architectures – languages (e.g., description, interconnection, definition), domainspecific

Accurate web application performance testing relies on the use of loading tests based on a realistic client behaviour load model. Unfortunately developing such load models and associated test plans and scripts is tedious and error-prone with most existing web performance testing tools providing limited client load modelling capabilities. We describe a new approach and toolset that we have developed, MaramaMTE+, which improves the ability to model realistic web client load behaviour, automatically generates complex web application testing plans and scripts, and integrates load behaviour modelling with a generic performance engineering tool. MaramaMTE+ uses a stochastic form chart as its client loading model. A 3 rd party web crawler application extracts structural information from a target web site, aggregating the collected data into a crawler database that is then used for form chart model generation. The performance engineer then augments this synthesized form chart with client loading probabilities. Realistic web loading tests for a 3 rd party web load testing tool are then automatically generated from this resultant stochastic form chart client load model. We describe the development of our MaramaMTE+ environment, example usage of the tool, and compare and contrast the results obtained from our generated performance load tests against hand-built 3 rd party tool load tests.

Domain-Specific Visual Languages (DSVLs) have captured the attention of the programming language world with their simplicity and high-level abstraction. This has encouraged many to use DSVLs as a way to write programs. With little or no programming knowledge, many end-users can program tasks that would be beyond them with conventional programming. Despite their benefits however, DSVLs need validation, just as conventional programs do. Tests are typically done manually and few DSVLs support testing processes inside the language or tool. Motivated by this, we propose a high-level visual test specification model that resides inside DSVL programs. This specification model enables users to design tests within their domains, providing a way to validate their development models.

Abstract—Cloud computing is an emerging technology paradigm that revolutionizes the computing landscape by providing on-demand delivery of software, platform, and infrastructure over the Internet. Yet, architecting, deploying, and configuring enterprise applications to run well on modern clouds remains a challenge due to associated complexities and non-trivial implications. The natural and presumably unbiased approach to these questions is thorough testing before moving applications to production settings. However, thorough testing of enterprise applications on modern clouds is cumbersome and error-prone due to a large number of relevant scenarios and difficulties in testing process. We address some of these challenges through Expertus—a flexible code generation framework for automated performance testing of distributed applications in Infrastructure as a Service (IaaS) clouds. Expertus uses a multi-pass compiler approach and leverages template-driven code generation to modularly incorporate different software applications on IaaS clouds. Expertus automatically handles complex configuration dependencies of software applications and significantly reduces human errors associated with manual approaches for software configuration and testing. To date, Expertus has been used to study three distributed applications on five IaaS clouds with over 10,000 different hardware, software, and virtualization configurations. The flexibility and extensibility of Expertus and our own experience on using it shows that new clouds, applications, and software packages can easily be incorporated.