Open-source development processes have emerged as an effective approach to reduce cycle-time and decrease design, implementation, and quality assurance costs for certain types of software, particularly systems infrastructure software, such as operating systems, compilers and language processing tools, editors, and distribution middleware. This paper presents two contributions to the study of open-source software engineering. First, we describe the key challenges of open-source software, such as controlling long-term maintenance and evolution costs, ensuring acceptable levels of quality, sustaining end-user confidence and good will, and ensuring the coherency of system-wide software and usability properties. We illustrate how well-organized open-source projects make it easier to address many of these challenges compared with traditional closedsource approaches to building software. Second, we present the goals and methodology of the Skoll project, which focuses on developing and empirically validating novel open-source software quality assurance and optimization techniques to resolve key open-source challenges. We summarize the experimental design of a long-term case study of two widely used open-source middleware projects---ACE and TAO---that we are using in the Skoll project to devise, deploy, and evaluate techniques for improving software quality through continuous distributed testing and profiling. These techniques are designed to leverage common open-source project assets, such as the technological sophistication and extensive computing resources of worldwide user communities, open access to source, and ubiquitous web access, that can improve the quality and performance of opensource software significantly. 1.