The increasing complexity of available infrastructures with specific features (caches, hyperthreading, dual core, etc.) or with complex architectures (hierarchical, parallel, distributed, etc.) makes models either extremely difficult to build or intractable. Hence, it raises the question: how to validate algorithms if a realistic analytic analysis is not possible any longer? As for some other sciences (physics, chemistry, biology, etc.), the answer partly falls in experimental validation. Nevertheless, experiment in computer science is a difficult subject that opens many questions: what an experiment is able to validate? What is a “good experiments”? How to build an experimental environment that allows for ”good experiments”? etc. In this paper we will provide some hints on this subject and show how some tools can help in performing “good experiments”. More precisely we will focus on three main experimental methodologies, namely real-scale experiments (with an emphasis on PlanetLab and Grid’5000), Emulation (with an emphasis on Wrekavoc:

Combinatorial optimization problems expressed as Boolean constraint satisfaction problems (BCSPs) arise in several contexts, ranging from the classical unate set-packing problems to the binate minimum cover problems, including the Haplotype Inference by Pure Parsimony (HIPP) problem. These problems are being solved under different formulations and in different formats. Results of experiments that are reported can be seldom compared and replicated. This paper is not about ‘the best BCSP solver’. Rather, it is a case study of how the scientific method can be applied to comparing the performance of not only BCSP solvers but also other solvers that address NP-hard problems. The approach is founded on two premises: (1) the introduction of instance isomorphs as families of equivalence classes, based on randomized replicas of a given reference instance, and (2) the use of isomorph classes for the design of reproducible experiments with BCSP solvers that includes performance testing hypotheses. We introduce a number of BCSP reference instances from different domains, generate isomorph classes and use various versions of cplex to characterize the solver performance and the isomorph classes themselves. This methodology may make it easier to (1) reliably improve the performance of combinatorial solvers and, (2) report results of experiments under the proposed schema. Categories and Subject Descriptors:

Computer scientists and software engineers seldom rely on using experimental methods despite frequent calls to do so. The problem may lie with the shortcomings of traditional experimental methods. We introduce a new form of experimental designs, synthetic designs, which address these shortcomings. Compared with classical experimental designs (between-subjects, within-subjects, and matched-subjects), synthetic designs can offer substantial reductions in sample sizes, cost, time and effort expended, increased statistical power, and fewer threats to validity (internal, external, and statistical conclusion). This new design is a variation of within-subjects design in which each system user serves in only a single treatment condition. System performance scores for all other treatment conditions are derived synthetically without repeated testing of each subject. This design, though not applicable in all situations, can be used in the development and testing of some computer systems provided that user behavior is unaffected by the version of computer system being used. We justify synthetic designs on three grounds: this design has been used successfully in the development of computerized mug shot systems, showing marked advantages over traditional designs; a detailed comparison with traditional designs showing their advantages on 17 of the 18 criteria considered; and an assessment showing these designs satisfy all the requirements of true experiments (albeit in a novel way).