Experimentation in Software Engineering, Second Edition

This textbook, which has become the leading source for empirical software engineering research through its earlier editions, provides an in-depth introduction to experimentation in software engineering. The focus is on the steps to go through when conducting experiments. It also introduces a structure for selecting a research design and an introduction to systematic literature reviews, surveys, and case study research. Experiments are valuable tools for all software engineers who evaluate and choose between different solutions, methods, techniques, languages, and tools. The book has three main parts. Part I introduces empirical research through a structure for selecting a research design and explains several essential areas related to empirical research. Moreover, it provides introductions to systematic literature reviews, experiments, surveys, and case studies. Part II devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendices. The book provides indispensable information regarding empirical studies, particularly for experiments, but also for systematic literature studies, surveys, and case studies. This new edition comprises substantial new material. This includes a chapter on selecting a research design, which helps put experimentation into a broader context of other research approaches. Furthermore, a new chapter on survey research has been added. Novel contributions concerning A/B testing, replications, open science, validity threats, paired comparison design and tools for statistical analysis have been added. Moreover, systematic literature studies, experiments, and case study research descriptions have been consolidated. The book has also been updated based on the evolution of software engineering since the previous edition's publication. The book is intended for students, teachers, researchers, and practitioners who would like to compare different options, for example, feature designs, or different development methods, techniques, languages, or tools. Teachers can use the book for undergraduate or graduate students; the book can be used as a course book on experimentation and empirical studies in general. Researchers will also benefit from the book by learning more about how to conduct empirical studies. Likewise, practitioners may use it as a “cookbook” when evaluating different solutions, alternatively for evaluating new methods or techniques before implementing them in their organisation. Endorsements Foreword from the First Edition Foreword from the First Edition Preface Software Engineering and Science Purpose Scope Target Audience Outline Exercises Acknowledgments Contents Part I Context 1 Introduction 1.1 Software Engineering Context 1.2 Science and Software Engineering 1.3 Exercises 2 Empirical Research 2.1 Selecting a Research Design 2.2 Strategy Phase 2.3 Tactical Phase 2.4 Operational Phase 2.5 Example Decision-Making Structure 2.6 Research Approach Comparison 2.7 Empirical Evaluation of Process Changes 2.8 Concluding Remarks 2.9 Exercises 3 Essential Areas in Empirical Research 3.1 Ethics 3.2 Replications 3.3 Theory in Software Engineering 3.4 Measurement 3.4.1 Basic Concepts 3.4.2 Scale Types 3.4.3 Objective and Subjective Measures 3.4.4 Direct or Indirect Measures 3.4.5 Measurements in Software Engineering 3.4.6 Measurements in Practice 3.5 Empiricism to Improve 3.5.1 Quality Improvement Paradigm 3.5.2 Experience Factory 3.5.3 Goal/Question/Metric Method 3.6 Empirically Based Technology Transfer 3.7 Exercises 4 Systematic Literature Studies 4.1 Definition of Systematic Literature Review 4.2 Planning the Review 4.3 Conducting the Review 4.4 Reporting the Review 4.5 Mapping Studies 4.6 Updating Systematic Literature Studies 4.7 Evolution of Systematic Literature Studies 4.8 Exercises 5 Surveys 5.1 Survey Characteristics 5.2 Survey Purposes 5.3 Survey Research Objective 5.4 Survey Design 5.5 Concluding Remarks 5.6 Exercises 6 Experiments 6.1 Experiment Principles 6.2 Variables, Treatments, Objects, and Subjects 6.3 Process 6.4 Overview 6.5 Exercises 7 Case Studies 7.1 Why Case Studies in Software Engineering? 7.2 Case Study Arrangements 7.3 Confounding Factors and Other Aspects 7.4 Case Study Research Process 7.5 Design and Planning 7.5.1 Case Study Planning 7.5.2 Case Study Protocol 7.6 Preparation and Data Collection 7.6.1 Interviews 7.6.2 Observations 7.6.3 Archival Analysis 7.6.4 Metrics 7.7 Data and Validity Analysis 7.7.1 Quantitative Data Analysis 7.7.2 Qualitative Data Analysis 7.7.3 Validity 7.8 Reporting 7.8.1 Characteristics 7.8.2 Structure 7.9 Exercises Part II Steps in the Experiment Process 8 Scoping 8.1 Scope Experiment 8.2 Example Experiment 8.3 Exercises 9 Planning 9.1 Context Selection 9.2 Hypothesis Formulation 9.3 Variables Selection 9.4 Selection of Subjects 9.5 Experiment Design 9.5.1 Choice of Experiment Design 9.5.2 General Design Principles 9.5.3 Standard Design Types 9.6 Instrumentation 9.7 Validity Evaluation 9.8 Detailed Description of Validity Threats 9.8.1 Conclusion Validity 9.8.2 Internal Validity 9.8.3 Construct Validity 9.8.4 External Validity 9.9 Priority Among Types of Validity Threats 9.10 Example Experiment 9.11 Exercises 10 Operation 10.1 Preparation 10.1.1 Commit Participants 10.1.2 Instrumentation Concerns 10.2 Execution 10.2.1 Data Collection 10.2.2 Experimental Environment 10.3 Data Validation 10.4 Example Operation 10.5 Exercises 11 Analysis and Interpretation 11.1 Descriptive Statistics 11.1.1 Measures of Central Tendency 11.1.2 Measures of Dispersion 11.1.3 Measures of Dependency 11.1.4 Graphical Visualization 11.2 Data Set Reduction 11.3 Hypothesis Testing 11.3.1 Basic Concept 11.3.2 Parametric and Non-parametric Tests 11.3.3 Overview of Tests 11.3.4 Tools for Data Analysis 11.3.5 t-Test 11.3.6 Mann-Whitney 11.3.7 F-Test 11.3.8 Paired t-Test 11.3.9 Wilcoxon 11.3.10 Sign Test 11.3.11 ANOVA (ANalysis Of VAriance) 11.3.12 Kruskal-Wallis 11.3.13 Chi-square 11.3.14 Model Adequacy Checking 11.3.15 Drawing Conclusions 11.4 Example Analysis 11.5 Exercises 12 Presentation and Package 12.1 Experiment Report Structure 12.2 Exercises Part III Example Experiments 13 Experiment Process Illustration 13.1 Scoping 13.1.1 Goal Definition 13.1.2 Summary of Scoping 13.2 Planning 13.2.1 Context Selection 13.2.2 Hypothesis Formulation 13.2.3 Variables Selection 13.2.4 Selection of Subjects 13.2.5 Experiment Design 13.2.6 Instrumentation 13.2.7 Validity Evaluation 13.3 Operation 13.3.1 Preparation 13.3.2 Execution 13.3.3 Data Validation 13.4 Analysis and Interpretation 13.4.1 Descriptive Statistics 13.4.2 Data Reduction 13.4.3 Hypothesis Testing 13.5 Summary 13.6 Conclusion 14 Are the Perspectives Really Different?: Further Experimentation on Scenario-Based Reading of Requirements 14.1 Introduction 14.2 Related Work 14.3 Research Questions 14.4 Experiment Planning 14.4.1 Variables 14.4.2 Hypotheses 14.4.3 Design 14.4.4 Threats to Validity 14.5 Experiment Operation 14.6 Data Analysis 14.6.1 Individual Performance for Different Perspectives 14.6.2 Defects Found by Different Perspectives 14.6.3 Is the Sample Size Large Enough? 14.6.4 Experience of Subjects 14.7 Interpretations of Results 14.8 Summary and Conclusions 14.9 Data on Individual Performance 14.10 Data on Defects Found by Perspectives 14.10.1 PG Document 14.10.2 ATM Document A Training Exercises A.1 Training A.1.1 Normally Distributed Data A.1.2 Experience A.1.3 Programming A.1.4 Design A.1.5 Inspections A.2 Reviewing A.3 Assignments A.3.1 Unit Test and Code Reviews A.3.2 Inspection Methods A.3.3 Requirements Notation B Statistical Tables References Index The purpose of Experimentation in Software Engineering: An Introduction is to introduce students, teachers, researchers, and practitioners to experimentation and experimental evaluation with a focus on software engineering. The objective is, in particular, to provide guidelines for performing experiments evaluating methods, techniques and tools in software engineering. The introduction is provided through a process perspective. The focus is on the steps that we go through to perform experiments and quasi-experiments. The process also includes other types of empirical studies. The motivation for the book emerged from the need for support we experienced when turning our software engineering research more experimental. Several books are available which either treat the subject in very general terms or focus on some specific part of experimentation; most focus on the statistical methods in experimentation. These are important, but there were few books elaborating on experimentation from a process perspective, none addressing experimentation in software engineering in particular. The scope of Experimentation in Software Engineering: An Introduction is primarily experiments in software engineering as a means for evaluating methods, techniques and tools. The book provides some information regarding empirical studies in general, including both case studies and surveys. The intention is to provide a brief understanding of these strategies and in particular to relate them to experimentation. Experimentation in Software Engineering: An Introduction is suitable for use as a textbook or a secondary text for graduate courses, and for researchers and practitioners interested in an empirical approach to software engineering.