Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice (Livelessons)

__“If this book had been available to Healthcare.gov’s contractors, and they read and followed its life cycle performance processes, there would not have been the enormous problems apparent in that application. In my 40+ years of experience in building leading-edge products, poor performance is the single most frequent cause of the failure or cancellation of software-intensive projects. This book provides techniques and skills necessary to implement performance engineering at the beginning of a project and manage it throughout the product’s life cycle. I cannot recommend it highly enough.”__ **–** __Don Shafer, CSDP, Technical Fellow, Athens Group, LLC__ Poor performance is a frequent cause of software project failure. Performance engineering can be extremely challenging. In **__Foundations of Software and System__** **__Performance Engineering,__** leading software performance expert Dr. André Bondi helps you create effective performance requirements up front, and then architect, develop, test, and deliver systems that meet them. Drawing on many years of experience at Siemens, AT&T Labs, Bell Laboratories, and two startups, Bondi offers practical guidance for every software stakeholder and development team participant. He shows you how to define and use metrics; plan for diverse workloads; evaluate scalability, capacity, and responsiveness; and test both individual components and entire systems. Throughout, Bondi helps you link performance engineering with everything else you do in the software life cycle, so you can achieve the right performance–now and in the future–at lower cost and with less pain. **This guide will help you** • Mitigate the business and engineering risk associated with poor system performance • Specify system performance requirements in business and engineering terms • Identify metrics for comparing performance requirements with actual performance • Verify the accuracy of measurements • Use simple mathematical models to make predictions, plan performance tests, and anticipate the impact of changes to the system or the load placed upon it • Avoid common performance and scalability mistakes • Clarify business and engineering needs to be satisfied by given levels of throughput and response time • Incorporate performance engineering into agile processes • Help stakeholders of a system make better performance-related decisions • Manage stakeholders’ expectations about system performance throughout the software life cycle, and deliver a software product with quality performance **André B. Bondi** is a senior staff engineer at Siemens Corp., Corporate Technologies in Princeton, New Jersey. His specialties include performance requirements, performance analysis, modeling, simulation, and testing. Bondi has applied his industrial and academic experience to the solution of performance issues in many problem domains. In addition to holding a doctorate in computer science and a master’s in statistics, he is a Certified Scrum Master. Cover 1 Title Page 6 Copyright Page 7 Contents 10 Preface 24 Acknowledgments 30 About the Author 32 Chapter 1 Why Performance Engineering? Why Performance Engineers? 34 1.1 Overview 34 1.2 The Role of Performance Requirements in Performance Engineering 37 1.3 Examples of Issues Addressed by Performance Engineering Methods 38 1.4 Business and Process Aspects of Performance Engineering 39 1.5 Disciplines and Techniques Used in Performance Engineering 41 1.6 Performance Modeling, Measurement, and Testing 43 1.7 Roles and Activities of a Performance Engineer 44 1.8 Interactions and Dependencies between Performance Engineering and Other Activities 46 1.9 A Road Map through the Book 48 1.10 Summary 50 Chapter 2 Performance Metrics 52 2.1 General 52 2.2 Examples of Performance Metrics 56 2.3 Useful Properties of Performance Metrics 57 2.4 Performance Metrics in Different Domains 59 2.4.1 Conveyor in a Warehouse 60 2.4.2 Fire Alarm Control Panel 61 2.4.3 Train Signaling and Departure Boards 62 2.4.4 Telephony 63 2.4.5 An Information Processing Example: Order Entry and Customer Relationship Management 63 2.5 Examples of Explicit and Implicit Metrics 65 2.6 Time Scale Granularity of Metrics 65 2.7 Performance Metrics for Systems with Transient, Bounded Loads 66 2.8 Summary 68 2.9 Exercises 68 Chapter 3 Basic Performance Analysis 70 3.1 How Performance Models Inform Us about Systems 70 3.2 Queues in Computer Systems and in Daily Life 71 3.3 Causes of Queueing 72 3.4 Characterizing the Performance of a Queue 75 3.5 Basic Performance Laws: Utilization Law, Little’s Law 78 3.5.1 Utilization Law 78 3.5.2 Little’s Law 80 3.6 A Single-Server Queue 82 3.7 Networks of Queues: Introduction and Elementary Performance Properties 85 3.7.1 System Features Described by Simple Queueing Networks 86 3.7.2 Quantifying Device Loadings and Flow through a Computer System 87 3.7.3 Upper Bounds on System Throughput 89 3.7.4 Lower Bounds on System Response Times 91 3.8 Open and Closed Queueing Network Models 91 3.8.1 Simple Single-Class Open Queueing Network Models 92 3.8.2 Simple Single-Class Closed Queueing Network Model 93 3.8.3 Performance Measures and Queueing Network Representation: A Qualitative View 95 3.9 Bottleneck Analysis for Single-Class Closed Queueing Networks 96 3.9.1 Asymptotic Bounds on Throughput and Response Time 96 3.9.2 The Impact of Asynchronous Activity on Performance Bounds 99 3.10 Regularity Conditions for Computationally Tractable Queueing Network Models 101 3.11 Mean Value Analysis of Single-Class Closed Queueing Network Models 102 3.12 Multiple-Class Queueing Networks 104 3.13 Finite Pool Sizes, Lost Calls, and Other Lost Work 108 3.14 Using Models for Performance Prediction 110 3.15 Limitations and Applicability of Simple Queueing Network Models 111 3.16 Linkage between Performance Models, Performance Requirements, and Performance Test Results 112 3.17 Applications of Basic Performance Laws to Capacity Planning and Performance Testing 113 3.18 Summary 113 3.19 Exercises 114 Chapter 4 Workload Identification and Characterization 118 4.1 Workload Identification 118 4.2 Reference Workloads for a System in Different Environments 120 4.3 Time-Varying Behavior 122 4.4 Mapping Application Domains to Computer System Workloads 124 4.4.1 Example: An Online Securities Trading System for Account Holders 124 4.4.2 Example: An Airport Conveyor System 125 4.4.3 Example: A Fire Alarm System 127 4.5 Numerical Specification of the Workloads 128 4.5.1 Example: An Online Securities Trading System for Account Holders 129 4.5.2 Example: An Airport Conveyor System 130 4.5.3 Example: A Fire Alarm System 131 4.6 Numerical Illustrations 132 4.6.1 Numerical Data for an Online Securities Trading System 133 4.6.2 Numerical Data for an Airport Conveyor System 134 4.6.3 Numerical Data for the Fire Alarm System 135 4.7 Summary 136 4.8 Exercises 136 Chapter 5 From Workloads to Business Aspects of Performance Requirements 138 5.1 Overview 138 5.2 Performance Requirements and Product Management 139 5.2.1 Sizing for Different Market Segments: Linking Workloads to Performance Requirements 140 5.2.2 Performance Requirements to Meet Market, Engineering, and Regulatory Needs 141 5.2.3 Performance Requirements to Support Revenue Streams 143 5.3 Performance Requirements and the Software Lifecycle 144 5.4 Performance Requirements and the Mitigation of Business Risk 145 5.5 Commercial Considerations and Performance Requirements 147 5.5.1 Performance Requirements, Customer Expectations, and Contracts 147 5.5.2 System Performance and the Relationship between Buyer and Supplier 147 5.5.3 Confidentiality 148 5.5.4 Performance Requirements and the Outsourcing of Software Development 149 5.5.5 Performance Requirements and the Outsourcing of Computing Services 149 5.6 Guidelines for Specifying Performance Requirements 149 5.6.1 Performance Requirements and Functional Requirements 150 5.6.2 Unambiguousness 150 5.6.3 Measurability 151 5.6.4 Verifiability 152 5.6.5 Completeness 152 5.6.6 Correctness 153 5.6.7 Mathematical Consistency 153 5.6.8 Testability 153 5.6.9 Traceability 154 5.6.10 Granularity and Time Scale 155 5.7 Summary 155 5.8 Exercises 156 Chapter 6 Qualitative and Quantitative Types of Performance Requirements 158 6.1 Qualitative Attributes Related to System Performance 159 6.2 The Concept of Sustainable Load 160 6.3 Formulation of Response Time Requirements 161 6.4 Formulation of Throughput Requirements 163 6.5 Derived and Implicit Performance Requirements 164 6.5.1 Derived Performance Requirements 165 6.5.2 Implicit Requirements 165 6.6 Performance Requirements Related to Transaction Failure Rates, Lost Calls, and Lost Packets 167 6.7 Performance Requirements Concerning Peak and Transient Loads 168 6.8 Summary 169 6.9 Exercises 170 Chapter 7 Eliciting, Writing, and Managing Performance Requirements 172 7.1 Elicitation and Gathering of Performance Requirements 173 7.2 Ensuring That Performance Requirements Are Enforceable 176 7.3 Common Patterns and Antipatterns for Performance Requirements 177 7.3.1 Response Time Pattern and Antipattern 177 7.3.2 “... All the Time/... of the Time” Antipattern 178 7.3.3 Resource Utilization Antipattern 179 7.3.4 Number of Users to Be Supported Pattern/ Antipattern 179 7.3.5 Pool Size Requirement Pattern 180 7.3.6 Scalability Antipattern 180 7.4 The Need for Mathematically Consistent Requirements: Ensuring That Requirements Conform to Basic Performance Laws 181 7.5 Expressing Performance Requirements in Terms of Parameters with Unknown Values 182 7.6 Avoidance of Circular Dependencies 182 7.7 External Performance Requirements and Their Implications for the Performance Requirements of Subsystems 183 7.8 Structuring Performance Requirements Documents 183 7.9 Layout of a Performance Requirement 186 7.10 Managing Performance Requirements: Responsibilities of the Performance Requirements Owner 188 7.11 Performance Requirements Pitfall: Transition from a Legacy System to a New System 189 7.12 Formulating Performance Requirements to Facilitate Performance Testing 191 7.13 Storage and Reporting of Performance Requirements 193 7.14 Summary 194 Chapter 8 System Measurement Techniques and Instrumentation 196 8.1 General 196 8.2 Distinguishing between Measurement and Testing 200 8.3 Validate, Validate, Validate; Scrutinize, Scrutinize, Scrutinize 201 8.4 Resource Usage Measurements 201 8.4.1 Measuring Processor Usage 202 8.4.2 Processor Utilization by Individual Processes 204 8.4.3 Disk Utilization 206 8.4.4 Bandwidth Utilization 207 8.4.5 Queue Lengths 208 8.5 Utilizations and the Averaging Time Window 208 8.6 Measurement of Multicore or Multiprocessor Systems 210 8.7 Measuring Memory-Related Activity 213 8.7.1 Memory Occupancy 214 8.7.2 Paging Activity 214 8.8 Measurement in Production versus Measurement for Performance Testing and Scalability 214 8.9 Measuring Systems with One Host and with Multiple Hosts 216 8.9.1 Clock Synchronization of Multiple Hosts 217 8.9.2 Gathering Measurements from Multiple Hosts 217 8.10 Measurements from within the Application 219 8.11 Measurements in Middleware 220 8.12 Measurements of Commercial Databases 221 8.13 Response Time Measurements 222 8.14 Code Profiling 223 8.15 Validation of Measurements Using Basic Properties of Performance Metrics 224 8.16 Measurement Procedures and Data Organization 225 8.17 Organization of Performance Data, Data Reduction, and Presentation 228 8.18 Interpreting Measurements in a Virtualized Environment 228 8.19 Summary 229 8.20 Exercises 229 Chapter 9 Performance Testing 232 9.1 Overview of Performance Testing 232 9.2 Special Challenges 235 9.3 Performance Test Planning and Performance Models 236 9.4 A Wrong Way to Evaluate Achievable System Throughput 241 9.5 Provocative Performance Testing 242 9.6 Preparing a Performance Test 243 9.6.1 Understanding the System 244 9.6.2 Pilot Testing, Playtime, and Performance Test Automation 246 9.6.3 Test Equipment and Test Software Must Be Tested, Too 246 9.6.4 Deployment of Load Drivers 247 9.6.5 Problems with Testing Financial Systems 249 9.7 Lab Discipline in Performance Testing 250 9.8 Performance Testing Challenges Posed by Systems with Multiple Hosts 251 9.9 Performance Testing Scripts and Checklists 252 9.10 Best Practices for Documenting Test Plans and Test Results 253 9.11 Linking the Performance Test Plan to Performance Requirements 255 9.12 The Role of Performance Tests in Detecting and Debugging Concurrency Issues 256 9.13 Planning Tests for System Stability 258 9.14 Prospective Testing When Requirements Are Unspecified 259 9.15 Structuring the Test Environment to Reflect the Scalability of the Architecture 261 9.16 Data Collection 262 9.17 Data Reduction and Presentation 263 9.18 Interpreting the Test Results 264 9.18.1 Preliminaries 264 9.18.2 Example: Services Use Cases 264 9.18.3 Example: Transaction System with High Failure Rate 268 9.18.4 Example: A System with Computationally Intense Transactions 270 9.18.5 Example: System Exhibiting Memory Leak and Deadlocks 274 9.19 Automating Performance Tests and the Analysis of the Outputs 277 9.20 Summary 279 9.21 Exercises 279 Chapter 10 System Understanding, Model Choice, and Validation 284 10.1 Overview 285 10.2 Phases of a Modeling Study 287 10.3 Example: A Conveyor System 289 10.4 Example: Modeling Asynchronous I/O 293 10.5 Systems with Load-Dependent or Time-Varying Behavior 299 10.5.1 Paged Virtual Memory Systems That Thrash 299 10.5.2 Applications with Increasing Processing Time per Unit of Work 300 10.5.3 Scheduled Movement of Load, Periodic Loads, and Critical Peaks 300 10.6 Summary 301 10.7 Exercises 303 Chapter 11 Scalability and Performance 306 11.1 What Is Scalability? 306 11.2 Scaling Methods 308 11.2.1 Scaling Up and Scaling Out 309 11.2.2 Vertical Scaling and Horizontal Scaling 309 11.3 Types of Scalability 310 11.3.1 Load Scalability 310 11.3.2 Space Scalability 312 11.3.3 Space-Time Scalability 313 11.3.4 Structural Scalability 314 11.3.5 Scalability over Long Distances and under Network Congestion 314 11.4 Interactions between Types of Scalability 315 11.5 Qualitative Analysis of Load Scalability and Examples 316 11.5.1 Serial Execution of Disjoint Transactions and the Inability to Exploit Parallel Resources 316 11.5.2 Busy Waiting on Locks 319 11.5.3 Coarse Granularity Locking 320 11.5.4 Ethernet and Token Ring: A Comparison 320 11.5.5 Museum Checkrooms 322 11.6 Scalability Limitations in a Development Environment 325 11.7 Improving Load Scalability 326 11.8 Some Mathematical Analyses 328 11.8.1 Comparison of Semaphores and Locks for Implementing Mutual Exclusion 329 11.8.2 Museum Checkroom 331 11.9 Avoiding Scalability Pitfalls 332 11.10 Performance Testing and Scalability 335 11.11 Summary 336 11.12 Exercises 337 Chapter 12 Performance Engineering Pitfalls 340 12.1 Overview 341 12.2 Pitfalls in Priority Scheduling 341 12.3 Transient CPU Saturation Is Not Always a Bad Thing 345 12.4 Diminishing Returns with Multiprocessors or Multiple Cores 347 12.5 Garbage Collection Can Degrade Performance 348 12.6 Virtual Machines: Panacea or Complication? 348 12.7 Measurement Pitfall: Delayed Time Stamping and Monitoring in Real-Time Systems 350 12.8 Pitfalls in Performance Measurement 351 12.9 Eliminating a Bottleneck Could Unmask a New One 352 12.10 Pitfalls in Performance Requirements Engineering 354 12.11 Organizational Pitfalls in Performance Engineering 354 12.12 Summary 355 12.13 Exercises 356 Chapter 13 Agile Processes and Performance Engineering 358 13.1 Overview 358 13.2 Performance Engineering under an Agile Development Process 360 13.2.1 Performance Requirements Engineering Considerations in an Agile Environment 361 13.2.2 Preparation and Alignment of Performance Testing with Sprints 362 13.2.3 Agile Interpretation and Application of Performance Test Results 363 13.2.4 Communicating Performance Test Results in an Agile Environment 364 13.3 Agile Methods in the Implementation and Execution of Performance Tests 365 13.3.1 Identification and Planning of Performance Tests and Instrumentation 365 13.3.2 Using Scrum When Implementing Performance Tests and Purpose-Built Instrumentation 366 13.3.3 Peculiar or Irregular Performance Test Results and Incorrect Functionality May Go Together 367 13.4 The Value of Playtime in an Agile Performance Testing Process 367 13.5 Summary 369 13.6 Exercises 369 Chapter 14 Working with Stakeholders to Learn, Influence, and Tell the Performance Engineering Story 372 14.1 Determining What Aspect of Performance Matters to Whom 373 14.2 Where Does the Performance Story Begin? 374 14.3 Identification of Performance Concerns, Drivers, and Stakeholders 377 14.4 Influencing the Performance Story 378 14.4.1 Using Performance Engineering Concerns to Affect the Architecture and Choice of Technology 378 14.4.2 Understanding the Impact of Existing Architectures and Prior Decisions on System Performance 379 14.4.3 Explaining Performance Concerns and Sharing and Developing the Performance Story with Different Stakeholders 380 14.5 Reporting on Performance Status to Different Stakeholders 386 14.6 Examples 387 14.7 The Role of a Capacity Management Engineer 388 14.8 Example: Explaining the Role of Measurement Intervals When Interpreting Measurements 389 14.9 Ensuring Ownership of Performance Concerns and Explanations by Diverse Stakeholders 393 14.10 Negotiating Choices for Design Changes and Recommendations for System Improvement among Stakeholders 393 14.11 Summary 395 14.12 Exercises 396 Chapter 15 Where to Learn More 400 15.1 Overview 400 15.2 Conferences and Journals 402 15.3 Texts on Performance Analysis 403 15.4 Queueing Theory 405 15.5 Discrete Event Simulation 405 15.6 Performance Evaluation of Specific Types of Systems 406 15.7 Statistical Methods 407 15.8 Performance Tuning 407 15.9 Summary 408 References 410 Index 418 A 418 B 419 C 420 D 422 E 423 F 423 G 424 H 424 I 424 J 425 K 425 L 425 M 427 N 429 O 429 P 430 Q 435 R 436 S 437 T 441 U 442 V 443 W 443 Why Performance Engineering? Why Performance Engineers? -- Performance Metrics -- Basic Performance Analysis -- Workload Identification And Characterization -- From Workloads To Business Aspects Of Performance Requirements -- Qualitative And Quantitative Types Of Performance Requirements -- Eliciting, Writing, And Managing Performance Requirements -- System Measurement Techniques And Instrumentation -- Performance Testing -- System Understanding, Model Choice, And Validation -- Scalability And Performance -- Performance Engineering Pitfalls -- Agile Processes And Performance Engineering -- Working With Stakeholders To Learn, Influence, And Tell The -- Performance Engineering Story -- Where To Learn More. André B. Bondi. Includes Bibliographical References And Index. The absence of clearly written performance requirements is the cause of much confusion and bad software architectures. This book's coverage of performance requirements engineering and domain-specific performance metrics at every stage of the software process addresses the problem. Application of the principles in this book will considerably mitigate the risks that performance post to the success of a software system and lead to a better quality product with wider acceptance. Focuses on the #1 cause of software project failure or cancellation: poor performance Covers the entire software lifecyc