CUDA Programming: A Developer's Guide to Parallel Computing with GPUs (Applications of Gpu Computing)

"If you need to learn CUDA but don't have experience with parallel computing, CUDA Programming: A Developer's Introduction offers a detailed guide to CUDA with a grounding in parallel fundamentals. It starts by introducing CUDA and bringing you up to speed on GPU parallelism and hardware, then delving into CUDA installation. Chapters on core concepts including threads, blocks, grids, and memory focus on both parallel and CUDA-specific issues. Later, the book demonstrates CUDA in practice for optimizing applications, adjusting to new hardware, and solving common problems. Comprehensive introduction to parallel programming with CUDA, for readers new to both. Detailed instructions help readers optimize the CUDA software development kit. Practical techniques illustrate working with memory, threads, algorithms, resources, and more. Covers CUDA on multiple hardware platforms: Mac, Linux and Windows with several NVIDIA chipsets. Each chapter includes exercises to test reader knowledge."-- From publisher description Front Cover 1 CUDA Programming: A Developer’s Guide to Parallel Computing with GPUs 4 Copyright 5 Contents 6 Preface 14 Chapter 1 - A Short History of Supercomputing 16 INTRODUCTION 16 VON NEUMANN ARCHITECTURE 17 CRAY 20 CONNECTION MACHINE 21 CELL PROCESSOR 22 MULTINODE COMPUTING 24 THE EARLY DAYS OF GPGPU CODING 26 THE DEATH OF THE SINGLE-CORE SOLUTION 27 NVIDIA AND CUDA 28 GPU HARDWARE 30 ALTERNATIVES TO CUDA 31 CONCLUSION 34 Chapter 2 - Understanding Parallelism with GPUs 36 INTRODUCTION 36 TRADITIONAL SERIAL CODE 36 SERIAL/PARALLEL PROBLEMS 38 CONCURRENCY 39 TYPES OF PARALLELISM 42 FLYNN’S TAXONOMY 45 SOME COMMON PARALLEL PATTERNS 46 CONCLUSION 51 Chapter 3 - CUDA Hardware Overview 52 PC ARCHITECTURE 52 GPU HARDWARE 57 CPUS AND GPUS 61 COMPUTE LEVELS 61 Chapter 4 - Setting Up CUDA 68 INTRODUCTION 68 INSTALLING THE SDK UNDER WINDOWS 68 VISUAL STUDIO 69 LINUX 73 MAC 77 INSTALLING A DEBUGGER 77 COMPILATION MODEL 81 ERROR HANDLING 82 CONCLUSION 83 Chapter 5 - Grids, Blocks, and Threads 84 WHAT IT ALL MEANS 84 THREADS 84 BLOCKS 93 GRIDS 98 WARPS 106 BLOCK SCHEDULING 110 A PRACTICAL EXAMPLE—HISTOGRAMS 112 CONCLUSION 118 Chapter 6 - Memory Handling with CUDA 122 INTRODUCTION 122 CACHES 123 REGISTER USAGE 126 SHARED MEMORY 135 CONSTANT MEMORY 165 GLOBAL MEMORY 182 TEXTURE MEMORY 215 CONCLUSION 217 Chapter 7 - Using CUDA in Practice 218 INTRODUCTION 218 SERIAL AND PARALLEL CODE 218 PROCESSING DATASETS 224 PROFILING 234 AN EXAMPLE USING AES 246 CONCLUSION 280 References 281 Chapter 8 - Multi-CPU and Multi-GPU Solutions 282 INTRODUCTION 282 LOCALITY 282 MULTI-CPU SYSTEMS 282 MULTI-GPU SYSTEMS 283 ALGORITHMS ON MULTIPLE GPUS 284 WHICH GPU? 285 SINGLE-NODE SYSTEMS 289 STREAMS 290 MULTIPLE-NODE SYSTEMS 305 CONCLUSION 316 Chapter 9 - Optimizing Your Application 320 STRATEGY 1: PARALLEL/SERIAL GPU/CPU PROBLEM BREAKDOWN 320 STRATEGY 2: MEMORY CONSIDERATIONS 335 STRATEGY 3: TRANSFERS 349 STRATEGY 4: THREAD USAGE, CALCULATIONS, AND DIVERGENCE 376 STRATEGY 5: ALGORITHMS 401 STRATEGY 6: RESOURCE CONTENTIONS 429 STRATEGY 7: SELF-TUNING APPLICATIONS 450 CONCLUSION 454 Chapter 10 - Libraries and SDK 456 INTRODUCTION 456 LIBRARIES 456 CUDA COMPUTING SDK 490 DIRECTIVE-BASED PROGRAMMING 506 WRITING YOUR OWN KERNELS 514 CONCLUSION 517 Chapter 11 - Designing GPU-Based Systems 518 INTRODUCTION 518 CPU PROCESSOR 520 GPU DEVICE 522 PCI-E BUS 524 GEFORCE CARDS 525 CPU MEMORY 525 AIR COOLING 527 LIQUID COOLING 528 DESKTOP CASES AND MOTHERBOARDS 532 MASS STORAGE 533 POWER CONSIDERATIONS 537 OPERATING SYSTEMS 540 CONCLUSION 541 Chapter 12 - Common Problems, Causes, and Solutions 542 INTRODUCTION 542 ERRORS WITH CUDA DIRECTIVES 542 PARALLEL PROGRAMMING ISSUES 551 ALGORITHMIC ISSUES 559 FINDING AND AVOIDING ERRORS 562 DEVELOPING FOR FUTURE GPUS 570 FURTHER RESOURCES 575 CONCLUSION 577 References 578 Index 580

If you need to learn CUDA but don't have experience with parallel computing, CUDA Programming: A Developer's Introduction offers a detailed guide to CUDA with a grounding in parallel fundamentals. It starts by introducing CUDA and bringing you up to speed on GPU parallelism and hardware, then delving into CUDA installation. Chapters on core concepts including threads, blocks, grids, and memory focus on both parallel and CUDA-specific issues. Later, the book demonstrates CUDA in practice for optimizing applications, adjusting to new hardware, and solving common problems.

• Comprehensive introduction to parallel programming with CUDA, for readers new to both
• Detailed instructions help readers optimize the CUDA software development kit
• Practical techniques illustrate working with memory, threads, algorithms, resources, and more
• Covers CUDA on multiple hardware platforms: Mac, Linux and Windows with several NVIDIA chipsets
• Each chapter includes exercises to test reader knowledge