GPU Parallel Program Development Using CUDA (Chapman & Hall/CRC Computational Science)

__GPU Parallel Program Development using CUDA__ teaches GPU programming by showing the differences among different families of GPUs. This approach prepares the reader for the next generation and future generations of GPUs. The book emphasizes concepts that will remain relevant for a long time, rather than concepts that are platform-specific. At the same time, the book also provides platform-dependent explanations that are as valuable as generalized GPU concepts. The book consists of three separate parts; it starts by explaining parallelism using CPU multi-threading in Part I. A few simple programs are used to demonstrate the concept of dividing a large task into multiple parallel sub-tasks and mapping them to CPU threads. Multiple ways of parallelizing the same task are analyzed and their pros/cons are studied in terms of both core and memory operation. Part II of the book introduces GPU massive parallelism. The same programs are parallelized on multiple Nvidia GPU platforms and the same performance analysis is repeated. Because the core and memory structures of CPUs and GPUs are different, the results differ in interesting ways. The end goal is to make programmers aware of all the good ideas, as well as the bad ideas, so readers can apply the good ideas and avoid the bad ideas in their own programs. Part III of the book provides pointer for readers who want to expand their horizons. It provides a brief introduction to popular CUDA libraries (such as cuBLAS, cuFFT, NPP, and Thrust),the OpenCL programming language, an overview of GPU programming using other programming languages and API libraries (such as Python, OpenCV, OpenGL, and Apple’s Swift and Metal,) and the deep learning library cuDNN. "GPU Parallel Program Development using CUDA teaches GPU programming by showing the differences among different families of GPUs. This approach prepares the reader for the next generation and future generations of GPUs. The book emphasizes concepts that will remain relevant for a long time, rather than concepts that are platform-specific. At the same time, the book also provides platform-dependent explanations that are as valuable as generalized GPU concepts. The book consists of three separate parts; it starts by explaining parallelism using CPU multi-threading in Part I. A few simple programs are used to demonstrate the concept of dividing a large task into multiple parallel sub-tasks and mapping them to CPU threads. Multiple ways of parallelizing the same task are analyzed and their pros/cons are studied in terms of both core and memory operation. Part II of the book introduces GPU massive parallelism. The same programs are parallelized on multiple Nvidia GPU platforms and the same performance analysis is repeated. Because the core and memory structures of CPUs and GPUs are different, the results differ in interesting ways. The end goal is to make programmers aware of all the good ideas, as well as the bad ideas, so readers can apply the good ideas and avoid the bad ideas in their own programs. Part III of the book provides pointer for readers who want to expand their horizons. It provides a brief introduction to popular CUDA libraries (such as cuBLAS, cuFFT, NPP, and Thrust),the OpenCL programming language, an overview of GPU programming using other programming languages and API libraries (such as Python, OpenCV, OpenGL, and Apple's Swift and Metal,) and the deep learning library cuDNN" -- Back cover Series Page......Page 3 Title Page......Page 6 Copyright......Page 7 Dedication......Page 8 Contents......Page 10 List of Figures......Page 20 List of Tables......Page 30 Preface......Page 34 About the Author......Page 36 Part I: Understanding CPU Parallelism......Page 38 1 Introduction to CPU Parallel Programming......Page 40 2 Developing Our First Parallel CPU Program......Page 64 3 Improving Our First Parallel CPU Program......Page 90 4 Understanding the Cores and Memory......Page 116 5 Thread Management and Synchronization......Page 144 Part II: GPU Programming Using CUDA......Page 172 6 Introduction to GPU Parallelism and CUDA......Page 174 7 CUDA Host/Device Programming Model......Page 222 8 Understanding GPU Hardware Architecture......Page 262 9 Understanding GPU Cores......Page 300 10 Understanding GPU Memory......Page 340 11 CUDA Streams......Page 382 Part III: More To Know......Page 418 12 CUDA Libraries......Page 420 13 Introduction to OpenCL......Page 434 14 Other GPU Programming Languages......Page 450 15 Deep Learning Using CUDA......Page 462 Bibliography......Page 472 Index......Page 476 This book provides a hands-on, class-tested introduction to CUDA and GPU programming. It begins by introducing CPU programming and the concepts of P-threads, thread programming, multi-tasking, and parallelism, and then interweaves those concepts into an introduction of GPU programming. Using Nvidia's new platform, based on Amazon EC2 and Web GPU, the book uses a standardized architecture, while also exploring other architectures and their differences. The book also covers GPU multi-threading and Global Memory, CUDA atomics and the use of libraries on GPUs. Example applications in image processing, face detection, and tumor detection are also included. This book teaches GPU programming by introducing CPU multi-threaded programming and bases GPU massively-parallel programming on this foundation. The differences among families of GPUs are also studied. The book also explores CUDA libraries, OpenCL, GPU programming with other languages and API libraries, and the deep learning library cuDNN.