MPI: A Message-Passing Interface Standard

Abstract History Contents List of Figures List of Tables Acknowledgments 1 Introduction to MPI 1.1 Overview and Goals 1.2 Background of MPI-1.0 1.3 Background of MPI-1.1, MPI-1.2, and MPI-2.0 1.4 Background of MPI-1.3 and MPI-2.1 1.5 Background of MPI-2.2 1.6 Background of MPI-3.0 1.7 Background of MPI-3.1 1.8 Background of MPI-4.0 1.9 Who Should Use This Standard? 1.10 What Platforms Are Targets for Implementation? 1.11 What Is Included in the Standard? 1.12 What Is Not Included in the Standard? 1.13 Organization of This Document 2 MPI Terms and Conventions 2.1 Document Notation 2.2 Naming Conventions 2.3 Procedure Specification 2.4 Semantic Terms 2.4.1 MPI Operations 2.4.2 MPI Procedures 2.4.3 MPI Datatypes 2.5 Datatypes 2.5.1 Opaque Objects 2.5.2 Array Arguments 2.5.3 State 2.5.4 Named Constants 2.5.5 Choice 2.5.6 Absolute Addresses and Relative Address Displacements 2.5.7 File Offsets 2.5.8 Counts 2.6 Language Binding 2.6.1 Deprecated and Removed Interfaces 2.6.2 Fortran Binding Issues 2.6.3 C Binding Issues 2.6.4 Functions and Macros 2.7 Processes 2.8 Error Handling 2.9 Implementation Issues 2.9.1 Independence of Basic Runtime Routines 2.9.2 Interaction with Signals 2.10 Examples 3 Point-to-Point Communication 3.1 Introduction 3.2 Blocking Send and Receive Operations 3.2.1 Blocking Send 3.2.2 Message Data 3.2.3 Message Envelope 3.2.4 Blocking Receive 3.2.5 Return Status 3.2.6 Passing MPI_STATUS_IGNORE for Status 3.2.7 Blocking Send-Receive 3.3 Datatype Matching and Data Conversion 3.3.1 Type Matching Rules Type MPI_CHARACTER 3.3.2 Data Conversion 3.4 Communication Modes 3.5 Semantics of Point-to-Point Communication 3.6 Buffer Allocation and Usage 3.6.1 Model Implementation of Buffered Mode 3.7 Nonblocking Communication 3.7.1 Communication Request Objects 3.7.2 Communication Initiation 3.7.3 Communication Completion 3.7.4 Semantics of Nonblocking Communications 3.7.5 Multiple Completions 3.7.6 Non-Destructive Test of status 3.8 Probe and Cancel 3.8.1 Probe 3.8.2 Matching Probe 3.8.3 Matched Receives 3.8.4 Cancel 3.9 Persistent Communication Requests 3.10 Null Processes 4 Partitioned Point-to-Point Communication 4.1 Introduction 4.2 Semantics of Partitioned Point-to-Point Communication 4.2.1 Communication Initialization and Starting with Partitioning 4.2.2 Communication Completion under Partitioning 4.2.3 Semantics of Communications in Partitioned Mode 4.3 Partitioned Communication Examples 4.3.1 Partition Communication with Threads/Tasks Using OpenMP 4.0 or later 4.3.2 Send-only Partitioning Example with Tasks and OpenMP version 4.0 or later 4.3.3 Send and Receive Partitioning Example with OpenMP version 4.0 or later 5 Datatypes 5.1 Derived Datatypes 5.1.1 Type Constructors with Explicit Addresses 5.1.2 Datatype Constructors 5.1.3 Subarray Datatype Constructor 5.1.4 Distributed Array Datatype Constructor 5.1.5 Address and Size Functions 5.1.6 Lower-Bound and Upper-Bound Markers 5.1.7 Extent and Bounds of Datatypes 5.1.8 True Extent of Datatypes 5.1.9 Commit and Free 5.1.10 Duplicating a Datatype 5.1.11 Use of General Datatypes in Communication 5.1.12 Correct Use of Addresses 5.1.13 Decoding a Datatype 5.1.14 Examples 5.2 Pack and Unpack 5.3 Canonical MPI_PACK and MPI_UNPACK 6 Collective Communication 6.1 Introduction and Overview 6.2 Communicator Argument 6.2.1 Specifics for Intra-Communicator Collective Operations 6.2.2 Applying Collective Operations to Inter-Communicators 6.2.3 Specifics for Inter-Communicator Collective Operations 6.3 Barrier Synchronization 6.4 Broadcast 6.4.1 Example using MPI_BCAST 6.5 Gather 6.5.1 Examples using MPI_GATHER, MPI_GATHERV 6.6 Scatter 6.6.1 Examples using MPI_SCATTER, MPI_SCATTERV 6.7 Gather-to-all 6.7.1 Example using MPI_ALLGATHER 6.8 All-to-All Scatter/Gather 6.9 Global Reduction Operations 6.9.1 Reduce 6.9.2 Predefined Reduction Operations 6.9.3 Signed Characters and Reductions 6.9.4 MINLOC and MAXLOC 6.9.5 User-Defined Reduction Operations Example of User-Defined Reduce 6.9.6 All-Reduce 6.9.7 Process-Local Reduction 6.10 Reduce-Scatter 6.10.1 MPI_REDUCE_SCATTER_BLOCK 6.10.2 MPI_REDUCE_SCATTER 6.11 Scan 6.11.1 Inclusive Scan 6.11.2 Exclusive Scan 6.11.3 Example using MPI_SCAN 6.12 Nonblocking Collective Operations 6.12.1 Nonblocking Barrier Synchronization 6.12.2 Nonblocking Broadcast Example using MPI_IBCAST 6.12.3 Nonblocking Gather 6.12.4 Nonblocking Scatter 6.12.5 Nonblocking Gather-to-all 6.12.6 Nonblocking All-to-All Scatter/Gather 6.12.7 Nonblocking Reduce 6.12.8 Nonblocking All-Reduce 6.12.9 Nonblocking Reduce-Scatter with Equal Blocks 6.12.10 Nonblocking Reduce-Scatter 6.12.11 Nonblocking Inclusive Scan 6.12.12 Nonblocking Exclusive Scan 6.13 Persistent Collective Operations 6.13.1 Persistent Barrier Synchronization 6.13.2 Persistent Broadcast 6.13.3 Persistent Gather 6.13.4 Persistent Scatter 6.13.5 Persistent Gather-to-all 6.13.6 Persistent All-to-All Scatter/Gather 6.13.7 Persistent Reduce 6.13.8 Persistent All-Reduce 6.13.9 Persistent Reduce-Scatter with Equal Blocks 6.13.10 Persistent Reduce-Scatter 6.13.11 Persistent Inclusive Scan 6.13.12 Persistent Exclusive Scan 6.14 Correctness 7 Groups, Contexts, Communicators, and Caching 7.1 Introduction 7.1.1 Features Needed to Support Libraries 7.1.2 MPI's Support for Libraries 7.2 Basic Concepts 7.2.1 Groups 7.2.2 Contexts 7.2.3 Intra-Communicators 7.2.4 Predefined Intra-Communicators 7.3 Group Management 7.3.1 Group Accessors 7.3.2 Group Constructors 7.3.3 Group Destructors 7.4 Communicator Management 7.4.1 Communicator Accessors 7.4.2 Communicator Constructors 7.4.3 Communicator Destructors 7.4.4 Communicator Info 7.5 Motivating Examples 7.5.1 Current Practice #1 7.5.2 Current Practice #2 7.5.3 (Approximate) Current Practice #3 7.5.4 Communication Safety Example 7.5.5 Library Example #1 7.5.6 Library Example #2 7.6 Inter-Communication 7.6.1 Inter-Communicator Accessors 7.6.2 Inter-Communicator Operations 7.6.3 Inter-Communication Examples Example 1: Three-Group ``Pipeline'' Example 2: Three-Group ``Ring'' 7.7 Caching 7.7.1 Functionality 7.7.2 Communicators 7.7.3 Windows 7.7.4 Datatypes 7.7.5 Error Class for Invalid Keyval 7.7.6 Attributes Example 7.8 Naming Objects 7.9 Formalizing the Loosely Synchronous Model 7.9.1 Basic Statements 7.9.2 Models of Execution Static Communicator Allocation Dynamic Communicator Allocation The General Case 8 Process Topologies 8.1 Introduction 8.2 Virtual Topologies 8.3 Embedding in MPI 8.4 Overview of the Functions 8.5 Topology Constructors 8.5.1 Cartesian Constructor 8.5.2 Cartesian Convenience Function: MPI_DIMS_CREATE 8.5.3 Graph Constructor 8.5.4 Distributed Graph Constructor 8.5.5 Topology Inquiry Functions 8.5.6 Cartesian Shift Coordinates 8.5.7 Partitioning of Cartesian Structures 8.5.8 Low-Level Topology Functions 8.6 Neighborhood Collective Communication 8.6.1 Neighborhood Gather 8.6.2 Neighbor Alltoall 8.7 Nonblocking Neighborhood Communication 8.7.1 Nonblocking Neighborhood Gather 8.7.2 Nonblocking Neighborhood Alltoall 8.8 Persistent Neighborhood Communication 8.8.1 Persistent Neighborhood Gather 8.8.2 Persistent Neighborhood Alltoall 8.9 An Application Example 9 MPI Environmental Management 9.1 Implementation Information 9.1.1 Version Inquiries 9.1.2 Environmental Inquiries Tag Values Host Rank IO Rank Clock Synchronization Inquire Processor Name 9.2 Memory Allocation 9.3 Error Handling 9.3.1 Error Handlers for Communicators 9.3.2 Error Handlers for Windows 9.3.3 Error Handlers for Files 9.3.4 Error Handlers for Sessions 9.3.5 Freeing Errorhandlers and Retrieving Error Strings 9.4 Error Codes and Classes 9.5 Error Classes, Error Codes, and Error Handlers 9.6 Timers and Synchronization 10 The Info Object 11 Process Initialization, Creation, and Management 11.1 Introduction 11.2 The World Model 11.2.1 Starting MPI Processes 11.2.2 Finalizing MPI 11.2.3 Determining Whether MPI Has Been Initialized When Using the World Model 11.2.4 Allowing User Functions at MPI Finalization 11.3 The Sessions Model 11.3.1 Session Creation and Destruction Methods 11.3.2 Processes Sets 11.3.3 Runtime Query Functions 11.3.4 Sessions Model Examples 11.4 Common Elements of Both Process Models 11.4.1 MPI Functionality that is Always Available 11.4.2 Aborting MPI Processes 11.5 Portable MPI Process Startup 11.6 MPI and Threads 11.6.1 General 11.6.2 Clarifications 11.7 The Dynamic Process Model 11.7.1 Starting Processes 11.7.2 The Runtime Environment 11.8 Process Manager Interface 11.8.1 Processes in MPI 11.8.2 Starting Processes and Establishing Communication 11.8.3 Starting Multiple Executables and Establishing Communication 11.8.4 Reserved Keys 11.8.5 Spawn Example 11.9 Establishing Communication 11.9.1 Names, Addresses, Ports, and All That 11.9.2 Server Routines 11.9.3 Client Routines 11.9.4 Name Publishing 11.9.5 Reserved Key Values 11.9.6 Client/Server Examples 11.10 Other Functionality 11.10.1 Universe Size 11.10.2 Singleton MPI Initialization 11.10.3 MPI_APPNUM 11.10.4 Releasing Connections 11.10.5 Another Way to Establish MPI Communication 12 One-Sided Communications 12.1 Introduction 12.2 Initialization 12.2.1 Window Creation 12.2.2 Window That Allocates Memory 12.2.3 Window That Allocates Shared Memory 12.2.4 Window of Dynamically Attached Memory 12.2.5 Window Destruction 12.2.6 Window Attributes 12.2.7 Window Info 12.3 Communication Calls 12.3.1 Put 12.3.2 Get 12.3.3 Examples for Communication Calls 12.3.4 Accumulate Functions Accumulate Function Get Accumulate Function Fetch and Op Function Compare and Swap Function 12.3.5 Request-based RMA Communication Operations 12.4 Memory Model 12.5 Synchronization Calls 12.5.1 Fence 12.5.2 General Active Target Synchronization 12.5.3 Lock 12.5.4 Flush and Sync 12.5.5 Assertions 12.5.6 Miscellaneous Clarifications 12.6 Error Handling 12.6.1 Error Handlers 12.6.2 Error Classes 12.7 Semantics and Correctness 12.7.1 Atomicity 12.7.2 Ordering 12.7.3 Progress 12.7.4 Registers and Compiler Optimizations 12.8 Examples 13 External Interfaces 13.1 Introduction 13.2 Generalized Requests 13.2.1 Examples 13.3 Associating Information with Status 14 I/O 14.1 Introduction 14.1.1 Definitions 14.2 File Manipulation 14.2.1 Opening a File 14.2.2 Closing a File 14.2.3 Deleting a File 14.2.4 Resizing a File 14.2.5 Preallocating Space for a File 14.2.6 Querying the Size of a File 14.2.7 Querying File Parameters 14.2.8 File Info Reserved File Hints 14.3 File Views 14.4 Data Access 14.4.1 Data Access Routines Positioning Synchronism Coordination Data Access Conventions 14.4.2 Data Access with Explicit Offsets 14.4.3 Data Access with Individual File Pointers 14.4.4 Data Access with Shared File Pointers Noncollective Operations Collective Operations Seek 14.4.5 Split Collective Data Access Routines 14.5 File Interoperability 14.5.1 Datatypes for File Interoperability 14.5.2 External Data Representation: external32 14.5.3 User-Defined Data Representations Extent Callback Datarep Conversion Functions 14.5.4 Matching Data Representations 14.6 Consistency and Semantics 14.6.1 File Consistency 14.6.2 Random Access vs. Sequential Files 14.6.3 Progress 14.6.4 Collective File Operations 14.6.5 Nonblocking Collective File Operations 14.6.6 Type Matching 14.6.7 Miscellaneous Clarifications 14.6.8 MPI_Offset Type 14.6.9 Logical vs. Physical File Layout 14.6.10 File Size 14.6.11 Examples Asynchronous I/O 14.7 I/O Error Handling 14.8 I/O Error Classes 14.9 Examples 14.9.1 Double Buffering with Split Collective I/O 14.9.2 Subarray Filetype Constructor 15 Tool Support 15.1 Introduction 15.2 Profiling Interface 15.2.1 Requirements 15.2.2 Discussion 15.2.3 Logic of the Design 15.2.4 Miscellaneous Control of Profiling 15.2.5 MPI Library Implementation 15.2.6 Complications Multiple Counting Linker Oddities Fortran Support Methods 15.2.7 Multiple Levels of Interception 15.3 The MPI Tool Information Interface 15.3.1 Verbosity Levels 15.3.2 Binding MPI Tool Information Interface Variables to MPI Objects 15.3.3 Convention for Returning Strings 15.3.4 Initialization and Finalization 15.3.5 Datatype System 15.3.6 Control Variables Control Variable Query Functions Handle Allocation and Deallocation Control Variable Access Functions 15.3.7 Performance Variables Performance Variable Classes Performance Variable Query Functions Performance Experiment Sessions Handle Allocation and Deallocation Starting and Stopping of Performance Variables Performance Variable Access Functions 15.3.8 Events Event Sources Callback Safety Requirements Event Type Query Functions Handle Allocation and Deallocation Handling Dropped Events Reading Event Data Reading Event Meta Data 15.3.9 Variable Categorization Category Query Functions Category Member Query Functions 15.3.10 Return Codes for the MPI Tool Information Interface 15.3.11 Profiling Interface 16 Deprecated Interfaces 16.1 Deprecated since MPI-2.0 16.2 Deprecated since MPI-2.2 16.3 Deprecated since MPI-4.0 17 Removed Interfaces 17.1 Removed MPI-1 Bindings 17.1.1 Overview 17.1.2 Removed MPI-1 Functions 17.1.3 Removed MPI-1 Datatypes 17.1.4 Removed MPI-1 Constants 17.1.5 Removed MPI-1 Callback Prototypes 17.2 C++ Bindings 18 Semantic Changes and Warnings 18.1 Semantic Changes 18.1.1 Semantic Changes Starting in MPI-4.0 18.2 Additional Warnings 18.2.1 Warnings Starting in MPI-4.0 19 Language Bindings 19.1 Support for Fortran 19.1.1 Overview 19.1.2 Fortran Support Through the mpi_f08 Module 19.1.3 Fortran Support Through the mpi Module 19.1.4 Fortran Support Through the mpif.h Include File 19.1.5 Interface Specifications, Procedure Names, and the Profiling Interface 19.1.6 MPI for Different Fortran Standard Versions 19.1.7 Requirements on Fortran Compilers 19.1.8 Additional Support for Fortran Register-Memory-Synchronization 19.1.9 Additional Support for Fortran Numeric Intrinsic Types Parameterized Datatypes with Specified Precision and Exponent Range Support for Size-specific MPI Datatypes Communication With Size-specific Types 19.1.10 Problems With Fortran Bindings for MPI 19.1.11 Problems Due to Strong Typing 19.1.12 Problems Due to Data Copying and Sequence Association with Subscript Triplets 19.1.13 Problems Due to Data Copying and Sequence Association with Vector Subscripts 19.1.14 Special Constants 19.1.15 Fortran Derived Types 19.1.16 Optimization Problems, an Overview 19.1.17 Problems with Code Movement and Register Optimization Nonblocking Operations Persistent Operations One-sided Communication MPI_BOTTOM and Combining Independent Variables in Datatypes Solutions The Fortran ASYNCHRONOUS Attribute Calling MPI_F_SYNC_REG A User Defined Routine Instead of MPI_F_SYNC_REG Module Variables and COMMON Blocks The (Poorly Performing) Fortran VOLATILE Attribute The Fortran TARGET Attribute 19.1.18 Temporary Data Movement and Temporary Memory Modification 19.1.19 Permanent Data Movement 19.1.20 Comparison with C 19.2 Support for Large Count and Large Byte Displacement 19.3 Language Interoperability 19.3.1 Introduction 19.3.2 Assumptions 19.3.3 Initialization 19.3.4 Transfer of Handles 19.3.5 Status 19.3.6 MPI Opaque Objects Datatypes Callback Functions Error Handlers Reduce Operations 19.3.7 Attributes 19.3.8 Extra-State 19.3.9 Constants 19.3.10 Interlanguage Communication A Language Bindings Summary A.1 Defined Values and Handles A.1.1 Defined Constants A.1.2 Types A.1.3 Prototype Definitions C Bindings Fortran 2008 Bindings with the mpi_f08 Module Fortran Bindings with mpif.h or the mpi Module A.1.4 Deprecated Prototype Definitions A.1.5 String Values Default Communicator Names Reserved Data Representations Process Set Names Info Keys Info Values A.2 Summary of the Semantics of all Op.-Related Routines A.3 C Bindings A.3.1 Point-to-Point Communication C Bindings A.3.2 Partitioned Communication C Bindings A.3.3 Datatypes C Bindings A.3.4 Collective Communication C Bindings A.3.5 Groups, Contexts, Communicators, and Caching C Bindings A.3.6 Process Topologies C Bindings A.3.7 MPI Environmental Management C Bindings A.3.8 The Info Object C Bindings A.3.9 Process Creation and Management C Bindings A.3.10 One-Sided Communications C Bindings A.3.11 External Interfaces C Bindings A.3.12 I/O C Bindings A.3.13 Language Bindings C Bindings A.3.14 Tools / Profiling Interface C Bindings A.3.15 Tools / MPI Tool Information Interface C Bindings A.3.16 Deprecated C Bindings A.4 Fortran 2008 Bindings with the mpi_f08 Module A.4.1 Point-to-Point Communication Fortran 2008 Bindings A.4.2 Partitioned Communication Fortran 2008 Bindings A.4.3 Datatypes Fortran 2008 Bindings A.4.4 Collective Communication Fortran 2008 Bindings A.4.5 Groups, Contexts, Communicators, and Caching Fortran 2008 Bindings A.4.6 Process Topologies Fortran 2008 Bindings A.4.7 MPI Environmental Management Fortran 2008 Bindings A.4.8 The Info Object Fortran 2008 Bindings A.4.9 Process Creation and Management Fortran 2008 Bindings A.4.10 One-Sided Communications Fortran 2008 Bindings A.4.11 External Interfaces Fortran 2008 Bindings A.4.12 I/O Fortran 2008 Bindings A.4.13 Language Bindings Fortran 2008 Bindings A.4.14 Tools / Profiling Interface Fortran 2008 Bindings A.4.15 Deprecated Fortran 2008 Bindings A.5 Fortran Bindings with mpif.h or the mpi Module A.5.1 Point-to-Point Communication Fortran Bindings A.5.2 Partitioned Communication Fortran Bindings A.5.3 Datatypes Fortran Bindings A.5.4 Collective Communication Fortran Bindings A.5.5 Groups, Contexts, Communicators, and Caching Fortran Bindings A.5.6 Process Topologies Fortran Bindings A.5.7 MPI Environmental Management Fortran Bindings A.5.8 The Info Object Fortran Bindings A.5.9 Process Creation and Management Fortran Bindings A.5.10 One-Sided Communications Fortran Bindings A.5.11 External Interfaces Fortran Bindings A.5.12 I/O Fortran Bindings A.5.13 Language Bindings Fortran Bindings A.5.14 Tools / Profiling Interface Fortran Bindings A.5.15 Deprecated Fortran Bindings B Change-Log B.1 Changes from Version 3.1 to Version 4.0 B.1.1 Fixes to Errata in Previous Versions of MPI B.1.2 Changes in MPI-4.0 B.2 Changes from Version 3.0 to Version 3.1 B.2.1 Fixes to Errata in Previous Versions of MPI B.2.2 Changes in MPI-3.1 B.3 Changes from Version 2.2 to Version 3.0 B.3.1 Fixes to Errata in Previous Versions of MPI B.3.2 Changes in MPI-3.0 B.4 Changes from Version 2.1 to Version 2.2 B.5 Changes from Version 2.0 to Version 2.1 Bibliography General Index Examples Index MPI Constant and Predefined Handle Index MPI Declarations Index MPI Callback Function Prototype Index MPI Function Index