============================================= Transfer of self-defined data types using MPI [TOC] ============================================= Let us return to the interfaces of `MPI_Send` and `MPI_Recv`: ---- CODE (type=cpp) ---------------------------------------------------------- int MPI_Send(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); int MPI_Recv(void* buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status* status); ------------------------------------------------------------------------------- The 1:1 message interchange operations `MPI_Send` and `MPI_Recv` support the transfer of arrays but insist that the array elements are located consecutively in memory. Hence, these operations do not allow us to transfer arbitrary objects of types `DenseVectorView` or `GeMatrixView` as the offsets between consecutive elements (which can be configured through `inc` or `incRow` and `incCol`, respectively) can be arbitrary. MPI allows us to define objects of type MPI_Datatype which permit us to support vector and matrix types which match those of our vector and matrix views. However, we should first learn a little bit more about data types in MPI. MPI comes with a set of elementary types $ET$ which includes, for example, `MPI_DOUBLE` and `MPI_INT`. Within the MPI library, a data type $T$ with the cardinality $n$ is a sequence of tuples $\left\{ (et_1, o_1), (et_2, o_2), \dots, (et_n, o_n) \right\}$ where $et_i \in ET$ and the associated offsets $o_i \in \mathbb{Z}$ for $i = 1, \dots, n$. The offsets are interpreted in bytes relatively to the start address. Two types $T_1$ and $T_2$ are considered as compatible to each other for `MPI_Send` and `MPI_Recv` if and only if the cardinalities $n_1$ and $n_2$ are equal and $et_{1_i} = et_{2_i}$ for all $i = 1, \dots, n_1$. Overlapping offsets are permitted for `MPI_Send`, in case of `MPI_Recv` their effect is undefined. Some examples, all of them assume that the underlying matrix is stored in row-major: * A row vector of the elementary type `MPI_DOUBLE` (8 bytes) has the data type $\left\{ (DOUBLE, 0), (DOUBLE, 8), (DOUBLE, 16), (DOUBLE, 24) \right\}$. * A column vector of length 3 out of a $5 \times 5$ matrix has the data type $\left\{ (DOUBLE, 0), (DOUBLE, 40), (DOUBLE, 80) \right\}$. * The trace of a $3 \times 3$ matrix has the data type $\left\{ (DOUBLE, 0), (DOUBLE, 32), (DOUBLE, 64) \right\}$. * The upper triangle of a $3 \times 3$ matrix: $\left\{ (DOUBLE, 0), (DOUBLE, 8), (DOUBLE, 16), (DOUBLE, 32), (DOUBLE, 40), (DOUBLE, 64) \right\}$. The type on the sending side must not be identical to the type on the receiving side. They just need to be compatible to each other. This means that the offsets are interpreted locally only. Hence, the receiving side is free to store transfered objects differently. Row vectors can be received as column vectors and matrices in row-major can be received in a column-major order. In MPI, multiple functions support the construction of data types. All of them construct a sequence of tuples as described above. Let us start with the type construction function `MPI_Type_vector`: ---- CODE (type=cpp) ---------------------------------------------------------- int MPI_Type_vector(int count, int blocklength, int stride, MPI_Datatype oldtype, MPI_Datatype* newtype); ------------------------------------------------------------------------------- The element type is always given as `oldtype`. The parameter `blocklength` specifies how many elements of this type are consecutively in memory. They are considered as a block. The offset of two consecutive blocks is specified as `stride` -- this value is implicitly multiplied with the size of the element type. In total, the data type covers `count` blocks. Whenever a data type is constructed using functions like `MPI_Type_vector` it is to be converted into a flat sequence of tuples using the `MPI_Type_commit` function before it can be used for actual transfers. ---- CODE (type=cpp) ---------------------------------------------------------- int MPI_Type_commit(MPI_Datatype* datatype); ------------------------------------------------------------------------------- Types which are no longer required can be released: ---- CODE (type=cpp) ---------------------------------------------------------- int MPI_Type_free(MPI_Datatype* datatype); ------------------------------------------------------------------------------- This does not affect any on-going communication using this data type nor any other types constructed from it. Internally, the MPI library uses here a mechanism similar to that of std::shared_ptr of C++ on base of reference counts. Exercise ======== The following source code is already prepared for a small test to transfer vectors. Construct the required MPI data types that allow to transfer a vector with a single `MPI_Send` and a single `MPI_Recv` operation. It is to be recommended to write a small function that constructs a matching `MPI_Datatype` object for a vector. Such a template function could be written as follows and put into a separate header file named _hpc/mpi/vector.hpp_. Then you need to add “-I.” to the compilation options (in front of the other “-I” option). ---- CODE (type=cpp) ---------------------------------------------------------- template typename Vector, Require>> = true> MPI_Datatype get_type(const Vector& vector) { MPI_Datatype datatype; /* ... */ return datatype; } ------------------------------------------------------------------------------- If you are looking for a general approach to map elementary types of C++ to elementary MPI data type objects, you are free to use `#include ` where you will find the template function `get_type` which supports all elementary types of MPI. If you have, for example, `double x;` then `hpc::mpi::get_type(x)` will return `MPI_DOUBLE`. The library for this session is to be found at _/home/numerik/pub/pp/ss19/lib_ on our hosts. By default, mpic++ selects on our machines in E.44 g++ version 6.3.0 which is shipped with Debian stretch. Some of our libraries need, however, a more recent version. You can select another C++ compiler using the `OMPI_CXX` environment variable: ---- SHELL (path=session06,hostname=heim) ------------------------------------- OMPI_CXX=g++-8.3 mpic++ --version ------------------------------------------------------------------------------- Source code =========== :import:session06/transfer_vectors.cpp [fold] :import:/home/numerik/pub/pp/ss19/lib/hpc/mpi/fundamental.hpp [fold] (Please note that `transfer_vectors` does not print a message if the tests succeed: No news are good news.) :navigate: up -> doc:index next -> doc:session06/page02