==============================================
Scatter and Gather with non-uniform partitions                      [TOC]
==============================================

Both, `MPI_Scatter` and `MPI_Gather`, require that each of the
processes works on exactly the same number of elements
(like rows of a matrix). What can be done if the number
of rows is not a multiply of the number of processes?

MPI offers for theses cases alternative operations
`MPI_Scatterv` and `MPI_Gatherv` where the number of
elements can be specified for each of the processes along
with the respective offsets:

---- CODE (type=cpp) ----------------------------------------------------------
int MPI_Scatterv(const void* sendbuf, int sendcounts[], const int displs[],
                 MPI_Datatype sendtype,
                 void* recvbuf, int recvcount, MPI_Datatype recvtype,
                 int root, MPI_Comm comm);

int MPI_Gatherv(const void* sendbuf, int sendcount, MPI_Datatype sendtype,
                void* recvbuf, const int recvcounts[], const int displs[],
                MPI_Datatype recvtype, int root, MPI_Comm comm);
-------------------------------------------------------------------------------

Exercise
========

It appears advantageous to offer functions for scattering
matrices by rows where all the parameters are computed in
dependence of an actual matrix. The partitioning can be done with
the help of a `hpc::aux::UniformSlices` object of the lecture library.
Such an object delivers the necessary values for the arrays
`sendcounts[]`/`recvcounts[]` and `displs[]`.

The two following functions are expected to copy from `A` to `B`.
In case of `scatter_by_row` the matrix `A` is to be partitioned
and the matrix `B` represents the local partition of `A`. In
case of `gather_by_row` the matrix `A` represents the local partition
and `B` is the matrix of the root process that collects all individual
partitions.

---- CODE (type=cpp) ----------------------------------------------------------
#include <type_traits>
#include <vector>
#include <mpi.h>
#include <hpc/aux/slices.hpp>
#include <hpc/matvec/gematrix.hpp>
#include <hpc/mpi/fundamental.hpp>

template<typename T,
   template<typename> class MA,
   template<typename> class MB,
   Require<Ge<MA<T>>, Ge<MB<T>>> = true>
int scatter_by_row(const MA<T>& A, MB<T>& B, int root, MPI_Comm comm) {

   /* ... */
}

template<typename T,
   template<typename> class MA,
   template<typename> class MB,
   Require<Ge<MA<T>>, Ge<MB<T>>> = true>
int gather_by_row(const MA<T>& A, MB<T>& B, int root, MPI_Comm comm) {

   /* ... */
}
-------------------------------------------------------------------------------

Test both functions with the help of the following test program:

:import:session07/scatter-gather3.cpp [fold]

:navigate: up    -> doc:index
           back  -> doc:session07/page02
           next  -> doc:session07/page04