==========================
Non-blocking communication                                    [TOC]
==========================

`MPI_Send` blocks the invoking process until the message has been
sent or buffered such that the message buffer can be re-used.
`MPI_Recv` blocks the caller until a matching message is
found in the incoming buffers or received. If multiple messages
are to be sent, blocking causes the latency periods to be added up.

Fortunately, all message transfer operations of MPI come
also in asynchronous variants. The asynchronous operations
`MPI_Isend` and `MPI_Irecv` can be used instead of
the blocking operations `MPI_Send` and `MPI_Recv`.
The `I` stands for _immediate_, i.e. these
operations do not block but return immediately. They just
initiate the communication which is then processed in the
background (in another thread). Eventually, we can check
whether the communication is completed or wait for that
event.

These are the signatures of `MPI_Isend` and `MPI_Irecv`:

---- CODE (type=cpp) ----------------------------------------------------------
int MPI_Isend(const void* buf, int count, MPI_Datatype datatype,
              int dest, int tag, MPI_Comm comm, MPI_Request* request);
int MPI_Irecv(void* buf, int count, MPI_Datatype datatype,
              int source, int tag, MPI_Comm comm, MPI_Request* request);
-------------------------------------------------------------------------------

The parameter lists are similar to `MPI_Send` and `MPI_Recv`. Just
a pointer to an `MPI_Request` object is added and the receiving
side passes no longer a pointer to an `MPI_Status` object.

The `MPI_Request` objects allow to test the current status or
to wait until the transfer is completed:

---- CODE (type=cpp) ----------------------------------------------------------
int MPI_Wait(MPI_Request* request, MPI_Status* status);
int MPI_Test(MPI_Request* request, int* flag, MPI_Status* status);
-------------------------------------------------------------------------------

`MPI_Wait` blocks until the associated operation is completed.
The status which is particularly interesting on the receiving
side, is stored within the `MPI_Status` object. `MPI_Test`
sets `*flag` to `true` if the operation is already completed.
In this case `*status` is set. Otherwise `*flag` remains `false`
and `*status` unchanged.

Exercise
========

Replace in your solution (or in the source code below) the
blocking communication between the Jacobi iterations by
non-blocking operations. In case you are using the source
code below, you would just need to adapt the function
`exchange_with_neighbors`.

Compare the execution times with and without blocking
communication.

Source code
===========

:import:session07/jacobi.cpp [fold]

Please do not forget to add `$(pkg-config --cflags gdk-pixbuf-2.0)`
to the compilation flags and `$(pkg-config --libs gdk-pixbuf-2.0)`
to the linkage flags. Or use a `Makefile`:

---- CODE (type=Makefile) -----------------------------------------------------
Sources := $(wildcard *.cpp)
Objects := $(patsubst %.cpp,%.o,$(Sources))
Targets := $(patsubst %.cpp,%,$(Sources))
System := $(shell uname -s)

ifeq ($(System), Linux)
CXX := OMPI_CXX=g++-8.3 mpic++ -Wno-literal-suffix
endif

ifeq ($(System), SunOS)
CXX := mpic++
endif

CC := $(CXX)
PKG_CONFIG := pkg-config
OPT := -g -O3
CPPFLAGS := -I/home/numerik/pub/pp/ss19/lib -std=c++17
CXXFLAGS := $(OPT) $(shell $(PKG_CONFIG) --cflags gdk-pixbuf-2.0)
LDFLAGS := $(OPT)
LDLIBS += $(shell $(PKG_CONFIG) --libs gdk-pixbuf-2.0)

.PHONY:	all clean depend realclean
all:	$(Targets)

$(Targets): %: %.o
	$(CXX) -o $@ $(LDFLAGS) $< $(LDLIBS)
depend:
	gcc-makedepend $(CPPFLAGS) $(Sources)
clean:
	rm -f $(Objects) a.out core
realclean:	clean
	rm -f $(Targets)
-------------------------------------------------------------------------------

:navigate: up    -> doc:index
           back  -> doc:session07/page04
           next  -> doc:session07/page06