FT-MPI provides four different error modes, which can be specified
while starting the application. These modes are called 'communicator
ABORT: like any other MPI application, FT-MPI can abort on an error.
BLANK: failed processes are not replaced, all surviving processes
have the same rank like before the crash and MPI_COMM_WORLD has
the same size like previously.
SHRINK: failed processes are not replaced, however the new communicator
after the crash has no 'holes' in its list of processes. Thus,
processes might have a new rank after recovery and the size
of MPI_COMM_WORLD has changed.
REBUILD: failed processes are respawned, surviving processes have the
same rank as previously. The REBUILD mode is the default,
and the best tested mode of FT-MPI.
The second parameter, the 'communication modes' indicates how messages,
which are on the 'fly' why an error occurs are treated.
CONT/CONTINUE: all operations which returned the error code MPI_SUCCESS
will finish properly, even if a process failure occurs
during the operation (unless the communication partner
NOOP/RESET: all ongoing messages are dropped. The asumption behind
this mode is, that on error the application returns to
its last constistent state, and all currently ongoing
operations are not of any interest.