A very common mishap while developing parallel applications is to assume an application running at small scales automatically translates into successful large scale runs. Such optimistic views are largely unproven in general, but tools exists to help with the validation process. However, adding resilience to a parallel application has a tendency to increase the likelihood of consistency errors, and unfortunately no tools to help through this process currently exist. In some cases, few common sense practices could save hours of debugging, and improve the quality of the parallel application.
As you work on you MPI fault tolerant application, you discover it runs fine for small scales and small data sets, but when increasing the number of processes or the computational load on the participating processes, spurious faults seems to be ‘injected’ for no good reason. It might be easy to blame it on the underlying libraries, but before we go there it is possible you are observing an inter-operability issue between the different layers of the resilient software stack. More precisely, you may be observing the effect of the lack of MPI progress on the failure detector within the MPI library.
MPI Progress (and lack thereof)
The MPI Standard does not mandate that asynchronous progress shall always happen. When you post any nonblocking communications, such as MPI_Irecv or MPI_Isend, you may be surprised to discover that, depending on the MPI library configuration and/or the network transport in use, no progress at all may have happened before you enter the corresponding completion function MPI_Wait, or in another MPI function. This is especially true for non-blocking collectives, for which asynchronous progress is a rarity, even with network hardware that would otherwise be capable of asynchronous progress.
The root cause is an optimization that helps obtain good results on raw latency/injection rate (e.g., in performance benchmarks), but is not an optimal choice for applications with long swath of computational code that does not perform any MPI calls (hence, does not trigger the progress engine of MPI). Achieving asynchronous progress for all operations often requires a progress thread, internal to the MPI implementation, which entails that most MPI calls have to be thread-safe, even if the user initializes with MPI_THREAD_SINGLE. It is also more difficult to implement. Hence, most MPI implementations defaults to an ‘user-call-progressed’ mode, that is, message transmission and reception management are both progressed when the user enters one of the ‘progressing’ MPI routines (e.g., MPI_Wait, MPI_Test, MPI_Recv, etc.) Upon, entering such a routine, any MPI operation currently ongoing may progress, including operations that are not referenced by the call.
For the same reasons that asynchronous progress is not the default behavior in Open MPI (latency performance optimization), the failure detector in ULFM is also call-progress based.
An example
Let’s look at the following code example to illustrate the progress issue.
#include#include #include static void compute(int how_long); int main(int argc, char* argv[]) { /* Send some large data to make the recv long enough to see something interesting */ int count = 4*1024*1024; int* send_buff = malloc(count * sizeof(*send_buff)); int* recv_buff = malloc(count * sizeof(*recv_buff)); /* Every process sends a token to the right on a ring (size tokens sent per * iteration, one per process per iteration) */ int left = MPI_PROC_NULL, right = MPI_PROC_NULL; int rank = MPI_PROC_NULL, size = 0; int tag = 42, iterations = 10, how_long = 5; MPI_Request req = MPI_REQUEST_NULL; double post_date, wait_date, complete_date; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); left = (rank+size-1)%size; /* left ring neighbor in circular space */ right = (rank+size+1)%size; /* right ring neighbor in circular space */ /**** Done with initialization, main loop now ************************/ printf("Entering test; computing silently for %d secondsn", how_long); for(int i = 0; i < iterations; i++) { MPI_Barrier(MPI_COMM_WORLD); post_date = MPI_Wtime(); MPI_Irecv(recv_buff, 512, MPI_INT, left, tag, MPI_COMM_WORLD, &req); MPI_Send (send_buff, 512, MPI_INT, right, tag, MPI_COMM_WORLD); compute(how_long); wait_date = MPI_Wtime(); MPI_Wait(&req, MPI_STATUS_IGNORE); complete_date = MPI_Wtime(); printf("Worked %g seconds before entering MPI_Wait and spend %g seconds waitingn", wait_date - post_date, complete_date - wait_date); } MPI_Finalize(); return 0; } #include static void compute(int how_long) { /* no real computation, just waiting doing nothing */ sleep(how_long); }
Without Fault Tolerance
The program will report how much of the time was overlapped between the compute operation and the reception of the large message. Note that this is the easy case: a contiguous receive. Hence, most high performance networks will achieve overlap. If we run on one of the networks that does not perform asynchronous progress, we may still observe that the majority (or the totality) of the communication time is visible in the duration of the MPI_Wait function. In essence, the lack of progress can cause disappointing performance with respect to communication/computation overlap.
With Fault Tolerance
The program will produce the following output:
mpirun -mca mpi_ft_detector_timeout 1 -np 10 sleeptest Entering test; computing silently for 5 seconds ... [saturn:62859] *** An error occurred in MPI_Send [saturn:62859] *** reported by process [3748724737,1] [saturn:62859] *** on communicator MPI_COMM_WORLD [saturn:62859] *** MPI_ERR_PROC_FAILED: Process Failure [saturn:62859] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [saturn:62859] *** and potentially your MPI job)
Our test program has aborted, because the MPI library has reported that a process has failed. This is manifestly a spurious error that is raised despite the fact that no process has failed. The root cause lies in the fact that, since the application stopped performing any MPI calls for longer than the detection timeout, there was no opportunity for progressing the MPI library, either to progress asynchronous communications (the aforementioned performance issue), or to progress the internal failure detector. Due to lack of progress, the dates at which heartbeat messages are sent and their reception managed can drift from the expectations set in the failure detector, and that drift can grow to make a process appear unresponsive for long enough to be reported as dead (hence resulting in the occurrence of a spurious error).
Tuning the Detector for your Use Case
Most communication intensive applications will do just well with the default settings for the failure detector. These settings favor latency and injection rate metrics by setting some sane defaults that will avoid triggering spurious faults in most communicating applications, while minimizing the performance impact on the most common communication patterns. It is however important to note that when spurious errors do get reported, it may be a simple matter of tuning the runtime parameters, or your application code to fix the error.
Solution 1: tune the detector
The detector in ULFM is highly configurable. You can change both the timeout, and the use of an internal progress thread within the MPI library that will ensure that the detector remains active when the application does not perform MPI progress. You can set the following mpirun parameters to control the detector:
-mca mpi_ft_detector_timeout number: number of seconds before the detector reports an error from an unresponsive process (floating point value). If you experience spurious errors, this can be a first step at weeding them out. The advantage of this approach is that it will leave micro-benchmarks performance unchanged; however, the timeout for detection of actual failures will be increased. Given that failures are somewhat rare, this can often be a beneficial trade-off.-mca mpi_ft_detector_thread [true/false]: defaults to false (to preserve micro-benchmark performance). When set to true, the detector executes in a MPI library internal thread, thus avoiding the dependence on user progress. The performance trade-off comes from the fact that this flag will act as-ifMPI_Inithad been called withMPI_THREAD_MULTIPLE, which has a mild impact on latency/injection rate. This is often a good trade-off for real applications.-mca mpi_ft_detector [true/false]: can be used to turn off the detector altogether. This approach can work with some networks that provide in-band error reporting (e.g., TCP), but the failure detection timeout is bounded by external parameters (TCP timeouts can reach into hours under certain circumstances), and some networks do not provide in-band detection, which can result in deadlocks in failure scenarios.
Solution 2: progress MPI
A different approach is to dedicate some resources to progressing MPI operations. A traditional way for achieving continuous progress from the user-code is to initialize MPI with MPI_THREAD_MULTIPLE, and dedicate a thread to MPI progress by parking the thread in an MPI_Recv for the entire duration of the program. This receive should never be matched by a send, except at the very end of the program, in order to release the progress thread and let it terminate.
This technique lets the user dedicate one of her own threads to ensure that the MPI progress engine is active at all times, and will avoid both spurious errors, as well as often improve performance in applications that are sensitive to communication/computation overlap (especially when using complex communication routines, like non-contiguous datatypes and non-blocking collective operations, which cannot be easily delegated to the network hardware). Note however that the raw latency/bandwidth in micro-benchmarks can be impacted, and one has to dedicate a CPU core to this background activity.
Conclusions
The lack of calls progressing the MPI library is a known adverse situation in HPC applications. In the fault-free world, this can translate into the application significantly under-performing when users expect communication/computation overlap. This issue can compound during fault-tolerant operation by causing drifts in the failure detector, resulting in spurious error reporting (i.e., the notification of failures about processes that have not failed). Fortunately, the ULFM implementation provides multiple parameters that permit circumventing this issue.