Debugging parallel programs is notoriously difficult. Parallel programs are subject not only to the usual kinds of bugs but also to new kinds having to do with timing and synchronization errors. Often, the program ``hangs,'' for example when a process is waiting for a message to arrive that is never sent or is sent with the wrong tag. Parallel bugs often disappear precisely when you adds code to try to identify the bug, which is particularly frustrating. In this section we discuss three approaches to parallel debugging.
Just as in sequential debugging, you often wish to trace interesting events in the program by printing trace messages. Usually you wish to identify a message by the rank of the process emitting it. This can be done explicitly by putting the rank in the trace message. As noted above, using the ``line labels'' option ( -l) with mpirun in the ch_p4mpd device in MPICH adds the rank automatically.
The TotalView(c) debugger from Etnus, Ltd. [1]
runs on a variety of platforms and interacts with many vendor implementations
of MPI, including MPICH on Linux clusters. For the ch_p4 device you
invoke TotalView with
mpirun -tv <other arguments>and with the ch_p4mpd device you use
totalview mpirun <other arguments>That is, again mpirun represents the parallel job as a whole. TotalView has special commands to display the message queues of an MPI process. It is possible to attach TotalView to a collection of processes that are already running in parallel; it is also possible to attach to just one of those processes.
The ch_p4mpd device version of MPICH features a ``parallel debugger'' that
consists simply of multiple copies of the gdb debugger, together with
a mechanism for redirecting stdin. The mpigdb command is a
version of mpirun that runs each user process under the control of
gdb and also takes control of stdin for gdb. The ` z'
command allows you to direct terminal input to any specified process or
to broadcast it to all processes. We demonstrate this by running the
example under this simple debugger.
donner% mpigdb -np 5 cpi # default is stdin bcast (mpigdb) b 29 # set breakpoint for all 0-4: Breakpoint 1 at 0x8049e93: file cpi.c, line 29. (mpigdb) r # run all 0-4: Starting program: /home/lusk/mpich/examples/basic/cpi 0: Breakpoint 1, main (argc=1, argv=0xbffffa84) at cpi.c:29 1-4: Breakpoint 1, main (argc=1, argv=0xbffffa74) at cpi.c:29 0-4: 29 n = 0; # all reach breakpoint (mpigdb) n # single step all 0: 38 if (n==0) n=100; else n=0; 1-4: 42 MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); (mpigdb) z 0 # limit stdin to rank 0 (mpigdb) n # single step process 0 0: 40 startwtime = MPI_Wtime(); (mpigdb) n # until caught up 0: 42 MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); (mpigdb) z # go back to bcast stdin (mpigdb) n # single step all ... # until interesting spot (mpigdb) n 0-4: 52 x = h * ((double)i - 0.5); (mpigdb) p x # bcast print command 0: $1 = 0.0050000000000000001 # 0's value of x 1: $1 = 0.014999999999999999 # 1's value of x 2: $1 = 0.025000000000000001 # 2's value of x 3: $1 = 0.035000000000000003 # 3's value of x 4: $1 = 0.044999999999999998 # 4's value of x (mpigdb) c # continue all 0: pi is approximately 3.141600986923, Error is 0.000008333333 0-4: Program exited normally. (mpigdb) q # quit donner%If the debugging process hangs (no mpigdb prompt) because the current process is waiting for action by another process, ctl-C will bring up a menu that allows you to switch processes. The mpigdb is not nearly as advanced as TotalView, but it is often useful, and it is freely distributed with MPICH.