memcpy

Determining delivered memory performance

`memcpy' is up to date.
Size (bytes) Time (sec)	Rate (MB/sec)
8	0.000000	17.271783
16	0.000001	18.047743
32	0.000001	33.160404
64	0.000001	63.387390
128	0.000001	108.778293
256	0.000001	173.637864
512	0.000002	248.607625
1024	0.000003	313.705030
2048	0.000013	157.836254
4096	0.000023	177.661522
8192	0.000042	194.859113
16384	0.000086	189.656806
32768	0.000174	188.421760
65536	0.000352	186.118683
131072	0.000710	184.533420
262144	0.001384	189.391375
524288	0.002765	189.630230
1048576	0.005536	189.403609
2097152	0.011130	188.426301
4194304	0.022280	188.256421

Determining delivered memory performance with unaligned data

`memcpy' is up to date.
Size (bytes) Time (sec)	Rate (MB/sec)
8	0.000000	17.171089
16	0.000003	4.853510
32	0.000004	8.423183
64	0.000004	15.436880
128	0.000005	27.253852
256	0.000006	45.984030
512	0.000007	70.614216
1024	0.000010	98.663366
2048	0.000013	159.462812
4096	0.000024	171.900069
8192	0.000048	172.159206
16384	0.000149	109.837207
32768	0.000386	84.860686
65536	0.000864	75.860651
131072	0.001724	76.012960
262144	0.003446	76.073879
524288	0.006846	76.583995
1048576	0.013754	76.238003
2097152	0.027498	76.266096
4194304	0.055049	76.191529

pingpong

Benchmarking point to point performance

`pingpong' is up to date.
Kind		n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000030	0.268657
Send/Recv	2	0.000035	0.457671
Send/Recv	4	0.000038	0.836936
Send/Recv	8	0.000041	1.572787
Send/Recv	16	0.000042	3.035382
Send/Recv	32	0.000046	5.524475
Send/Recv	64	0.000055	9.392184
Send/Recv	128	0.000073	14.081860
Send/Recv	256	0.000126	16.268174
Send/Recv	512	0.000192	21.333171
Send/Recv	1024	0.000401	20.413494
Send/Recv	2048	0.000656	24.980385
Send/Recv	4096	0.001193	27.456902
Send/Recv	8192	0.002272	28.838893
Send/Recv	16384	0.004436	29.550409
Send/Recv	32768	0.008807	29.766006
Send/Recv	65536	0.017622	29.752087
Send/Recv	131072	0.035077	29.893721
Send/Recv	262144	0.070244	29.855393
Send/Recv	524288	0.140226	29.911012
Send/Recv	1048576	0.273352	30.687971

Benchmarking point to point performance with nonblocking operations

`pingpong' is up to date.
Kind		n	time (sec)	Rate (MB/sec)
Isend/Irecv	1	0.000044	0.179782
Isend/Irecv	2	0.000048	0.336702
Isend/Irecv	4	0.000054	0.593883
Isend/Irecv	8	0.000055	1.162066
Isend/Irecv	16	0.000057	2.250118
Isend/Irecv	32	0.000061	4.226659
Isend/Irecv	64	0.000068	7.479915
Isend/Irecv	128	0.000086	11.974717
Isend/Irecv	256	0.000137	14.909306
Isend/Irecv	512	0.000203	20.210125
Isend/Irecv	1024	0.000410	19.994029
Isend/Irecv	2048	0.000663	24.697581
Isend/Irecv	4096	0.001199	27.329281
Isend/Irecv	8192	0.002280	28.740706
Isend/Irecv	16384	0.004465	29.356223
Isend/Irecv	32768	0.008825	29.703284
Isend/Irecv	65536	0.017657	29.692581
Isend/Irecv	131072	0.035186	29.800533
Isend/Irecv	262144	0.070343	29.813146
Isend/Irecv	524288	0.140353	29.884032
Isend/Irecv	1048576	0.274087	30.605605

Benchmarking point to point performance with nonblocking operations, head-to-head

`pingpong' is up to date.
Kind				n	time (sec)	Rate (MB/sec)
head-to-head Isend/Irecv	1	0.000067	0.238037
head-to-head Isend/Irecv	2	0.000070	0.460258
head-to-head Isend/Irecv	4	0.000097	0.658788
head-to-head Isend/Irecv	8	0.000098	1.299527
head-to-head Isend/Irecv	16	0.000114	2.253765
head-to-head Isend/Irecv	32	0.000118	4.333797
head-to-head Isend/Irecv	64	0.000126	8.126199
head-to-head Isend/Irecv	128	0.000151	13.575175
head-to-head Isend/Irecv	256	0.000265	15.479103
head-to-head Isend/Irecv	512	0.000344	23.827835
head-to-head Isend/Irecv	1024	0.000482	33.987995
head-to-head Isend/Irecv	2048	0.000750	43.673007
head-to-head Isend/Irecv	4096	0.001385	47.317280
head-to-head Isend/Irecv	8192	0.002593	50.548192
head-to-head Isend/Irecv	16384	0.005138	51.023493
head-to-head Isend/Irecv	32768	0.010032	52.260486
head-to-head Isend/Irecv	65536	0.020117	52.124996
head-to-head Isend/Irecv	131072	0.037852	55.403923
head-to-head Isend/Irecv	262144	0.079523	52.743047
head-to-head Isend/Irecv	524288	0.156833	53.487450
head-to-head Isend/Irecv	1048576	0.307424	54.573544

Benchmarking point to point performance with unaligned data

`pingpong' is up to date.
Kind char		n	time (sec)	Rate (MB/sec)
Send/Recv		1	0.000031	0.032188
Send/Recv		2	0.000031	0.064279
Send/Recv		4	0.000031	0.129318
Send/Recv		8	0.000031	0.259705
Send/Recv		16	0.000036	0.446286
Send/Recv		32	0.000039	0.817346
Send/Recv		64	0.000040	1.587984
Send/Recv		128	0.000044	2.913112
Send/Recv		256	0.000048	5.346432
Send/Recv		512	0.000059	8.705280
Send/Recv		1024	0.000074	13.776960
Send/Recv		2048	0.000127	16.182616
Send/Recv		4096	0.000195	21.053761
Send/Recv		8192	0.000393	20.844613
Send/Recv		16384	0.000669	24.503836
Send/Recv		32768	0.001204	27.206880
Send/Recv		65536	0.002298	28.520593
Send/Recv		131072	0.004493	29.175406
Send/Recv		262144	0.008879	29.523481
Send/Recv		524288	0.017741	29.552494
Send/Recv		1048576	0.035440	29.587560
Kind double		n	time (sec)	Rate (MB/sec)
Send/Recv		1	0.000031	0.254581
Send/Recv		2	0.000037	0.436061
Send/Recv		4	0.000040	0.804994
Send/Recv		8	0.000041	1.542600
Send/Recv		16	0.000044	2.917905
Send/Recv		32	0.000048	5.363843
Send/Recv		64	0.000056	9.182840
Send/Recv		128	0.000071	14.510025
Send/Recv		256	0.000124	16.487398
Send/Recv		512	0.000195	21.010312
Send/Recv		1024	0.000390	20.995066
Send/Recv		2048	0.000660	24.817435
Send/Recv		4096	0.001197	27.365671
Send/Recv		8192	0.002273	28.835868
Send/Recv		16384	0.004432	29.573359
Send/Recv		32768	0.008770	29.889506
Send/Recv		65536	0.017600	29.789747
Send/Recv		131072	0.035257	29.741059
Send/Recv		262144	0.070140	29.899647
Send/Recv		524288	0.140708	29.808663
Send/Recv		1048576	0.280591	29.896190
Kind int		n	time (sec)	Rate (MB/sec)
Send/Recv		1	0.000033	0.244278
Send/Recv		2	0.000038	0.422886
Send/Recv		4	0.000042	0.765353
Send/Recv		8	0.000044	1.468980
Send/Recv		16	0.000046	2.767894
Send/Recv		32	0.000049	5.184180
Send/Recv		64	0.000058	8.887212
Send/Recv		128	0.000073	14.063334
Send/Recv		256	0.000126	16.190241
Send/Recv		512	0.000194	21.079594
Send/Recv		1024	0.000391	20.967041
Send/Recv		2048	0.000662	24.754297
Send/Recv		4096	0.001192	27.489509
Send/Recv		8192	0.002272	28.847594
Send/Recv		16384	0.004435	29.556963
Send/Recv		32768	0.008748	29.965467
Send/Recv		65536	0.017554	29.867077
Send/Recv		131072	0.035196	29.792294
Send/Recv		262144	0.070102	29.915866
Send/Recv		524288	0.140629	29.825308
Send/Recv		1048576	0.274875	30.517848

Benchmarking point to point performance with contention

`pingpong' is up to date.
Kind (np=2)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000031	0.261174
Send/Recv	2	0.000036	0.447222
Send/Recv	4	0.000039	0.820140
Send/Recv	8	0.000040	1.583975
Send/Recv	16	0.000043	2.996697
Send/Recv	32	0.000047	5.465288
Send/Recv	64	0.000054	9.405553
Send/Recv	128	0.000073	14.080623
Send/Recv	256	0.000127	16.133226
Send/Recv	512	0.000194	21.118462
Send/Recv	1024	0.000391	20.952657
Send/Recv	2048	0.000635	25.800442
Send/Recv	4096	0.001168	28.051628
Send/Recv	8192	0.002213	29.609695
Send/Recv	16384	0.004316	30.371257
Send/Recv	32768	0.008561	30.622107
Send/Recv	65536	0.017157	30.557958
Send/Recv	131072	0.034188	30.671238
Send/Recv	262144	0.068369	30.674072
Send/Recv	524288	0.136323	30.767333
Send/Recv	1048576	0.273436	30.678522
Kind (np=4)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000031	0.261748
Send/Recv	2	0.000036	0.443476
Send/Recv	4	0.000039	0.820807
Send/Recv	8	0.000040	1.585655
Send/Recv	16	0.000043	3.000225
Send/Recv	32	0.000047	5.455826
Send/Recv	64	0.000055	9.280897
Send/Recv	128	0.000074	13.874714
Send/Recv	256	0.000128	15.938031
Send/Recv	512	0.000206	19.849647
Send/Recv	1024	0.000399	20.526297
Send/Recv	2048	0.000650	25.201394
Send/Recv	4096	0.001195	27.413090
Send/Recv	8192	0.002266	28.923100
Send/Recv	16384	0.004431	29.577834
Send/Recv	32768	0.008795	29.807716
Send/Recv	65536	0.017574	29.833395
Send/Recv	131072	0.034976	29.979612
Send/Recv	262144	0.070270	29.844327
Send/Recv	524288	0.139833	29.995173
Send/Recv	1048576	0.279571	30.005301
Kind (np=8)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000030	0.264762
Send/Recv	2	0.000035	0.452902
Send/Recv	4	0.000039	0.827118
Send/Recv	8	0.000040	1.609354
Send/Recv	16	0.000042	3.017650
Send/Recv	32	0.000047	5.500511
Send/Recv	64	0.000055	9.343826
Send/Recv	128	0.000074	13.908814
Send/Recv	256	0.000127	16.111792
Send/Recv	512	0.000206	19.891302
Send/Recv	1024	0.000399	20.519402
Send/Recv	2048	0.000649	25.254068
Send/Recv	4096	0.001192	27.494321
Send/Recv	8192	0.002273	28.832843
Send/Recv	16384	0.004438	29.534435
Send/Recv	32768	0.008771	29.886713
Send/Recv	65536	0.017475	30.002134
Send/Recv	131072	0.035034	29.930404
Send/Recv	262144	0.070130	29.903573
Send/Recv	524288	0.140126	29.932304
Send/Recv	1048576	0.279444	30.018966
Kind (np=16)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000030	0.263838
Send/Recv	2	0.000036	0.449495
Send/Recv	4	0.000039	0.822926
Send/Recv	8	0.000040	1.591931
Send/Recv	16	0.000042	3.014968
Send/Recv	32	0.000047	5.503068
Send/Recv	64	0.000055	9.355529
Send/Recv	128	0.000072	14.162290
Send/Recv	256	0.000128	16.047829
Send/Recv	512	0.000203	20.142002
Send/Recv	1024	0.000414	19.808879
Send/Recv	2048	0.000682	24.037770
Send/Recv	4096	0.001254	26.137775
Send/Recv	8192	0.002385	27.481951
Send/Recv	16384	0.004673	28.048319
Send/Recv	32768	0.009231	28.396891
Send/Recv	65536	0.018481	28.369689
Send/Recv	131072	0.037032	28.315625
Send/Recv	262144	0.074114	28.296160
Send/Recv	524288	0.148102	28.320451
Send/Recv	1048576	0.295799	28.359135
Kind (np=32)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000030	0.263472
Send/Recv	2	0.000036	0.450567
Send/Recv	4	0.000039	0.825901
Send/Recv	8	0.000040	1.589881
Send/Recv	16	0.000042	3.020713
Send/Recv	32	0.000047	5.483464
Send/Recv	64	0.000055	9.352303
Send/Recv	128	0.000072	14.190700
Send/Recv	256	0.000127	16.109274
Send/Recv	512	0.000203	20.202698
Send/Recv	1024	0.000413	19.836039
Send/Recv	2048	0.000678	24.177844
Send/Recv	4096	0.001250	26.206927
Send/Recv	8192	0.002388	27.448505
Send/Recv	16384	0.004707	27.848340
Send/Recv	32768	0.009319	28.130304
Send/Recv	65536	0.018526	28.300115
Send/Recv	131072	0.037033	28.314485
Send/Recv	262144	0.074192	28.266375
Send/Recv	524288	0.147904	28.358199
Send/Recv	1048576	0.295961	28.343644

barrier

Benchmarking collective barrier

`barrier' is up to date.
Kind	np	time (sec)
Barrier	1	0.000001
Barrier	2	0.000002
Barrier	4	0.000002
Barrier	8	0.000002
Barrier	16	0.000002
Barrier	32	0.000002

Benchmarking collective Allreduce

`barrier' is up to date.
Kind		np	time (sec)
Allreduce	1	0.000025
Allreduce	2	0.000098
Allreduce	4	0.000127
Allreduce	8	0.000167
Allreduce	16	0.000194
Allreduce	32	0.000232

vector

Comparing the performance of MPI vector datatypes

`vector' is up to date.
Kind	n	stride	time (sec)	Rate (MB/sec)
Vector	1000	24	0.000606	13.197222
Struct	1000	24	0.019555	0.409099
User	1000	24	0.000735	10.883781
User(add)	1000	24	0.000727	10.999323

circulate

Pipelining pitfalls

`circulate' is up to date.
For n = 20000, m = 20000, T_comm = 0.000910, T_compute = 0.061264, sum = 0.062174, T_both = 0.062272
For n = 500, m = 500, T_comm = 0.000098, T_compute = 0.001495, sum = 0.001593, T_both = 0.001553

3way

Exploring the cost of synchronization delays

`bad' is up to date.
mppexec: Non-power of 2 PE partitions are not currently supported
mppexec: Non-power of 2 PE partitions are not currently supported
mppexec: Non-power of 2 PE partitions are not currently supported
mppexec: Non-power of 2 PE partitions are not currently supported
mppexec: Non-power of 2 PE partitions are not currently supported

jacobi

Jacobi Iteration - Example Parallel Mesh

`jacobi' is up to date.
send/recv: 6 iterations in 0.000409 secs (1.026802 MFlops); diffnorm 0.008134, m=7 n=4 np=1
send/recv: 7 iterations in 0.072540 secs (5.533608 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
send/recv: 24 iterations in 0.008229 secs (0.816625 MFlops); diffnorm 0.009895, m=7 n=10 np=4
send/recv: 25 iterations in 0.332666 secs (17.237731 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
send/recv: 25 iterations in 0.010323 secs (2.712409 MFlops); diffnorm 0.036615, m=7 n=34 np=16
send/recv: 25 iterations in 0.341091 secs (67.247830 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
send/recv: 25 iterations in 0.011190 secs (5.004683 MFlops); diffnorm 0.055291, m=7 n=66 np=32
send/recv: 25 iterations in 0.345478 secs (132.787628 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Shift up and down

`jacobi' is up to date.
shift/sendrecv: 6 iterations in 0.000416 secs (1.010676 MFlops); diffnorm 0.008134, m=7 n=4 np=1
shift/sendrecv: 7 iterations in 0.072530 secs (5.534345 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
shift/sendrecv: 24 iterations in 0.007532 secs (0.892176 MFlops); diffnorm 0.009895, m=7 n=10 np=4
shift/sendrecv: 25 iterations in 0.331143 secs (17.316995 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
shift/sendrecv: 25 iterations in 0.009481 secs (2.953243 MFlops); diffnorm 0.036615, m=7 n=34 np=16
shift/sendrecv: 25 iterations in 0.340187 secs (67.426466 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
shift/sendrecv: 25 iterations in 0.010381 secs (5.394665 MFlops); diffnorm 0.055291, m=7 n=66 np=32
shift/sendrecv: 25 iterations in 0.344120 secs (133.311475 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Exchange head-to-head

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
head-to-head sendrecv: 6 iterations in 0.000549 secs (0.764545 MFlops); diffnorm 0.008134, m=7 n=4 np=1
head-to-head sendrecv: 7 iterations in 0.072570 secs (5.531315 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
head-to-head sendrecv: 24 iterations in 0.007432 secs (0.904182 MFlops); diffnorm 0.009895, m=7 n=10 np=4
head-to-head sendrecv: 25 iterations in 0.333759 secs (17.181273 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
head-to-head sendrecv: 25 iterations in 0.009585 secs (2.921260 MFlops); diffnorm 0.036615, m=7 n=34 np=16
head-to-head sendrecv: 25 iterations in 0.339352 secs (67.592344 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
head-to-head sendrecv: 25 iterations in 0.010437 secs (5.365321 MFlops); diffnorm 0.055291, m=7 n=66 np=32
head-to-head sendrecv: 25 iterations in 0.342970 secs (133.758516 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Nonblocking send/recv

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
irecv/isend: 6 iterations in 0.000502 secs (0.837266 MFlops); diffnorm 0.008134, m=7 n=4 np=1
irecv/isend: 7 iterations in 0.072673 secs (5.523498 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
irecv/isend: 24 iterations in 0.007725 secs (0.869943 MFlops); diffnorm 0.009895, m=7 n=10 np=4
irecv/isend: 25 iterations in 0.336572 secs (17.037666 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
irecv/isend: 25 iterations in 0.009385 secs (2.983516 MFlops); diffnorm 0.036615, m=7 n=34 np=16
irecv/isend: 25 iterations in 0.340403 secs (67.383615 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
irecv/isend: 25 iterations in 0.010643 secs (5.261716 MFlops); diffnorm 0.055291, m=7 n=66 np=32
irecv/isend: 25 iterations in 0.346529 secs (132.385059 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Nonblocking send/recv for receiver pull

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
isend/irecv: 6 iterations in 0.000496 secs (0.846113 MFlops); diffnorm 0.008134, m=7 n=4 np=1
isend/irecv: 7 iterations in 0.072604 secs (5.528749 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
isend/irecv: 24 iterations in 0.008758 secs (0.767272 MFlops); diffnorm 0.009895, m=7 n=10 np=4
isend/irecv: 25 iterations in 0.333078 secs (17.216368 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
isend/irecv: 25 iterations in 0.010598 secs (2.641918 MFlops); diffnorm 0.036615, m=7 n=34 np=16
isend/irecv: 25 iterations in 0.339030 secs (67.656598 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
isend/irecv: 25 iterations in 0.011630 secs (4.815245 MFlops); diffnorm 0.055291, m=7 n=66 np=32
isend/irecv: 25 iterations in 0.341810 secs (134.212497 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Synchronous send

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
ssend/irecv: 6 iterations in 0.000466 secs (0.901251 MFlops); diffnorm 0.008134, m=7 n=4 np=1
ssend/irecv: 7 iterations in 0.072641 secs (5.525912 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
ssend/irecv: 24 iterations in 0.009916 secs (0.677717 MFlops); diffnorm 0.009895, m=7 n=10 np=4
ssend/irecv: 25 iterations in 0.291280 secs (19.686889 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
ssend/irecv: 25 iterations in 0.011663 secs (2.400677 MFlops); diffnorm 0.036615, m=7 n=34 np=16
ssend/irecv: 25 iterations in 0.297938 secs (76.987732 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
ssend/irecv: 25 iterations in 0.011995 secs (4.668489 MFlops); diffnorm 0.055291, m=7 n=66 np=32
ssend/irecv: 25 iterations in 0.306152 secs (149.844528 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Ready send

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064bd0
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bb8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000000038
   t12:0x000000000000000d	 t13:0x000000400006a350

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000006
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70dba30e2b4   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70dba30e2b4
   f03:0x41a9a70dba30a4a1   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3f11580000000000   f11:0x3f80a88865903544
   f12:0x3f80a88865903545   f13:0x3f1157ffffffffff   f14:0x3f1157ffffffffff
   f15:0xbbd0000000000000   f16:0x3f11580000000000   f17:0x3ff0000000000000
   f18:0x3fd4400000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x404ebc414efb607a   f23:0x3f88fccc98561dcf
   f24:0x3f70a888658bd115   f25:0x4060000000000000   f26:0x3cb0000000000000
   f27:0x3c40000000000000   f28:0x40009937258f3ca1   f29:0xbff0eed847a3911c
   f30:0x405ebc414efb607a			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70696 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064bb0
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bd8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000008010
   t12:0x000000000000000d	 t13:0x000000400006a350

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000007
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70dc5308e8e   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70dc5308e8e
   f03:0x41a9a70dc50b09ed   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3f08f40000000000   f11:0x3f7c41fb66456959
   f12:0x3f7c41fb6645695b   f13:0x3f08f40000000000   f14:0x3f08f40000000002
   f15:0x0000000000000000   f16:0x3f08f40000000000   f17:0x3ff0000000000000
   f18:0x3fd4740000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x40521e7211ce9219   f23:0x3f85317c8cb40f03
   f24:0x3f6c41fb66456958   f25:0x4060000000000000   f26:0x3cb0000000000000
   f27:0x3c30000000000000   f28:0x3ff9271b63a6a929   f29:0xbff0eed847a3911c
   f30:0x40621e7211ce9219			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70710 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064bb0
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064b98   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000000038
   t12:0x000000000000000d	 t13:0x000000400006a350

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000018
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70dcfc2893e   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70dcfc2893e
   f03:0x41a9a70dcfbe8527   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3f19aab76ce5406a   f11:0x3f8443d652b0baab
   f12:0x3f8443d652b0baad   f13:0x3f19aab76ce5406a   f14:0x3f19aab76ce5406d
   f15:0x0000000000000000   f16:0x3f19aab76ce5406a   f17:0x3fe88aa67be6be00
   f18:0x3fc44ba66463f200   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x404943e760423bf0   f23:0x3f8e65c17c08b84c
   f24:0x3f7443d652affb41   f25:0x4060000000000000   f26:0x3cb0000000000000
   f27:0x3c40000000000000   f28:0x40102edb29a92532   f29:0xbff0eed847a3911c
   f30:0x405943e760423bf0			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70725 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064b90
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bb8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000008010
   t12:0xfffffffffffffc02	 t13:0x0000000000000000

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000019
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70ddb32138b   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70ddb32138b
   f03:0x41a9a70dda80df55   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3f93bbce179cee76   f11:0x3fc1c4de05aa0587
   f12:0x3fc1c4de05aa0589   f13:0x3f93bbce179cee77   f14:0x3f93bbce179cee79
   f15:0x3c50000000000000   f16:0x3f93bbce179cee76   f17:0x3fe87b96d44c38a0
   f18:0x3fc35d32eee31900   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x400cd06fd2bbe320   f23:0x3fcaa74d087f06f8
   f24:0x3fb1c4de05aa02e1   f25:0x4020000000000000   f26:0x3cb0000000000000
   f27:0x3c80000000000000   f28:0x4003fa4e9e17d999   f29:0xbff0eed847a3911c
   f30:0x401cd06fd2bbe320			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70739 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064b90
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bb8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000000038
   t12:0x000000000000000d	 t13:0x000000400006a350

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000019
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70de5f5795e   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70de5f5795e
   f03:0x41a9a70de5f09063   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3f55f739f73981d7   f11:0x3fa2bf3e43707ca5
   f12:0x3fa2bf3e43707ca7   f13:0x3f55f739f73981d8   f14:0x3f55f739f73981da
   f15:0x3c10000000000000   f16:0x3f55f739f73981d7   f17:0x3fe8708df2950fe0
   f18:0x3fc40525a336bc00   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x402b4f9be68119bb   f23:0x3fac1edd6523c7ea
   f24:0x3f92bf3e43669689   f25:0x4040000000000000   f26:0x3cb0000000000000
   f27:0x3c60000000000000   f28:0x4007f827604dbc5b   f29:0xbff0eed847a3911c
   f30:0x403b4f9be68119bb			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70754 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064b90
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bb8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000008010
   t12:0xfffffffffffffc02	 t13:0x0000000000000000

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000019
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70df19ffab7   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70df19ffab7
   f03:0x41a9a70df0ed5dab   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3fcc238004d7933a   f11:0x3fde01ddd195cb85
   f12:0x3fde01ddd195cb87   f13:0x3fcc238004d79338   f14:0x3fcc238004d7933a
   f15:0xbc90000000000000   f16:0x3fcc238004d7933a   f17:0x3fe84e2b32eaea00
   f18:0x3fc2b904b872d000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x3ff110014e93f95f   f23:0x3fe681665d2e64d6
   f24:0x3fce01ddd18dfc4d   f25:0x4000000000000000   f26:0x3cb0000000000000
   f27:0x3c90000000000000   f28:0x3ffaf0cb49c11ca2   f29:0xbff0eed847a3911c
   f30:0x400110014e93f95f			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70769 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064b90
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bb8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000000038
   t12:0x000000000000000d	 t13:0x000000400006a350

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000019
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70dfd9411f6   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70dfd9411f6
   f03:0x41a9a70dfd8ec16f   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3f690b25a0f0d78e   f11:0x3fac4f138de942c4
   f12:0x3fac4f138de942c5   f13:0x3f690b25a0f0d78b   f14:0x3f690b25a0f0d78b
   f15:0xbc38000000000000   f16:0x3f690b25a0f0d78e   f17:0x3fe8708df2950fe0
   f18:0x3fc40525a336bc00   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x402216108b4184e2   f23:0x3fb53b4eaa6ef214
   f24:0x3f9c4f138de942c7   f25:0x4030000000000000   f26:0x3cb0000000000000
   f27:0x3c60000000000000   f28:0x3ff931328352a223   f29:0xbff0eed847a3911c
   f30:0x403216108b4184e2			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70785 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000000	pa1: 0x0000000000001600  pa2: 0x0000000000000000

   pc: 0x000000200003b83c	 sp: 0x00000060ffffec40   fp: 0x00000060ffffed40
   v0: 0x0000000000000000	 ra: 0x00000020000020bc   ci: 0x00002d0000000101

   a0: 0x00000060ffffeeb8	 a1: 0x0000000000000000   a2: 0x0000004000064b90
   a3: 0x0000004000064800	 a4: 0x0000ffffffffffff   a5: 0x0000000000000030

   t0: 0x0000000000000000	 t1: 0x0000004000029430   t2: 0x00000000000000fe
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000550
   t6: 0x0000000000000000	 t7: 0x0000004000064bb8   t8: 0x0000000000000001
   t9: 0x0000000000000001	 t10:0x0000000000000000   t11:0x0000000000008010
   t12:0xfffffffffffffc02	 t13:0x0000000000000000

   s0: 0x00000060ffffee78	 s1: 0x0000000000000000   s2: 0x0000000000000019
   s3: 0x0000004000000398	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70e098bc15c   f01:0x3e3c58c6d8a67ba9   f02:0x41a9a70e098bc15c
   f03:0x41a9a70e08d9a7d4   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x3fcc5b88017f6edc   f11:0x3fde1faf456b2a54
   f12:0x3fde1faf456b2a56   f13:0x3fcc5b88017f6edb   f14:0x3fcc5b88017f6edd
   f15:0xbc80000000000000   f16:0x3fcc5b88017f6edc   f17:0x3fe84e2b32eaea00
   f18:0x3fc2b904b872d000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x3ff0ff1d9d169a6c   f23:0x3fe697c3740e04fb
   f24:0x3fce1faf4561bf42   f25:0x4000000000000000   f26:0x3cb0000000000000
   f27:0x3c90000000000000   f28:0x3ffb17f6b55f4196   f29:0xbff0eed847a3911c
   f30:0x4000ff1d9d169a6c			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 70804 Operand range error

Jacobi Iteration - Overlapping communication

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
isend/overlap: 6 iterations in 0.000484 secs (0.867360 MFlops); diffnorm 0.008134, m=7 n=4 np=1
isend/overlap: 7 iterations in 0.074253 secs (5.405953 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
isend/overlap: 24 iterations in 0.007536 secs (0.891735 MFlops); diffnorm 0.009895, m=7 n=10 np=4
isend/overlap: 25 iterations in 0.339793 secs (16.876142 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
isend/overlap: 25 iterations in 0.009446 secs (2.964210 MFlops); diffnorm 0.036615, m=7 n=34 np=16
isend/overlap: 25 iterations in 0.346255 secs (66.244785 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
isend/overlap: 25 iterations in 0.010722 secs (5.222755 MFlops); diffnorm 0.055291, m=7 n=66 np=32
isend/overlap: 25 iterations in 0.350676 secs (130.819421 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Overlapping communication (sends first)

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
send first/overlap: 6 iterations in 0.000491 secs (0.855409 MFlops); diffnorm 0.008134, m=7 n=4 np=1
send first/overlap: 7 iterations in 0.074393 secs (5.395778 MFlops); diffnorm 0.006899, m=4098 n=4 np=1
send first/overlap: 24 iterations in 0.008690 secs (0.773263 MFlops); diffnorm 0.009895, m=7 n=10 np=4
send first/overlap: 25 iterations in 0.339618 secs (16.884834 MFlops); diffnorm 0.138820, m=4098 n=10 np=4
send first/overlap: 25 iterations in 0.010563 secs (2.650848 MFlops); diffnorm 0.036615, m=7 n=34 np=16
send first/overlap: 25 iterations in 0.344820 secs (66.520505 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
send first/overlap: 25 iterations in 0.011356 secs (4.931225 MFlops); diffnorm 0.055291, m=7 n=66 np=32
send first/overlap: 25 iterations in 0.347709 secs (131.935737 MFlops); diffnorm 0.470684, m=4098 n=66 np=32

Jacobi Iteration - Persistent send/recv

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000009
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000001
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000480
   t6: 0x000000400006fc70	 t7: 0x000000400006fd70   t8: 0x000000400006fd70
   t9: 0x000000400006fd70	 t10:0x00000060fffff72d   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x000000400006fc80

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f43059679   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f43059679   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x4000000000000000   f11:0x3ff0000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71134 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000009
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000001
   t3: 0x0000000000000001	 t4: 0x0000000000000000   t5: 0x0000004000000480
   t6: 0x000000400008fbf0	 t7: 0x00000040000afc70   t8: 0x00000040000afc70
   t9: 0x00000040000afc70	 t10:0x00000060fffff72a   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x000000400008fc00

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f4e1058ee   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f4e1058ee   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x4000000000000000   f11:0x3ff0000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71148 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000000
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000000
   t3: 0x0000000000000001	 t4: 0x0000000000000008   t5: 0x0000000000000001
   t6: 0x000000560003f500	 t7: 0x0000004000070db0   t8: 0x0000004000070db0
   t9: 0x0000004000070db0	 t10:0x00000060fffff72d   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x0000004000070cc0

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f58ef6b16   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f58ef6b16   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x0000000000000002   f11:0x4010000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71163 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000000
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000000
   t3: 0x0000000000000001	 t4: 0x0000000000000008   t5: 0x0000000000000001
   t6: 0x000000560003f500	 t7: 0x00000040000b0cb0   t8: 0x00000040000b0cb0
   t9: 0x00000040000b0cb0	 t10:0x00000060fffff72a   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x0000004000090c40

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f643dd5d8   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f643dd5d8   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x0000000000000002   f11:0x4010000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71177 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000000
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000000
   t3: 0x0000000000000001	 t4: 0x0000000000000008   t5: 0x0000000000000001
   t6: 0x000000560003f500	 t7: 0x000000400006ff70   t8: 0x000000400006ff70
   t9: 0x000000400006ff70	 t10:0x00000060fffff72d   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x000000400006fe80

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f6f0e246e   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f6f0e246e   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x0000000000000002   f11:0x4030000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71192 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000000
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000000
   t3: 0x0000000000000001	 t4: 0x0000000000000008   t5: 0x0000000000000001
   t6: 0x000000560003f500	 t7: 0x00000040000afe70   t8: 0x00000040000afe70
   t9: 0x00000040000afe70	 t10:0x00000060fffff72a   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x000000400008fe00

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f7a05aa68   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f7a05aa68   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x0000000000000002   f11:0x4030000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71207 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000000
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000000
   t3: 0x0000000000000001	 t4: 0x0000000000000008   t5: 0x0000000000000001
   t6: 0x000000560003f500	 t7: 0x00000040000700f0   t8: 0x00000040000700f0
   t9: 0x00000040000700f0	 t10:0x00000060fffff72d   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x0000004000070000

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f854f4f4b   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f854f4f4b   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x0000000000000002   f11:0x4040000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71223 Operand range error

Agent printing core file information:
user exiting after receiving signal 11
Exit message came from virtual PE 0, logical PE 0x0
Register dump

  pa0: 0x0000000000000068	pa1: 0x0000000000001610  pa2: 0x0000000000000000

   pc: 0x000000200003e69c	 sp: 0x00000060ffffea00   fp: 0x00000060ffffeb80
   v0: 0x0000000000000000	 ra: 0x0000002000044184   ci: 0x000b120000000101

   a0: 0x00000060ffffee88	 a1: 0x00000060ffffee88   a2: 0x0000000000000000
   a3: 0x00000040000002d0	 a4: 0x0000000000000001   a5: 0x0000000000000002

   t0: 0x00000060ffffee88	 t1: 0x0000004000029430   t2: 0x0000000000000000
   t3: 0x0000000000000001	 t4: 0x0000000000000008   t5: 0x0000000000000001
   t6: 0x000000560003f500	 t7: 0x00000040000afff0   t8: 0x00000040000afff0
   t9: 0x00000040000afff0	 t10:0x00000060fffff72a   t11:0x00000040000003d5
   t12:0x000000000000000d	 t13:0x000000400008ff80

   s0: 0x0000000000000000	 s1: 0x00000060ffffee88   s2: 0x000000400001ee40
   s3: 0x0000000000000000	 s4: 0x0000000000000000   s5: 0x0000000000000000

   f00:0x41a9a70f917176df   f01:0x3e3c58c6d8a67ba9   f02:0x0000000000000000
   f03:0x41a9a70f917176df   f04:0x0000000000000000   f05:0x0000000000000000
   f06:0x0000000000000000   f07:0x0000000000000000   f08:0x0000000000000000
   f09:0x0000000000000000   f10:0x0000000000000002   f11:0x4040000000000000
   f12:0x0000000000000000   f13:0x0000000000000000   f14:0x0000000000000000
   f15:0x0000000000000000   f16:0x0000000000000000   f17:0x0000000000000000
   f18:0x0000000000000000   f19:0x0000000000000000   f20:0x0000000000000000
   f21:0x0000000000000000   f22:0x0000000000000000   f23:0x0000000000000000
   f24:0x0000000000000000   f25:0x0000000000000000   f26:0x0000000000000000
   f27:0x0000000000000000   f28:0x0000000000000000   f29:0x0000000000000000
   f30:0x0000000000000000			   fpcr:0x8900000000000000


Agent finished printing core file information.
User core dump completed (./mppcore)
/u4/gropp/mpiexmpl/maint/envs/t3d-epcc/mpirun[74]: 71242 Operand range error