Results for MPI perforance tests on sp2

Contents

memcpy

Determining delivered memory performance

	mpicc -o memcpy -O memcpy.c
Size (bytes) Time (sec)	Rate (MB/sec)
4	0.000000	18.985126
8	0.000000	37.941395
16	0.000000	76.032619
32	0.000000	70.972320
64	0.000001	109.090656
128	0.000001	149.132699
256	0.000001	182.660843
512	0.000002	205.801349
1024	0.000005	219.709235
2048	0.000009	227.390348
4096	0.000018	231.436011
8192	0.000035	233.497731
16384	0.000070	234.552805
32768	0.000139	235.082143
65536	0.000278	235.349463
131072	0.000574	228.418066
262144	0.001592	164.619353
524288	0.003165	165.676686
1048576	0.006346	165.228959
2097152	0.012861	163.057514
Running job under PBS

Determining delivered memory performance with unaligned data

	mpicc -o memcpy memcpy.c
Size (bytes) Time (sec)	Rate (MB/sec)
4	0.000000	15.642843
8	0.000000	29.534463
16	0.000000	56.050671
32	0.000001	56.050669
64	0.000001	86.871068
128	0.000001	119.808007
256	0.000002	147.840661
512	0.000003	167.424287
1024	0.000006	179.298258
2048	0.000011	185.889342
4096	0.000022	189.365272
8192	0.000043	191.148339
16384	0.000085	192.051572
32768	0.000170	192.512289
65536	0.000340	192.745864
131072	0.000698	187.880932
262144	0.001853	141.450967
524288	0.003683	142.344819
1048576	0.007368	142.308116
2097152	0.014944	140.334516
Running job under PBS

pingpong

Benchmarking point to point performance

	mpicc -o pingpong -O pingpong.c
Kind		n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000053	0.150089
Send/Recv	2	0.000054	0.298215
Send/Recv	4	0.000054	0.596500
Send/Recv	8	0.000054	1.184371
Send/Recv	16	0.000055	2.310024
Send/Recv	32	0.000071	3.630687
Send/Recv	64	0.000084	6.099959
Send/Recv	128	0.000101	10.126798
Send/Recv	256	0.000141	14.563124
Send/Recv	512	0.000213	19.230048
Send/Recv	1024	0.000423	19.361279
Send/Recv	2048	0.000674	24.323040
Send/Recv	4096	0.001171	27.989792
Send/Recv	8192	0.002097	31.246490
Send/Recv	16384	0.004030	32.524977
Send/Recv	32768	0.007784	33.679180
Send/Recv	65536	0.015430	33.977823
Send/Recv	131072	0.030680	34.178072
Send/Recv	262144	0.061428	34.140008
Send/Recv	524288	0.122645	34.198834
Send/Recv	1048576	0.244523	34.305987
Running job under PBS

Benchmarking point to point performance with nonblocking operations

	mpicc -o pingpong -O pingpong.c
Kind		n	time (sec)	Rate (MB/sec)
Isend/Irecv	1	0.000063	0.125993
Isend/Irecv	2	0.000063	0.252664
Isend/Irecv	4	0.000063	0.504331
Isend/Irecv	8	0.000064	1.000644
Isend/Irecv	16	0.000065	1.968846
Isend/Irecv	32	0.000081	3.141229
Isend/Irecv	64	0.000092	5.575469
Isend/Irecv	128	0.000112	9.133245
Isend/Irecv	256	0.000153	13.353618
Isend/Irecv	512	0.000225	18.239913
Isend/Irecv	1024	0.000420	19.483886
Isend/Irecv	2048	0.000685	23.914758
Isend/Irecv	4096	0.001165	28.116783
Isend/Irecv	8192	0.002098	31.235881
Isend/Irecv	16384	0.004051	32.358663
Isend/Irecv	32768	0.007869	33.313720
Isend/Irecv	65536	0.015441	33.954058
Isend/Irecv	131072	0.030745	34.105911
Isend/Irecv	262144	0.061168	34.284962
Isend/Irecv	524288	0.122713	34.179858
Isend/Irecv	1048576	0.244238	34.346092
Running job under PBS

Benchmarking point to point performance with nonblocking operations, head-to-head

	mpicc -o pingpong -O pingpong.c
Kind				n	time (sec)	Rate (MB/sec)
head-to-head Isend/Irecv	1	0.000078	0.204488
head-to-head Isend/Irecv	2	0.000078	0.407942
head-to-head Isend/Irecv	4	0.000079	0.810322
head-to-head Isend/Irecv	8	0.000079	1.615431
head-to-head Isend/Irecv	16	0.000081	3.148363
head-to-head Isend/Irecv	32	0.000097	5.289917
head-to-head Isend/Irecv	64	0.000111	9.196503
head-to-head Isend/Irecv	128	0.000132	15.482894
head-to-head Isend/Irecv	256	0.000171	23.922904
head-to-head Isend/Irecv	512	0.000285	28.756468
head-to-head Isend/Irecv	1024	0.000560	29.238868
head-to-head Isend/Irecv	2048	0.000963	34.022586
head-to-head Isend/Irecv	4096	0.001630	40.214152
head-to-head Isend/Irecv	8192	0.003025	43.322428
head-to-head Isend/Irecv	16384	0.005868	44.672340
head-to-head Isend/Irecv	32768	0.011529	45.477457
head-to-head Isend/Irecv	65536	0.022791	46.008286
head-to-head Isend/Irecv	131072	0.045532	46.059315
head-to-head Isend/Irecv	262144	0.090737	46.224759
head-to-head Isend/Irecv	524288	0.181549	46.205866
head-to-head Isend/Irecv	1048576	0.363547	46.148720
Running job under PBS

Benchmarking point to point performance with unaligned data

	mpicc -o pingpong -O pingpong.c
Kind char		n	time (sec)	Rate (MB/sec)
Send/Recv		1	0.000053	0.018697
Send/Recv		2	0.000053	0.037477
Send/Recv		4	0.000053	0.075319
Send/Recv		8	0.000053	0.149838
Send/Recv		16	0.000053	0.302274
Send/Recv		32	0.000054	0.593275
Send/Recv		64	0.000054	1.179579
Send/Recv		128	0.000057	2.264127
Send/Recv		256	0.000071	3.590043
Send/Recv		512	0.000087	5.891832
Send/Recv		1024	0.000107	9.547780
Send/Recv		2048	0.000141	14.499118
Send/Recv		4096	0.000210	19.485052
Send/Recv		8192	0.000410	19.967704
Send/Recv		16384	0.000672	24.398197
Send/Recv		32768	0.001156	28.357368
Send/Recv		65536	0.002106	31.121664
Send/Recv		131072	0.004029	32.535675
Send/Recv		262144	0.007798	33.618711
Send/Recv		524288	0.015488	33.851813
Send/Recv		1048576	0.030682	34.175107
Kind double		n	time (sec)	Rate (MB/sec)
Send/Recv		1	0.000053	0.149679
Send/Recv		2	0.000053	0.299734
Send/Recv		4	0.000054	0.593250
Send/Recv		8	0.000054	1.182612
Send/Recv		16	0.000056	2.304716
Send/Recv		32	0.000071	3.623020
Send/Recv		64	0.000086	5.967946
Send/Recv		128	0.000103	9.981203
Send/Recv		256	0.000142	14.424649
Send/Recv		512	0.000207	19.823354
Send/Recv		1024	0.000407	20.106769
Send/Recv		2048	0.000684	23.938342
Send/Recv		4096	0.001158	28.290344
Send/Recv		8192	0.002119	30.932905
Send/Recv		16384	0.004069	32.210457
Send/Recv		32768	0.007951	32.971029
Send/Recv		65536	0.015507	33.809382
Send/Recv		131072	0.030882	33.953783
Send/Recv		262144	0.061395	34.158449
Send/Recv		524288	0.122433	34.257915
Send/Recv		1048576	0.245376	34.186701
Kind int		n	time (sec)	Rate (MB/sec)
Send/Recv		1	0.000053	0.074976
Send/Recv		2	0.000053	0.149942
Send/Recv		4	0.000054	0.298942
Send/Recv		8	0.000054	0.596341
Send/Recv		16	0.000054	1.178917
Send/Recv		32	0.000055	2.321520
Send/Recv		64	0.000071	3.630004
Send/Recv		128	0.000084	6.094201
Send/Recv		256	0.000102	10.004071
Send/Recv		512	0.000145	14.141208
Send/Recv		1024	0.000211	19.401973
Send/Recv		2048	0.000419	19.560066
Send/Recv		4096	0.000669	24.486165
Send/Recv		8192	0.001159	28.282715
Send/Recv		16384	0.002124	30.860439
Send/Recv		32768	0.004062	32.269040
Send/Recv		65536	0.007892	33.216738
Send/Recv		131072	0.015544	33.729800
Send/Recv		262144	0.030750	34.100282
Send/Recv		524288	0.061239	34.245436
Send/Recv		1048576	0.122615	34.206981
Running job under PBS

Benchmarking point to point performance with contention

	mpicc -o pingpong -O pingpong.c
Kind (np=2)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000054	0.149366
Send/Recv	2	0.000054	0.296665
Send/Recv	4	0.000054	0.594678
Send/Recv	8	0.000054	1.181097
Send/Recv	16	0.000056	2.290092
Send/Recv	32	0.000071	3.584928
Send/Recv	64	0.000085	6.027016
Send/Recv	128	0.000100	10.224116
Send/Recv	256	0.000142	14.442878
Send/Recv	512	0.000210	19.512888
Send/Recv	1024	0.000422	19.406569
Send/Recv	2048	0.000680	24.089690
Send/Recv	4096	0.001163	28.170564
Send/Recv	8192	0.002098	31.243885
Send/Recv	16384	0.004012	32.667039
Send/Recv	32768	0.007813	33.554218
Send/Recv	65536	0.015561	33.692003
Send/Recv	131072	0.030591	34.277481
Send/Recv	262144	0.061389	34.161628
Send/Recv	524288	0.122552	34.224667
Send/Recv	1048576	0.245654	34.148081
Running job under PBS
Kind (np=4)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000054	0.148377
Send/Recv	2	0.000054	0.298231
Send/Recv	4	0.000054	0.589707
Send/Recv	8	0.000054	1.175876
Send/Recv	16	0.000056	2.290547
Send/Recv	32	0.000071	3.618911
Send/Recv	64	0.000086	5.969627
Send/Recv	128	0.000101	10.144532
Send/Recv	256	0.000141	14.510244
Send/Recv	512	0.000212	19.331010
Send/Recv	1024	0.000416	19.675755
Send/Recv	2048	0.000676	24.254625
Send/Recv	4096	0.001160	28.248276
Send/Recv	8192	0.002121	30.902458
Send/Recv	16384	0.004045	32.399757
Send/Recv	32768	0.007877	33.278618
Send/Recv	65536	0.015517	33.788655
Send/Recv	131072	0.030811	34.032479
Send/Recv	262144	0.061475	34.113629
Send/Recv	524288	0.122883	34.132372
Send/Recv	1048576	0.245223	34.208087
Running job under PBS
Kind (np=8)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000054	0.148451
Send/Recv	2	0.000054	0.296799
Send/Recv	4	0.000054	0.587943
Send/Recv	8	0.000055	1.173255
Send/Recv	16	0.000056	2.274496
Send/Recv	32	0.000072	3.570874
Send/Recv	64	0.000086	5.971542
Send/Recv	128	0.000103	9.950717
Send/Recv	256	0.000142	14.386233
Send/Recv	512	0.000213	19.270756
Send/Recv	1024	0.000420	19.500122
Send/Recv	2048	0.000679	24.119833
Send/Recv	4096	0.001164	28.144551
Send/Recv	8192	0.002122	30.886619
Send/Recv	16384	0.004038	32.457222
Send/Recv	32768	0.007872	33.300231
Send/Recv	65536	0.015558	33.698473
Send/Recv	131072	0.030774	34.073882
Send/Recv	262144	0.061564	34.064832
Send/Recv	524288	0.122607	34.209301
Send/Recv	1048576	0.246236	34.067417
Running job under PBS
Command to clear nodes took too long.
Please contact system administrator.
Kind (np=16)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000054	0.147868
Send/Recv	2	0.000054	0.295488
Send/Recv	4	0.000055	0.586556
Send/Recv	8	0.000055	1.166706
Send/Recv	16	0.000056	2.271948
Send/Recv	32	0.000072	3.568466
Send/Recv	64	0.000087	5.918733
Send/Recv	128	0.000105	9.778657
Send/Recv	256	0.000146	14.056682
Send/Recv	512	0.000225	18.208489
Send/Recv	1024	0.000442	18.554403
Send/Recv	2048	0.000718	22.826492
Send/Recv	4096	0.001253	26.160770
Send/Recv	8192	0.002282	28.715994
Send/Recv	16384	0.004405	29.758402
Send/Recv	32768	0.008663	30.260100
Send/Recv	65536	0.017099	30.661686
Send/Recv	131072	0.034008	30.833182
Send/Recv	262144	0.068180	30.759016
Send/Recv	524288	0.135852	30.874015
Send/Recv	1048576	0.270954	30.959542
Running job under PBS
Kind (np=32)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000055	0.146436
Send/Recv	2	0.000055	0.293164
Send/Recv	4	0.000055	0.580721
Send/Recv	8	0.000055	1.153623
Send/Recv	16	0.000058	2.202097
Send/Recv	32	0.000074	3.438773
Send/Recv	64	0.000088	5.805757
Send/Recv	128	0.000108	9.478347
Send/Recv	256	0.000156	13.103704
Send/Recv	512	0.000334	12.263935
Send/Recv	1024	0.000503	16.283452
Send/Recv	2048	0.000829	19.762379
Send/Recv	4096	0.001534	21.359406
Send/Recv	8192	0.002848	23.013458
Send/Recv	16384	0.005491	23.869518
Send/Recv	32768	0.010778	24.321602
Send/Recv	65536	0.021273	24.645701
Send/Recv	131072	0.042271	24.806096
Send/Recv	262144	0.083974	24.973866
Send/Recv	524288	0.168072	24.955330
Send/Recv	1048576	0.336926	24.897471
Running job under PBS
Kind (np=64)	n	time (sec)	Rate (MB/sec)
Send/Recv	1	0.000055	0.146161
Send/Recv	2	0.000054	0.294086
Send/Recv	4	0.000055	0.585374
Send/Recv	8	0.000055	1.156581
Send/Recv	16	0.000056	2.300315
Send/Recv	32	0.000071	3.595709
Send/Recv	64	0.000086	5.974503
Send/Recv	128	0.000108	9.522260
Send/Recv	256	0.000228	8.984426
Send/Recv	512	0.000314	13.050301
Send/Recv	1024	0.000488	16.776142
Send/Recv	2048	0.000796	20.594557
Send/Recv	4096	0.001487	22.030759
Send/Recv	8192	0.002909	22.531996
Send/Recv	16384	0.005741	22.829176
Send/Recv	32768	0.011942	21.951156
Send/Recv	65536	0.024091	21.763245
Send/Recv	131072	0.047464	22.092173
Send/Recv	262144	0.094512	22.189308
Send/Recv	524288	0.182551	22.976102
Send/Recv	1048576	0.370842	22.620428

barrier

Benchmarking collective barrier

	mpicc -o barrier -O barrier.c
Kind	np	time (sec)
Barrier	1	0.000009
Running job under PBS
Barrier	2	0.000083
Running job under PBS
Barrier	4	0.000148
Running job under PBS
Barrier	8	0.000217
Running job under PBS
Barrier	16	0.000297
Running job under PBS
Barrier	32	0.000391
Running job under PBS
Barrier	64	0.000564

Benchmarking collective Allreduce

	mpicc -o barrier -O barrier.c
Kind		np	time (sec)
Allreduce	1	0.000019
Running job under PBS
Allreduce	2	0.000107
Running job under PBS
Allreduce	4	0.000182
Running job under PBS
Allreduce	8	0.000265
Running job under PBS
Allreduce	16	0.000362
Running job under PBS
Allreduce	32	0.000476
Running job under PBS
Allreduce	64	0.000667

vector

Comparing the performance of MPI vector datatypes

	mpicc -o vector -O vector.c
Kind	n	stride	time (sec)	Rate (MB/sec)
Vector	1000	24	0.002031	3.939534
Struct	1000	24	0.002665	3.002139
User	1000	24	0.000536	14.933010
User(add)	1000	24	0.000534	14.980330
Running job under PBS

circulate

Pipelining pitfalls

	mpicc -c -O  circulate.c
	mpicc -o circulate -O circulate.o -lm
For n = 20000, m = 20000, T_comm = 0.012568, T_compute = 0.025471, sum = 0.038039, T_both = 0.031753
Running job under PBS
For n = 500, m = 500, T_comm = 0.000149, T_compute = 0.000630, sum = 0.000779, T_both = 0.000771
Running job under PBS

3way

Exploring the cost of synchronization delays

	mpicc -c -O  bad.c
	mpicc -o bad -O bad.o  -lm
[2] Litsize = 8, Time for first send = 0.000058, for second = 0.000030
Running job under PBS
[2] Litsize = 9, Time for first send = 0.000051, for second = 0.000030
Running job under PBS
[2] Litsize = 511, Time for first send = 0.000173, for second = 0.000138
Running job under PBS
[2] Litsize = 512, Time for first send = 0.000172, for second = 0.000141
Running job under PBS
[2] Litsize = 513, Time for first send = 0.000341, for second = 0.001797
Running job under PBS

jacobi

Jacobi Iteration - Example Parallel Mesh

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
send/recv: 25 iterations in 0.014018 secs (1.997457 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
send/recv: 25 iterations in 0.858319 secs (26.723861 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
send/recv: 25 iterations in 0.017718 secs (3.160579 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
send/recv: 25 iterations in 1.814446 secs (25.283312 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
send/recv: 25 iterations in 0.022098 secs (5.068343 MFlops); diffnorm 0.080560, m=7 n=130 np=64
send/recv: 25 iterations in 3.511754 secs (26.126662 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Shift up and down

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
shift/sendrecv: 25 iterations in 0.014787 secs (1.893517 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
shift/sendrecv: 25 iterations in 0.144997 secs (158.193181 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
shift/sendrecv: 25 iterations in 0.019105 secs (2.931166 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
shift/sendrecv: 25 iterations in 0.175082 secs (262.021187 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
shift/sendrecv: 25 iterations in 0.024481 secs (4.575037 MFlops); diffnorm 0.080560, m=7 n=130 np=64
shift/sendrecv: 25 iterations in 0.168200 secs (545.483704 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Exchange head-to-head

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
head-to-head sendrecv: 25 iterations in 0.014324 secs (1.954696 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.153932 secs (149.011736 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.018076 secs (3.098103 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.181382 secs (252.920530 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.023036 secs (4.861855 MFlops); diffnorm 0.080560, m=7 n=130 np=64
head-to-head sendrecv: 25 iterations in 0.172437 secs (532.080162 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Nonblocking send/recv

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
irecv/isend: 25 iterations in 0.013462 secs (2.079867 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
irecv/isend: 25 iterations in 0.147150 secs (155.878532 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
irecv/isend: 25 iterations in 0.020551 secs (2.724965 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
irecv/isend: 25 iterations in 0.178002 secs (257.722404 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
irecv/isend: 25 iterations in 0.023265 secs (4.814161 MFlops); diffnorm 0.080560, m=7 n=130 np=64
irecv/isend: 25 iterations in 0.175803 secs (521.894369 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Nonblocking send/recv for receiver pull

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
isend/irecv: 25 iterations in 0.014331 secs (1.953823 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
isend/irecv: 25 iterations in 0.139625 secs (164.280095 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
isend/irecv: 25 iterations in 0.019518 secs (2.869161 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
isend/irecv: 25 iterations in 0.168687 secs (271.954566 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
isend/irecv: 25 iterations in 0.024794 secs (4.517204 MFlops); diffnorm 0.080560, m=7 n=130 np=64
isend/irecv: 25 iterations in 0.155092 secs (591.586929 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Synchronous send

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
ssend/irecv: 25 iterations in 0.019663 secs (1.423976 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
ssend/irecv: 25 iterations in 0.258036 secs (88.893005 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
ssend/irecv: 25 iterations in 0.023209 secs (2.412857 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
ssend/irecv: 25 iterations in 0.175369 secs (261.593047 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
ssend/irecv: 25 iterations in 0.031128 secs (3.598090 MFlops); diffnorm 0.080560, m=7 n=130 np=64
b1313.nas.nasa.gov b1311.nas.nasa.gov b1309.nas.nasa.gov b1307.nas.nasa.gov b1305.nas.nasa.gov b1303.nas.nasa.gov b1215.nas.nasa.gov b1213.nas.nasa.gov b1211.nas.nasa.gov b1209.nas.nasa.gov b1207.nas.nasa.gov b1205.nas.nasa.gov b1203.nas.nasa.gov b1115.nas.nasa.gov b1113.nas.nasa.gov b1111.nas.nasa.gov b1109.nas.nasa.gov b1107.nas.nasa.gov b1105.nas.nasa.gov b1103.nas.nasa.gov b1015.nas.nasa.gov b1013.nas.nasa.gov b1011.nas.nasa.gov b1009.nas.nasa.gov b1007.nas.nasa.gov b1005.nas.nasa.gov b1003.nas.nasa.gov b1001.nas.nasa.gov b0915.nas.nasa.gov b0913.nas.nasa.gov b0911.nas.nasa.gov b0909.nas.nasa.gov b0907.nas.nasa.gov b0905.nas.nasa.gov b0903.nas.nasa.gov b0815.nas.nasa.gov b0813.nas.nasa.gov b0811.nas.nasa.gov b0809.nas.nasa.gov b0807.nas.nasa.gov b0805.nas.nasa.gov b0803.nas.nasa.gov b0801.nas.nasa.gov b0715.nas.nasa.gov b0713.nas.nasa.gov b0711.nas.nasa.gov b0709.nas.nasa.gov b0707.nas.nasa.gov b0705.nas.nasa.gov b0703.nas.nasa.gov b0615.nas.nasa.gov b0613.nas.nasa.gov b0611.nas.nasa.gov b0609.nas.nasa.gov b0607.nas.nasa.gov b0605.nas.nasa.gov b0603.nas.nasa.gov b0515.nas.nasa.gov b0513.nas.nasa.gov b0511.nas.nasa.gov b0509.nas.nasa.gov b0507.nas.nasa.gov b0505.nas.nasa.gov b0503.nas.nasa.gov ERROR: 0031-616  gethostbyname failed for home node

Jacobi Iteration - Ready send

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
rsend: 25 iterations in 0.014087 secs (1.987666 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
rsend: 25 iterations in 0.138943 secs (165.085986 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
rsend: 25 iterations in 0.017959 secs (3.118214 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
rsend: 25 iterations in 0.167202 secs (274.369185 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
rsend: 25 iterations in 0.022795 secs (4.913315 MFlops); diffnorm 0.080560, m=7 n=130 np=64
rsend: 25 iterations in 0.146721 secs (625.338290 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Overlapping communication

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
isend/overlap: 25 iterations in 0.014547 secs (1.924819 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
isend/overlap: 25 iterations in 0.157001 secs (146.098316 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
isend/overlap: 25 iterations in 0.017324 secs (3.232566 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
isend/overlap: 25 iterations in 0.179038 secs (256.232067 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
isend/overlap: 25 iterations in 0.022999 secs (4.869682 MFlops); diffnorm 0.080560, m=7 n=130 np=64
isend/overlap: 25 iterations in 0.180448 secs (508.459651 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Overlapping communication (sends first)

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
send first/overlap: 25 iterations in 0.014333 secs (1.953568 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
send first/overlap: 25 iterations in 0.141071 secs (162.595765 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
send first/overlap: 25 iterations in 0.018143 secs (3.086586 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
send first/overlap: 25 iterations in 0.170800 secs (268.590714 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
send first/overlap: 25 iterations in 0.025103 secs (4.461600 MFlops); diffnorm 0.080560, m=7 n=130 np=64
send first/overlap: 25 iterations in 0.176234 secs (520.617055 MFlops); diffnorm 0.474303, m=4098 n=130 np=64

Jacobi Iteration - Persistent send/recv

	mpicc -c -O  jacobi.c
	mpicc -c -O  cmdline.c
	mpicc -c -O  setupmesh.c
	mpicc -c -O  exchng.c
	mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
p021 p020 p019 p018 p016 p015 p014 p013 p012 p008 p007 p006 p005 p004 p003 p002 Running job under PBS
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 1
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 0
ERROR: 0031-250  task 11: Terminated
ERROR: 0031-250  task 12: Terminated
ERROR: 0031-250  task 13: Terminated
ERROR: 0031-250  task 14: Terminated
ERROR: 0031-250  task 15: Terminated
ERROR: 0031-250  task 3: Terminated
ERROR: 0031-250  task 1: Terminated
ERROR: 0031-250  task 2: Terminated
ERROR: 0031-250  task 8: Terminated
ERROR: 0031-250  task 9: Terminated
ERROR: 0031-250  task 10: Terminated
ERROR: 0031-250  task 0: Terminated
ERROR: 0031-250  task 4: Terminated
ERROR: 0031-250  task 5: Terminated
ERROR: 0031-250  task 6: Terminated
ERROR: 0031-250  task 7: Terminated
p021 p020 p019 p018 p016 p015 p014 p013 p012 p008 p007 p006 p005 p004 p003 p002 Running job under PBS
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 0
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 1
ERROR: 0031-250  task 0: Terminated
ERROR: 0031-250  task 1: Terminated
ERROR: 0031-250  task 2: Terminated
ERROR: 0031-250  task 3: Terminated
ERROR: 0031-250  task 4: Terminated
ERROR: 0031-250  task 5: Terminated
ERROR: 0031-250  task 6: Terminated
ERROR: 0031-250  task 7: Terminated
ERROR: 0031-250  task 8: Terminated
ERROR: 0031-250  task 9: Terminated
ERROR: 0031-250  task 10: Terminated
ERROR: 0031-250  task 11: Terminated
ERROR: 0031-250  task 12: Terminated
ERROR: 0031-250  task 13: Terminated
ERROR: 0031-250  task 14: Terminated
ERROR: 0031-250  task 15: Terminated
Running job under PBS
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 1
=>> PBS: job killed: walltime 308 exceeded limit 300
ERROR: 0031-250  task 30: Terminated
ERROR: 0031-250  task 31: Terminated
ERROR: 0031-250  task 27: Terminated
ERROR: 0031-250  task 28: Terminated
ERROR: 0031-250  task 29: Terminated
ERROR: 0031-250  task 19: Terminated
ERROR: 0031-250  task 25: Terminated
ERROR: 0031-250  task 26: Terminated
ERROR: 0031-250  task 22: Terminated
ERROR: 0031-250  task 23: Terminated
ERROR: 0031-250  task 24: Terminated
ERROR: 0031-250  task 0: Terminated
ERROR: 0031-250  task 1: Terminated
ERROR: 0031-250  task 2: Terminated
ERROR: 0031-250  task 3: Terminated
ERROR: 0031-250  task 4: Terminated
ERROR: 0031-250  task 5: Terminated
ERROR: 0031-250  task 6: Terminated
ERROR: 0031-250  task 7: Terminated
ERROR: 0031-250  task 8: Terminated
ERROR: 0031-250  task 9: Terminated
ERROR: 0031-250  task 10: Terminated
ERROR: 0031-250  task 11: Terminated
ERROR: 0031-250  task 14: Terminated
ERROR: 0031-250  task 15: Terminated
ERROR: 0031-250  task 16: Terminated
ERROR: 0031-250  task 17: Terminated
ERROR: 0031-250  task 18: Terminated
ERROR: 0031-250  task 20: Terminated
ERROR: 0031-250  task 21: Terminated
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 1
=>> PBS: job killed: walltime 329 exceeded limit 300
sh: 77614 Terminated
ERROR: 0031-250  task 31: Terminated
ERROR: 0031-250  task 29: Terminated
ERROR: 0031-250  task 30: Terminated
ERROR: 0031-250  task 10: Terminated
ERROR: 0031-250  task 11: Terminated
ERROR: 0031-250  task 12: Terminated
ERROR: 0031-250  task 13: Terminated
ERROR: 0031-250  task 14: Terminated
ERROR: 0031-250  task 15: Terminated
ERROR: 0031-250  task 16: Terminated
ERROR: 0031-250  task 17: Terminated
ERROR: 0031-250  task 18: Terminated
ERROR: 0031-250  task 19: Terminated
ERROR: 0031-250  task 21: Terminated
ERROR: 0031-250  task 22: Terminated
ERROR: 0031-250  task 23: Terminated
ERROR: 0031-250  task 24: Terminated
ERROR: 0031-250  task 25: Terminated
ERROR: 0031-250  task 26: Terminated
ERROR: 0031-250  task 27: Terminated
ERROR: 0031-250  task 28: Terminated
ERROR: 0031-250  task 7: Terminated
ERROR: 0031-250  task 8: Terminated
ERROR: 0031-250  task 9: Terminated
ERROR: 0031-250  task 0: Terminated
ERROR: 0031-250  task 1: Terminated
ERROR: 0031-250  task 2: Terminated
ERROR: 0031-250  task 3: Terminated
ERROR: 0031-250  task 4: Terminated
ERROR: 0031-250  task 5: Terminated
ERROR: 0031-250  task 6: Terminated
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 4
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 2
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 3
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 7
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 5
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 6
=>> PBS: job killed: walltime 308 exceeded limit 300
sh: 101720 Terminated
ERROR: 0031-250  task 63: Terminated
ERROR: 0031-250  task 61: Terminated
ERROR: 0031-250  task 62: Terminated
ERROR: 0031-250  task 58: Terminated
ERROR: 0031-250  task 59: Terminated
ERROR: 0031-250  task 60: Terminated
ERROR: 0031-250  task 56: Terminated
ERROR: 0031-250  task 57: Terminated
ERROR: 0031-250  task 1: Terminated
ERROR: 0031-250  task 2: Terminated
ERROR: 0031-250  task 23: Terminated
ERROR: 0031-250  task 24: Terminated
ERROR: 0031-250  task 25: Terminated
ERROR: 0031-250  task 26: Terminated
ERROR: 0031-250  task 27: Terminated
ERROR: 0031-250  task 28: Terminated
ERROR: 0031-250  task 29: Terminated
ERROR: 0031-250  task 30: Terminated
ERROR: 0031-250  task 31: Terminated
ERROR: 0031-250  task 32: Terminated
ERROR: 0031-250  task 33: Terminated
ERROR: 0031-250  task 34: Terminated
ERROR: 0031-250  task 35: Terminated
ERROR: 0031-250  task 36: Terminated
ERROR: 0031-250  task 37: Terminated
ERROR: 0031-250  task 38: Terminated
ERROR: 0031-250  task 39: Terminated
ERROR: 0031-250  task 40: Terminated
ERROR: 0031-250  task 41: Terminated
ERROR: 0031-250  task 42: Terminated
ERROR: 0031-250  task 43: Terminated
ERROR: 0031-250  task 44: Terminated
ERROR: 0031-250  task 45: Terminated
ERROR: 0031-250  task 46: Terminated
ERROR: 0031-250  task 47: Terminated
ERROR: 0031-250  task 49: Terminated
ERROR: 0031-250  task 50: Terminated
ERROR: 0031-250  task 52: Terminated
ERROR: 0031-250  task 53: Terminated
ERROR: 0031-250  task 54: Terminated
ERROR: 0031-250  task 55: Terminated
ERROR: 0031-250  task 0: Terminated
ERROR: 0031-250  task 8: Terminated
ERROR: 0031-250  task 9: Terminated
ERROR: 0031-250  task 10: Terminated
ERROR: 0031-250  task 11: Terminated
ERROR: 0031-250  task 12: Terminated
ERROR: 0031-250  task 16: Terminated
ERROR: 0031-250  task 17: Terminated
ERROR: 0031-250  task 18: Terminated
ERROR: 0031-250  task 19: Terminated
ERROR: 0031-250  task 20: Terminated
ERROR: 0031-250  task 21: Terminated
ERROR: 0031-250  task 22: Terminated
ERROR: 0031-250  task 3: Terminated
ERROR: 0031-250  task 4: Terminated
ERROR: 0031-250  task 5: Terminated
ERROR: 0031-250  task 6: Terminated
ERROR: 0031-250  task 7: Terminated
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 3
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 5
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 6
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 4
ERROR: 0032-158 Persistent request already active  (6) in MPI_Startall, task 7
ERROR: 0032-158 Persistent request already active  (7) in MPI_Startall, task 2
=>> PBS: job killed: walltime 336 exceeded limit 300
sh: 76886 Terminated
ERROR: 0031-250  task 63: Terminated
ERROR: 0031-250  task 61: Terminated
ERROR: 0031-250  task 62: Terminated
ERROR: 0031-250  task 56: Terminated
ERROR: 0031-250  task 57: Terminated
ERROR: 0031-250  task 58: Terminated
ERROR: 0031-250  task 59: Terminated
ERROR: 0031-250  task 60: Terminated
ERROR: 0031-250  task 53: Terminated
ERROR: 0031-250  task 54: Terminated
ERROR: 0031-250  task 55: Terminated
ERROR: 0031-250  task 21: Terminated
ERROR: 0031-250  task 22: Terminated
ERROR: 0031-250  task 23: Terminated
ERROR: 0031-250  task 28: Terminated
ERROR: 0031-250  task 31: Terminated
ERROR: 0031-250  task 32: Terminated
ERROR: 0031-250  task 33: Terminated
ERROR: 0031-250  task 34: Terminated
ERROR: 0031-250  task 35: Terminated
ERROR: 0031-250  task 36: Terminated
ERROR: 0031-250  task 37: Terminated
ERROR: 0031-250  task 38: Terminated
ERROR: 0031-250  task 39: Terminated
ERROR: 0031-250  task 40: Terminated
ERROR: 0031-250  task 41: Terminated
ERROR: 0031-250  task 42: Terminated
ERROR: 0031-250  task 43: Terminated
ERROR: 0031-250  task 44: Terminated
ERROR: 0031-250  task 45: Terminated
ERROR: 0031-250  task 46: Terminated
ERROR: 0031-250  task 47: Terminated
ERROR: 0031-250  task 48: Terminated
ERROR: 0031-250  task 49: Terminated
ERROR: 0031-250  task 50: Terminated
ERROR: 0031-250  task 51: Terminated
ERROR: 0031-250  task 52: Terminated
ERROR: 0031-250  task 4: Terminated
ERROR: 0031-250  task 5: Terminated
ERROR: 0031-250  task 6: Terminated
ERROR: 0031-250  task 7: Terminated
ERROR: 0031-250  task 8: Terminated
ERROR: 0031-250  task 9: Terminated
ERROR: 0031-250  task 10: Terminated
ERROR: 0031-250  task 11: Terminated
ERROR: 0031-250  task 12: Terminated
ERROR: 0031-250  task 13: Terminated
ERROR: 0031-250  task 14: Terminated
ERROR: 0031-250  task 15: Terminated
ERROR: 0031-250  task 17: Terminated
ERROR: 0031-250  task 18: Terminated
ERROR: 0031-250  task 0: Terminated
ERROR: 0031-250  task 1: Terminated
ERROR: 0031-250  task 2: Terminated
ERROR: 0031-250  task 3: Terminated
ERROR: 0031-250  task 19: Terminated
ERROR: 0031-250  task 20: Terminated