1-LastTest.log

1-LastTest.log (text/plain), 49.32 KB, created by Agostino Sarubbo on 2021-10-07 09:27:13 UTC

Start testing: Oct 07 10:50 CEST
----------------------------------------------------------
1/21 Testing: test_scr_param
1/21 Test: test_scr_param
Command: "/var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/scripts/test/test_scr_param"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/scripts/test
"test_scr_param" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
<end of output>
Test time =   0.03 sec
----------------------------------------------------------
Test Passed.
"test_scr_param" end time: Oct 07 10:50 CEST
"test_scr_param" time elapsed: 00:00:00
----------------------------------------------------------

4/21 Testing: parallel_test_api_start
4/21 Test: parallel_test_api_start
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_api" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_api_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api  restart=start
Running ./test_api  restart=start
Running ./test_api  restart=start
Running ./test_api  restart=start
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[localhost:00047] PMIX ERROR: UNREACHABLE in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2193
[localhost:00047] PMIX ERROR: UNREACHABLE in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2193
[localhost:00047] PMIX ERROR: UNREACHABLE in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2193
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00047] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00047] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.82 sec
----------------------------------------------------------
Test Failed.
"parallel_test_api_start" end time: Oct 07 10:50 CEST
"parallel_test_api_start" time elapsed: 00:00:00
----------------------------------------------------------

12/21 Testing: parallel_test_api_multiple_start
12/21 Test: parallel_test_api_multiple_start
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_api_multiple" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_api_multiple_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api_multiple  restart=start
Running ./test_api_multiple  restart=start
Running ./test_api_multiple  restart=start
Running ./test_api_multiple  restart=start
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00051] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00051] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.89 sec
----------------------------------------------------------
Test Failed.
"parallel_test_api_multiple_start" end time: Oct 07 10:50 CEST
"parallel_test_api_multiple_start" time elapsed: 00:00:00
----------------------------------------------------------

20/21 Testing: parallel_test_ckpt_F_start
20/21 Test: parallel_test_ckpt_F_start
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_ckpt_F" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_ckpt_F_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt_F  restart=start
Running ./test_ckpt_F  restart=start
Running ./test_ckpt_F  restart=start
Running ./test_ckpt_F  restart=start
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
[localhost:00055] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00055] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.90 sec
----------------------------------------------------------
Test Failed.
"parallel_test_ckpt_F_start" end time: Oct 07 10:50 CEST
"parallel_test_ckpt_F_start" time elapsed: 00:00:00
----------------------------------------------------------

8/21 Testing: parallel_test_config_start
8/21 Test: parallel_test_config_start
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_config" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_config_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_config  restart=start
Running ./test_config  restart=start
Running ./test_config  restart=start
Running ./test_config  restart=start
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'STORE'. Expected '/dev/shm' but got '/dev/shm/foo'
Failed to get 'STORE'. Expected '/dev/shm' but got '/dev/shm/foo'
Failed to get 'STORE'. Expected '/dev/shm' but got '/dev/shm/foo'
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[localhost:00049] PMIX ERROR: UNREACHABLE in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2193
[localhost:00049] PMIX ERROR: UNREACHABLE in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2193
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00049] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00049] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.95 sec
----------------------------------------------------------
Test Failed.
"parallel_test_config_start" end time: Oct 07 10:50 CEST
"parallel_test_config_start" time elapsed: 00:00:00
----------------------------------------------------------

16/21 Testing: parallel_test_ckpt_start
16/21 Test: parallel_test_ckpt_start
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_ckpt" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_ckpt_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt  restart=start
Running ./test_ckpt  restart=start
Running ./test_ckpt  restart=start
Running ./test_ckpt  restart=start
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
[localhost:00053] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00053] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.93 sec
----------------------------------------------------------
Test Failed.
"parallel_test_ckpt_start" end time: Oct 07 10:50 CEST
"parallel_test_ckpt_start" time elapsed: 00:00:00
----------------------------------------------------------

5/21 Testing: parallel_test_api_restart
5/21 Test: parallel_test_api_restart
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_api" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_api_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api  restart=restart
Running ./test_api  restart=restart
Running ./test_api  restart=restart
Running ./test_api  restart=restart
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
Restarting
Restarting
Restarting
[localhost:00267] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00267] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00265] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00265] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00266] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00266] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00268] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00268] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00112] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00112] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.47 sec
----------------------------------------------------------
Test Failed.
"parallel_test_api_restart" end time: Oct 07 10:50 CEST
"parallel_test_api_restart" time elapsed: 00:00:00
----------------------------------------------------------

17/21 Testing: parallel_test_ckpt_restart
17/21 Test: parallel_test_ckpt_restart
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_ckpt" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_ckpt_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt  restart=restart
Running ./test_ckpt  restart=restart
Running ./test_ckpt  restart=restart
Running ./test_ckpt  restart=restart
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[localhost:00360] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00360] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00362] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00362] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00361] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00361] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[localhost:00359] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00359] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
Restarting
Restarting
Restarting
Restarting
[localhost:00245] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00245] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.44 sec
----------------------------------------------------------
Test Failed.
"parallel_test_ckpt_restart" end time: Oct 07 10:50 CEST
"parallel_test_ckpt_restart" time elapsed: 00:00:00
----------------------------------------------------------

13/21 Testing: parallel_test_api_multiple_restart
13/21 Test: parallel_test_api_multiple_restart
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_api_multiple" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_api_multiple_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api_multiple  restart=restart
Running ./test_api_multiple  restart=restart
Running ./test_api_multiple  restart=restart
Running ./test_api_multiple  restart=restart
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
Restarting
Restarting
Restarting
[localhost:00365] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00365] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
[localhost:00364] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00364] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[localhost:00366] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00366] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00363] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00363] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00212] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00212] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.49 sec
----------------------------------------------------------
Test Failed.
"parallel_test_api_multiple_restart" end time: Oct 07 10:50 CEST
"parallel_test_api_multiple_restart" time elapsed: 00:00:00
----------------------------------------------------------

21/21 Testing: parallel_test_ckpt_F_restart
21/21 Test: parallel_test_ckpt_F_restart
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_ckpt_F" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_ckpt_F_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt_F  restart=restart
Running ./test_ckpt_F  restart=restart
Running ./test_ckpt_F  restart=restart
Running ./test_ckpt_F  restart=restart
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[localhost:00370] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00370] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00371] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00371] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00372] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00372] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[localhost:00373] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00373] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
Restarting
Restarting
Restarting
Restarting
[localhost:00225] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00225] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.47 sec
----------------------------------------------------------
Test Failed.
"parallel_test_ckpt_F_restart" end time: Oct 07 10:50 CEST
"parallel_test_ckpt_F_restart" time elapsed: 00:00:00
----------------------------------------------------------

9/21 Testing: parallel_test_config_restart
9/21 Test: parallel_test_config_restart
Command: "/usr/bin/mpirun" "-np" "4" "run_test.sh" "./test_config" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"parallel_test_config_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_config  restart=restart
Running ./test_config  restart=restart
Running ./test_config  restart=restart
Running ./test_config  restart=restart
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'DEBUG'. Expected '1' but got '(null)'
Failed to get 'STORE'. Expected '/dev/shm' but got '/dev/shm/foo'
Failed to get 'STORE'. Expected '/dev/shm' but got '/dev/shm/foo'
Failed to get 'STORE'. Expected '/dev/shm' but got '/dev/shm/foo'
SCR v1.2.0 ABORT: rank 1 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 2 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
SCR v1.2.0 ABORT: rank 3 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
Restarting
Restarting
Restarting
[localhost:00430] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00430] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
[localhost:00431] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00431] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
[localhost:00429] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00429] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
[localhost:00432] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00432] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
[localhost:00244] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:00244] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
<end of output>
Test time =   0.50 sec
----------------------------------------------------------
Test Failed.
"parallel_test_config_restart" end time: Oct 07 10:50 CEST
"parallel_test_config_restart" time elapsed: 00:00:00
----------------------------------------------------------

6/21 Testing: serial_test_config_start
6/21 Test: serial_test_config_start
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_config" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_config_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_config  restart=start
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.73 sec
----------------------------------------------------------
Test Failed.
"serial_test_config_start" end time: Oct 07 10:50 CEST
"serial_test_config_start" time elapsed: 00:00:01
----------------------------------------------------------

2/21 Testing: serial_test_api_start
2/21 Test: serial_test_api_start
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_api" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_api_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api  restart=start
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.77 sec
----------------------------------------------------------
Test Failed.
"serial_test_api_start" end time: Oct 07 10:50 CEST
"serial_test_api_start" time elapsed: 00:00:01
----------------------------------------------------------

10/21 Testing: serial_test_api_multiple_start
10/21 Test: serial_test_api_multiple_start
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_api_multiple" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_api_multiple_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api_multiple  restart=start
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.91 sec
----------------------------------------------------------
Test Failed.
"serial_test_api_multiple_start" end time: Oct 07 10:50 CEST
"serial_test_api_multiple_start" time elapsed: 00:00:01
----------------------------------------------------------

14/21 Testing: serial_test_ckpt_start
14/21 Test: serial_test_ckpt_start
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_ckpt" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_ckpt_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt  restart=start
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.94 sec
----------------------------------------------------------
Test Failed.
"serial_test_ckpt_start" end time: Oct 07 10:50 CEST
"serial_test_ckpt_start" time elapsed: 00:00:01
----------------------------------------------------------

18/21 Testing: serial_test_ckpt_F_start
18/21 Test: serial_test_ckpt_F_start
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_ckpt_F" "start"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_ckpt_F_start" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt_F  restart=start
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.94 sec
----------------------------------------------------------
Test Failed.
"serial_test_ckpt_F_start" end time: Oct 07 10:50 CEST
"serial_test_ckpt_F_start" time elapsed: 00:00:01
----------------------------------------------------------

7/21 Testing: serial_test_config_restart
7/21 Test: serial_test_config_restart
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_config" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_config_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_config  restart=restart
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
[localhost:00471] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00471] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.44 sec
----------------------------------------------------------
Test Failed.
"serial_test_config_restart" end time: Oct 07 10:50 CEST
"serial_test_config_restart" time elapsed: 00:00:01
----------------------------------------------------------

3/21 Testing: serial_test_api_restart
3/21 Test: serial_test_api_restart
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_api" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_api_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api  restart=restart
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
[localhost:00470] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00470] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.41 sec
----------------------------------------------------------
Test Failed.
"serial_test_api_restart" end time: Oct 07 10:50 CEST
"serial_test_api_restart" time elapsed: 00:00:01
----------------------------------------------------------

11/21 Testing: serial_test_api_multiple_restart
11/21 Test: serial_test_api_multiple_restart
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_api_multiple" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_api_multiple_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_api_multiple  restart=restart
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[localhost:00503] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00503] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
Restarting
<end of output>
Test time =   1.55 sec
----------------------------------------------------------
Test Failed.
"serial_test_api_multiple_restart" end time: Oct 07 10:50 CEST
"serial_test_api_multiple_restart" time elapsed: 00:00:01
----------------------------------------------------------

19/21 Testing: serial_test_ckpt_F_restart
19/21 Test: serial_test_ckpt_F_restart
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_ckpt_F" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_ckpt_F_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt_F  restart=restart
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
[localhost:00505] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00505] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.51 sec
----------------------------------------------------------
Test Failed.
"serial_test_ckpt_F_restart" end time: Oct 07 10:50 CEST
"serial_test_ckpt_F_restart" time elapsed: 00:00:01
----------------------------------------------------------

15/21 Testing: serial_test_ckpt_restart
15/21 Test: serial_test_ckpt_restart
Command: "/usr/bin/mpirun" "-np" "1" "run_test.sh" "./test_ckpt" "restart"
Directory: /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0_pre1_build/examples
"serial_test_ckpt_restart" start time: Oct 07 10:50 CEST
Output:
----------------------------------------------------------
Running ./test_ckpt  restart=restart
SCR v1.2.0 ABORT: rank 0 on localhost: Failed to record username @ /var/tmp/portage/sys-cluster/scr-3.0_pre1/work/scr-3.0rc1/src/scr.c:670
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Restarting
[localhost:00504] OPAL ERROR: Unreachable in file /var/tmp/portage/sys-cluster/openmpi-4.0.5-r1/work/openmpi-4.0.5/opal/mca/pmix/pmix3x/pmix3x_client.c at line 112
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost:00504] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
+ '[' 0 -gt 1 ']'
+ rm -rf '/dev/shm//scr.*/'
+ rm -rf .scr/
+ '[' 0 -eq 1 ']'
+ exit 0
<end of output>
Test time =   1.52 sec
----------------------------------------------------------
Test Failed.
"serial_test_ckpt_restart" end time: Oct 07 10:50 CEST
"serial_test_ckpt_restart" time elapsed: 00:00:01
----------------------------------------------------------

End testing: Oct 07 10:50 CEST