Project

General

Profile

Bug #61574

Updated by Rishabh Dave 8 months ago

<pre> 
 "[Makefile:59: mdtest] Error 1" (undefined reference to `fi_strerror') 
 </pre> 

 Seen in two instances: 
 https://pulpito.ceph.com/yuriw-2023-05-28_14:46:14-fs-reef-release-distro-default-smithi/7288920 
 https://pulpito.ceph.com/yuriw-2023-05-28_14:46:14-fs-reef-release-distro-default-smithi/7289015 

 <pre> 
 2023-05-28T19:19:13.308 INFO:teuthology.orchestra.run.smithi139.stdout:mpicc -DLinux -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D__USE_LARGEFILE64=1    -g -o mdtest mdtest.c -lm 
 2023-05-28T19:19:13.309 INFO:teuthology.orchestra.run.smithi151.stderr:HTTP request sent, awaiting response... 200 OK 
 2023-05-28T19:19:13.309 INFO:teuthology.orchestra.run.smithi151.stderr:Length: 36252 (35K) [application/octet-stream] 
 2023-05-28T19:19:13.310 INFO:teuthology.orchestra.run.smithi151.stderr:Saving to: 'mdtest-1.9.3.tgz' 
 2023-05-28T19:19:13.310 INFO:teuthology.orchestra.run.smithi151.stderr: 
 2023-05-28T19:19:13.323 INFO:teuthology.orchestra.run.smithi151.stderr:       0K .......... .......... .......... .....                  100% 1.68M=0.02s 
 2023-05-28T19:19:13.323 INFO:teuthology.orchestra.run.smithi151.stderr: 
 2023-05-28T19:19:13.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-05-28 19:19:13 (1.68 MB/s) - 'mdtest-1.9.3.tgz' saved [36252/36252] 
 2023-05-28T19:19:13.324 INFO:teuthology.orchestra.run.smithi151.stderr: 
 2023-05-28T19:19:13.330 INFO:teuthology.orchestra.run.smithi151.stdout:COPYRIGHT 
 2023-05-28T19:19:13.330 INFO:teuthology.orchestra.run.smithi151.stdout:Makefile 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:README 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:RELEASE_LOG 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:mdtest.1 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:mdtest.c 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/ 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/env_to_db.tcsh 
 2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/mdtest_wrapper.py 
 2023-05-28T19:19:13.332 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/paramCatch.py 
 2023-05-28T19:19:13.332 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/tester.py 
 2023-05-28T19:19:13.332 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/WRAPPER_README 
 2023-05-28T19:19:13.335 INFO:teuthology.orchestra.run.smithi151.stdout:mpicc -DLinux -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D__USE_LARGEFILE64=1    -g -o mdtest mdtest.c -lm 
 2023-05-28T19:19:13.854 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c: In function 'main': 
 2023-05-28T19:19:13.862 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:2492:54: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=] 
 2023-05-28T19:19:13.862 INFO:teuthology.orchestra.run.smithi139.stderr: 2492 |         fprintf( stdout, "read_bytes                : %llu\n", read_bytes ); 
 2023-05-28T19:19:13.863 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                     ~~~^       ~~~~~~~~~~ 
 2023-05-28T19:19:13.863 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                        |       | 
 2023-05-28T19:19:13.863 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                        |       size_t {aka long unsigned int} 
 2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                        long long unsigned int 
 2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                     %lu 
 2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:2510:54: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=] 
 2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr: 2510 |         fprintf( stdout, "write_bytes               : %llu\n", write_bytes ); 
 2023-05-28T19:19:13.865 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                     ~~~^       ~~~~~~~~~~~ 
 2023-05-28T19:19:13.865 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                        |       | 
 2023-05-28T19:19:13.865 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                        |       size_t {aka long unsigned int} 
 2023-05-28T19:19:13.866 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                        long long unsigned int 
 2023-05-28T19:19:13.866 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                                     %lu 
 2023-05-28T19:19:13.896 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c: In function 'create_remove_items_helper': 
 2023-05-28T19:19:13.896 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:670:37: warning: '%s' directive writing up to 4095 bytes into a region of size 4090 [-Wformat-overflow=] 
 2023-05-28T19:19:13.896 INFO:teuthology.orchestra.run.smithi139.stderr:    670 |           sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i); 
 2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                       ^~               ~~~~~~~ 
 2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:670:9: note: 'sprintf' output 8 or more bytes (assuming 4103) into a destination of size 4096 
 2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:    670 |           sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i); 
 2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:        |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:414:37: warning: '%s' directive writing up to 4095 bytes into a region of size 4090 [-Wformat-overflow=] 
 2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:    414 |           sprintf(curr_item, "%s/file.%s%llu", path, mk_name, itemNum+i); 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                       ^~               ~~~~~~~ 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:414:9: note: 'sprintf' output 8 or more bytes (assuming 4103) into a destination of size 4096 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:    414 |           sprintf(curr_item, "%s/file.%s%llu", path, mk_name, itemNum+i); 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:        |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:376:36: warning: '%s' directive writing up to 4095 bytes into a region of size 4091 [-Wformat-overflow=] 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:    376 |           sprintf(curr_item, "%s/dir.%s%llu", path, rm_name, itemNum+i); 
 2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                      ^~               ~~~~~~~ 
 2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:376:9: note: 'sprintf' output 7 or more bytes (assuming 4102) into a destination of size 4096 
 2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:    376 |           sprintf(curr_item, "%s/dir.%s%llu", path, rm_name, itemNum+i); 
 2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:        |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:341:36: warning: '%s' directive writing up to 4095 bytes into a region of size 4091 [-Wformat-overflow=] 
 2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:    341 |           sprintf(curr_item, "%s/dir.%s%llu", path, mk_name, itemNum+i); 
 2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                      ^~               ~~~~~~~ 
 2023-05-28T19:19:13.900 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:341:9: note: 'sprintf' output 7 or more bytes (assuming 4102) into a destination of size 4096 
 2023-05-28T19:19:13.901 INFO:teuthology.orchestra.run.smithi139.stderr:    341 |           sprintf(curr_item, "%s/dir.%s%llu", path, mk_name, itemNum+i); 
 2023-05-28T19:19:13.901 INFO:teuthology.orchestra.run.smithi139.stderr:        |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:13.907 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c: In function 'collective_helper': 
 2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:820:45: warning: '%s' directive writing up to 4095 bytes into a region of size 4090 [-Wformat-overflow=] 
 2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:    820 |                   sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i); 
 2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:        |                                               ^~               ~~~~~~~ 
 2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:820:17: note: 'sprintf' output 8 or more bytes (assuming 4103) into a destination of size 4096 
 2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:    820 |                   sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i); 
 2023-05-28T19:19:13.909 INFO:teuthology.orchestra.run.smithi139.stderr:        |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 ... 
 ... 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2848:47: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=] 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: 2848 |                       sprintf(unique_mk_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:        |                                                 ^~~~~~~~~~~~~ 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2848:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: 2848 |                       sprintf(unique_mk_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:        |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: 2849 |                               (rank+(0*nstride))%i); 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:        |                               ~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2850:50: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=] 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: 2850 |                       sprintf(unique_chdir_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:        |                                                    ^~~~~~~~~~~~~ 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2850:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: 2850 |                       sprintf(unique_chdir_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:        |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: 2851 |                               (rank+(1*nstride))%i); 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:        |                               ~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2852:49: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=] 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: 2852 |                       sprintf(unique_stat_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:        |                                                   ^~~~~~~~~~~~~ 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2852:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: 2852 |                       sprintf(unique_stat_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:        |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: 2853 |                               (rank+(2*nstride))%i); 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr:        |                               ~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2854:49: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=] 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: 2854 |                       sprintf(unique_read_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr:        |                                                   ^~~~~~~~~~~~~ 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2854:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096 
 2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: 2854 |                       sprintf(unique_read_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:        |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: 2855 |                               (rank+(3*nstride))%i); 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:        |                               ~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2856:47: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=] 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: 2856 |                       sprintf(unique_rm_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:        |                                                 ^~~~~~~~~~~~~ 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2856:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096 
 2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: 2856 |                       sprintf(unique_rm_dir, "%s/mdtest_tree.%d.0", testdir, 
 2023-05-28T19:19:14.065 INFO:teuthology.orchestra.run.smithi151.stderr:        |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:14.065 INFO:teuthology.orchestra.run.smithi151.stderr: 2857 |                               (rank+(4*nstride))%i); 
 2023-05-28T19:19:14.065 INFO:teuthology.orchestra.run.smithi151.stderr:        |                               ~~~~~~~~~~~~~~~~~~~~~ 
 2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDI_OFI_do_irecv.constprop.0': 
 2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x1803): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1878): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDIG_do_irecv.constprop.0': 
 2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x33e3): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3509): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0': 
 2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x15ed): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o):(.text+0x1aad): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `get_first_non_io_obj_by_pci': 
 2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x191): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x226): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_init': 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x291): undefined reference to `hwloc_bitmap_alloc' 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2a4): undefined reference to `hwloc_topology_init' 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2b2): undefined reference to `hwloc_topology_set_io_types_filter' 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2be): undefined reference to `hwloc_topology_load' 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2ee): undefined reference to `hwloc_get_proc_cpubind' 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_finalize': 
 2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x320): undefined reference to `hwloc_topology_destroy' 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x32c): undefined reference to `hwloc_bitmap_free' 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_leaf': 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x389): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x394): undefined reference to `hwloc_bitmap_iszero' 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3ac): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3d0): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_depth': 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x469): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_ancestor': 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x4b2): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_type': 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x60f): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x655): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x664): undefined reference to `hwloc_bitmap_isequal' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6de): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_name.part.0': 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x777): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x782): undefined reference to `hwloc_bitmap_iszero' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x799): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x7c0): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x8d0): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x9e9): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xa46): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb8f): undefined reference to `hwloc_obj_type_is_normal' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xba8): undefined reference to `hwloc_obj_type_is_normal' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xbd9): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xbf6): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xc6b): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xc87): undefined reference to `hwloc_get_nbobjs_by_depth' 
 2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xcb0): undefined reference to `hwloc_bitmap_isset' 
 2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_mem_bind': 
 2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0xf7c): undefined reference to `hwloc_topology_get_support' 
 2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfbb): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfc3): undefined reference to `hwloc_bitmap_alloc' 
 2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfd8): undefined reference to `hwloc_bitmap_or' 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1016): undefined reference to `hwloc_bitmap_free' 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x104d): undefined reference to `hwloc_set_area_membind' 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1058): undefined reference to `hwloc_bitmap_free' 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_node_mem': 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x10ed): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1116): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_name': 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x116f): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1188): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x11a0): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_pci': 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x11f9): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1211): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0': 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x1dad): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2245): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_anysrc_try_cancel_partner': 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x28b5): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_eager.constprop.0': 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x33fd): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_pipeline.constprop.0': 
 2023-05-28T19:19:15.885 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x409b): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.885 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o):(.text+0x4be3): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.886 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_init_local': 
 2023-05-28T19:19:15.886 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x49b2): undefined reference to `fi_fabric' 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x5215): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_mpi_finalize_hook': 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x5ffb): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x66d0): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x67fd): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6875): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6a35): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6aad): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o):(.text+0x6b25): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `win_init': 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0xa58): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb68): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb80): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xcb6): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xe52): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfb8): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1095): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1106): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x120b): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x12f3): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_attach_hook': 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x1cdd): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_free_hook': 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x2595): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x260d): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2685): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_events.o):(.text+0x1bf1): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.891 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_nic.o): in function `MPIDI_OFI_init_multi_nic': 
 2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x44a): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x491): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x4fc): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `find_provider': 
 2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x94): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb8): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x278): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x296): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x30d): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x341): undefined reference to `fi_version' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x355): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x415): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x444): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x451): undefined reference to `fi_version' 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x461): undefined reference to `fi_version' 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x480): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x4c9): undefined reference to `fi_version' 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_find_provider_cleanup': 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x781): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_get_required_version': 
 2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x841): undefined reference to `fi_version' 
 2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_root_ctx': 
 2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x31c): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3b5): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x45d): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x555): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.896 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_all_ctx': 
 2023-05-28T19:19:15.896 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0xb35): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o):(.text+0xdf6): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDI_OFI_do_irecv.constprop.0': 
 2023-05-28T19:19:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x1803): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1878): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDIG_do_irecv.constprop.0': 
 2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x33e3): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3509): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0': 
 2023-05-28T19:19:15.979 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x15ed): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.979 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o):(.text+0x1aad): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.982 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `get_first_non_io_obj_by_pci': 
 2023-05-28T19:19:15.982 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x191): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x226): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_init': 
 2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x291): undefined reference to `hwloc_bitmap_alloc' 
 2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2a4): undefined reference to `hwloc_topology_init' 
 2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2b2): undefined reference to `hwloc_topology_set_io_types_filter' 
 2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2be): undefined reference to `hwloc_topology_load' 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2ee): undefined reference to `hwloc_get_proc_cpubind' 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_finalize': 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x320): undefined reference to `hwloc_topology_destroy' 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x32c): undefined reference to `hwloc_bitmap_free' 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_leaf': 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x389): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x394): undefined reference to `hwloc_bitmap_iszero' 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3ac): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3d0): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_depth': 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x469): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_ancestor': 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x4b2): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_type': 
 2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x60f): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x655): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x664): undefined reference to `hwloc_bitmap_isequal' 
 2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6de): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_name.part.0': 
 2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x777): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x782): undefined reference to `hwloc_bitmap_iszero' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x799): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x7c0): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x8d0): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x9e9): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xa46): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb8f): undefined reference to `hwloc_obj_type_is_normal' 
 2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xba8): undefined reference to `hwloc_obj_type_is_normal' 
 2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xbd9): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xbf6): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xc6b): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xc87): undefined reference to `hwloc_get_nbobjs_by_depth' 
 2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xcb0): undefined reference to `hwloc_bitmap_isset' 
 2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_mem_bind': 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0xf7c): undefined reference to `hwloc_topology_get_support' 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfbb): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfc3): undefined reference to `hwloc_bitmap_alloc' 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfd8): undefined reference to `hwloc_bitmap_or' 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1016): undefined reference to `hwloc_bitmap_free' 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x104d): undefined reference to `hwloc_set_area_membind' 
 2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1058): undefined reference to `hwloc_bitmap_free' 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_node_mem': 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x10ed): undefined reference to `hwloc_get_type_depth' 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1116): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_name': 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x116f): undefined reference to `hwloc_get_obj_by_depth' 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1188): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x11a0): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_pci': 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x11f9): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1211): undefined reference to `hwloc_bitmap_isincluded' 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0': 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x1dad): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2245): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_anysrc_try_cancel_partner': 
 2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x28b5): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_eager.constprop.0': 
 2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x33fd): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_pipeline.constprop.0': 
 2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x409b): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o):(.text+0x4be3): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_init_local': 
 2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x49b2): undefined reference to `fi_fabric' 
 2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x5215): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_mpi_finalize_hook': 
 2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x5ffb): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x66d0): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x67fd): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6875): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6a35): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6aad): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.995 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o):(.text+0x6b25): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:15.995 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `win_init': 
 2023-05-28T19:19:15.995 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0xa58): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb68): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb80): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xcb6): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xe52): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.997 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfb8): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.997 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1095): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.997 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1106): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x120b): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x12f3): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_attach_hook': 
 2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x1cdd): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_free_hook': 
 2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x2595): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x260d): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2685): undefined reference to `fi_strerror' 
 2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_events.o):(.text+0x1bf1): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 2 
 2023-05-28T19:19:16.003 INFO:teuthology.orchestra.run.smithi139.stderr:collect2: error: ld returned 1 exit status 
 2023-05-28T19:19:16.004 INFO:teuthology.orchestra.run.smithi139.stderr:make: *** [Makefile:59: mdtest] Error 1 
 2023-05-28T19:19:16.005 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_nic.o): in function `MPIDI_OFI_init_multi_nic': 
 2023-05-28T19:19:16.005 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x44a): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:16.005 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x491): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:16.006 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x4fc): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:16.006 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `find_provider': 
 2023-05-28T19:19:16.006 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x94): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb8): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x278): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x296): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x30d): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x341): undefined reference to `fi_version' 
 2023-05-28T19:19:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x355): undefined reference to `fi_strerror' 
 2023-05-28T19:19:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x415): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x444): undefined reference to `fi_getinfo' 
 2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x451): undefined reference to `fi_version' 
 2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x461): undefined reference to `fi_version' 
 2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x480): undefined reference to `fi_dupinfo' 
 2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x4c9): undefined reference to `fi_version' 
 2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_find_provider_cleanup': 
 2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x781): undefined reference to `fi_freeinfo' 
 2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_get_required_version': 
 2023-05-28T19:19:16.011 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x841): undefined reference to `fi_version' 
 2023-05-28T19:19:16.011 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_root_ctx': 
 2023-05-28T19:19:16.011 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x31c): undefined reference to `fi_strerror' 
 2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3b5): undefined reference to `fi_strerror' 
 2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x45d): undefined reference to `fi_strerror' 
 2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x555): undefined reference to `fi_strerror' 
 2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_all_ctx': 
 2023-05-28T19:19:16.013 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0xb35): undefined reference to `fi_strerror' 
 2023-05-28T19:19:16.013 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o):(.text+0xdf6): more undefined references to `fi_strerror' follow 
 2023-05-28T19:19:16.013 ERROR:teuthology.run_tasks:Saw exception from tasks. 
 Traceback (most recent call last): 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 105, in run_tasks 
     manager = run_one_task(taskname, ctx=ctx, config=config) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 84, in run_one_task 
     return task(**kwargs) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 149, in task 
     p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1]) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 84, in __exit__ 
     for result in self: 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 98, in __next__ 
     resurrect_traceback(result) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 30, in resurrect_traceback 
     raise exc.exc_info[1] 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 23, in capture_traceback 
     return func(*args, **kwargs) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 60, in _exec_host 
     tor.wait([r]) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 479, in wait 
     proc.wait() 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 161, in wait 
     self._raise_for_status() 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 181, in _raise_for_status 
     raise CommandFailedError( 
 teuthology.exceptions.CommandFailedError: Command failed on smithi139 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 
 2023-05-28T19:19:16.085 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1812c80acea040cc8c3d57b8cd2daa20 
 Traceback (most recent call last): 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 105, in run_tasks 
     manager = run_one_task(taskname, ctx=ctx, config=config) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 84, in run_one_task 
     return task(**kwargs) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 149, in task 
     p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1]) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 84, in __exit__ 
     for result in self: 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 98, in __next__ 
     resurrect_traceback(result) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 30, in resurrect_traceback 
     raise exc.exc_info[1] 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 23, in capture_traceback 
     return func(*args, **kwargs) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 60, in _exec_host 
     tor.wait([r]) 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 479, in wait 
     proc.wait() 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 161, in wait 
     self._raise_for_status() 
   File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 181, in _raise_for_status 
     raise CommandFailedError( 
 teuthology.exceptions.CommandFailedError: Command failed on smithi139 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 

 /<pre>

Back