Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2024-03-15T10:30:31Z
Ceph
Redmine
CephFS - Bug #64946 (New): qa: unable to locate package libcephfs1
https://tracker.ceph.com/issues/64946
2024-03-15T10:30:31Z
Rishabh Dave
<p><a class="external" href="https://pulpito.ceph.com/yuriw-2024-03-08_16:17:06-fs-wip-yuri10-testing-2024-03-07-1242-reef-distro-default-smithi/7586963">https://pulpito.ceph.com/yuriw-2024-03-08_16:17:06-fs-wip-yuri10-testing-2024-03-07-1242-reef-distro-default-smithi/7586963</a><br /><a class="external" href="https://pulpito.ceph.com/yuriw-2024-03-08_16:17:06-fs-wip-yuri10-testing-2024-03-07-1242-reef-distro-default-smithi/7586967">https://pulpito.ceph.com/yuriw-2024-03-08_16:17:06-fs-wip-yuri10-testing-2024-03-07-1242-reef-distro-default-smithi/7586967</a></p>
<pre>
2024-03-09T12:04:23.348 INFO:teuthology.orchestra.run.smithi174.stderr:E: Unable to locate package libcephfs1
</pre>
<pre>
2024-03-09T12:04:23.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.1-738-gdd302ccd-1focal ceph-mds=18.2.1-738-gdd302ccd-1focal ceph-common=18.2.1-738-gdd302ccd-1focal ceph-fuse=18.2.1-738-gdd302ccd-1focal ceph-test=18.2.1-738-gdd302ccd-1focal radosgw=18.2.1-738-gdd302ccd-1focal python-ceph=18.2.1-738-gdd302ccd-1focal libcephfs1=18.2.1-738-gdd302ccd-1focal libcephfs-java=18.2.1-738-gdd302ccd-1focal libcephfs-jni=18.2.1-738-gdd302ccd-1focal librados2=18.2.1-738-gdd302ccd-1focal librbd1=18.2.1-738-gdd302ccd-1focal rbd-fuse=18.2.1-738-gdd302ccd-1focal python3-cephfs=18.2.1-738-gdd302ccd-1focal cephfs-shell=18.2.1-738-gdd302ccd-1focal cephfs-top=18.2.1-738-gdd302ccd-1focal cephfs-mirror=18.2.1-738-gdd302ccd-1focal
2024-03-09T12:04:23.236 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists...
2024-03-09T12:04:23.317 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree...
2024-03-09T12:04:23.318 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information...
2024-03-09T12:04:23.347 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead.
2024-03-09T12:04:23.347 INFO:teuthology.orchestra.run.smithi174.stderr:E: Version '18.2.1-738-gdd302ccd-1focal' for 'python-ceph' was not found
2024-03-09T12:04:23.348 INFO:teuthology.orchestra.run.smithi174.stderr:E: Unable to locate package libcephfs1
2024-03-09T12:04:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 100
</pre>
CephFS - Bug #64856 (New): mds crashes when extracting from a tar is cancelled
https://tracker.ceph.com/issues/64856
2024-03-12T08:15:47Z
Rishabh Dave
<p>On fresh <code>vstart</code> cluster following commands were run -</p>
<pre>
./bin/ceph config set mon mon_allow_pool_delete true
./bin/ceph config set mds mds_cache_memory_limit 1G
./bin/ceph config set mds mds_health_cache_threshold 1.000001
./bin/ceph fs set a allow_standby_replay true
sleep 2
./bin/ceph status | grep "hot standby"
./bin/ceph-fuse cephfs1
tar -xv -f linux-6.7.9.tar.xz
</pre>
<p>This extraction takes some time. After 2-5 seconds into extracting, when it is cancelled through <code>ctrl-c</code>, the <code>tar</code> command hangs and prompt never returns. Hitting <code>ctrl-c</code> multiple times has no effect; <code>tar</code> commands remain stuck and prompt doesn't return. Hitting enter too few times doesn't help. As per <code>ps</code> output, tar process is still running. Killing <code>tar</code> command using <code>kill <tar-pid></code>, <code>sudo kill <tar-pid></code> and <code>sudo kill -9 <tar-pid></code> has no effect.</p>
<p>After unmount CephFS using <code>sudo umount -lf <cephfs-mntp></code>, the prompt returns, <code>ps</code> doesn't report <code>tar</code> anymore and MDS crashes instantaneously. In some cases, <code>tar</code> doesn't hang but MDS definitely crashes everytime.</p>
<p>Reproducing this issue was successfully also when MDS cache size was set to 50M, 100M, 500M, 1G, 2G and 4G.</p>
<p>Reproduciblity: 10/10 times, but minutely lesser with 2G, even lesser with 4G. I've reproduced this around 70 times with different cache sizes on the main branch as well as on feature branch on which this issue was discovered.</p>
CephFS - Bug #64538 (Fix Under Review): cephfs-shell: hangs and then aborts
https://tracker.ceph.com/issues/64538
2024-02-22T12:31:25Z
Rishabh Dave
<p>When cephfs-shell is launched, it prints a deprecation warning, hangs for it and then aborts -</p>
<pre>
(venv) neerajpratapsingh:build$ source vstart_environment.sh && source venv/bin/activate && python3 ../src/tools/cephfs/shell/cephfs-shell
/home/neerajpratapsingh/ceph/build/../src/tools/cephfs/shell/cephfs-shell:19: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import packaging # type: ignore
</pre>
<p>Neeraj hit this issue. I couldn't reproduce it but it most likely occurred on his machine due to relatively newer version of Python and pip.</p>
CephFS - Bug #64127 (Fix Under Review): mds: passing multiple caps to "fs authorize" cmd causes M...
https://tracker.ceph.com/issues/64127
2024-01-23T13:53:40Z
Rishabh Dave
<p>The command <code>ceph fs authorize b client.air / rw root_squash /volumes rw</code> fails, when it is run for a client that already contains caps for another FS.</p>
<p>Adding caps for a FS to a client is fine. Adding caps for 2nd caps is also fine -</p>
<pre>
$ ./bin/ceph fs volume create b
$ ./bin/ceph fs authorize a client.x / rw
$ ./bin/ceph fs authorize b client.x / rw
[client.x]
key = AQCDj69lh8JqNRAAkaZfZ64v27d5NZUVB7F8Pg==
caps mds = "allow rw fsname=a, allow rw fsname=b"
caps mon = "allow r fsname=a, allow r fsname=b"
caps osd = "allow rw tag cephfs data=a, allow rw tag cephfs data=b"
updated caps for client.x
</pre>
<p>But when following command is run, command hangs -<br /><pre>
$ ./bin/ceph fs authorize b client.x / rw root_squash /volumes rw
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2024-01-23T14:45:25.827+0530 7ffa7937d6c0 -1 WARNING: all dangerous and experimental features are enabled.
2024-01-23T14:45:25.836+0530 7ffa7937d6c0 -1 WARNING: all dangerous and experimental features are enabled.
</pre></p>
<p>After this <code>ceph status</code> also hangs because MON daemons have crashed.</p>
<p>Traceback from MON logs -<br /><pre>
ceph version 19.0.0-772-gd7e2a322a54 (d7e2a322a5490d1fc9395471e25c3a98499975ea) squid (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x125) [0x7f2392c79a3e]
2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2392c79c60]
3: (MDSAuthCaps::merge(MDSAuthCaps)+0x78) [0x560a9b8b057a]
4: (AuthMonitor::caps_update AuthMonitor::_merge_caps<MDSAuthCaps>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream&)+0x106) [0x560a9b86abba]
5: (AuthMonitor::_gen_wanted_caps(EntityAuth&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream&)+0xde) [0x560a9b857344]
6: (AuthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x238e) [0x560a9b861c96]
7: (AuthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x8e) [0x560a9b8630ba]
8: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x853) [0x560a9b9184df]
9: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x20df) [0x560a9b7e8b9b]
10: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x85d) [0x560a9b7ed203]
11: (Monitor::_ms_dispatch(Message*)+0xada) [0x560a9b7eeeea]
12: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x5d1) [0x560a9b7ef773]
13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1045) [0x560a9b7ed9eb]
14: (Monitor::_ms_dispatch(Message*)+0xada) [0x560a9b7eeeea]
15: (Monitor::ms_dispatch(Message*)+0x32) [0x560a9b826932]
16: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x2a) [0x560a9b7f3386]
17: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xa4) [0x7f2392dac4b4]
18: (DispatchQueue::entry()+0x43d) [0x7f2392da8893]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2392e5061b]
20: (Thread::entry_wrapper()+0x3f) [0x7f2392c5489f]
21: (Thread::_entry_func(void*)+0x9) [0x7f2392c548b7]
22: /lib64/libc.so.6(+0x8c947) [0x7f23914ae947]
23: /lib64/libc.so.6(+0x112860) [0x7f2391534860]
</pre></p>
<p>Following are few log entries before traceback -<br /><pre>
-5> 2024-01-23T15:36:45.715+0530 7f2389bf46c0 10 mon.a@0(leader).paxosservice(auth 1..13) dispatch 0x560aa40a5200 mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885 con 0x560aa404e1e0
-4> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 5 mon.a@0(leader).paxos(paxos active c 1..229) is_readable = 1 - now=2024-01-23T15:36:45.717565+0530 lease_expire=2024-01-23T15:36:50.530042+0530 has v0 lc 229
-3> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 10 mon.a@0(leader).auth v13 preprocess_query mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885
-2> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 10 mon.a@0(leader).auth v13 prepare_update mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885
-1> 2024-01-23T15:36:45.725+0530 7f2389bf46c0 -1 /home/rishabh/repos/ceph/test2/src/mds/MDSAuthCaps.cc: In function 'bool MDSAuthCaps::merge(MDSAuthCaps)' thread 7f2389bf46c0 time 2024-01-23T15:36:45.717766+0530
/home/rishabh/repos/ceph/test2/src/mds/MDSAuthCaps.cc: 387: FAILED ceph_assert(newcap.grants.size() == 1)
</pre></p>
CephFS - Bug #63764 (In Progress): Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cep...
https://tracker.ceph.com/issues/63764
2023-12-08T04:30:54Z
Venky Shankar
vshankar@redhat.com
<p>/a/vshankar-2023-12-06_15:12:46-fs-wip-vshankar-testing-20231206.125818-testing-default-smithi/7480346</p>
<pre>
2023-12-06T22:49:02.292 INFO:tasks.cephfs.caps_helper:test read perm: read file /home/ubuntu/cephtest/mnt.0/testdir/testfile and expect data "self.path = /home/ubuntu/cephtest/mnt.0/testdir/testfile
cephfs_name = cephfs
cephfs_mntpt = /
hostfs_mntpt = /home/ubuntu/cephtest/mnt.0"
2023-12-06T22:49:02.293 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-12-06T22:49:02.293 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.0 && exec cat /home/ubuntu/cephtest/mnt.0/testdir/testfile)
2023-12-06T22:49:02.357 INFO:teuthology.orchestra.run.smithi167.stderr:cat: /home/ubuntu/cephtest/mnt.0/testdir/testfile: Permission denied
2023-12-06T22:49:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1
</pre>
CephFS - Bug #63514 (Fix Under Review): mds: avoid sending inode/stray counters as part of health...
https://tracker.ceph.com/issues/63514
2023-11-13T15:09:04Z
Venky Shankar
vshankar@redhat.com
<p>Otherwise a health warning from a s-r daemon looks like</p>
<pre>
sh-4.4$ ceph health detail
HEALTH_WARN 1 MDSs report oversized cache
[WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache
cephfs(mds.0): MDS cache is too large (6GB/4GB); 0 inodes in use by clients, 0 stray files
sh-4.4$
</pre>
<p>Which is a wee <em>bit</em> confusing - so, avoid sending those counters as part of the warning.</p>
CephFS - Bug #63233 (New): mon|client|mds: valgrind reports possible leaks in the MDS
https://tracker.ceph.com/issues/63233
2023-10-18T12:24:10Z
Venky Shankar
vshankar@redhat.com
<p>/a/vshankar-2023-10-14_01:51:22-fs-wip-vshankar-testing-20231013.093215-testing-default-smithi/7427332</p>
<pre>
2023-10-16T04:24:05.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq
2023-10-16T04:24:05.842 DEBUG:teuthology.orchestra.run.smithi062:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq
2023-10-16T04:24:05.886 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.b.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.886 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.d.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.f.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mon.b.log: <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mon.c.log: <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mds.a.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mds.c.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mon.a.log: <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.979 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.a.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.c.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.980 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.b.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.d.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.f.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.981 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.982 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
</pre>
CephFS - Bug #62126 (New): test failure: suites/blogbench.sh stops running
https://tracker.ceph.com/issues/62126
2023-07-24T12:39:03Z
Rishabh Dave
<p>I found this failure during running integration tests for few CephFS PRs. This failure occurred even after running the buggy PR from the batch of PRs that were under testing. And this PR even occurred against main branch (IOW, no PRs were on the testing branch this time).<br /><a class="external" href="https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/7347715">https://pulpito.ceph.com/rishabh-2023-07-21_21:30:53-fs-wip-rishabh-2023Jul13-base-2-testing-default-smithi/7347715</a></p>
<pre>
2023-07-21T22:18:55.848 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0...
</pre>
<pre>
2023-07-21T22:15:30.678 INFO:tasks.workunit.client.0.smithi073.stdout:
2023-07-21T22:15:30.678 INFO:tasks.workunit.client.0.smithi073.stdout:Final score for writes: 425
2023-07-21T22:15:30.678 INFO:tasks.workunit.client.0.smithi073.stdout:Final score for reads : 1132026
2023-07-21T22:15:30.679 INFO:tasks.workunit.client.0.smithi073.stdout:
</pre>
CephFS - Bug #61791 (Triaged): snaptest-git-ceph.sh test timed out (job dead)
https://tracker.ceph.com/issues/61791
2023-06-26T06:32:17Z
Venky Shankar
vshankar@redhat.com
<p><a class="external" href="https://pulpito.ceph.com/vshankar-2023-06-20_10:07:44-fs-wip-vshankar-testing-20230620.052303-testing-default-smithi/7308843/">https://pulpito.ceph.com/vshankar-2023-06-20_10:07:44-fs-wip-vshankar-testing-20230620.052303-testing-default-smithi/7308843/</a></p>
<pre>
2023-06-20T13:27:06.641 INFO:tasks.workunit.client.0.smithi017.stderr:+ mkdir .snap/v0.4
2023-06-20T13:27:06.673 INFO:tasks.workunit.client.0.smithi017.stderr:+ for v in $versions
2023-06-20T13:27:06.673 INFO:tasks.workunit.client.0.smithi017.stderr:+ '[' 5 -eq 48 ']'
2023-06-20T13:27:06.673 INFO:tasks.workunit.client.0.smithi017.stderr:+ ver=v0.5
2023-06-20T13:27:06.674 INFO:tasks.workunit.client.0.smithi017.stderr:+ echo v0.5
2023-06-20T13:27:06.674 INFO:tasks.workunit.client.0.smithi017.stdout:v0.5
2023-06-20T13:27:06.674 INFO:tasks.workunit.client.0.smithi017.stderr:+ git reset --hard v0.5
</pre>
<p>And after ~6hrs</p>
<pre>
2023-06-20T19:07:12.096 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: <Greenlet at 0x7f2965f059d0: copy_file_to(<paramiko.ChannelFile from <paramiko.Channel 489 (, <Logger tasks.workunit.client.0.smithi017.stderr (, None, False)>
/pre>
The client log, however, does not have anything after: 2023-06-20T13:27:47.055+0000
./remote/smithi017/log/ceph-client.0.140320.log.gz
<pre>
2023-06-20T13:27:47.055+0000 7fd6a5ffb700 10 client.4995 caps on 0x1000000458b.head pAsLsXsFsc wants -
2023-06-20T13:27:47.055+0000 7fd6a5ffb700 10 client.4995 path #0x1000000458b
2023-06-20T13:27:47.055+0000 7fd6a5ffb700 10 client.4995 snaprealm snaprealm(0x10000000e63 nref=445 c=1fd seq=1fd parent=0x1 my_snaps=[1fd,1fc,1fb,1fa] cached_snapc=1fd=[1fd,1fc,1fb,1fa] last_modified=2023-06-20T13:27:06.646939+0000 change_attr=4)
2023-06-20T13:27:47.055+0000 7fd6a5ffb700 10 client.4995 caps on 0x1000000458a.head pAsLsXsFsc wants -
2023-06-20T13:27:47.055+0000 7fd6a5ffb700 10 client.4995 path #0x1000000458a
</pre></pre>
CephFS - Bug #61574 (Pending Backport): qa: build failure for mdtest project
https://tracker.ceph.com/issues/61574
2023-06-02T09:20:41Z
Kotresh Hiremath Ravishankar
<pre>
"[Makefile:59: mdtest] Error 1" (undefined reference to `fi_strerror')
</pre>
<p>Seen in two instances:<br /><a class="external" href="https://pulpito.ceph.com/yuriw-2023-05-28_14:46:14-fs-reef-release-distro-default-smithi/7288920">https://pulpito.ceph.com/yuriw-2023-05-28_14:46:14-fs-reef-release-distro-default-smithi/7288920</a><br /><a class="external" href="https://pulpito.ceph.com/yuriw-2023-05-28_14:46:14-fs-reef-release-distro-default-smithi/7289015">https://pulpito.ceph.com/yuriw-2023-05-28_14:46:14-fs-reef-release-distro-default-smithi/7289015</a></p>
<pre>
2023-05-28T19:19:13.308 INFO:teuthology.orchestra.run.smithi139.stdout:mpicc -DLinux -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D__USE_LARGEFILE64=1 -g -o mdtest mdtest.c -lm
2023-05-28T19:19:13.309 INFO:teuthology.orchestra.run.smithi151.stderr:HTTP request sent, awaiting response... 200 OK
2023-05-28T19:19:13.309 INFO:teuthology.orchestra.run.smithi151.stderr:Length: 36252 (35K) [application/octet-stream]
2023-05-28T19:19:13.310 INFO:teuthology.orchestra.run.smithi151.stderr:Saving to: 'mdtest-1.9.3.tgz'
2023-05-28T19:19:13.310 INFO:teuthology.orchestra.run.smithi151.stderr:
2023-05-28T19:19:13.323 INFO:teuthology.orchestra.run.smithi151.stderr: 0K .......... .......... .......... ..... 100% 1.68M=0.02s
2023-05-28T19:19:13.323 INFO:teuthology.orchestra.run.smithi151.stderr:
2023-05-28T19:19:13.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-05-28 19:19:13 (1.68 MB/s) - 'mdtest-1.9.3.tgz' saved [36252/36252]
2023-05-28T19:19:13.324 INFO:teuthology.orchestra.run.smithi151.stderr:
2023-05-28T19:19:13.330 INFO:teuthology.orchestra.run.smithi151.stdout:COPYRIGHT
2023-05-28T19:19:13.330 INFO:teuthology.orchestra.run.smithi151.stdout:Makefile
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:README
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:RELEASE_LOG
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:mdtest.1
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:mdtest.c
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/env_to_db.tcsh
2023-05-28T19:19:13.331 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/mdtest_wrapper.py
2023-05-28T19:19:13.332 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/paramCatch.py
2023-05-28T19:19:13.332 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/tester.py
2023-05-28T19:19:13.332 INFO:teuthology.orchestra.run.smithi151.stdout:scripts/WRAPPER_README
2023-05-28T19:19:13.335 INFO:teuthology.orchestra.run.smithi151.stdout:mpicc -DLinux -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D__USE_LARGEFILE64=1 -g -o mdtest mdtest.c -lm
2023-05-28T19:19:13.854 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c: In function 'main':
2023-05-28T19:19:13.862 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:2492:54: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=]
2023-05-28T19:19:13.862 INFO:teuthology.orchestra.run.smithi139.stderr: 2492 | fprintf( stdout, "read_bytes : %llu\n", read_bytes );
2023-05-28T19:19:13.863 INFO:teuthology.orchestra.run.smithi139.stderr: | ~~~^ ~~~~~~~~~~
2023-05-28T19:19:13.863 INFO:teuthology.orchestra.run.smithi139.stderr: | | |
2023-05-28T19:19:13.863 INFO:teuthology.orchestra.run.smithi139.stderr: | | size_t {aka long unsigned int}
2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr: | long long unsigned int
2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr: | %lu
2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:2510:54: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 3 has type 'size_t' {aka 'long unsigned int'} [-Wformat=]
2023-05-28T19:19:13.864 INFO:teuthology.orchestra.run.smithi139.stderr: 2510 | fprintf( stdout, "write_bytes : %llu\n", write_bytes );
2023-05-28T19:19:13.865 INFO:teuthology.orchestra.run.smithi139.stderr: | ~~~^ ~~~~~~~~~~~
2023-05-28T19:19:13.865 INFO:teuthology.orchestra.run.smithi139.stderr: | | |
2023-05-28T19:19:13.865 INFO:teuthology.orchestra.run.smithi139.stderr: | | size_t {aka long unsigned int}
2023-05-28T19:19:13.866 INFO:teuthology.orchestra.run.smithi139.stderr: | long long unsigned int
2023-05-28T19:19:13.866 INFO:teuthology.orchestra.run.smithi139.stderr: | %lu
2023-05-28T19:19:13.896 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c: In function 'create_remove_items_helper':
2023-05-28T19:19:13.896 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:670:37: warning: '%s' directive writing up to 4095 bytes into a region of size 4090 [-Wformat-overflow=]
2023-05-28T19:19:13.896 INFO:teuthology.orchestra.run.smithi139.stderr: 670 | sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i);
2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~ ~~~~~~~
2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:670:9: note: 'sprintf' output 8 or more bytes (assuming 4103) into a destination of size 4096
2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr: 670 | sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i);
2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:414:37: warning: '%s' directive writing up to 4095 bytes into a region of size 4090 [-Wformat-overflow=]
2023-05-28T19:19:13.897 INFO:teuthology.orchestra.run.smithi139.stderr: 414 | sprintf(curr_item, "%s/file.%s%llu", path, mk_name, itemNum+i);
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~ ~~~~~~~
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:414:9: note: 'sprintf' output 8 or more bytes (assuming 4103) into a destination of size 4096
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr: 414 | sprintf(curr_item, "%s/file.%s%llu", path, mk_name, itemNum+i);
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:376:36: warning: '%s' directive writing up to 4095 bytes into a region of size 4091 [-Wformat-overflow=]
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr: 376 | sprintf(curr_item, "%s/dir.%s%llu", path, rm_name, itemNum+i);
2023-05-28T19:19:13.898 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~ ~~~~~~~
2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:376:9: note: 'sprintf' output 7 or more bytes (assuming 4102) into a destination of size 4096
2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr: 376 | sprintf(curr_item, "%s/dir.%s%llu", path, rm_name, itemNum+i);
2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:341:36: warning: '%s' directive writing up to 4095 bytes into a region of size 4091 [-Wformat-overflow=]
2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr: 341 | sprintf(curr_item, "%s/dir.%s%llu", path, mk_name, itemNum+i);
2023-05-28T19:19:13.899 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~ ~~~~~~~
2023-05-28T19:19:13.900 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:341:9: note: 'sprintf' output 7 or more bytes (assuming 4102) into a destination of size 4096
2023-05-28T19:19:13.901 INFO:teuthology.orchestra.run.smithi139.stderr: 341 | sprintf(curr_item, "%s/dir.%s%llu", path, mk_name, itemNum+i);
2023-05-28T19:19:13.901 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:13.907 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c: In function 'collective_helper':
2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:820:45: warning: '%s' directive writing up to 4095 bytes into a region of size 4090 [-Wformat-overflow=]
2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr: 820 | sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i);
2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~ ~~~~~~~
2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr:mdtest.c:820:17: note: 'sprintf' output 8 or more bytes (assuming 4103) into a destination of size 4096
2023-05-28T19:19:13.908 INFO:teuthology.orchestra.run.smithi139.stderr: 820 | sprintf(curr_item, "%s/file.%s%llu", path, rm_name, itemNum+i);
2023-05-28T19:19:13.909 INFO:teuthology.orchestra.run.smithi139.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
...
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2848:47: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=]
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: 2848 | sprintf(unique_mk_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2848:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: 2848 | sprintf(unique_mk_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.060 INFO:teuthology.orchestra.run.smithi151.stderr: 2849 | (rank+(0*nstride))%i);
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: | ~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2850:50: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=]
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: 2850 | sprintf(unique_chdir_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2850:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: 2850 | sprintf(unique_chdir_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.061 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: 2851 | (rank+(1*nstride))%i);
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: | ~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2852:49: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=]
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: 2852 | sprintf(unique_stat_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2852:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: 2852 | sprintf(unique_stat_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.062 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: 2853 | (rank+(2*nstride))%i);
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: | ~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2854:49: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=]
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: 2854 | sprintf(unique_read_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2854:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096
2023-05-28T19:19:14.063 INFO:teuthology.orchestra.run.smithi151.stderr: 2854 | sprintf(unique_read_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: 2855 | (rank+(3*nstride))%i);
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: | ~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2856:47: warning: '/mdtest_tree.' directive writing 13 bytes into a region of size between 1 and 4096 [-Wformat-overflow=]
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: 2856 | sprintf(unique_rm_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:mdtest.c:2856:21: note: 'sprintf' output between 17 and 4122 bytes into a destination of size 4096
2023-05-28T19:19:14.064 INFO:teuthology.orchestra.run.smithi151.stderr: 2856 | sprintf(unique_rm_dir, "%s/mdtest_tree.%d.0", testdir,
2023-05-28T19:19:14.065 INFO:teuthology.orchestra.run.smithi151.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:14.065 INFO:teuthology.orchestra.run.smithi151.stderr: 2857 | (rank+(4*nstride))%i);
2023-05-28T19:19:14.065 INFO:teuthology.orchestra.run.smithi151.stderr: | ~~~~~~~~~~~~~~~~~~~~~
2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDI_OFI_do_irecv.constprop.0':
2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x1803): undefined reference to `fi_strerror'
2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1878): undefined reference to `fi_strerror'
2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDIG_do_irecv.constprop.0':
2023-05-28T19:19:15.869 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x33e3): undefined reference to `fi_strerror'
2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3509): undefined reference to `fi_strerror'
2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0':
2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x15ed): undefined reference to `fi_strerror'
2023-05-28T19:19:15.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o):(.text+0x1aad): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `get_first_non_io_obj_by_pci':
2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x191): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x226): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.875 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_init':
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x291): undefined reference to `hwloc_bitmap_alloc'
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2a4): undefined reference to `hwloc_topology_init'
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2b2): undefined reference to `hwloc_topology_set_io_types_filter'
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2be): undefined reference to `hwloc_topology_load'
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2ee): undefined reference to `hwloc_get_proc_cpubind'
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_finalize':
2023-05-28T19:19:15.876 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x320): undefined reference to `hwloc_topology_destroy'
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x32c): undefined reference to `hwloc_bitmap_free'
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_leaf':
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x389): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x394): undefined reference to `hwloc_bitmap_iszero'
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3ac): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3d0): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.877 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_depth':
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x469): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_ancestor':
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x4b2): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_type':
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x60f): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x655): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.878 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x664): undefined reference to `hwloc_bitmap_isequal'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6de): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_name.part.0':
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x777): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x782): undefined reference to `hwloc_bitmap_iszero'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x799): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x7c0): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x8d0): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.879 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x9e9): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xa46): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb8f): undefined reference to `hwloc_obj_type_is_normal'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xba8): undefined reference to `hwloc_obj_type_is_normal'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xbd9): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xbf6): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xc6b): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.880 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xc87): undefined reference to `hwloc_get_nbobjs_by_depth'
2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xcb0): undefined reference to `hwloc_bitmap_isset'
2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_mem_bind':
2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0xf7c): undefined reference to `hwloc_topology_get_support'
2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfbb): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfc3): undefined reference to `hwloc_bitmap_alloc'
2023-05-28T19:19:15.881 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfd8): undefined reference to `hwloc_bitmap_or'
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1016): undefined reference to `hwloc_bitmap_free'
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x104d): undefined reference to `hwloc_set_area_membind'
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1058): undefined reference to `hwloc_bitmap_free'
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_node_mem':
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x10ed): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1116): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.882 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_name':
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x116f): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1188): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x11a0): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_pci':
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x11f9): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1211): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.883 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0':
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x1dad): undefined reference to `fi_strerror'
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2245): undefined reference to `fi_strerror'
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_anysrc_try_cancel_partner':
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x28b5): undefined reference to `fi_strerror'
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_eager.constprop.0':
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x33fd): undefined reference to `fi_strerror'
2023-05-28T19:19:15.884 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_pipeline.constprop.0':
2023-05-28T19:19:15.885 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x409b): undefined reference to `fi_strerror'
2023-05-28T19:19:15.885 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o):(.text+0x4be3): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.886 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_init_local':
2023-05-28T19:19:15.886 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x49b2): undefined reference to `fi_fabric'
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x5215): undefined reference to `fi_strerror'
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_mpi_finalize_hook':
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x5ffb): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x66d0): undefined reference to `fi_strerror'
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x67fd): undefined reference to `fi_strerror'
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6875): undefined reference to `fi_strerror'
2023-05-28T19:19:15.887 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6a35): undefined reference to `fi_strerror'
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x6aad): undefined reference to `fi_strerror'
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o):(.text+0x6b25): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `win_init':
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0xa58): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb68): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb80): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.888 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xcb6): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xe52): undefined reference to `fi_strerror'
2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xfb8): undefined reference to `fi_strerror'
2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1095): undefined reference to `fi_strerror'
2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x1106): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x120b): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.889 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x12f3): undefined reference to `fi_strerror'
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_attach_hook':
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x1cdd): undefined reference to `fi_strerror'
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_free_hook':
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x2595): undefined reference to `fi_strerror'
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x260d): undefined reference to `fi_strerror'
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x2685): undefined reference to `fi_strerror'
2023-05-28T19:19:15.890 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_events.o):(.text+0x1bf1): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.891 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_nic.o): in function `MPIDI_OFI_init_multi_nic':
2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x44a): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x491): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x4fc): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `find_provider':
2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x94): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.892 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0xb8): undefined reference to `fi_getinfo'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x278): undefined reference to `fi_getinfo'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x296): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x30d): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x341): undefined reference to `fi_version'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x355): undefined reference to `fi_strerror'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x415): undefined reference to `fi_getinfo'
2023-05-28T19:19:15.893 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x444): undefined reference to `fi_getinfo'
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x451): undefined reference to `fi_version'
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x461): undefined reference to `fi_version'
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x480): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x4c9): undefined reference to `fi_version'
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_find_provider_cleanup':
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x781): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_get_required_version':
2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x841): undefined reference to `fi_version'
2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_root_ctx':
2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0x31c): undefined reference to `fi_strerror'
2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x3b5): undefined reference to `fi_strerror'
2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x45d): undefined reference to `fi_strerror'
2023-05-28T19:19:15.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: (.text+0x555): undefined reference to `fi_strerror'
2023-05-28T19:19:15.896 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_all_ctx':
2023-05-28T19:19:15.896 INFO:teuthology.orchestra.run.smithi139.stderr:(.text+0xb35): undefined reference to `fi_strerror'
2023-05-28T19:19:15.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o):(.text+0xdf6): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDI_OFI_do_irecv.constprop.0':
2023-05-28T19:19:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x1803): undefined reference to `fi_strerror'
2023-05-28T19:19:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1878): undefined reference to `fi_strerror'
2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-recv.o): in function `MPIDIG_do_irecv.constprop.0':
2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x33e3): undefined reference to `fi_strerror'
2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3509): undefined reference to `fi_strerror'
2023-05-28T19:19:15.978 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0':
2023-05-28T19:19:15.979 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x15ed): undefined reference to `fi_strerror'
2023-05-28T19:19:15.979 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-send.o):(.text+0x1aad): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.982 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `get_first_non_io_obj_by_pci':
2023-05-28T19:19:15.982 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x191): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x226): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_init':
2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x291): undefined reference to `hwloc_bitmap_alloc'
2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2a4): undefined reference to `hwloc_topology_init'
2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2b2): undefined reference to `hwloc_topology_set_io_types_filter'
2023-05-28T19:19:15.983 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2be): undefined reference to `hwloc_topology_load'
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2ee): undefined reference to `hwloc_get_proc_cpubind'
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPII_hwtopo_finalize':
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x320): undefined reference to `hwloc_topology_destroy'
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x32c): undefined reference to `hwloc_bitmap_free'
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_leaf':
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x389): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.984 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x394): undefined reference to `hwloc_bitmap_iszero'
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3ac): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3d0): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_depth':
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x469): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_ancestor':
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x4b2): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_type':
2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x60f): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x655): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x664): undefined reference to `hwloc_bitmap_isequal'
2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6de): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_obj_by_name.part.0':
2023-05-28T19:19:15.986 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x777): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x782): undefined reference to `hwloc_bitmap_iszero'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x799): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x7c0): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x8d0): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x9e9): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xa46): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb8f): undefined reference to `hwloc_obj_type_is_normal'
2023-05-28T19:19:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xba8): undefined reference to `hwloc_obj_type_is_normal'
2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xbd9): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xbf6): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xc6b): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xc87): undefined reference to `hwloc_get_nbobjs_by_depth'
2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xcb0): undefined reference to `hwloc_bitmap_isset'
2023-05-28T19:19:15.988 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_mem_bind':
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0xf7c): undefined reference to `hwloc_topology_get_support'
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfbb): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfc3): undefined reference to `hwloc_bitmap_alloc'
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfd8): undefined reference to `hwloc_bitmap_or'
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1016): undefined reference to `hwloc_bitmap_free'
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x104d): undefined reference to `hwloc_set_area_membind'
2023-05-28T19:19:15.989 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1058): undefined reference to `hwloc_bitmap_free'
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_get_node_mem':
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x10ed): undefined reference to `hwloc_get_type_depth'
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1116): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_name':
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x116f): undefined reference to `hwloc_get_obj_by_depth'
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1188): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x11a0): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpir_hwtopo.o): in function `MPIR_hwtopo_is_dev_close_by_pci':
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x11f9): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1211): undefined reference to `hwloc_bitmap_isincluded'
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_rdma_read.constprop.0':
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x1dad): undefined reference to `fi_strerror'
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2245): undefined reference to `fi_strerror'
2023-05-28T19:19:15.991 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_anysrc_try_cancel_partner':
2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x28b5): undefined reference to `fi_strerror'
2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_eager.constprop.0':
2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x33fd): undefined reference to `fi_strerror'
2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o): in function `MPIDI_OFI_do_am_isend_pipeline.constprop.0':
2023-05-28T19:19:15.992 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x409b): undefined reference to `fi_strerror'
2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-mpidig_pt2pt_callbacks.o):(.text+0x4be3): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_init_local':
2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x49b2): undefined reference to `fi_fabric'
2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x5215): undefined reference to `fi_strerror'
2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o): in function `MPIDI_OFI_mpi_finalize_hook':
2023-05-28T19:19:15.993 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x5ffb): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x66d0): undefined reference to `fi_strerror'
2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x67fd): undefined reference to `fi_strerror'
2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6875): undefined reference to `fi_strerror'
2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6a35): undefined reference to `fi_strerror'
2023-05-28T19:19:15.994 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x6aad): undefined reference to `fi_strerror'
2023-05-28T19:19:15.995 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_init.o):(.text+0x6b25): more undefined references to `fi_strerror' follow
2023-05-28T19:19:15.995 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `win_init':
2023-05-28T19:19:15.995 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0xa58): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb68): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb80): undefined reference to `fi_dupinfo'
2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xcb6): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xe52): undefined reference to `fi_strerror'
2023-05-28T19:19:15.997 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xfb8): undefined reference to `fi_strerror'
2023-05-28T19:19:15.997 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1095): undefined reference to `fi_strerror'
2023-05-28T19:19:15.997 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x1106): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x120b): undefined reference to `fi_freeinfo'
2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x12f3): undefined reference to `fi_strerror'
2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_attach_hook':
2023-05-28T19:19:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x1cdd): undefined reference to `fi_strerror'
2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_win.o): in function `MPIDI_OFI_mpi_win_free_hook':
2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x2595): undefined reference to `fi_strerror'
2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x260d): undefined reference to `fi_strerror'
2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x2685): undefined reference to `fi_strerror'
2023-05-28T19:19:15.999 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_events.o):(.text+0x1bf1): more undefined references to `fi_strerror' follow
2023-05-28T19:19:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 2
2023-05-28T19:19:16.003 INFO:teuthology.orchestra.run.smithi139.stderr:collect2: error: ld returned 1 exit status
2023-05-28T19:19:16.004 INFO:teuthology.orchestra.run.smithi139.stderr:make: *** [Makefile:59: mdtest] Error 1
2023-05-28T19:19:16.005 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-ofi_nic.o): in function `MPIDI_OFI_init_multi_nic':
2023-05-28T19:19:16.005 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x44a): undefined reference to `fi_dupinfo'
2023-05-28T19:19:16.005 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x491): undefined reference to `fi_freeinfo'
2023-05-28T19:19:16.006 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x4fc): undefined reference to `fi_dupinfo'
2023-05-28T19:19:16.006 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `find_provider':
2023-05-28T19:19:16.006 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x94): undefined reference to `fi_dupinfo'
2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0xb8): undefined reference to `fi_getinfo'
2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x278): undefined reference to `fi_getinfo'
2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x296): undefined reference to `fi_freeinfo'
2023-05-28T19:19:16.007 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x30d): undefined reference to `fi_freeinfo'
2023-05-28T19:19:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x341): undefined reference to `fi_version'
2023-05-28T19:19:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x355): undefined reference to `fi_strerror'
2023-05-28T19:19:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x415): undefined reference to `fi_getinfo'
2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x444): undefined reference to `fi_getinfo'
2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x451): undefined reference to `fi_version'
2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x461): undefined reference to `fi_version'
2023-05-28T19:19:16.009 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x480): undefined reference to `fi_dupinfo'
2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x4c9): undefined reference to `fi_version'
2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_find_provider_cleanup':
2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x781): undefined reference to `fi_freeinfo'
2023-05-28T19:19:16.010 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_get_required_version':
2023-05-28T19:19:16.011 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x841): undefined reference to `fi_version'
2023-05-28T19:19:16.011 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_root_ctx':
2023-05-28T19:19:16.011 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0x31c): undefined reference to `fi_strerror'
2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x3b5): undefined reference to `fi_strerror'
2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x45d): undefined reference to `fi_strerror'
2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: (.text+0x555): undefined reference to `fi_strerror'
2023-05-28T19:19:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_all_ctx':
2023-05-28T19:19:16.013 INFO:teuthology.orchestra.run.smithi151.stderr:(.text+0xb35): undefined reference to `fi_strerror'
2023-05-28T19:19:16.013 INFO:teuthology.orchestra.run.smithi151.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o):(.text+0xdf6): more undefined references to `fi_strerror' follow
2023-05-28T19:19:16.013 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 84, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi139 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-05-28T19:19:16.085 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1812c80acea040cc8c3d57b8cd2daa20
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 84, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi139 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
/<pre></pre>
CephFS - Bug #61399 (Pending Backport): qa: build failure for ior
https://tracker.ceph.com/issues/61399
2023-05-24T14:11:53Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2023-05-24T02:32:38.769 INFO:teuthology.orchestra.run.smithi092.stdout:Making all in src
2023-05-24T02:32:38.771 INFO:teuthology.orchestra.run.smithi092.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/ior/src'
2023-05-24T02:32:38.772 INFO:teuthology.orchestra.run.smithi092.stdout:make all-am
2023-05-24T02:32:38.775 INFO:teuthology.orchestra.run.smithi092.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/ior/src'
2023-05-24T02:32:38.803 INFO:teuthology.orchestra.run.smithi092.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT ior.o -MD -MP -MF .deps/ior.Tpo -c -o ior.o ior.c
2023-05-24T02:32:39.027 INFO:teuthology.orchestra.run.smithi092.stderr:ior.c: In function 'CountTasksPerNode':
2023-05-24T02:32:39.027 INFO:teuthology.orchestra.run.smithi092.stderr:ior.c:476:42: warning: '%s' directive writing up to 1023 bytes into a region of size between 1004 and 1014 [-Wformat-overflow=]
2023-05-24T02:32:39.027 INFO:teuthology.orchestra.run.smithi092.stderr: 476 | sprintf(tmp, "task %d on %s", rank, localhost);
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr: | ^~ ~~~~~~~~~
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr:In file included from /usr/include/stdio.h:894,
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr: from ior.c:15:
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 11 and 1044 bytes into a destination of size 1024
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-24T02:32:39.028 INFO:teuthology.orchestra.run.smithi092.stderr: 39 | __glibc_objsize (__s), __fmt,
2023-05-24T02:32:39.029 INFO:teuthology.orchestra.run.smithi092.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-24T02:32:39.029 INFO:teuthology.orchestra.run.smithi092.stderr: 40 | __va_arg_pack ());
2023-05-24T02:32:39.029 INFO:teuthology.orchestra.run.smithi092.stderr: | ~~~~~~~~~~~~~~~~~
2023-05-24T02:32:39.183 INFO:teuthology.orchestra.run.smithi172.stdout:mv -f .deps/ior.Tpo .deps/ior.Po
2023-05-24T02:32:39.186 INFO:teuthology.orchestra.run.smithi172.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT utilities.o -MD -MP -MF .deps/utilities.Tpo -c -o utilities.o utilities.c
2023-05-24T02:32:39.383 INFO:teuthology.orchestra.run.smithi172.stdout:mv -f .deps/utilities.Tpo .deps/utilities.Po
2023-05-24T02:32:39.385 INFO:teuthology.orchestra.run.smithi172.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT parse_options.o -MD -MP -MF .deps/parse_options.Tpo -c -o parse_options.o parse_options.c
2023-05-24T02:32:39.603 INFO:teuthology.orchestra.run.smithi172.stdout:mv -f .deps/parse_options.Tpo .deps/parse_options.Po
2023-05-24T02:32:39.607 INFO:teuthology.orchestra.run.smithi172.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT aiori-POSIX.o -MD -MP -MF .deps/aiori-POSIX.Tpo -c -o aiori-POSIX.o aiori-POSIX.c
2023-05-24T02:32:39.752 INFO:teuthology.orchestra.run.smithi092.stdout:mv -f .deps/ior.Tpo .deps/ior.Po
2023-05-24T02:32:39.755 INFO:teuthology.orchestra.run.smithi092.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT utilities.o -MD -MP -MF .deps/utilities.Tpo -c -o utilities.o utilities.c
2023-05-24T02:32:39.776 INFO:teuthology.orchestra.run.smithi172.stdout:mv -f .deps/aiori-POSIX.Tpo .deps/aiori-POSIX.Po
2023-05-24T02:32:39.778 INFO:teuthology.orchestra.run.smithi172.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT aiori-MPIIO.o -MD -MP -MF .deps/aiori-MPIIO.Tpo -c -o aiori-MPIIO.o aiori-MPIIO.c
2023-05-24T02:32:39.848 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c: In function 'MPIIO_Xfer':
2023-05-24T02:32:39.849 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c:236:24: warning: assignment to 'int (*)(struct ADIOI_FileD *, void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, void *, int, int, MPI_Status *)'} from incompatible pointer type 'int (*)(struct ADIOI_FileD *, const void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, const void *, int, int, MPI_Status *)'} [-Wincompatible-pointer-types]
2023-05-24T02:32:39.849 INFO:teuthology.orchestra.run.smithi172.stderr: 236 | Access = MPI_File_write;
2023-05-24T02:32:39.849 INFO:teuthology.orchestra.run.smithi172.stderr: | ^
2023-05-24T02:32:39.849 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c:237:27: warning: assignment to 'int (*)(struct ADIOI_FileD *, MPI_Offset, void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, long int, void *, int, int, MPI_Status *)'} from incompatible pointer type 'int (*)(struct ADIOI_FileD *, MPI_Offset, const void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, long int, const void *, int, int, MPI_Status *)'} [-Wincompatible-pointer-types]
2023-05-24T02:32:39.850 INFO:teuthology.orchestra.run.smithi172.stderr: 237 | Access_at = MPI_File_write_at;
2023-05-24T02:32:39.850 INFO:teuthology.orchestra.run.smithi172.stderr: | ^
2023-05-24T02:32:39.850 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c:238:28: warning: assignment to 'int (*)(struct ADIOI_FileD *, void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, void *, int, int, MPI_Status *)'} from incompatible pointer type 'int (*)(struct ADIOI_FileD *, const void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, const void *, int, int, MPI_Status *)'} [-Wincompatible-pointer-types]
2023-05-24T02:32:39.851 INFO:teuthology.orchestra.run.smithi172.stderr: 238 | Access_all = MPI_File_write_all;
2023-05-24T02:32:39.851 INFO:teuthology.orchestra.run.smithi172.stderr: | ^
2023-05-24T02:32:39.851 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c:239:31: warning: assignment to 'int (*)(struct ADIOI_FileD *, MPI_Offset, void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, long int, void *, int, int, MPI_Status *)'} from incompatible pointer type 'int (*)(struct ADIOI_FileD *, MPI_Offset, const void *, int, MPI_Datatype, MPI_Status *)' {aka 'int (*)(struct ADIOI_FileD *, long int, const void *, int, int, MPI_Status *)'} [-Wincompatible-pointer-types]
2023-05-24T02:32:39.851 INFO:teuthology.orchestra.run.smithi172.stderr: 239 | Access_at_all = MPI_File_write_at_all;
2023-05-24T02:32:39.852 INFO:teuthology.orchestra.run.smithi172.stderr: | ^
2023-05-24T02:32:39.871 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c: In function 'MPIIO_SetVersion':
2023-05-24T02:32:39.871 INFO:teuthology.orchestra.run.smithi172.stderr:aiori-MPIIO.c:378:38: warning: ' (version=' directive writing 10 bytes into a region of size between 1 and 1024 [-Wformat-overflow=]
2023-05-24T02:32:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 378 | sprintf(test->apiVersion, "%s (version=%d, subversion=%d)",
2023-05-24T02:32:39.872 INFO:teuthology.orchestra.run.smithi172.stderr: | ^~~~~~~~~~
2023-05-24T02:32:39.872 INFO:teuthology.orchestra.run.smithi172.stderr:In file included from /usr/include/stdio.h:894,
2023-05-24T02:32:39.872 INFO:teuthology.orchestra.run.smithi172.stderr: from aiori-MPIIO.c:19:
2023-05-24T02:32:39.872 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 27 and 1070 bytes into a destination of size 1024
2023-05-24T02:32:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
2023-05-24T02:32:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-24T02:32:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 39 | __glibc_objsize (__s), __fmt,
2023-05-24T02:32:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-24T02:32:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 40 | __va_arg_pack ());
2023-05-24T02:32:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: | ~~~~~~~~~~~~~~~~~
2023-05-24T02:32:39.959 INFO:teuthology.orchestra.run.smithi092.stdout:mv -f .deps/utilities.Tpo .deps/utilities.Po
2023-05-24T02:32:39.963 INFO:teuthology.orchestra.run.smithi092.stdout:mpicc -DHAVE_CONFIG_H -I. -g -O2 -MT parse_options.o -MD -MP -MF .deps/parse_options.Tpo -c -o parse_options.o parse_options.c
2023-05-24T02:32:39.963 INFO:teuthology.orchestra.run.smithi172.stdout:mv -f .deps/aiori-MPIIO.Tpo .deps/aiori-MPIIO.Po
2023-05-24T02:32:39.968 INFO:teuthology.orchestra.run.smithi172.stdout:mpicc -g -O2 -o ior ior.o utilities.o parse_options.o aiori-POSIX.o aiori-MPIIO.o -lm
2023-05-24T02:32:39.988 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: utilities.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: multiple definition of `ncmpi_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: first defined here
2023-05-24T02:32:39.988 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: utilities.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: multiple definition of `hdf5_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: first defined here
2023-05-24T02:32:39.988 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: utilities.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: multiple definition of `mpiio_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: first defined here
2023-05-24T02:32:39.988 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: utilities.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: multiple definition of `posix_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: first defined here
2023-05-24T02:32:39.990 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: parse_options.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: multiple definition of `ncmpi_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: first defined here
2023-05-24T02:32:39.990 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: parse_options.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: multiple definition of `hdf5_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: first defined here
2023-05-24T02:32:39.991 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: parse_options.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: multiple definition of `mpiio_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: first defined here
2023-05-24T02:32:39.991 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: parse_options.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: multiple definition of `posix_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: first defined here
2023-05-24T02:32:39.992 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-POSIX.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: multiple definition of `ncmpi_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: first defined here
2023-05-24T02:32:39.992 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-POSIX.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: multiple definition of `hdf5_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: first defined here
2023-05-24T02:32:39.992 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-POSIX.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: multiple definition of `mpiio_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: first defined here
2023-05-24T02:32:39.992 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-POSIX.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: multiple definition of `posix_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: first defined here
2023-05-24T02:32:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-MPIIO.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: multiple definition of `ncmpi_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:68: first defined here
2023-05-24T02:32:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-MPIIO.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: multiple definition of `hdf5_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:67: first defined here
2023-05-24T02:32:39.994 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-MPIIO.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: multiple definition of `mpiio_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:66: first defined here
2023-05-24T02:32:39.994 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: aiori-MPIIO.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: multiple definition of `posix_aiori'; ior.o:/home/ubuntu/cephtest/ior/src/aiori.h:65: first defined here
2023-05-24T02:32:40.186 INFO:teuthology.orchestra.run.smithi092.stdout:mv -f .deps/parse_options.Tpo .deps/parse_options.Po
...
2023-05-24T02:32:40.249 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x451): undefined reference to `fi_version'
2023-05-24T02:32:40.249 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x461): undefined reference to `fi_version'
2023-05-24T02:32:40.249 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x480): undefined reference to `fi_dupinfo'
2023-05-24T02:32:40.249 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x4c9): undefined reference to `fi_version'
2023-05-24T02:32:40.249 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_find_provider_cleanup':
2023-05-24T02:32:40.250 INFO:teuthology.orchestra.run.smithi172.stderr:(.text+0x781): undefined reference to `fi_freeinfo'
2023-05-24T02:32:40.250 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_provider.o): in function `MPIDI_OFI_get_required_version':
2023-05-24T02:32:40.250 INFO:teuthology.orchestra.run.smithi172.stderr:(.text+0x841): undefined reference to `fi_version'
2023-05-24T02:32:40.250 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_root_ctx':
2023-05-24T02:32:40.250 INFO:teuthology.orchestra.run.smithi172.stderr:(.text+0x31c): undefined reference to `fi_strerror'
2023-05-24T02:32:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x3b5): undefined reference to `fi_strerror'
2023-05-24T02:32:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x45d): undefined reference to `fi_strerror'
2023-05-24T02:32:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: (.text+0x555): undefined reference to `fi_strerror'
2023-05-24T02:32:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o): in function `MPIDI_OFI_addr_exchange_all_ctx':
2023-05-24T02:32:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:(.text+0xb35): undefined reference to `fi_strerror'
2023-05-24T02:32:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-init_addrxchg.o):(.text+0xdf6): more undefined references to `fi_strerror' follow
2023-05-24T02:32:40.308 INFO:teuthology.orchestra.run.smithi172.stderr:collect2: error: ld returned 1 exit status
2023-05-24T02:32:40.309 INFO:teuthology.orchestra.run.smithi172.stderr:make[2]: *** [Makefile:299: ior] Error 1
2023-05-24T02:32:40.309 INFO:teuthology.orchestra.run.smithi172.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/ior/src'
2023-05-24T02:32:40.309 INFO:teuthology.orchestra.run.smithi172.stderr:make[1]: *** [Makefile:208: all] Error 2
2023-05-24T02:32:40.310 INFO:teuthology.orchestra.run.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/ior/src'
2023-05-24T02:32:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 2
2023-05-24T02:32:40.311 INFO:teuthology.orchestra.run.smithi172.stderr:make: *** [Makefile:261: all-recursive] Error 1
2023-05-24T02:32:40.311 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/run_tasks.py", line 84, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi172 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-05-24T02:32:40.383 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4da8bbc47d444b9c91169957e0eb1d99
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/run_tasks.py", line 84, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi172 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284202/teuthology.log</p>
CephFS - Bug #58244 (Triaged): Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.T...
https://tracker.ceph.com/issues/58244
2022-12-12T08:56:41Z
Venky Shankar
vshankar@redhat.com
<p>/a/vshankar-2022-12-12_06:30:28-fs-wip-vshankar-testing-20221209.043803-testing-default-smithi/7112851/</p>
<p>The test expects that post running `scan_links` (`cephfs-data-scan`) would rebuild the inode table. The test pins a directory to rank 1 and ensures the subtree is migrated by checking the inode number of a file created inside the (pinned) directory. For some reason, only rank 0 MDS inode table was rebuilt.</p>
CephFS - Bug #57682 (Triaged): client: ERROR: test_reconnect_after_blocklisted
https://tracker.ceph.com/issues/57682
2022-09-27T12:54:22Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2022-09-18T03:29:20.394 INFO:teuthology.orchestra.run.smithi093.stderr:Traceback (most recent call last):
2022-09-18T03:29:20.395 INFO:teuthology.orchestra.run.smithi093.stderr: File "<string>", line 39, in <module>
2022-09-18T03:29:20.395 INFO:teuthology.orchestra.run.smithi093.stderr:BrokenPipeError: [Errno 108] Cannot send after transport endpoint shutdown
...
2022-09-18T03:29:29.438 INFO:tasks.cephfs_test_runner:======================================================================
2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-09-18T03:29:29.440 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_b72a2ce17ad60da863086f3b9f8648dc49f11b79/qa/tasks/cephfs/test_client_recovery.py", line 725, in test_reconnect_after_blocklisted
2022-09-18T03:29:29.440 INFO:tasks.cephfs_test_runner: rproc.wait()
2022-09-18T03:29:29.440 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
2022-09-18T03:29:29.441 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2022-09-18T03:29:29.441 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-09-18T03:29:29.441 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2022-09-18T03:29:29.442 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi093 with status 1: 'python3 -c \'\nimport os\nimport sys\nimport fcntl\nimport errno\nimport time\n\nfd1 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDWR | os.O_CREAT, 0O666)\nfd2 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDONLY)\nfd3 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDWR | os.O_CREAT, 0O666)\nfd4 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDONLY)\n\nos.write(fd1, b\'"\'"\'content\'"\'"\')\nos.read(fd2, 1);\n\nos.write(fd3, b\'"\'"\'content\'"\'"\')\nos.read(fd4, 1);\nfcntl.flock(fd4, fcntl.LOCK_SH | fcntl.LOCK_NB)\n\nprint("blocklist")\nsys.stdout.flush()\n\nsys.stdin.readline()\n\n# wait for mds to close session\ntime.sleep(10);\n\n# trigger \'"\'"\'open session\'"\'"\' message. kclient relies on \'"\'"\'session reject\'"\'"\' message\n# to detect if itself is blocklisted\ntry:\n os.stat("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1")\nexcept:\n pass\n\n# wait for auto reconnect\ntime.sleep(10);\n\ntry:\n os.write(fd1, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EBADF:\n raise\nelse:\n raise RuntimeError("write() failed to raise error")\n\nos.read(fd2, 1);\n\ntry:\n os.read(fd4, 1)\nexcept OSError as e:\n if e.errno != errno.EIO:\n raise\nelse:\n raise RuntimeError("read() failed to raise error")\n\''
</pre>
<p>From: /ceph/teuthology-archive/teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/teuthology.log</p>
<pre>
2022-09-18T03:29:20.392+0000 7fad6b7fe700 10 client.9507 waiting for caps 0x10000000000.head(faked_ino=0 nref=5 ll_ref=2 cap_refs={4=0,1024=0,2048=0,4096=0,8192=0} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2022-09-18T03:28:58.828547+0000 mtime=2022-09-18T03:28:58.836968+0000 ctime=2022-09-18T03:28:58.836968+0000 caps=- objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 0] parents=0x1.head["testfile_reconnect_after_blocklisted.1"] 0x7fad7c0097c0) need AsFw want Fb
2022-09-18T03:29:20.392+0000 7fad6b7fe700 20 client.9507 put_inode on 0x10000000000.head(faked_ino=0 nref=6 ll_ref=2 cap_refs={4=0,1024=0,2048=0,4096=0,8192=0} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2022-09-18T03:28:58.828547+0000 mtime=2022-09-18T03:28:58.836968+0000 ctime=2022-09-18T03:28:58.836968+0000 caps=- objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 0] parents=0x1.head["testfile_reconnect_after_blocklisted.1"] 0x7fad7c0097c0) n = 1
2022-09-18T03:29:20.392+0000 7fad6b7fe700 3 client.9507 ll_write 0x7fad64003140 7~7 = -108
</pre>
<p>From: /ceph/teuthology-archive/teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/remote/smithi093/log/ceph-client.0.165556.log.gz</p>
<p>Seen in many other runs:</p>
<pre>
/repdonnell-2022-09-23_19:31:41-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7042346/teuthology.log:2022-09-24T06:44:10.444 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-23_19:31:41-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7042346/teuthology.log:2022-09-24T06:44:10.457 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-25_13:20:24-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044004/teuthology.log:2022-09-25T14:56:02.280 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-25_13:20:24-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044004/teuthology.log:2022-09-25T14:56:02.298 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044462/teuthology.log:2022-09-26T17:16:48.337 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044462/teuthology.log:2022-09-26T17:16:48.349 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_19:11:10-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044502/teuthology.log:2022-09-26T19:43:49.312 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_19:11:10-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044502/teuthology.log:2022-09-26T19:43:49.321 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-27_02:27:48-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044558/teuthology.log:2022-09-27T03:00:24.834 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-27_02:27:48-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044558/teuthology.log:2022-09-27T03:00:24.841 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/teuthology.log:2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/teuthology.log:2022-09-18T03:29:29.444 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_04:17:04-fs-pacific-distro-default-smithi/7030309/teuthology.log:2022-09-17T13:46:04.269 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_04:17:04-fs-pacific-distro-default-smithi/7030309/teuthology.log:2022-09-17T13:46:04.286 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-27_04:17:03-fs-pacific-distro-default-smithi/7045022/teuthology.log:2022-09-27T07:43:33.096 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-27_04:17:03-fs-pacific-distro-default-smithi/7045022/teuthology.log:2022-09-27T07:43:33.102 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
vshankar-2022-09-06_06:13:42-fs-wip-vshankar-testing-20220901-100101-testing-default-smithi/7014009/teuthology.log:2022-09-06T11:55:08.106 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
vshankar-2022-09-06_06:13:42-fs-wip-vshankar-testing-20220901-100101-testing-default-smithi/7014009/teuthology.log:2022-09-06T11:55:08.112 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
</pre>
CephFS - Bug #48075 (Triaged): qa: AssertionError: 12582912 != 'infinite'
https://tracker.ceph.com/issues/48075
2020-11-02T18:13:50Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2020-11-02T03:18:46.969 INFO:teuthology.orchestra.run.smithi042:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvolume_0000000000352808/.snap/snapshot_0000000000072850/ab868765-4126-4019-b0d3-04d6b4a4e3a0)
2020-11-02T03:18:46.999 INFO:teuthology.orchestra.run.smithi042.stderr:volumes/_nogroup/subvolume_0000000000352808/.snap/snapshot_0000000000072850/ab868765-4126-4019-b0d3-04d6b4a4e3a0: ceph.quota.max_bytes: No such attribute
...
2020-11-02T03:18:48.704 INFO:tasks.cephfs_test_runner:======================================================================
2020-11-02T03:18:48.704 INFO:tasks.cephfs_test_runner:FAIL: test_subvolume_clone_inherit_snapshot_namespace_and_size (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2020-11-02T03:18:48.704 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-11-02T03:18:48.705 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-11-02T03:18:48.705 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 2247, in test_subvolume_clone_inherit_snapshot_namespace_and_size
2020-11-02T03:18:48.705 INFO:tasks.cephfs_test_runner: self._verify_clone(subvolume, snapshot, clone)
2020-11-02T03:18:48.705 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 151, in _verify_clone
2020-11-02T03:18:48.705 INFO:tasks.cephfs_test_runner: self._verify_clone_root(path1, path2, clone, clone_group, clone_pool)
2020-11-02T03:18:48.706 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 119, in _verify_clone_root
2020-11-02T03:18:48.706 INFO:tasks.cephfs_test_runner: self.assertEqual(clone_info["bytes_quota"], "infinite" if src_quota is None else int(src_quota))
2020-11-02T03:18:48.706 INFO:tasks.cephfs_test_runner:AssertionError: 12582912 != 'infinite'
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2020-11-02_02:29:29-fs:volumes-master-distro-basic-smithi/5581161/teuthology.log</p>
<p>Unusual difference in behavior with the fuse client here.</p>
<p>I will disable this check for now with the fs/kcephfs merge.</p>
CephFS - Bug #42022 (Need More Info): mgr/volumes: "Timed out after 30s waiting for ./volumes/_de...
https://tracker.ceph.com/issues/42022
2019-09-24T08:34:46Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2019-09-22T12:58:08.384 INFO:tasks.cephfs_test_runner:======================================================================
2019-09-22T12:58:08.384 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_snapshot_create_and_rm (tasks.cephfs.test_volumes.TestVolumes)
2019-09-22T12:58:08.384 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-09-22T12:58:08.384 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-09-22T12:58:08.385 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20190922.042910/qa/tasks/cephfs/test_volumes.py", line 436, in test_subvolume_snapshot_create_and_rm
2019-09-22T12:58:08.385 INFO:tasks.cephfs_test_runner: self._wait_for_trash_empty()
2019-09-22T12:58:08.385 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20190922.042910/qa/tasks/cephfs/test_volumes.py", line 84, in _wait_for_trash_empty
2019-09-22T12:58:08.385 INFO:tasks.cephfs_test_runner: self.mount_a.wait_for_dir_empty(trashdir)
2019-09-22T12:58:08.385 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20190922.042910/qa/tasks/cephfs/mount.py", line 267, in wait_for_dir_empty
2019-09-22T12:58:08.385 INFO:tasks.cephfs_test_runner: i, dirname, self.client_id))
2019-09-22T12:58:08.386 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out after 30s waiting for ./volumes/_deleting to become empty from 0
2019-09-22T12:58:08.386 INFO:tasks.cephfs_test_runner:
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2019-09-22_10:05:30-fs-wip-pdonnell-testing-20190922.042910-distro-basic-smithi/4326053/teuthology.log</p>