Ceph » RADOS

2019-11-21T15:12:02.079 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: [ RUN      ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1
2019-11-21T15:12:02.079 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: *** Error in `ceph_test_rados_api_watch_notify_pp': double free or corruption (fasttop): 0x00005647753bfb50 ***
2019-11-21T15:12:02.079 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: ======= Backtrace: =========
2019-11-21T15:12:02.079 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/libc.so.6(+0x81679)[0x7f0a911e7679]
2019-11-21T15:12:02.080 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/librados.so.2(+0xc5cdb)[0x7f0a9c5cccdb]
2019-11-21T15:12:02.080 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/librados.so.2(+0xe38b9)[0x7f0a9c5ea8b9]
2019-11-21T15:12:02.080 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/librados.so.2(+0xe72de)[0x7f0a9c5ee2de]
2019-11-21T15:12:02.081 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/librados.so.2(+0xe760a)[0x7f0a9c5ee60a]
2019-11-21T15:12:02.081 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /usr/lib64/ceph/libceph-common.so.2(_ZN13DispatchQueue5entryEv+0x472)[0x7f0a932e8752]
2019-11-21T15:12:02.081 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /usr/lib64/ceph/libceph-common.so.2(_ZN13DispatchQueue14DispatchThread5entryEv+0xd)[0x7f0a9338477d]
2019-11-21T15:12:02.081 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/libpthread.so.0(+0x7e65)[0x7f0a9c2f2e65]
2019-11-21T15:12:02.081 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: /lib64/libc.so.6(clone+0x6d)[0x7f0a9126488d]
2019-11-21T15:12:02.081 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: ======= Memory map: ========
2019-11-21T15:12:02.083 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: 564774abe000-564774b6a000 r-xp 00000000 08:01 9081                       /usr/bin/ceph_test_rados_api_watch_notify_pp
2019-11-21T15:12:02.083 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: 564774d6a000-564774d6d000 r--p 000ac000 08:01 9081                       /usr/bin/ceph_test_rados_api_watch_notify_pp
2019-11-21T15:12:02.083 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: 564774d6d000-564774d6e000 rw-p 000af000 08:01 9081                       /usr/bin/ceph_test_rados_api_watch_notify_pp
2019-11-21T15:12:02.083 INFO:tasks.workunit.client.0.smithi106.stdout:      api_watch_notify_pp: 564775363000-564775fd9000 rw-p 00000000 00:00 0                          [heap]
...

/a/yuriw-2019-11-21_01:58:26-rados-wip-yuri-master_baselline_11.20.19-distro-basic-smithi/4527961

Same run,

2019-11-21T14:53:41.241 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: [ RUN      ] LibRadosWatchNotify.Watch3Timeout
2019-11-21T14:53:41.241 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: waiting up to 64 for osd to time us out ...
2019-11-21T14:53:41.241 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: watch_notify2_test_errcb cookie 94473989630560 err -107
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: watch_notify2_test_cb from 4517 notify_id 197568495622 cookie 94473989630560
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: watch_notify2_test_cb from 4517 notify_id 197568495623 cookie 94473989630560
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: *** Error in `ceph_test_rados_api_watch_notify': malloc(): memory corruption: 0x000055ec7097475b ***
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ======= Backtrace: =========
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: /lib64/libc.so.6(+0x82e86)[0x7fedcc91de86]
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: /lib64/libc.so.6(__libc_malloc+0x4c)[0x7fedcc920adc]
2019-11-21T14:53:41.242 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: /lib64/libstdc++.so.6(_Znwm+0x1d)[0x7fedcd1dfecd]
2019-11-21T14:53:41.243 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing7MessageC1Ev+0x13)[0x55ec6fbb5543]
2019-11-21T14:53:41.243 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8internal18StreamableToStringIxEESsRKT_+0x29)[0x55ec6fbd6389]
2019-11-21T14:53:41.243 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8internal27PrettyUnitTestResultPrinter9OnTestEndERKNS_8TestInfoE+0xa5)[0x55ec6fbc0ef5]
2019-11-21T14:53:41.243 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8internal17TestEventRepeater9OnTestEndERKNS_8TestInfoE+0x45)[0x55ec6fbb3ce5]
2019-11-21T14:53:41.243 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8TestInfo3RunEv+0xf8)[0x55ec6fbd12f8]
2019-11-21T14:53:41.244 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing9TestSuite3RunEv+0xb5)[0x55ec6fbd1415]
2019-11-21T14:53:41.244 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8internal12UnitTestImpl11RunAllTestsEv+0x453)[0x55ec6fbd1903]
2019-11-21T14:53:41.244 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc+0x4a)[0x55ec6fbdba7a]
2019-11-21T14:53:41.244 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(_ZN7testing8UnitTest3RunEv+0x8e)[0x55ec6fbd1b0e]
2019-11-21T14:53:41.245 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(main+0x39)[0x55ec6fb94f09]
2019-11-21T14:53:41.245 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fedcc8bd505]
2019-11-21T14:53:41.245 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ceph_test_rados_api_watch_notify(+0x32697)[0x55ec6fb95697]
2019-11-21T14:53:41.245 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: ======= Memory map: ========
2019-11-21T14:53:41.245 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 55ec6fb63000-55ec6fc01000 r-xp 00000000 08:01 9079                       /usr/bin/ceph_test_rados_api_watch_notify
2019-11-21T14:53:41.246 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 55ec6fe00000-55ec6fe03000 r--p 0009d000 08:01 9079                       /usr/bin/ceph_test_rados_api_watch_notify
2019-11-21T14:53:41.246 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 55ec6fe03000-55ec6fe04000 rw-p 000a0000 08:01 9079                       /usr/bin/ceph_test_rados_api_watch_notify
2019-11-21T14:53:41.246 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 55ec707b8000-55ec709d8000 rw-p 00000000 00:00 0                          [heap]
2019-11-21T14:53:41.246 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7fed94000000-7fed94021000 rw-p 00000000 00:00 0
2019-11-21T14:53:41.246 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7fed94021000-7fed98000000 ---p 00000000 00:00 0
2019-11-21T14:53:41.247 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7fed9c000000-7fed9c021000 rw-p 00000000 00:00 0
2019-11-21T14:53:41.247 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7fed9c021000-7feda0000000 ---p 00000000 00:00 0
2019-11-21T14:53:41.247 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7feda0000000-7feda0021000 rw-p 00000000 00:00 0
2019-11-21T14:53:41.247 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7feda0021000-7feda4000000 ---p 00000000 00:00 0
2019-11-21T14:53:41.247 INFO:tasks.workunit.client.0.smithi175.stdout:         api_watch_notify: 7feda4000000-7feda4036000 rw-p 00000000 00:00 0

/a/yuriw-2019-11-21_01:58:26-rados-wip-yuri-master_baselline_11.20.19-distro-basic-smithi/4527923

and

2019-11-21T15:32:38.369 INFO:tasks.workunit.client.0.smithi156.stdout:         api_watch_notify: [ RUN      ] LibRadosWatchNotify.AioWatchDelete2
2019-11-21T15:32:38.370 INFO:tasks.workunit.client.0.smithi156.stdout:         api_watch_notify: double free or corruption (fasttop)
2019-11-21T15:32:38.370 INFO:tasks.workunit.client.0.smithi156.stderr:bash: line 1: 13158 Aborted                 (core dumped) ceph_test_rados_api_watch_notify 2>&1
2019-11-21T15:32:38.370 INFO:tasks.workunit.client.0.smithi156.stderr:     13159 Done                    | tee ceph_test_rados_api_watch_notify.log
2019-11-21T15:32:38.371 INFO:tasks.workunit.client.0.smithi156.stderr:     13160 Done                    | sed "s/^/         api_watch_notify: /" 
...

/a/yuriw-2019-11-21_01:58:26-rados-wip-yuri-master_baselline_11.20.19-distro-basic-smithi/4528003