Bug #16454
Updated by Yuri Weinstein almost 8 years ago
Run: http://pulpito.ceph.com/yuriw-2016-06-22_17:09:03-rados-wip-yuri-testing_2016_6_22-distro-basic-smithi/ Job: 271597 Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-06-22_17:09:03-rados-wip-yuri-testing_2016_6_22-distro-basic-smithi/271597/teuthology.log <pre> 2016-06-22T19:35:44.015 INFO:teuthology.orchestra.run.smithi018.stderr:instructing pg 0.17 on osd.3 to deep-scrub 2016-06-22T19:35:44.935 INFO:tasks.ceph.osd.3.smithi058.stderr:2016-06-23 02:35:45.072538 7f72b7515700 7f72b7512016-06-22T19:35:55.337 INFO:teuthology.orchestra.run.smithi058.stderr:mkdir: cannot create directory ‘/var/lib/ceph/osd/ceph-3/fuse/0.17_head/all/#0:e80963ad:::benchmark_data_smithi018_12314_object498:head#’: File exists 2016-06-22T19:35:55.338 INFO:teuthology.orchestra.run.smithi058:Running: "sudo cp /tmp/tmp5DM_1F '/var/lib/ceph/osd/ceph-3/fuse/0.17_head/all/#0:e80963ad:::benchmark_data_smithi018_12314_object498:head#/data'" 2016-06-22T19:35:55.417 INFO:tasks.ceph.osd.3.smithi058.stderr:tcmalloc: large alloc 4294967296 bytes == 0x7f72e035e000 @ 0x7f72d3b28c4c 0x7f72d3b2b547 0x7f72d3b49a02 0x7f72d49faef5 0x7f72d49fb1a5 0x7f72d49fb1d4 0x7f72d4607819 0x7f72d2e6965c 0x7f72d2e697e8 0x7f72d2e72389 0x7f72d2e6ea6c 0x7f72d2e671c8 0x7f72d4606e57 0x7f72d460ce5d 0x7f72d2c47182 0x7f72d0b5a47d (nil) 2016-06-22T19:35:56.914 INFO:tasks.ceph.osd.3.smithi058.stderr:*** Caught signal (Segmentation fault) ** 2016-06-22T19:35:56.914 INFO:tasks.ceph.osd.3.smithi058.stderr: in thread 7f72ca6c9700 thread_name:safe_timer 2016-06-22T19:35:56.916 INFO:tasks.ceph.osd.3.smithi058.stderr: ceph version 10.2.0-2691-gc95c188 (c95c188f6c06bf62d75e1a5b4ecf3b5c27939915) 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 1: (()+0x94aa22) [0x7f72d48f0a22] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 2: (()+0x10340) [0x7f72d2c4f340] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 3: (SafeTimer::timer_thread()+0xe6) [0x7f72d49da7b6] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 4: (SafeTimerThread::entry()+0xd) [0x7f72d49dc15d] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 5: (()+0x8182) [0x7f72d2c47182] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 6: (clone()+0x6d) [0x7f72d0b5a47d] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr:2016-06-23 02:35:57.054426 7f72ca6c9700 -1 *** Caught signal (Segmentation fault) ** 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: in thread 7f72ca6c9700 thread_name:safe_timer 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: ceph version 10.2.0-2691-gc95c188 (c95c188f6c06bf62d75e1a5b4ecf3b5c27939915) 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: 1: (()+0x94aa22) [0x7f72d48f0a22] 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: 2: (()+0x10340) [0x7f72d2c4f340] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 3: (SafeTimer::timer_thread()+0xe6) [0x7f72d49da7b6] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 4: (SafeTimerThread::entry()+0xd) [0x7f72d49dc15d] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 5: (()+0x8182) [0x7f72d2c47182] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 6: (clone()+0x6d) [0x7f72d0b5a47d] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.5700 -1 log_channel(cluster) log [ERR] : 0.17 shard 3: soid 0:e80963ad:::benchmark_data_smithi018_12314_object498:head data_digest 0xf3ab4886 != known data_digest 0xc527f3f0 from auth shard 2, size 4097 != known size 4096 2016-06-22T19:35:44.977 INFO:tasks.ceph.osd.3.smithi058.stderr:2016-06-23 02:35:45.115320 7f72b4d10700 -1 log_channel(cluster) log [ERR] : 0.17 deep-scrub 0 missing, 1 inconsistent objects 2016-06-22T19:35:44.978 INFO:tasks.ceph.osd.3.smithi058.stderr:2016-06-23 02:35:45.115332 7f72b4d10700 -1 log_channel(cluster) log [ERR] : 0.17 deep-scrub 1 errors </pre> and then <pre> 2016-06-22T19:35:55.337 INFO:teuthology.orchestra.run.smithi058.stderr:mkdir: cannot create directory ‘/var/lib/ceph/osd/ceph-3/fuse/0.17_head/all/#0:e80963ad:::benchmark_data_smithi018_12314_object498:head#’: File exists 2016-06-22T19:35:55.338 INFO:teuthology.orchestra.run.smithi058:Running: "sudo cp /tmp/tmp5DM_1F '/var/lib/ceph/osd/ceph-3/fuse/0.17_head/all/#0:e80963ad:::benchmark_data_smithi018_12314_object498:head#/data'" 2016-06-22T19:35:55.417 INFO:tasks.ceph.osd.3.smithi058.stderr:tcmalloc: large alloc 4294967296 bytes == 0x7f72e035e000 @ 0x7f72d3b28c4c 0x7f72d3b2b547 0x7f72d3b49a02 0x7f72d49faef5 0x7f72d49fb1a5 0x7f72d49fb1d4 0x7f72d4607819 0x7f72d2e6965c 0x7f72d2e697e8 0x7f72d2e72389 0x7f72d2e6ea6c 0x7f72d2e671c8 0x7f72d4606e57 0x7f72d460ce5d 0x7f72d2c47182 0x7f72d0b5a47d (nil) 2016-06-22T19:35:56.914 INFO:tasks.ceph.osd.3.smithi058.stderr:*** Caught signal (Segmentation fault) ** 2016-06-22T19:35:56.914 INFO:tasks.ceph.osd.3.smithi058.stderr: in thread 7f72ca6c9700 thread_name:safe_timer 2016-06-22T19:35:56.916 INFO:tasks.ceph.osd.3.smithi058.stderr: ceph version 10.2.0-2691-gc95c188 (c95c188f6c06bf62d75e1a5b4ecf3b5c27939915) 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 1: (()+0x94aa22) [0x7f72d48f0a22] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 2: (()+0x10340) [0x7f72d2c4f340] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 3: (SafeTimer::timer_thread()+0xe6) [0x7f72d49da7b6] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 4: (SafeTimerThread::entry()+0xd) [0x7f72d49dc15d] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 5: (()+0x8182) [0x7f72d2c47182] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr: 6: (clone()+0x6d) [0x7f72d0b5a47d] 2016-06-22T19:35:56.917 INFO:tasks.ceph.osd.3.smithi058.stderr:2016-06-23 02:35:57.054426 7f72ca6c9700 -1 *** Caught signal (Segmentation fault) ** 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: in thread 7f72ca6c9700 thread_name:safe_timer 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: ceph version 10.2.0-2691-gc95c188 (c95c188f6c06bf62d75e1a5b4ecf3b5c27939915) 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: 1: (()+0x94aa22) [0x7f72d48f0a22] 2016-06-22T19:35:56.918 INFO:tasks.ceph.osd.3.smithi058.stderr: 2: (()+0x10340) [0x7f72d2c4f340] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 3: (SafeTimer::timer_thread()+0xe6) [0x7f72d49da7b6] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 4: (SafeTimerThread::entry()+0xd) [0x7f72d49dc15d] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 5: (()+0x8182) [0x7f72d2c47182] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: 6: (clone()+0x6d) [0x7f72d0b5a47d] 2016-06-22T19:35:56.919 INFO:tasks.ceph.osd.3.smithi058.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. </pre> @Sam complete guess - related to https://github.com/ceph/ceph/pull/9456 ?