Bug #12746
Updated by Loïc Dachary over 8 years ago
"/a/sage-2015-08-19_19:18:53-rados-wip-newstore-misc-distro-basic-multi/1023449":http://pulpito.ceph.com/sage-2015-08-19_19:18:53-rados-wip-newstore-misc-distro-basic-multi/1023449/ "/a/sage-2015-08-19_19:18:53-rados-wip-newstore-misc-distro-basic-multi/1023449":sage-2015-08-19_19:18:53-rados-wip-newstore-misc-distro-basic-multi/1023449 <pre> 2015-08-20T12:33:43.537 INFO:tasks.repair_test:starting repair test type 1 2015-08-20T12:33:43.537 INFO:tasks.repair_test:Choosing replica 2015-08-20T12:33:43.538 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2015-08-20T12:33:43.770 INFO:teuthology.orchestra.run.mira012.stderr:dumped all in format json 2015-08-20T12:33:43.789 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-08-20T12:33:43.986 INFO:teuthology.orchestra.run.mira012.stderr:2015-08-20 12:33:43.987052 7f9c7657d700 0 monclient: hunting for new mon 2015-08-20T12:33:44.107 INFO:tasks.repair_test:doing put 2015-08-20T12:33:44.107 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p repair_pool_1 put repair_test_obj /etc/hosts' 2015-08-20T12:33:44.550 INFO:tasks.repair_test:corrupting object 2015-08-20T12:33:44.550 INFO:tasks.repair_test:injecting mdata err on object 2015-08-20T12:33:44.550 INFO:teuthology.orchestra.run.burnupi48:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok injectmdataerr repair_pool_1 repair_test_obj' 2015-08-20T12:33:44.978 INFO:tasks.repair_test:scrubbing 2015-08-20T12:33:44.979 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-08-20T12:33:45.223 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2015-08-20T12:33:45.464 INFO:teuthology.orchestra.run.mira012.stderr:dumped all in format json 2015-08-20T12:33:45.477 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-08-20T12:33:45.782 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg scrub 2.0' 2015-08-20T12:33:46.018 INFO:teuthology.orchestra.run.mira012.stderr:instructing pg 2.0 on osd.3 to scrub 2015-08-20T12:33:46.032 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-08-20T12:33:46.283 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2015-08-20T12:33:46.393 INFO:tasks.ceph.osd.3.burnupi48.stderr:2015-08-20 15:33:46.359305 7fcf5b944700 -1 log_channel(cluster) log [ERR] : 2.0 shard 5: soid 2/3df68405/repair_test_obj/head candidate had a read error, size 18446744073709551615 != known size 221, missing attr _, missing attr snapset 2015-08-20T12:33:46.393 INFO:tasks.ceph.osd.3.burnupi48.stderr:2015-08-20 15:33:46.359582 7fcf5b944700 -1 log_channel(cluster) log [ERR] : 2.0 scrub 0 missing, 1 inconsistent objects 2015-08-20T12:33:46.394 INFO:tasks.ceph.osd.3.burnupi48.stderr:2015-08-20 15:33:46.359593 7fcf5b944700 -1 log_channel(cluster) log [ERR] : 2.0 scrub 1 errors 2015-08-20T12:33:46.508 INFO:teuthology.orchestra.run.mira012.stderr:dumped all in format json 2015-08-20T12:33:46.521 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-08-20T12:33:46.826 INFO:teuthology.orchestra.run.mira012:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2015-08-20T12:33:47.062 INFO:teuthology.orchestra.run.mira012.stderr:dumped all in format json 2015-08-20T12:33:47.076 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task return fn(**kwargs) File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/repair_test.py", line 301, in task repair_test_1(ctx, mdataerr, choose_replica, "scrub") File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/repair_test.py", line 98, in repair_test_1 assert ctx.manager.pg_inconsistent(pool, 0) AssertionError 2015-08-20T12:33:47.109 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/search?q=44c1e7cfa4d34b6987b5aaa5a8cb18d1 AssertionError </pre>