Actions
Bug #21408
closedosd: "fsck error: free extent 0x2000~2000 intersects allocated blocks"
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/luminous-x
Component(RADOS):
BlueStore
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2017-09-15_17:30:33-upgrade:luminous-x-master-distro-basic-smithi/
Jobs: many
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2017-09-15_17:30:33-upgrade:luminous-x-master-distro-basic-smithi/1636248/teuthology.log
2017-09-15T17:40:55.564 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2' 2017-09-15T17:40:55.565 INFO:tasks.ceph.osd.1.smithi100.stdout:starting osd.1 at - osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal 2017-09-15T17:40:55.567 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:55.569051 7f74c94631c0 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:55.567 INFO:tasks.ceph.osd.2:Started 2017-09-15T17:40:55.567 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0' 2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.611529 7fb15f2ab1c0 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.611674 7fb15f2ab1c0 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stdout:starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.614455 7fb15f2ab1c0 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:55.639 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:55.641517 7f707df44700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:55.649 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:55.651011 7f707df44700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:55.829 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:55.831611 7fcb0de8f1c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) fsck error: free extent 0x2000~2000 intersects allocated blocks 2017-09-15T17:40:55.894 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:55.896752 7f74c94631c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: free extent 0x2000~2000 intersects allocated blocks 2017-09-15T17:40:55.925 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.927309 7fb15f2ab1c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: free extent 0x2000~2000 intersects allocated blocks 2017-09-15T17:40:56.340 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:56.341993 7fcb0de8f1c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _mount fsck found 1 errors 017-09-15T17:40:56.340 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:56.342000 7fcb0de8f1c0 -1 osd.0 0 OSD:init: unable to mount object store 2017-09-15T17:40:56.340 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:56.342007 7fcb0de8f1c0 -1 [0;31m ** ERROR: osd init failed: (5) Input/output error[0m 2017-09-15T17:40:56.412 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:56.413972 7f74c94631c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _mount fsck found 1 errors 2017-09-15T17:40:56.412 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:56.413980 7f74c94631c0 -1 osd.1 0 OSD:init: unable to mount object store 2017-09-15T17:40:56.412 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:56.413987 7f74c94631c0 -1 [0;31m ** ERROR: osd init failed: (5) Input/output error[0m 2017-09-15T17:40:56.448 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:56.449997 7fb15f2ab1c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors 2017-09-15T17:40:56.448 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:56.450004 7fb15f2ab1c0 -1 osd.2 0 OSD:init: unable to mount object store 2017-09-15T17:40:56.448 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:56.450010 7fb15f2ab1c0 -1 [0;31m ** ERROR: osd init failed: (5) Input/output error[0m 2017-09-15T17:40:56.500 INFO:tasks.ceph.osd.0.smithi100.stderr:daemon-helper: command failed with exit status 1 2017-09-15T17:40:56.543 INFO:tasks.ceph.osd.1.smithi100.stderr:daemon-helper: command failed with exit status 1 2017-09-15T17:40:56.595 INFO:tasks.ceph.osd.2.smithi100.stderr:daemon-helper: command failed with exit status 1 2017-09-15T17:40:57.625 INFO:teuthology.orchestra.run.smithi100.stderr:marked down osd.0. 2017-09-15T17:40:57.642 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1' 2017-09-15T17:40:57.805 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:57.807445 7ff0677d5700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:57.825 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:57.827287 7ff0677d5700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:57.914 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1 is already down. <2017-09-15T17:40:57.924 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2' 2017-09-15T17:40:57.994 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:57.995836 7f45653c4700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:58.004 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:58.005996 7f45653c4700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:58.107 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2 is already down. 2017-09-15T17:40:58.118 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2017-09-15T17:40:58.118 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2017-09-15T17:40:58.118 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json' 2017-09-15T17:40:58.211 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:58.213075 7fed8ec92700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:58.221 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:58.223046 7fed8ec92700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:40:58.352 INFO:teuthology.orchestra.run.smithi100.stdout: 2017-09-15T17:40:58.352 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":5,"active_gid":44097,"active_name":"x","active_addr":"-","available":false,"standbys":[],"modules":["restful","status"],"available_modules":["balancer","dashboard","prometheus","restful","status","zabbix"]} 2017-09-15T17:41:01.353 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json' 2017-09-15T17:41:01.468 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:41:01.470538 7f2762a87700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:41:01.490 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:41:01.492581 7f2762a87700 -1 WARNING: all dangerous and experimental features are enabled. 2017-09-15T17:41:01.656 INFO:teuthology.orchestra.run.smithi100.stdout: 2017-09-15T17:41:01.656 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":6,"active_gid":44097,"active_name":"x","active_addr":"172.21.15.100:6800/124301","available":true,"standbys":[],"modules":["restful","status"],"available_modules":["balancer","dashboard","prometheus","restful","status","zabbix"]} 2017-09-15T17:41:01.656 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2017-09-15T17:41:01.657 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1378, in restart healthy(ctx=ctx, config=dict(cluster=cluster)) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1246, in healthy ceph_cluster=cluster_name, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 924, in wait_until_osds_up daemon.check_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon.py", line 151, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 205, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 177, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed on smithi100 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' /pre>
Updated by Sage Weil over 6 years ago
- Status changed from New to In Progress
- Assignee set to Sage Weil
Updated by Greg Farnum over 6 years ago
- Subject changed from "daemon-helper kill ceph-osd -f --cluster ceph -i 1" in upgrade:luminous-x-master to osd: "fsck error: free extent 0x2000~2000 intersects allocated blocks"
- Category set to Correctness/Safety
- Component(RADOS) BlueStore added
Updated by Sage Weil over 6 years ago
- Status changed from In Progress to Fix Under Review
Updated by Sage Weil over 6 years ago
- Status changed from Fix Under Review to Resolved
Actions