Bug #25148
"ceph session ls" produces unparseable json when run against ceph-mds.a.asok when mds.a not running
0%
Description
What?¶
Apparent race condition in upgrade test
Where?¶
It happened in the upgrade test upgrade:luminous-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/luminous.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/filestore-xfs.yaml supported-all-distro/centos_latest.yaml}
Analysis (such as it is)¶
mds.a is stopped:
2018-07-29T03:37:40.288 INFO:tasks.ceph.mds.a:Stopped 2018-07-29T03:37:40.288 INFO:teuthology.orchestra.run.smithi134:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'
The command "ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok session ls" is run on the stopped mds's asok:
2018-07-29T03:37:40.290 INFO:teuthology.orchestra.run.smithi134:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok session ls'
mds.a is started, but not fast enough
2018-07-29T03:37:40.330 INFO:tasks.ceph.mds.a:Started 2018-07-29T03:37:40.482 INFO:tasks.ceph.mds.a.smithi134.stdout:starting mds.a at
The result of "ceph session ls" comes in, and it isn't parseable as JSON:
2018-07-29T03:37:40.530 INFO:teuthology.orchestra.run.smithi134.stdout:"mds_not_active" 2018-07-29T03:37:40.531 INFO:teuthology.orchestra.run.smithi134.stdout:failed 2018-07-29T03:37:40.531 INFO:tasks.cephfs.filesystem:_json_asok output: "mds_not_active" failed 2018-07-29T03:37:40.531 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 55, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 63, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/full_sequential.py", line 30, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/sequential.py", line 48, in task mgr.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/ceph_fuse.py", line 136, in task mount.mount() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/fuse_mount.py", line 33, in mount return self._mount(mount_path, mount_fs_name) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/fuse_mount.py", line 161, in _mount self.gather_mount_info() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/fuse_mount.py", line 170, in gather_mount_info sessions = self.fs.rank_asok(['session', 'ls']) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/filesystem.py", line 920, in rank_asok return self.json_asok(command, 'mds', info['name']) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/filesystem.py", line 180, in json_asok return json.loads(response_data) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 367, in decode raise ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 2 column 1 - line 2 column 7 (char 17 - 23) 2018-07-29T03:37:40.572 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=69a609d9c5cb4f04b2d8d2a04f1ec98d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 55, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 63, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/full_sequential.py", line 30, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/sequential.py", line 48, in task mgr.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/ceph_fuse.py", line 136, in task mount.mount() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/fuse_mount.py", line 33, in mount return self._mount(mount_path, mount_fs_name) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/fuse_mount.py", line 161, in _mount self.gather_mount_info() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/fuse_mount.py", line 170, in gather_mount_info sessions = self.fs.rank_asok(['session', 'ls']) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/filesystem.py", line 920, in rank_asok return self.json_asok(command, 'mds', info['name']) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-smithfarm-testing/qa/tasks/cephfs/filesystem.py", line 180, in json_asok return json.loads(response_data) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 367, in decode raise ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 2 column 1 - line 2 column 7 (char 17 - 23)
Related issues
History
#1 Updated by Nathan Cutler over 5 years ago
- Description updated (diff)
#2 Updated by Nathan Cutler over 5 years ago
- Description updated (diff)
#3 Updated by Nathan Cutler over 5 years ago
- Related to Bug #17093: test failure due to extra data in json parsing added
#4 Updated by Patrick Donnelly over 5 years ago
- Assignee set to Patrick Donnelly
- Target version set to v14.0.0
- Source set to Development
- Backport set to mimic,luminous
- Component(FS) MDS, qa-suite added
- Labels (FS) qa added
#5 Updated by Patrick Donnelly about 5 years ago
- Target version changed from v14.0.0 to v15.0.0
#6 Updated by Patrick Donnelly about 4 years ago
- Target version deleted (
v15.0.0)
#7 Updated by Patrick Donnelly 5 months ago
- Status changed from New to Won't Fix - EOL