Project

General

Profile

Actions

Bug #10513

closed

ceph_test_librbd_fsx fails with thrasher

Added by Loïc Dachary over 9 years ago. Updated over 9 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/loic-2015-01-08_10:36:47-rbd-giant-backports-testing-basic-vps/690698/

s)
2015-01-08T03:31:52.223 INFO:teuthology.orchestra.run.vpm020.stdout:1368 write    0xcdd426c thru    0xcdda38b    (0x6120 bytes)
2015-01-08T03:31:52.245 INFO:teuthology.orchestra.run.vpm020.stdout:1370 write    0x1c01757 thru    0x1c111a4    (0xfa4e bytes)
2015-01-08T03:31:52.359 INFO:teuthology.orchestra.run.vpm020.stdout:1371 punch    from 0xbd27283 to 0xbd28661, (0x13de bytes)
2015-01-08T03:31:52.360 INFO:teuthology.orchestra.run.vpm020.stdout:1372 read    0x8ec1a40 thru    0x8ed14b4    (0xfa75 bytes)
2015-01-08T03:31:52.362 INFO:teuthology.orchestra.run.vpm020.stdout:1376 read    0x2c2dc2f thru    0x2c3a85c    (0xcc2e bytes)
2015-01-08T03:31:52.363 INFO:teuthology.orchestra.run.vpm020.stdout:1377 write    0x49bbbaa thru    0x49c3cee    (0x8145 bytes)
2015-01-08T03:31:52.381 INFO:teuthology.orchestra.run.vpm020.stdout:1378 clone    18 order 24 su 65536 sc 10
2015-01-08T03:31:53.488 INFO:teuthology.orchestra.run.vpm020.stdout:leaving image image_client.0-clone17 intact
2015-01-08T03:31:54.094 INFO:tasks.thrashosds.thrasher:in_osds:  [0, 5, 4, 3, 2, 1]  out_osds:  [] dead_osds:  [] live_osds:  [1, 0, 2, 3, 5, 4]
2015-01-08T03:31:54.094 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-01-08T03:31:54.094 INFO:tasks.thrashosds.thrasher:inject_pause on 2
2015-01-08T03:31:54.094 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3
2015-01-08T03:31:54.094 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False
2015-01-08T03:31:54.095 INFO:teuthology.orchestra.run.vpm020:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3'
2015-01-08T03:31:55.767 INFO:teuthology.orchestra.run.vpm020.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17
2015-01-08T03:31:57.236 INFO:teuthology.orchestra.run.vpm020.stdout:1379 trunc    from 0xcdda38c to 0x356fbb0
2015-01-08T03:31:57.276 INFO:teuthology.orchestra.run.vpm020.stdout:1380 punch    from 0x2dc498a to 0x2dd41a9, (0xf81f bytes)
2015-01-08T03:31:58.230 INFO:teuthology.orchestra.run.vpm020.stdout:1381 write    0x9812d9b thru    0x981dbff    (0xae65 bytes)
2015-01-08T03:31:58.256 INFO:teuthology.orchestra.run.vpm020.stdout:1382 read    0x41e63cb thru    0x41eaaac    (0x46e2 bytes)
2015-01-08T03:31:58.260 INFO:teuthology.orchestra.run.vpm020.stdout:1384 punch    from 0x56cf68e to 0x56d8789, (0x90fb bytes)
2015-01-08T03:31:58.262 INFO:teuthology.orchestra.run.vpm020.stdout:1385 punch    from 0x34d0564 to 0x34de4ae, (0xdf4a bytes)
2015-01-08T03:31:58.503 INFO:teuthology.orchestra.run.vpm020.stdout:1386 read    0x14bc432 thru    0x14cc124    (0xfcf3 bytes)
2015-01-08T03:31:58.516 INFO:teuthology.orchestra.run.vpm020.stdout:1387 trunc    from 0x981dc00 to 0x4b15e63
2015-01-08T03:31:59.360 INFO:teuthology.orchestra.run.vpm020.stdout:1388 write    0x9a9f9e9 thru    0x9aa4022    (0x463a bytes)
2015-01-08T03:31:59.564 INFO:teuthology.orchestra.run.vpm020.stdout:1389 read    0x8ab5000 thru    0x8ab9bf3    (0x4bf4 bytes)
2015-01-08T03:31:59.588 INFO:teuthology.orchestra.run.vpm020.stdout:1390 write    0xc97e2a0 thru    0xc980f4e    (0x2caf bytes)
2015-01-08T03:31:59.607 INFO:teuthology.orchestra.run.vpm020.stdout:1391 write    0x83e230d thru    0x83eb9c6    (0x96ba bytes)
2015-01-08T03:31:59.926 INFO:teuthology.orchestra.run.vpm020.stdout:1392 read    0x393a1d1 thru    0x3941192    (0x6fc2 bytes)
2015-01-08T03:31:59.929 INFO:teuthology.orchestra.run.vpm020.stdout:1393 read    0xa4b4684 thru    0xa4bf84a    (0xb1c7 bytes)
2015-01-08T03:31:59.956 INFO:teuthology.orchestra.run.vpm020.stdout:1394 write    0xea94054 thru    0xea94507    (0x4b4 bytes)
2015-01-08T03:31:59.973 INFO:teuthology.orchestra.run.vpm020.stdout:1398 read    0x4b1ef26 thru    0x4b26525    (0x7600 bytes)
2015-01-08T03:31:59.975 INFO:teuthology.orchestra.run.vpm020.stdout:1399 clone    19 order 21 su 32768 sc 14
2015-01-08T03:32:04.230 INFO:teuthology.orchestra.run.vpm020.stdout:truncating image image_client.0-clone18 from 0xea94508 (overlap 0x356fbb0) to 0x557f91
2015-01-08T03:32:04.549 INFO:tasks.thrashosds.thrasher:in_osds:  [0, 5, 4, 3, 2, 1]  out_osds:  [] dead_osds:  [] live_osds:  [1, 0, 2, 3, 5, 4]
2015-01-08T03:32:04.549 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-01-08T03:32:04.549 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 5, 4, 3, 2, 1]
2015-01-08T03:32:04.549 INFO:teuthology.orchestra.run.vpm020:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 4'
2015-01-08T03:32:04.973 INFO:teuthology.orchestra.run.vpm020.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18
2015-01-08T03:32:08.495 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_giant/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/rbd_fsx.py", line 82, in _run_one_client
    remote.run(args=args)
  File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_giant/teuthology/orchestra/run.py", line 103, in wait
    raise CommandCrashedError(command=self.command)
CommandCrashedError: Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 2000 pool_client.0 image_client.0'

Actions

Also available in: Atom PDF