Project

General

Profile

Bug #1508

iozone stuck on kernel rbd mount

Added by Josh Durgin about 8 years ago. Updated almost 8 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
09/06/2011
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Logs are in teuthology:~teuthworker/archive/nightly_coverage_2011-09-05/647

2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:Error writing block 9407, fd= 3
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:     Children see throughput for  1 initial writers  =       0.00 KB/sec
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:     Parent sees throughput for  1 initial writers   =       0.00 KB/sec
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:     Min throughput per process                      =       0.00 KB/sec
2011-09-06T10:53:13.862 INFO:teuthology.task.workunit.client.0.out:     Max throughput per process                      =       0.00 KB/sec
2011-09-06T10:53:13.862 INFO:teuthology.task.workunit.client.0.out:     Avg throughput per process                      =       0.00 KB/sec
2011-09-06T10:53:13.862 INFO:teuthology.task.workunit.client.0.out:     Min xfer                                        =       0.00 KB
2011-09-06T10:53:42.483 INFO:teuthology.task.workunit.client.0.out:
2011-09-06T10:53:42.484 INFO:teuthology.task.workunit.client.0.out:Child 0

The osd logs show the journal being full for a while:

2011-09-06 10:35:41.333815 7f55ecc31700 journal check_for_full at 1101824 : JOURNAL FULL 1101824 >= 147455 (max_size 104857600 start 1249280)
2011-09-06 10:35:41.768625 7f55ecc31700 journal check_for_full at 1101824 : JOURNAL FULL 1101824 >= 147455 (max_size 104857600 start 1249280)
2011-09-06 10:35:47.044935 7f55ecc31700 journal check_for_full at 638976 : JOURNAL FULL 638976 >= 462847 (max_size 104857600 start 1101824)
2011-09-06 10:35:47.530908 7f55ecc31700 journal check_for_full at 638976 : JOURNAL FULL 638976 >= 462847 (max_size 104857600 start 1101824)
...
2011-09-06 10:52:58.039352 7f55ecc31700 journal check_for_full at 38965248 : JOURNAL FULL 38965248 >= 356351 (max_size 104857600 start 39321600)
2011-09-06 10:53:03.719543 7f55ecc31700 journal check_for_full at 38797312 : JOURNAL FULL 38797312 >= 167935 (max_size 104857600 start 38965248)
2011-09-06 10:53:04.047012 7f55ecc31700 journal check_for_full at 38797312 : JOURNAL FULL 38797312 >= 167935 (max_size 104857600 start 38965248)
2011-09-06 10:53:13.895644 7f55ecc31700 journal check_for_full at 38612992 : JOURNAL FULL 38612992 >= 184319 (max_size 104857600 start 38797312)

History

#1 Updated by Sage Weil about 8 years ago

  • translation missing: en.field_position set to 6

#2 Updated by Sage Weil about 8 years ago

  • translation missing: en.field_position deleted (12)
  • translation missing: en.field_position set to 13

#3 Updated by Sage Weil about 8 years ago

  • Assignee set to Sage Weil

#4 Updated by Josh Durgin about 8 years ago

ffsb killed the rbd node in teuthology:~teuthworker/archive/nightly_coverage_2011-09-19/323 - probably the same bug.

#5 Updated by Sage Weil about 8 years ago

this doesn't include commit:935b639a049053d0ccbcf7422f2f9cd221642f58 (kernel), which i hope is responsible for all of these rbd issues (any osd connection blip was crashing the client).

on a related note, it's annoying that we aren't getting proper BUG messages on the console for these crashes. we may want to adjust the .config on the kernel gitbuilder to enable some debugging options.

#6 Updated by Sage Weil about 8 years ago

  • Target version changed from v0.36 to v0.37

#7 Updated by Sage Weil about 8 years ago

  • Target version changed from v0.37 to v0.38

#8 Updated by Sage Weil about 8 years ago

  • Assignee deleted (Sage Weil)

#9 Updated by Sage Weil about 8 years ago

  • Status changed from New to Need More Info

#10 Updated by Sage Weil about 8 years ago

  • Target version changed from v0.38 to v0.39

#11 Updated by Sage Weil almost 8 years ago

  • Target version changed from v0.39 to v0.40

#12 Updated by Sage Weil almost 8 years ago

  • Status changed from Need More Info to Can't reproduce

haven't seen this recently

Also available in: Atom PDF