Bug #1508: iozone stuck on kernel rbd mount - Ceph - Ceph

Actions

Copy link

Bug #1508

closed

iozone stuck on kernel rbd mount

Added by Josh Durgin over 12 years ago. Updated over 12 years ago.

Status:

Can't reproduce

Priority:

Normal

Assignee:

Category:

Target version:

v0.40

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Logs are in teuthology:~teuthworker/archive/nightly_coverage_2011-09-05/647

2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:Error writing block 9407, fd= 3
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:     Children see throughput for  1 initial writers  =       0.00 KB/sec
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:     Parent sees throughput for  1 initial writers   =       0.00 KB/sec
2011-09-06T10:53:13.861 INFO:teuthology.task.workunit.client.0.out:     Min throughput per process                      =       0.00 KB/sec
2011-09-06T10:53:13.862 INFO:teuthology.task.workunit.client.0.out:     Max throughput per process                      =       0.00 KB/sec
2011-09-06T10:53:13.862 INFO:teuthology.task.workunit.client.0.out:     Avg throughput per process                      =       0.00 KB/sec
2011-09-06T10:53:13.862 INFO:teuthology.task.workunit.client.0.out:     Min xfer                                        =       0.00 KB
2011-09-06T10:53:42.483 INFO:teuthology.task.workunit.client.0.out:
2011-09-06T10:53:42.484 INFO:teuthology.task.workunit.client.0.out:Child 0

The osd logs show the journal being full for a while:

2011-09-06 10:35:41.333815 7f55ecc31700 journal check_for_full at 1101824 : JOURNAL FULL 1101824 >= 147455 (max_size 104857600 start 1249280)
2011-09-06 10:35:41.768625 7f55ecc31700 journal check_for_full at 1101824 : JOURNAL FULL 1101824 >= 147455 (max_size 104857600 start 1249280)
2011-09-06 10:35:47.044935 7f55ecc31700 journal check_for_full at 638976 : JOURNAL FULL 638976 >= 462847 (max_size 104857600 start 1101824)
2011-09-06 10:35:47.530908 7f55ecc31700 journal check_for_full at 638976 : JOURNAL FULL 638976 >= 462847 (max_size 104857600 start 1101824)
...
2011-09-06 10:52:58.039352 7f55ecc31700 journal check_for_full at 38965248 : JOURNAL FULL 38965248 >= 356351 (max_size 104857600 start 39321600)
2011-09-06 10:53:03.719543 7f55ecc31700 journal check_for_full at 38797312 : JOURNAL FULL 38797312 >= 167935 (max_size 104857600 start 38965248)
2011-09-06 10:53:04.047012 7f55ecc31700 journal check_for_full at 38797312 : JOURNAL FULL 38797312 >= 167935 (max_size 104857600 start 38965248)
2011-09-06 10:53:13.895644 7f55ecc31700 journal check_for_full at 38612992 : JOURNAL FULL 38612992 >= 184319 (max_size 104857600 start 38797312)

Actions

Copy link

Updated by Sage Weil over 12 years ago

Translation missing: en.field_position set to 6

Actions

Copy link

Updated by Sage Weil over 12 years ago

Translation missing: en.field_position deleted (12)
Translation missing: en.field_position set to 13

Actions

Copy link

Updated by Sage Weil over 12 years ago

Assignee set to Sage Weil

Actions

Copy link

Updated by Josh Durgin over 12 years ago

ffsb killed the rbd node in teuthology:~teuthworker/archive/nightly_coverage_2011-09-19/323 - probably the same bug.

Actions

Copy link

Updated by Sage Weil over 12 years ago

this doesn't include commit:935b639a049053d0ccbcf7422f2f9cd221642f58 (kernel), which i hope is responsible for all of these rbd issues (any osd connection blip was crashing the client).

on a related note, it's annoying that we aren't getting proper BUG messages on the console for these crashes. we may want to adjust the .config on the kernel gitbuilder to enable some debugging options.

Actions

Copy link