Project

General

Profile

Actions
Copy link

Bug #8226

closed

0.80~rc1: RBD read errors (ENXIO)

Added by Dmitry Smirnov almost 10 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Ilya Dryomov
Category:
rbd
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

With Ceph_0.80~rc1 and Linux_3.14.1 I'm getting read errors on RBD devices.

from '/var/log/messages':

Apr 26 20:06:32 deblab kernel: [ 7964.417701] rbd: rbd0: read 30000 at d17400000 (0)
Apr 26 20:06:32 deblab kernel: [ 7964.417701] 
Apr 26 20:06:32 deblab kernel: [ 7964.417708] rbd: rbd0:   result -6 xferred 30000
Apr 26 20:06:32 deblab kernel: [ 7964.417708] 
Apr 26 20:06:32 deblab kernel: [ 7964.419285] rbd: rbd0: read 40000 at d17430000 (30000)
Apr 26 20:06:32 deblab kernel: [ 7964.419285] 
Apr 26 20:06:32 deblab kernel: [ 7964.419293] rbd: rbd0:   result -6 xferred 40000
Apr 26 20:06:32 deblab kernel: [ 7964.419293] 
Apr 26 20:06:32 deblab kernel: [ 7964.419596] rbd: rbd0: read 1000 at d17400000 (0)
Apr 26 20:06:32 deblab kernel: [ 7964.419596] 
Apr 26 20:06:32 deblab kernel: [ 7964.419602] rbd: rbd0:   result -6 xferred 1000

I have 'ext4' file system on RBD and attempt to read some files ends with "Input/output error".
I confirmed this problem with 'md5sum' and 'badblocks' utilities.

Cluster is degraded/recovering after replacing some OSDs, 1 PG inconsistent:
"1 active+degraded+remapped+inconsistent+wait_backfill"

All pools configured with "replicated size 4 min_size 2".
There are no "unfound" objects.

I wonder if this is an error or expected behaviour?

Before 0.80~rc1 (and with Linux-3.13) I've seen no read errors on RBD devices whatsoever.

Is this correct that even for inconsistent PG there shall be no read errors as long as correct object can be found?

Files

Download all files
crushmap (904 Bytes) crushmap Dmitry Smirnov, 05/02/2014 03:51 PM
osdmap (9.09 KB) osdmap Dmitry Smirnov, 05/02/2014 03:51 PM
32514.osdmap (15.8 KB) 32514.osdmap Dmitry Smirnov, 05/09/2014 12:21 AM
osdmap.32513 (15.6 KB) osdmap.32513 Dmitry Smirnov, 05/09/2014 12:48 AM
osdmap.32550 (15.6 KB) osdmap.32550 Dmitry Smirnov, 05/09/2014 12:51 AM
patch (563 Bytes) patch Ilya Dryomov, 05/09/2014 08:11 AM
crushmap (1.48 KB) crushmap Ruslan Usifov, 02/24/2016 11:26 PM
osdmap (25.1 KB) osdmap Ruslan Usifov, 02/24/2016 11:27 PM
Actions
Copy link
 #1

Updated by Ian Colle almost 10 years ago

  • Assignee set to Josh Durgin
  • Priority changed from Normal to Urgent
Actions
Copy link
 #2

Updated by Josh Durgin almost 10 years ago

RBD devices should never report I/O errors. Generally they block and retry instead of reporting an error to the upper layers. libceph/krbd may still be letting some errors through though, which would be a bug. -6 is EAGAIN, which the osd returns for a misdirected op (which means there may be a bug in the kernel client's sending, or the osd's interpretation, of the crush map). If you've enabled primary affinity, it would be: http://tracker.ceph.com/issues/7954 . Otherwise, could you attach your crushmap (ceph osd getcrushmap -o crushmap) and osdmap (ceph osd getmap -o osdmap)?

Actions
Copy link
 #3

Updated by Dmitry Smirnov almost 10 years ago

Perhaps it's too late to provide maps -- I had RBD read errors on three clients while cluster was severely degraded (3 or 4 OSDs lost but enough replicas around).
It has recovered ever since; I've added new OSDs etc.
I use simplest cluster topology: 11 OSDs on 5 servers (under "root default", no crush customisations).

Besides Linux-3.14.1 was crashing on BTRFS snapshots and causing loss of OSDs so
I retreated back to Linux-3.13.10 and so far hadn't seen read errors on RBD devices.

I'm not sure what "primary affinity" is and I think this issue is different from #7954 as I couldn't find "misdirected client" in logs...

Thanks.

Actions
Copy link
 #4

Updated by Josh Durgin almost 10 years ago

If you can reproduce, it'd be useful to get osd logs with debug osd = 20, debug ms = 1 when the I/O errors occur. Did you have any cache pools or erasure coded pools when the errors occurred? There were some crush changes in the kernel client to accomodate those in 3.14 and some fixes that will go into 3.15 for those cases.

Actions
Copy link
 #5

Updated by Dmitry Smirnov almost 10 years ago

Understood, I'll report debug logs if it happen again. I only have simplest replicated 'rbd' pool. Read errors occurred when cluster was severely degraded, that's pretty much all I can say... Thank you.

Actions
Copy link Download all files
 #6

Updated by Dmitry Smirnov almost 10 years ago

Damn it just happened again, roughly at the same time when just one OSD went down on healthy cluster.

Linux-3.14.2: 

May  3 08:41:18 deblab kernel: [70680.187835] rbd: rbd0: write 7000 at 180de10000 (210000)
May  3 08:41:18 deblab kernel: [70680.187835] 
May  3 08:41:18 deblab kernel: [70680.187843] rbd: rbd0:   result -6 xferred 7000
May  3 08:41:18 deblab kernel: [70680.187843] 
May  3 08:41:18 deblab kernel: [70680.187854] EXT4-fs warning (device rbd0): ext4_end_bio:317: I/O error writing to inode 6293383 (offset 0 size 28672 starting block 25222679)
May  3 08:41:18 deblab kernel: [70680.187858] buffer_io_error: 73 callbacks suppressed
May  3 08:41:19 deblab kernel: [70680.200812] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.200812] 
May  3 08:41:19 deblab kernel: [70680.200818] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.200818] 
May  3 08:41:19 deblab kernel: [70680.202791] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.202791] 
May  3 08:41:19 deblab kernel: [70680.202797] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.202797] 
May  3 08:41:19 deblab kernel: [70680.215985] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.215985] 
May  3 08:41:19 deblab kernel: [70680.215992] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.215992] 
May  3 08:41:19 deblab kernel: [70680.222145] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.222145] 
May  3 08:41:19 deblab kernel: [70680.222150] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.222150] 
May  3 08:41:19 deblab kernel: [70680.234530] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.234530] 
May  3 08:41:19 deblab kernel: [70680.234536] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.234536] 
May  3 08:41:19 deblab kernel: [70680.236383] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.236383] 
May  3 08:41:19 deblab kernel: [70680.236388] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.236388] 
May  3 08:41:19 deblab kernel: [70680.249980] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.249980] 
May  3 08:41:19 deblab kernel: [70680.249986] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.249986] 
May  3 08:41:19 deblab kernel: [70680.253400] rbd: rbd0: read 1000 at 180ddf1000 (1f1000)
May  3 08:41:19 deblab kernel: [70680.253400] 
May  3 08:41:19 deblab kernel: [70680.253405] rbd: rbd0:   result -6 xferred 1000
May  3 08:41:19 deblab kernel: [70680.253405] 
May  3 08:41:19 deblab kernel: [70680.499972] rbd: rbd0: write 7000 at 180de17000 (217000)
May  3 08:41:19 deblab kernel: [70680.499972] 
May  3 08:41:19 deblab kernel: [70680.499979] rbd: rbd0:   result -6 xferred 7000
May  3 08:41:19 deblab kernel: [70680.499979] 
May  3 08:41:19 deblab kernel: [70680.499988] EXT4-fs warning (device rbd0): ext4_end_bio:317: I/O error writing to inode 6293346 (offset 0 size 28672 starting block 25222686)

Warnings from Ceph (much more of such warnings were logged):

2014-05-03 08:34:02.678437 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1288324 pg 2.9e5d4055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:35:16.747708 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1288702 pg 2.9e5d4055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:35:16.747776 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1288703 pg 2.9e5d4055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:36:01.946549 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1288883 pg 2.9e5d4055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:17.531428 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290169 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:17.532158 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290170 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:17.544823 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290171 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.004019 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290260 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.016986 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290261 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.019003 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290262 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.032173 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290264 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.038421 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290298 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.050713 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290299 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.052624 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290300 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.066101 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290301 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.069615 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290302 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154 
2014-05-03 08:41:19.316120 osd.3 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1290303 pg 2.edd52055 to osd.3 not [0,3,6,9] in e29154/29154

Attaching maps...

Actions
Copy link
 #7

Updated by Dmitry Smirnov almost 10 years ago

Forgot to mention that shortly before error appear I did

ceph osd reweight-by-utilization 150
which changed 'reweight' value of OSD.7 from "1" to "0.6408" and re-balancing began.
Some time later I found OSD.4 down and kernel RBD clients reporting I/O errors all over the place.

I'm afraid I may have lost some data as file systems on RBD are inconsistent...

Actions
Copy link
 #8

Updated by Sage Weil almost 10 years ago

  • Subject changed from 0.80~rc1: RBD read errors to 0.80~rc1: RBD read errors (ENXIO)
Actions
Copy link
 #9

Updated by Dmitry Smirnov almost 10 years ago

A little update: `badblocks` show multiple "bad blocks" or RBD devices.
Ceph logs the following on every failed read:

2014-05-03 10:39:29.083984 osd.5 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1327425 pg 2.1afad8b2 to osd.5 not [8,5,6,9] in e29407/29407 
2014-05-03 10:39:29.084477 osd.5 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1327426 pg 2.1afad8b2 to osd.5 not [8,5,6,9] in e29407/29407 
2014-05-03 10:39:40.072685 osd.5 [WRN] client.2033685 192.168.0.7:0/2319955223 misdirected client.2033685.1:195699 pg 2.df26c535 to osd.5 not [9,5,6,8] in e29409/29409

Restarting all monitors have no effect.

Actions
Copy link
 #10

Updated by Sage Weil almost 10 years ago

Can you try setting hte osd weights all to 1.0? (ceph osd reweight N 1) My best guess is that the bug is in that code as you were doing reweight-by-utilization recently, and our test coverage there is a litle weeak

Actions
Copy link
 #11

Updated by Dmitry Smirnov almost 10 years ago

That was the first thin I did (setting weight of OSD.7 back to 1) but it didn't help.
All OSDs have reweight value 1 but errors are still there.
The only thing that seems to help is "ceph osd out 5".
I've taken OSD.5 down and checked the integrity of its file system (btrfs). HDD is healthy, there are no problems detected whatsoever.
Errors reappear as soon as I started it again and set as "in".
`badblocks` can read from RBD device only while OSD.5 is "out". Why could that be?

Actions
Copy link
 #12

Updated by Dmitry Smirnov almost 10 years ago

I re-built OSD.4 from scratch and re-added it.
While data is flowing back to OSD.4 I run `badblocks` and found read errors.

Ceph reporting "misdirected" errors on OSD.4 (!):

2014-05-03 11:50:41.387532 osd.4 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1478707 pg 2.c012255 to osd.4 not [2,4,9] in e29634/29634 
2014-05-03 11:50:41.387893 osd.4 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1478708 pg 2.c012255 to osd.4 not [2,4,9] in e29634/29634 
2014-05-03 11:50:41.388263 osd.4 [WRN] client.2037836 192.168.0.204:0/4029990440 misdirected client.2037836.1:1478709 pg 2.c012255 to osd.4 not [2,4,9] in e29634/29634

Any ideas? I'd very much like to restore cluster back to usable state...

Actions
Copy link
 #13

Updated by Dmitry Smirnov almost 10 years ago

After recovery to "active+clean" cluster became operational again. I/O errors on RBD devices are gone for now.
Could it be that recovery from "reweight" took time?

Actions
Copy link
 #14

Updated by Dmitry Smirnov almost 10 years ago

I had another incident of "misdirected client" during recovery after replacement of another OSD...

All pools were created in 0.72.2 so they were not using flag "hashpspool".
I'm going to set this flag to see if it make any difference...

Also I recall that I moved one OSD (I can't recall which one) between hosts some days before problem manifested for the first time.

Actions
Copy link
 #15

Updated by Ilya Dryomov almost 10 years ago

Can you build your own debug kernels? If so, I'll come up with a debug
patch to at least try to isolate the problem.

Actions
Copy link
 #16

Updated by Dmitry Smirnov almost 10 years ago

In theory I could but I have so little time for this that I can't promise anything...
Let's see if issue comes back again on "hashpspool" pools...

Actually I might also try to build DKMS version of ceph-client for Linux_3.14.
If successful I hope it would be much easier to debug unless your patches are going to touch mainline kernel...

Actions
Copy link
 #17

Updated by Ilya Dryomov almost 10 years ago

No, only rbd.ko and libceph.ko.

Actions
Copy link
 #18

Updated by Ilya Dryomov almost 10 years ago

Dmitry, a patch in #8300 should fix this.

Actions
Copy link
 #19

Updated by Sage Weil almost 10 years ago

  • Status changed from New to 15
Actions
Copy link
 #20

Updated by Dmitry Smirnov almost 10 years ago

I built DKMS modules [libceph,ceph,rbd] based on HEAD ("ceph: reserve caps for file layout/lock MDS requests") of "for-linus" branch of "ceph-client" repository.
I'm running 'em on Linux-3.14 and I just had this problem again.

Patch from #8300 is included and it does NOT fix this problem.
Ilya, let's try your debug patches. This problem is killing me...

Thanks.

Actions
Copy link
 #21

Updated by Ilya Dryomov almost 10 years ago

Can you provide the latest batch of [WRN] misdirected client errors and the corresponding osdmaps?

E.g. for ... in e29634/29635,

ceph osd getmap 29634 -o ~/osdmap.29634
ceph osd getmap 29635 -o ~/osdmap.29635

Actions
Copy link
 #22

Updated by Dmitry Smirnov almost 10 years ago 

2014-05-09 17:16:47.795595 osd.11 [WRN] client.2827502 192.168.0.2:0/4166584294 misdirected client.2827502.1:55573 pg 2.303172d6 to osd.11 in e32514, client e32513 pg 2.56 features 2990539024962

There will be more...

Actions
Copy link
 #23

Updated by Ilya Dryomov almost 10 years ago

ceph osd getmap 32513 -o ~/osdmap.32513?

Actions
Copy link
 #24

Updated by Dmitry Smirnov almost 10 years ago

Ilya Dryomov wrote:

ceph osd getmap 32513 -o ~/osdmap.32513?

I see, you need both maps... This one taken just now I hope it's not too late...

Actions
Copy link
 #25

Updated by Dmitry Smirnov almost 10 years ago 

2014-05-09 17:50:47.442004 osd.11 [WRN] client.2827502 192.168.0.2:0/4166584294 misdirected client.2827502.1:106137 pg 2.303172d6 to osd.11 in e32550, client e32550 pg 2.56 features 2990539024962

It's the same "e32550" twice...

Actions
Copy link
 #26

Updated by Ilya Dryomov almost 10 years ago

OK, I think I have enough to chew on for now. I'll need a few hours to process this.

Actions
Copy link
 #27

Updated by Ilya Dryomov almost 10 years ago

  • File patch patch added
  • Status changed from 15 to 12
  • Assignee changed from Josh Durgin to Ilya Dryomov

This patch fixes it for me, but I want you to confirm.

Actions
Copy link
 #28

Updated by Dmitry Smirnov almost 10 years ago

Fantastic, Ilya, your patch seems to fix the issue.
For about ~30 min. I couldn't reproduce the problem while usually I need just about 1...2 min. for that.
Please allow me to test longer -- I have 4 hosts and 6 RBD devices affected so I shall deploy fix and report back to you soon.

I can't thank you enough for this very important fix.
Thank you, thank you. :)

Actions
Copy link
 #29

Updated by Ilya Dryomov almost 10 years ago

Hi Dmitry,

How did your tests go?

Actions
Copy link
 #30

Updated by Dmitry Smirnov almost 10 years ago

  • Project changed from rbd to Linux kernel client
  • Category set to rbd

So far so good, I think we can declare this bug as fixed.

Please backport to 3.14 if possible. Thanks.

P.S. Hah, Ilya, I got "The issue has been updated by an other user while you were editing it" so we were editing it at the same time. :)

Actions
Copy link
 #31

Updated by Ilya Dryomov almost 10 years ago

  • Status changed from 12 to Resolved
[PATCH] crush: decode and initialize chooseleaf_vary_r
Actions
Copy link
 #32

Updated by Ruslan Usifov about 8 years ago

Perhaps we have simular problems when cluster rebalance happens(we change placement groups count).

We run on

Linux server4 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

With Infernalis ceph release.

In our syslog we have follow messages:

Feb 24 22:48:05 server4 kernel: [1233872.246524] rbd: rbd0: read 1000 at 36803e0000 (3e0000)
Feb 24 22:48:05 server4 kernel: [1233872.246531] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.246534] blk_update_request: I/O error, dev rbd0, sector 457187072
Feb 24 22:48:05 server4 kernel: [1233872.251958] rbd: rbd0: read 1000 at 36803e2000 (3e2000)
Feb 24 22:48:05 server4 kernel: [1233872.251960] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.251961] blk_update_request: I/O error, dev rbd0, sector 457187088
Feb 24 22:48:05 server4 kernel: [1233872.257317] rbd: rbd0: read 1000 at 36803e3000 (3e3000)
Feb 24 22:48:05 server4 kernel: [1233872.257319] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.257323] blk_update_request: I/O error, dev rbd0, sector 457187096
Feb 24 22:48:05 server4 kernel: [1233872.262639] rbd: rbd0: read 1000 at 36803e4000 (3e4000)
Feb 24 22:48:05 server4 kernel: [1233872.262641] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.262643] blk_update_request: I/O error, dev rbd0, sector 457187104
Feb 24 22:48:05 server4 kernel: [1233872.273156] rbd: rbd0: read 1000 at 36803e6000 (3e6000)
Feb 24 22:48:05 server4 kernel: [1233872.273158] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.278388] rbd: rbd0: read 1000 at 36803e7000 (3e7000)
Feb 24 22:48:05 server4 kernel: [1233872.278389] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.278396] blk_update_request: I/O error, dev rbd0, sector 457187128
Feb 24 22:48:05 server4 kernel: [1233872.283422] rbd: rbd0: read 1000 at 36803e8000 (3e8000)
Feb 24 22:48:05 server4 kernel: [1233872.283423] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.283425] blk_update_request: I/O error, dev rbd0, sector 457187136
Feb 24 22:48:05 server4 kernel: [1233872.288427] rbd: rbd0: read 1000 at 36803e9000 (3e9000)
Feb 24 22:48:05 server4 kernel: [1233872.288429] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.288430] blk_update_request: I/O error, dev rbd0, sector 457187144
Feb 24 22:48:05 server4 kernel: [1233872.293197] rbd: rbd0: read 1000 at 36803ea000 (3ea000)
Feb 24 22:48:05 server4 kernel: [1233872.293199] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.293200] blk_update_request: I/O error, dev rbd0, sector 457187152
Feb 24 22:48:05 server4 kernel: [1233872.297930] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.297940] rbd: rbd0: read 1000 at 36803f1000 (3f1000)
Feb 24 22:48:05 server4 kernel: [1233872.297941] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.297950] rbd: rbd0: read 1000 at 36803f2000 (3f2000)
Feb 24 22:48:05 server4 kernel: [1233872.297952] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.297961] rbd: rbd0: read 1000 at 36803f3000 (3f3000)
Feb 24 22:48:05 server4 kernel: [1233872.297962] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.297971] rbd: rbd0: read 1000 at 36803f4000 (3f4000)
Feb 24 22:48:05 server4 kernel: [1233872.297972] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:05 server4 kernel: [1233872.297981] rbd: rbd0: read 1000 at 36803f5000 (3f5000)
Feb 24 22:48:05 server4 kernel: [1233872.297983] rbd: rbd0:   result -6 xferred 1000
Feb 24 22:48:11 server4 kernel: [1233878.367247] rbd: rbd0: read 1000 at 36803e0000 (3e0000)
Feb 24 22:48:11 server4 kernel: [1233878.367256] blk_update_request: 22 callbacks suppressed
Feb 24 22:48:11 server4 kernel: [1233878.381653] rbd: rbd0: read 1000 at 36803e4000 (3e4000)
Actions
Copy link Download all files
 #33

Updated by Ruslan Usifov about 8 years ago

Our crash, and osd maps in attachments. Also we see that this happens alltime when rebalance happens. In our cluster rebalance happens 4 times, and all that 4 time we see rbd IO errors

Actions
Copy link
 #34

Updated by Ilya Dryomov about 8 years ago

Hi Ruslan,

The issue this ticket is for is fixed. Please open a new ticket.

Is there anything else in dmesg? What is general state of the cluster, any warnings in ceph -s, osd logs?

Do you use rbd snapshots and/or clones?

How exactly are you increasing the number of PGs (commands you are issuing)? Are those errors transient or do they persist?

Can you reproduce with "debug ms = 1" set on the OSDs?

Actions
Copy link
 #35

Updated by Ruslan Usifov about 8 years ago

Hello

We create new issue http://tracker.ceph.com/issues/14901

We increase pg_count from 512 to 1024, and see rbd errors at first time. Nextime one of the osd in our cluster was fail due HDD failure which triggered the procedure of ceph rebalance, and we see rbd IO errors again. And at final step yesterday we aditionly add 9 new osd, and rebalance heppens again, and as a result we see rbd IO errors again. To fix rbd and filesistem failure we umount filesistem, umap rbd, then map rbd and mount file system

We doesn't use snapshots and/or clones. Our rbd image created with simple command

```
rbd create rbd/backup
```

Actions
Copy link

Also available in: Atom PDF