Project

General

Profile

Actions
Copy link

Bug #15946

closed

xfs layering on rbd block device was corrupted

Added by Shinobu Kinjo almost 8 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Ilya Dryomov
Category:
rbd
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
krbd
Crash signature (v1):
Crash signature (v2):

Description

Just to tracking purpose.
Forcefully unmounted xfs file system was never been recovered.

During FIO, when I interrupted this process and unmounted filesystem, filesystem was corrupted. And it's never been recovered, even I did `xfs_repair -L`.

[root@cluster-stack01 ~]# uname -a
Linux cluster-stack01.fullstack.go 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@cluster-stack01 ~]# ls /mnt/rbd/testing01/
ls: reading directory /mnt/rbd/testing01/: Structure needs cleaning

Files

xfs_repair_-L.txt (231 KB) xfs_repair_-L.txt Shinobu Kinjo, 05/20/2016 05:47 AM
Actions
Copy link
 #1

Updated by Shinobu Kinjo almost 8 years ago

  1. Call Trace*
    May 19 04:27:29 cluster-stack01 kernel: Key type ceph registered
    May 19 04:27:29 cluster-stack01 kernel: libceph: loaded (mon/osd proto 15/24)
    May 19 04:27:29 cluster-stack01 kernel: rbd: loaded (major 252)
    May 19 04:27:29 cluster-stack01 kernel: libceph: client24177 fsid 97692aec-1c52-4022-a01e-0127eb554a5e
    May 19 04:27:29 cluster-stack01 kernel: libceph: mon0 172.16.0.2:6789 session established
    May 19 04:27:29 cluster-stack01 systemd-udevd: failed to execute '/usr/bin/ceph-rbdnamer' '/usr/bin/ceph-rbdnamer rbd0': No such file or directory
    May 19 04:27:29 cluster-stack01 kernel: rbd0: unknown partition table
    May 19 04:27:29 cluster-stack01 kernel: rbd: rbd0: added with size 0x500000000
    May 19 04:27:31 cluster-stack01 kernel: XFS (rbd0): Mounting V4 Filesystem
    May 19 04:27:34 cluster-stack01 kernel: XFS (rbd0): Starting recovery (logdev: internal)
    May 19 04:27:34 cluster-stack01 kernel: XFS (rbd0): Ending recovery (logdev: internal)
    May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): corrupt dinode 16384, (btree extents).
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75000: 42 4d 41 50 00 00 00 10 ff ff ff ff ff ff ff ff BMAP............
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75010: ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ................
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75020: 00 00 00 00 90 00 00 01 00 00 00 00 00 00 02 00 ................
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75030: 00 00 00 00 90 40 00 05 00 00 00 00 00 00 0c 00 ...............
    May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Internal error xfs_bmap_read_extents(1) at line 1317 of file fs/xfs/libxfs/xfs_bmap.c. Caller xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: CPU: 1 PID: 2545 Comm: ls Not tainted 3.10.0-327.18.2.el7.x86_64 #1
    May 19 04:27:41 cluster-stack01 kernel: Hardware name: LENOVO 20ARS1BG00/20ARS1BG00, BIOS GJET79WW (2.29 ) 09/03/2014
    May 19 04:27:41 cluster-stack01 kernel: ffff88030c179800 0000000029229d86 ffff880034e879c0 ffffffff81635a0c
    May 19 04:27:41 cluster-stack01 kernel: ffff880034e879d8 ffffffffa036ce5b ffffffffa035e495 ffff880034e87a10
    May 19 04:27:41 cluster-stack01 kernel: ffffffffa036ceb5 0000052500002540 ffff880035f38000 ffff880035f38040
    May 19 04:27:41 cluster-stack01 kernel: Call Trace:
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81635a0c>] dump_stack+0x19/0x1b
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ce5b>] xfs_error_report+0x3b/0x40 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ceb5>] xfs_corruption_error+0x55/0x80 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033d715>] xfs_bmap_read_extents+0x3c5/0x3d0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033e9bd>] xfs_bmapi_read+0x2cd/0x2f0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff8117059e>] ? __rmqueue+0xee/0x460
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81189a19>] ? zone_statistics+0x89/0xa0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa03491b9>] xfs_dabuf_map.constprop.16+0x229/0x390 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa034c36a>] xfs_da_reada_buf+0x4a/0xc0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa034fad7>] xfs_dir3_data_readahead+0x17/0x20 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036e2c2>] xfs_dir_open+0x52/0x70 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811dc367>] do_dentry_open+0x1a7/0x2e0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036e270>] ? xfs_file_mmap+0x40/0x40 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811dc599>] vfs_open+0x39/0x70
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811ebb8d>] do_last+0x1ed/0x1270
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff8116bbd5>] ? filemap_fault+0x215/0x410
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811ee8f2>] path_openat+0xc2/0x490
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f00bb>] do_filp_open+0x4b/0xb0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811fcc47>] ? __alloc_fd+0xa7/0x130
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811dda63>] do_sys_open+0xf3/0x1f0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811ddb94>] SyS_openat+0x14/0x20
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff816461c9>] system_call_fastpath+0x16/0x1b
    May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Corruption detected. Unmount and run xfs_repair
    May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): corrupt dinode 16384, (btree extents).
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75000: 42 4d 41 50 00 00 00 10 ff ff ff ff ff ff ff ff BMAP............
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75010: ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ................
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75020: 00 00 00 00 90 00 00 01 00 00 00 00 00 00 02 00 ................
    May 19 04:27:41 cluster-stack01 kernel: ffff880035b75030: 00 00 00 00 90 40 00 05 00 00 00 00 00 00 0c 00 .....    ..........
    May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Internal error xfs_bmap_read_extents(1) at line 1317 of file fs/xfs/libxfs/xfs_bmap.c. Caller xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: CPU: 1 PID: 2545 Comm: ls Not tainted 3.10.0-327.18.2.el7.x86_64 #1
    May 19 04:27:41 cluster-stack01 kernel: Hardware name: LENOVO 20ARS1BG00/20ARS1BG00, BIOS GJET79WW (2.29 ) 09/03/2014
    May 19 04:27:41 cluster-stack01 kernel: ffff88030c179800 0000000029229d86 ffff880034e87c50 ffffffff81635a0c
    May 19 04:27:41 cluster-stack01 kernel: ffff880034e87c68 ffffffffa036ce5b ffffffffa035e495 ffff880034e87ca0
    May 19 04:27:41 cluster-stack01 kernel: ffffffffa036ceb5 0000052500002540 ffff880035f38000 ffff880035f38040
    May 19 04:27:41 cluster-stack01 kernel: Call Trace:
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81635a0c>] dump_stack+0x19/0x1b
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ce5b>] xfs_error_report+0x3b/0x40 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ceb5>] xfs_corruption_error+0x55/0x80 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033d715>] xfs_bmap_read_extents+0x3c5/0x3d0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f25d0>] ? fillonedir+0xe0/0xe0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] xfs_iread_extents+0x75/0xd0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033d9f5>] xfs_bmap_last_extent+0x75/0x80 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033db05>] xfs_bmap_last_offset+0x55/0xa0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff8116931b>] ? unlock_page+0x2b/0x30
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81193181>] ? __do_fault+0x401/0x510
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa034d334>] xfs_dir2_isblock+0x34/0x80 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036c5f8>] xfs_readdir+0xb8/0x1a0 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f25d0>] ? fillonedir+0xe0/0xe0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036e312>] xfs_file_readdir+0x32/0x40 [xfs]
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f24c0>] vfs_readdir+0xb0/0xe0
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f28e5>] SyS_getdents+0x95/0x120
    May 19 04:27:41 cluster-stack01 kernel: [<ffffffff816461c9>] system_call_fastpath+0x16/0x1b
    May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Corruption detected. Unmount and run xfs_repair
Actions
Copy link
 #2

Updated by Shinobu Kinjo almost 8 years ago

[root@cluster-stack01 ~]# !mount
mount /dev/rbd0 /mnt/rbd/testing01
[root@cluster-stack01 ~]# ls /mnt/rbd/testing01/
ls: reading directory /mnt/rbd/testing01/: Structure needs cleaning

Attached `xfs_repair -L /dev/rbd0`

Actions
Copy link
 #3

Updated by Shinobu Kinjo almost 8 years ago

Actions
Copy link
 #4

Updated by Ilya Dryomov about 5 years ago

  • Category set to rbd
  • Status changed from New to Need More Info
  • Assignee set to Ilya Dryomov

Shinobu, IIRC from our IRC conversation, there weren't any additional details available beyond what's in this ticket.

Did you see it repeat itself, either on that kernel or newer kernels? If not, I'm going to close.

Actions
Copy link
 #5

Updated by Ilya Dryomov over 4 years ago

  • Status changed from Need More Info to Closed

An old kernel, no evidence that this is an rbd bug.

Actions
Copy link

Also available in: Atom PDF