Actions
Bug #15946
closedxfs layering on rbd block device was corrupted
Added by Shinobu Kinjo almost 8 years ago. Updated over 4 years ago.
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
krbd
Crash signature (v1):
Crash signature (v2):
Description
Just to tracking purpose.
Forcefully unmounted xfs file system was never been recovered.
During FIO, when I interrupted this process and unmounted filesystem, filesystem was corrupted. And it's never been recovered, even I did `xfs_repair -L`.
[root@cluster-stack01 ~]# uname -a
Linux cluster-stack01.fullstack.go 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@cluster-stack01 ~]# ls /mnt/rbd/testing01/
ls: reading directory /mnt/rbd/testing01/: Structure needs cleaning
Files
| xfs_repair_-L.txt (231 KB) xfs_repair_-L.txt | Shinobu Kinjo, 05/20/2016 05:47 AM |
Updated by Shinobu Kinjo almost 8 years ago
- Call Trace*
May 19 04:27:29 cluster-stack01 kernel: Key type ceph registered
May 19 04:27:29 cluster-stack01 kernel: libceph: loaded (mon/osd proto 15/24)
May 19 04:27:29 cluster-stack01 kernel: rbd: loaded (major 252)
May 19 04:27:29 cluster-stack01 kernel: libceph: client24177 fsid 97692aec-1c52-4022-a01e-0127eb554a5e
May 19 04:27:29 cluster-stack01 kernel: libceph: mon0 172.16.0.2:6789 session established
May 19 04:27:29 cluster-stack01 systemd-udevd: failed to execute '/usr/bin/ceph-rbdnamer' '/usr/bin/ceph-rbdnamer rbd0': No such file or directory
May 19 04:27:29 cluster-stack01 kernel: rbd0: unknown partition table
May 19 04:27:29 cluster-stack01 kernel: rbd: rbd0: added with size 0x500000000
May 19 04:27:31 cluster-stack01 kernel: XFS (rbd0): Mounting V4 Filesystem
May 19 04:27:34 cluster-stack01 kernel: XFS (rbd0): Starting recovery (logdev: internal)
May 19 04:27:34 cluster-stack01 kernel: XFS (rbd0): Ending recovery (logdev: internal)
May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): corrupt dinode 16384, (btree extents).
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75000: 42 4d 41 50 00 00 00 10 ff ff ff ff ff ff ff ff BMAP............
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75010: ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ................
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75020: 00 00 00 00 90 00 00 01 00 00 00 00 00 00 02 00 ................
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75030: 00 00 00 00 90 40 00 05 00 00 00 00 00 00 0c 00 .........................
May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Internal error xfs_bmap_read_extents(1) at line 1317 of file fs/xfs/libxfs/xfs_bmap.c. Caller xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: CPU: 1 PID: 2545 Comm: ls Not tainted 3.10.0-327.18.2.el7.x86_64 #1
May 19 04:27:41 cluster-stack01 kernel: Hardware name: LENOVO 20ARS1BG00/20ARS1BG00, BIOS GJET79WW (2.29 ) 09/03/2014
May 19 04:27:41 cluster-stack01 kernel: ffff88030c179800 0000000029229d86 ffff880034e879c0 ffffffff81635a0c
May 19 04:27:41 cluster-stack01 kernel: ffff880034e879d8 ffffffffa036ce5b ffffffffa035e495 ffff880034e87a10
May 19 04:27:41 cluster-stack01 kernel: ffffffffa036ceb5 0000052500002540 ffff880035f38000 ffff880035f38040
May 19 04:27:41 cluster-stack01 kernel: Call Trace:
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81635a0c>] dump_stack+0x19/0x1b
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ce5b>] xfs_error_report+0x3b/0x40 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ceb5>] xfs_corruption_error+0x55/0x80 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033d715>] xfs_bmap_read_extents+0x3c5/0x3d0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033e9bd>] xfs_bmapi_read+0x2cd/0x2f0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff8117059e>] ? __rmqueue+0xee/0x460
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81189a19>] ? zone_statistics+0x89/0xa0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa03491b9>] xfs_dabuf_map.constprop.16+0x229/0x390 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa034c36a>] xfs_da_reada_buf+0x4a/0xc0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa034fad7>] xfs_dir3_data_readahead+0x17/0x20 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036e2c2>] xfs_dir_open+0x52/0x70 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811dc367>] do_dentry_open+0x1a7/0x2e0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036e270>] ? xfs_file_mmap+0x40/0x40 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811dc599>] vfs_open+0x39/0x70
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811ebb8d>] do_last+0x1ed/0x1270
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff8116bbd5>] ? filemap_fault+0x215/0x410
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811ee8f2>] path_openat+0xc2/0x490
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f00bb>] do_filp_open+0x4b/0xb0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811fcc47>] ? __alloc_fd+0xa7/0x130
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811dda63>] do_sys_open+0xf3/0x1f0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811ddb94>] SyS_openat+0x14/0x20
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff816461c9>] system_call_fastpath+0x16/0x1b
May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Corruption detected. Unmount and run xfs_repair
May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): corrupt dinode 16384, (btree extents).
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75000: 42 4d 41 50 00 00 00 10 ff ff ff ff ff ff ff ff BMAP............
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75010: ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ................
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75020: 00 00 00 00 90 00 00 01 00 00 00 00 00 00 02 00 ................
May 19 04:27:41 cluster-stack01 kernel: ffff880035b75030: 00 00 00 00 90 40 00 05 00 00 00 00 00 00 0c 00 .....
May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Internal error xfs_bmap_read_extents(1) at line 1317 of file fs/xfs/libxfs/xfs_bmap.c. Caller xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: CPU: 1 PID: 2545 Comm: ls Not tainted 3.10.0-327.18.2.el7.x86_64 #1
May 19 04:27:41 cluster-stack01 kernel: Hardware name: LENOVO 20ARS1BG00/20ARS1BG00, BIOS GJET79WW (2.29 ) 09/03/2014
May 19 04:27:41 cluster-stack01 kernel: ffff88030c179800 0000000029229d86 ffff880034e87c50 ffffffff81635a0c
May 19 04:27:41 cluster-stack01 kernel: ffff880034e87c68 ffffffffa036ce5b ffffffffa035e495 ffff880034e87ca0
May 19 04:27:41 cluster-stack01 kernel: ffffffffa036ceb5 0000052500002540 ffff880035f38000 ffff880035f38040
May 19 04:27:41 cluster-stack01 kernel: Call Trace:
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81635a0c>] dump_stack+0x19/0x1b
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ce5b>] xfs_error_report+0x3b/0x40 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036ceb5>] xfs_corruption_error+0x55/0x80 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033d715>] xfs_bmap_read_extents+0x3c5/0x3d0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] ? xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f25d0>] ? fillonedir+0xe0/0xe0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa035e495>] xfs_iread_extents+0x75/0xd0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033d9f5>] xfs_bmap_last_extent+0x75/0x80 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa033db05>] xfs_bmap_last_offset+0x55/0xa0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff8116931b>] ? unlock_page+0x2b/0x30
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff81193181>] ? __do_fault+0x401/0x510
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa034d334>] xfs_dir2_isblock+0x34/0x80 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036c5f8>] xfs_readdir+0xb8/0x1a0 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f25d0>] ? fillonedir+0xe0/0xe0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffffa036e312>] xfs_file_readdir+0x32/0x40 [xfs]
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f24c0>] vfs_readdir+0xb0/0xe0
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff811f28e5>] SyS_getdents+0x95/0x120
May 19 04:27:41 cluster-stack01 kernel: [<ffffffff816461c9>] system_call_fastpath+0x16/0x1b
May 19 04:27:41 cluster-stack01 kernel: XFS (rbd0): Corruption detected. Unmount and run xfs_repair
Updated by Shinobu Kinjo almost 8 years ago
[root@cluster-stack01 ~]# !mount
mount /dev/rbd0 /mnt/rbd/testing01
[root@cluster-stack01 ~]# ls /mnt/rbd/testing01/
ls: reading directory /mnt/rbd/testing01/: Structure needs cleaning
Attached `xfs_repair -L /dev/rbd0`
Updated by Shinobu Kinjo almost 8 years ago
- File xfs_repair_-L.txt xfs_repair_-L.txt added
Updated by Ilya Dryomov about 5 years ago
- Category set to rbd
- Status changed from New to Need More Info
- Assignee set to Ilya Dryomov
Shinobu, IIRC from our IRC conversation, there weren't any additional details available beyond what's in this ticket.
Did you see it repeat itself, either on that kernel or newer kernels? If not, I'm going to close.
Updated by Ilya Dryomov over 4 years ago
- Status changed from Need More Info to Closed
An old kernel, no evidence that this is an rbd bug.
Actions