Project

General

Profile

Actions
Copy link

Bug #63259

open

mds: failed to store backtrace and force file system read-only

Added by Xiubo Li 7 months ago. Updated 4 months ago.

Status:
Fix Under Review
Priority:
High
Assignee:
Venky Shankar
Category:
Correctness/Safety
Target version:
Ceph - v19.0.0
% Done:

0%

Source:
Tags:
Backport:
quincy,reef
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
55421
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429549/

From teuthology file:

2023-10-17T00:51:05.801 INFO:tasks.workunit.client.0.smithi114.stderr:+ set -e
2023-10-17T00:51:05.802 INFO:tasks.workunit.client.0.smithi114.stderr:+ sudo rsync -av /tmp/multiple_rsync_payload.271406 payload.1
2023-10-17T00:51:05.855 INFO:tasks.workunit.client.0.smithi114.stdout:sending incremental file list
2023-10-17T00:51:05.865 INFO:tasks.workunit.client.0.smithi114.stderr:rsync: mkdir "/home/ubuntu/cephtest/mnt.0/client.0/tmp/payload.1" failed: Read-only file system (30)
2023-10-17T00:51:05.866 INFO:tasks.workunit.client.0.smithi114.stderr:rsync error: error in file IO (code 11) at main.c(664) [Receiver=3.1.3]
2023-10-17T00:51:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 11
2023-10-17T00:51:05.966 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0...

And from mds.b:

2023-10-17T00:50:12.583+0000 7fa035298700 10 mds.0.cache.ino(0x1000000c567) clear_dirty_parent
2023-10-17T00:50:12.583+0000 7fa035298700  1 mds.0.cache.ino(0x1000000c566) store backtrace error -2 v 36221
2023-10-17T00:50:12.583+0000 7fa035298700 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000c566 object, pool 2, errno -2
2023-10-17T00:50:12.583+0000 7fa035298700 -1 mds.0.14 unhandled write error (2) No such file or directory, force readonly...
2023-10-17T00:50:12.583+0000 7fa035298700  1 mds.0.cache force file system read-only
2023-10-17T00:50:12.583+0000 7fa035298700  0 log_channel(cluster) log [WRN] : force file system read-only
2023-10-17T00:50:12.583+0000 7fa035298700 10 mds.0.server force_clients_readonly
2023-10-17T00:50:12.583+0000 7fa035298700 10 mds.0.14 send_message_client client.15078 192.168.0.1:0/1906848847 client_session(force_ro) v5
2023-10-17T00:50:12.583+0000 7fa035298700  1 -- [v2:172.21.15.139:6838/1590276002,v1:172.21.15.139:6839/1590276002] --> 192.168.0.1:0/1906848847 -- client_session(force_ro) v5 -- 0x5630e2d58e00 con 0x5630ce089800
2023-10-17T00:50:12.583+0000 7fa035298700 10 mds.0.locker eval 3648 [inode 0x1000000c568 [...2,head] /volumes/_nogroup/sv_1/327a5477-e5c2-4ade-b24e-f477c29c079e/client.0/tmp/ auth v828 ap=1 DIRTYPARENT f() n(v0 1=0+1) (iauth excl) (inest lock) (ifile excl) (ixattr excl) (iversion lock) caps={15078=pAsxLsXsxFsx/-@1},l=15078 | request=0 dirfrag=0 caps=1 dirtyparent=1 dirty=0 authpin=1 0x5630dadb6580]
2023-10-17T00:50:12.583+0000 7fa035298700 10 mds.0.locker eval want loner: client.-1 but failed to set it
2023-10-17T00:50:12.583+0000 7fa035298700  7 mds.0.locker file_eval wanted= loner_wanted= other_wanted=  filelock=(ifile excl) on [inode 0x1000000c568 [...2,head] /volumes/_nogroup/sv_1/327a5477-e5c2-4ade-b24e-f477c29c079e/client.0/tmp/ auth v828 ap=1 DIRTYPARENT f() n(v0 1=0+1) (iauth excl) (inest lock) (ifile excl) (ixattr excl) (iversion lock) caps={15078=pAsxLsXsxFsx/-@1},l=15078(-1) | request=0 dirfrag=0 caps=1 dirtyparent=1 dirty=0 authpin=1 0x5630dadb6580]

It failed to store the backtrace and force the cephfs to be readonly.

Actions
Copy link

Also available in: Atom PDF