Project

General

Profile

Bug #43407

mds crash after update to v14.2.5

Added by Marco Savoca 3 months ago. Updated 3 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
12/23/2019
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature:

Description

All MDS crashed and not able to restart after update from v14.2.4 to v14.2.5

systemctl status:

ceph-mds@ceph1.service - Ceph metadata server daemon
   Loaded: loaded (/lib/systemd/system/ceph-mds@.service; indirect; vendor preset: enabled)
   Active: failed (Result: signal) since Sun 2019-12-22 23:15:58 UTC; 3s ago
  Process: 7922 ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id ceph1 --setuser ceph --setgroup ceph (code=killed, signal=SEGV)
 Main PID: 7922 (code=killed, signal=SEGV)

Dez 22 23:15:58 ceph1.savoca.de systemd[1]: ceph-mds@ceph1.service: Service hold-off time over, scheduling restart.
Dez 22 23:15:58 ceph1.savoca.de systemd[1]: ceph-mds@ceph1.service: Scheduled restart job, restart counter is at 3.
Dez 22 23:15:58 ceph1.savoca.de systemd[1]: Stopped Ceph metadata server daemon.
Dez 22 23:15:58 ceph1.savoca.de systemd[1]: ceph-mds@ceph1.service: Start request repeated too quickly.
Dez 22 23:15:58 ceph1.savoca.de systemd[1]: ceph-mds@ceph1.service: Failed with result 'signal'.
Dez 22 23:15:58 ceph1.savoca.de systemd[1]: Failed to start Ceph metadata server daemon.

ceph -s:

cluster:
    id:     3ac4d7e5-98ca-4b75-bb88-02386caa5793
    health: HEALTH_WARN
            1 filesystem is degraded
            insufficient standby MDS daemons available
            85 daemons have recently crashed

  services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 22h)
    mgr: ceph2(active, since 22h), standbys: ceph3, ceph1
    mds: media:1/1 {0=ceph1=up:replay(laggy or crashed)}
    osd: 6 osds: 6 up (since 22h), 6 in (since 8w)

  data:
    pools:   12 pools, 181 pgs
    objects: 2.22M objects, 8.3 TiB
    usage:   12 TiB used, 9.4 TiB / 22 TiB avail
    pgs:     181 active+clean

Excerpt of ceph-mds.log:

2019-12-22 12:20:17.347 7fce967872c0  0 set uid:gid to 64045:64045 (ceph:ceph)
2019-12-22 12:20:17.347 7fce967872c0  0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7040
2019-12-22 12:20:17.347 7fce967872c0  0 pidfile_write: ignore empty --pid-file
2019-12-22 12:20:17.351 7fce848d7700  1 mds.ceph1 Updating MDS map to version 331 from mon.0
2019-12-22 12:20:21.811 7fce848d7700  1 mds.ceph1 Updating MDS map to version 332 from mon.0
2019-12-22 12:20:21.811 7fce848d7700  1 mds.ceph1 Map has assigned me to become a standby
2019-12-22 12:20:21.815 7fce848d7700  1 mds.ceph1 Updating MDS map to version 333 from mon.0
2019-12-22 12:20:21.815 7fce848d7700  1 mds.0.333 handle_mds_map i am now mds.0.333
2019-12-22 12:20:21.815 7fce848d7700  1 mds.0.333 handle_mds_map state change up:boot --> up:replay
2019-12-22 12:20:21.815 7fce848d7700  1 mds.0.333 replay_start
2019-12-22 12:20:21.815 7fce848d7700  1 mds.0.333  recovery set is 
2019-12-22 12:20:21.815 7fce848d7700  1 mds.0.333  waiting for osdmap 6044 (which blacklists prior instance)
2019-12-22 12:20:21.819 7fce7d8c9700  0 mds.0.cache creating system inode with ino:0x100
2019-12-22 12:20:21.819 7fce7d8c9700  0 mds.0.cache creating system inode with ino:0x1
2019-12-22 12:20:22.055 7fce7c8c7700 -1 log_channel(cluster) log [ERR] :  replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2)
2019-12-22 12:20:22.055 7fce7c8c7700  0 mds.0.journal journal subtrees: {0x1=[],0x100=[]}
2019-12-22 12:20:22.055 7fce7c8c7700  0 mds.0.journal journal ambig_subtrees: 
2019-12-22 12:20:22.059 7fce7c8c7700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fce7c8c7700 thread_name:md_log_replay

 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)
 1: (()+0x12890) [0x7fce8d26d890]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ceph-mds.ceph1.log View (100 KB) Marco Savoca, 12/22/2019 11:36 PM

ceph-mds.ceph2.log.gz (450 KB) Marco Savoca, 01/02/2020 09:18 PM

History

#1 Updated by Patrick Donnelly 3 months ago

Were you using multiple MDS before?

Can you increase MDS debugging:

ceph config set mds debug_mds 10

and restart the MDS daemon?

#2 Updated by Marco Savoca 3 months ago

Yes I had 3 filesystems (namespaces), one for every mds daemon, and the setup was working up to the update to v14.2.5.

After the crash I deleted 2 of the 3 filesystems.

Attached the new full ceph-mds.log (some private Informations about folders and files are deleted).

By the way: cephfs-journal-tool journal inspect states a clean journal (beside this strange error message)

2020-01-02 20:56:14.129 7ff4a4110700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol
Overall journal integrity: OK

With this summary
2020-01-02 20:56:46.956 7f69e9fb6700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol
Events by type:
NOOP: 0
SESSION: 6
SUBTREEMAP: 3
UPDATE: 1030
Errors: 0

#3 Updated by Zheng Yan 3 months ago

mds shows there are some ENoOp log events. This means some region of mds log was erased by cephfs-journal-tools. Why did you do that

1. Try back mds journal:
cephfs-journal-tool journal export backup.bin

2. recover journal events:
cephfs-journal-tool journal export backup.bin

3. truncate journal
cephfs-journal-tool journal reset

#4 Updated by Marco Savoca 3 months ago

2. recover journal events:
cephfs-journal-tool journal export backup.bin

Do you mean
cephfs-journal-tool event apply

or

cephfs-journal-tool event recover_dentries summary ?

The only manipulative action on the journal, that i did was following:

sudo cephfs-journal-tool event splice --type OPEN summary

bacause i supposed the open events to be problematic. I did a backup of the journal before.

But all this actions were after the degrading of the fs and the crash of the mds.

So the question remains:
Why the the mds daemons crahed after the update?

By the way: I dont really care about data loss, due to a solid backup strategy.

Do you think, that deleting the filesystem and the associated pools will solve the problem an let the daemons restart normally?

#5 Updated by Marco Savoca 3 months ago

Status update:
I have tried
cephfs-journal-tool event recover_dentries summary
followed with
cephfs-journal-tool journal reset

This did the job, so that I was able to restart the mds daemons.

Someone an idea why the journal got corrupted?

#6 Updated by Patrick Donnelly 3 months ago

  • Assignee set to Patrick Donnelly

#7 Updated by Patrick Donnelly 3 months ago

  • Status changed from New to Triaged
  • Assignee changed from Patrick Donnelly to Zheng Yan

#8 Updated by Zheng Yan 3 months ago

  • Assignee deleted (Zheng Yan)

The first ESubtreeMap in the journal was wrong. It should also contains dir 0x1

-4719> 2020-01-02 20:14:44.029 7f66a05f5700 10 mds.0.log _replay 808964827~520 / 812539516 2019-12-14 15:02:06.893542: ESubtreeMap 1 subtrees , 0 ambiguous [metablob 0x100, 1 dirs]

No idea now this can happens.

What steps you did when upgrading mds?

Also available in: Atom PDF