Bug #17236
closedMDS goes damaged on blacklist (failed to read JournalPointer: -108 ((108) Cannot send after transport endpoint shutdown)
0%
Description
OSD log:
2016-09-05 19:15:04.996890 7f3a92cea700 10 osd.1 pg_epoch: 9 pg[2.7( v 8'7692 (7'4629,8'7692] local-les=7 n=376 ec=6 les/c/f 7/7/0 6/6/6) [1,2] r=0 lpr=6 luod=8'7691 lua=8'7689 crt=8'7688 lcod 8'7690 mlcod 8'7688 active+clean] do_op 172.21.5.140:6808/12206 is blacklisted remote/mira037/log/ceph-osd.1.log.gz:2016-09-05 19:15:18.303488 7f3a92cea700 1 -- 172.21.5.140:6804/11520 >> 172.21.8.106:6808/19233 conn(0x55d18aee9000 sd=66 :6804 s=STATE_OPEN pgs=44 cs=1 l=1). == tx == 0x55d18c1e3e40 osd_op_reply(4 400.00000000 [read 0~0] v0'0 uv0 ack = -108 ((108) Cannot send after transport endpoint shutdown)) v7
There is at least one case here where r!=0 is being taken to mean damage, but we should be just respawning when seeing EBLACKLISTED. Almost everywhere else MDSIOContext handles this, but JournalPointer doesn't use it because it works outside of the MDS lock.
Updated by John Spray over 7 years ago
- Priority changed from Normal to Urgent
Promoting to urgent because I've seen this fail the fs suite more than once.
I looked further into logs this time and found that the original blacklist was happening because the MDS was re-using a PID.
It's respawning again and again with the same PID, because teuthology runs daemons with "-f" -- usually we get a new PID when daemonizing. I have no idea why we've only just starting seeing this issue.
Updated by John Spray over 7 years ago
- Status changed from New to Fix Under Review
Updated by John Spray over 7 years ago
As for the mystery of why this started happening, I think the MDS failures are triggered by #17308
Updated by John Spray over 7 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Loïc Dachary over 7 years ago
- Copied to Backport #17478: jewel: MDS goes damaged on blacklist (failed to read JournalPointer: -108 ((108) Cannot send after transport endpoint shutdown) added
Updated by Nathan Cutler about 7 years ago
- Status changed from Pending Backport to Resolved