Project

General

Profile

Actions
Copy link

Bug #40775

closed

/src/include/xlist.h: 77: FAILED assert(_size == 0)

Added by Xiaoxi Chen almost 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Xiaoxi Chen
Category:
-
Target version:
Ceph - v15.0.0
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
nautilus,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v13.2.5
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
crash
Pull request ID:
29136
Crash signature (v1):
Crash signature (v2):

Description

It seems like we handle the inode ref wrongly? the number looks like overflow.

-12> 2019-07-13 00:49:44.582 7ffbc5adf700  1 client.14834497 WARNING: ll_forget on 0x1000239aa9b 1, which only has ll_ref=-2147483646
   -11> 2019-07-13 00:49:44.582 7ffbc42dc700  3 client.14834497 ll_readlink 0x1000239aa9b.head
   -10> 2019-07-13 00:49:44.582 7ffbc42dc700  3 client.14834497 ll_readlink 0x1000239aa9b.head = 23
    -9> 2019-07-13 00:49:44.582 7ffbc42dc700  1 client.14834497 WARNING: ll_forget on 0x1000239aa9b 1, which only has ll_ref=0

  -40> 2019-07-13 00:49:44.582 7ffbc42dc700  3 client.14834497 ll_readlink 0x1000239aa9b.head
   -39> 2019-07-13 00:49:44.582 7ffbc42dc700  3 client.14834497 ll_readlink 0x1000239aa9b.head = 23
   -38> 2019-07-13 00:49:44.582 7ffbcdaef700  3 client.14834497 ll_lookup 0x10002372a42.head oracle
   -37> 2019-07-13 00:49:44.582 7ffbcdaef700  3 client.14834497 may_lookup 0x55e37fd19080 = 0
   -36> 2019-07-13 00:49:44.582 7ffbcdaef700  3 client.14834497 ll_lookup 0x10002372a42.head oracle -> 0 (1000239aa9b)
   -35> 2019-07-13 00:49:44.582 7ffbcbaeb700  3 client.14834497 ll_getattr 0x1000239aa8e.head = 0
   -34> 2019-07-13 00:49:44.582 7ffbcd2ee700  3 client.14834497 ll_lookup 0x10002372a42.head oracle
   -33> 2019-07-13 00:49:44.582 7ffbcd2ee700  3 client.14834497 may_lookup 0x55e37fd19080 = 0
   -32> 2019-07-13 00:49:44.582 7ffbcd2ee700  3 client.14834497 ll_lookup 0x10002372a42.head oracle -> 0 (1000239aa9b)
   -31> 2019-07-13 00:49:44.582 7ffbce2f0700  3 client.14834497 ll_readlink 0x1000239aa9b.head
   -30> 2019-07-13 00:49:44.582 7ffbce2f0700  3 client.14834497 ll_readlink 0x1000239aa9b.head = 23
   -29> 2019-07-13 00:49:44.582 7ffbcaae9700  3 client.14834497 ll_lookup 0x10002372a42.head oracle
   -28> 2019-07-13 00:49:44.582 7ffbcaae9700  3 client.14834497 may_lookup 0x55e37fd19080 = 0
   -27> 2019-07-13 00:49:44.582 7ffbcaae9700  3 client.14834497 ll_lookup 0x10002372a42.head oracle -> 0 (1000239aa9b)
   -26> 2019-07-13 00:49:44.582 7ffbcdaef700  3 client.14834497 ll_readlink 0x1000239aa9b.head
   -25> 2019-07-13 00:49:44.582 7ffbcdaef700  3 client.14834497 ll_readlink 0x1000239aa9b.head = 23
   -24> 2019-07-13 00:49:44.582 7ffbceaf1700  3 client.14834497 ll_getattr 0x1000239aa8e.head = 0
   -23> 2019-07-13 00:49:44.582 7ffbcc2ec700  3 client.14834497 ll_lookup 0x10002372a42.head oracle
   -22> 2019-07-13 00:49:44.582 7ffbcc2ec700  3 client.14834497 may_lookup 0x55e37fd19080 = 0
   -21> 2019-07-13 00:49:44.582 7ffbcc2ec700  3 client.14834497 ll_lookup 0x10002372a42.head oracle -> 0 (1000239aa9b)
   -20> 2019-07-13 00:49:44.582 7ffbccaed700  3 client.14834497 ll_lookup 0x10002372a42.head oracle
   -19> 2019-07-13 00:49:44.582 7ffbccaed700  3 client.14834497 may_lookup 0x55e37fd19080 = 0
   -18> 2019-07-13 00:49:44.582 7ffbccaed700  3 client.14834497 ll_lookup 0x10002372a42.head oracle -> 0 (1000239aa9b)
   -17> 2019-07-13 00:49:44.582 7ffbcfaf3700  3 client.14834497 ll_lookup 0x10002372a42.head oracle
   -16> 2019-07-13 00:49:44.582 7ffbcfaf3700  3 client.14834497 may_lookup 0x55e37fd19080 = 0
   -15> 2019-07-13 00:49:44.582 7ffbcfaf3700  3 client.14834497 ll_lookup 0x10002372a42.head oracle -> 0 (1000239aa9b)
   -14> 2019-07-13 00:49:44.582 7ffbc5adf700  3 client.14834497 ll_readlink 0x1000239aa9b.head
   -13> 2019-07-13 00:49:44.582 7ffbc5adf700  3 client.14834497 ll_readlink 0x1000239aa9b.head = 23
   -12> 2019-07-13 00:49:44.582 7ffbc5adf700  1 client.14834497 WARNING: ll_forget on 0x1000239aa9b 1, which only has ll_ref=-2147483646
   -11> 2019-07-13 00:49:44.582 7ffbc42dc700  3 client.14834497 ll_readlink 0x1000239aa9b.head
   -10> 2019-07-13 00:49:44.582 7ffbc42dc700  3 client.14834497 ll_readlink 0x1000239aa9b.head = 23
    -9> 2019-07-13 00:49:44.582 7ffbc42dc700  1 client.14834497 WARNING: ll_forget on 0x1000239aa9b 1, which only has ll_ref=0
    -8> 2019-07-13 00:49:44.582 7ffbd8304700  1 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e390a80a00 :-1 s=STATE_OPEN pgs=534570 cs=1 l=1).read_bulk peer close file descriptor 2
    -7> 2019-07-13 00:49:44.582 7ffbd8304700  1 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e390a80a00 :-1 s=STATE_OPEN pgs=534570 cs=1 l=1).read_until read failed
    -6> 2019-07-13 00:49:44.582 7ffbd8304700  1 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e390a80a00 :-1 s=STATE_OPEN pgs=534570 cs=1 l=1).process read tag failed
    -5> 2019-07-13 00:49:44.582 7ffbd8304700  1 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e390a80a00 :-1 s=STATE_OPEN pgs=534570 cs=1 l=1).fault on lossy channel, failing
    -4> 2019-07-13 00:49:44.582 7ffbd8304700  2 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e390a80a00 :-1 s=STATE_OPEN pgs=534570 cs=1 l=1)._stop
    -3> 2019-07-13 00:49:44.582 7ffbd42fc700  1 client.14834497.objecter ms_handle_reset 0x55e390a80a00 session 0x55e380c6e9a0 osd.262
    -2> 2019-07-13 00:49:44.582 7ffbd42fc700  1 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e390a80a00 :-1 s=STATE_CLOSED pgs=534570 cs=1 l=1).mark_down
    -1> 2019-07-13 00:49:44.582 7ffbd8304700  2 -- 10.20.75.48:0/2528003948 >> 10.75.6.20:6809/924194 conn(0x55e38ca80400 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
     0> 2019-07-13 00:49:44.582 7ffbc42dc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/include/xlist.h: In function 'xlist<T>::~xlist() [with T = Dentry*]' thread 7ffbc42dc700 time 2019-07-13 00:49:44.586521
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/include/xlist.h: 77: FAILED assert(_size == 0)

 ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7ffbdff2afbf]
 2: (()+0x26d187) [0x7ffbdff2b187]
 3: (Inode::~Inode()+0x839) [0x55e37e9773a9]
 4: (Client::put_inode(Inode*, int)+0x1a0) [0x55e37e8f8a50]
 5: (Client::_ll_put(Inode*, int)+0xdb) [0x55e37e902b9b]
 6: (Client::_ll_forget(Inode*, int)+0x1b9) [0x55e37e902f89]
 7: (Client::ll_forget(Inode*, int)+0x32) [0x55e37e9031a2]
 8: (()+0x545c8) [0x55e37e8d85c8]
 9: (()+0x16b6b) [0x7ffbe8951b6b]
 10: (()+0x13401) [0x7ffbe894e401]
 11: (()+0x7dd5) [0x7ffbddf61dd5]
 12: (clone()+0x6d) [0x7ffbdce3aead]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-client.etl.log
--- end dump of recent events ---
2019-07-13 00:49:44.606 7ffbc42dc700 -1 *** Caught signal (Aborted) **
 in thread 7ffbc42dc700 thread_name:ceph-fuse

Files

log.tar.gz (62.5 KB) log.tar.gz Xiaoxi Chen, 07/13/2019 11:34 PM

Related issues 2 (0 open — 2 closed)

Copied to CephFS - Backport #40874: nautilus: /src/include/xlist.h: 77: FAILED assert(_size == 0)ResolvedXiaoxi ChenActions
Copied to CephFS - Backport #40875: mimic: /src/include/xlist.h: 77: FAILED assert(_size == 0)ResolvedXiaoxi ChenActions
Actions
Copy link

Also available in: Atom PDF