Ceph » CephFS

I still don't figure out the cause of the crash, infinite loop in iterate_session_caps(), BUG_ON(session->s_nr_caps > 0) or BUG_ON(!list_empty(&session->s_cap_flushing))? please upload ceph.ko

Zheng Yan wrote:

I still don't figure out that root cause of the crash, infinite loop in iterate_session_caps(), BUG_ON(session->s_nr_caps > 0) or BUG_ON(!list_empty(&session->s_cap_flushing))?

bah, i rebooted the machine. next time i'll gather more info from kdb. the dump.txt is above, but remove_session_caps+0x33 doesn't line up with the current kernel builds

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-07-12_01:01:16-kernel-master-testing-basic/63639$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 365b57b1317524bb0cdd15859a224ba1ab58d1d7
machine_type: plana
nuke-on-error: true
overrides:
  admin_socket:
    branch: master
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: cf8f16d7433b86b0bdfc192f719f3029f04996a6
  install:
    ceph:
      sha1: cf8f16d7433b86b0bdfc192f719f3029f04996a6
  s3tests:
    branch: master
  workunit:
    sha1: cf8f16d7433b86b0bdfc192f719f3029f04996a6
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph: null
- kclient: null
- workunit:
    clients:
      all:
      - suites/fsync-tester.sh

I need to know which line caused the crash. looks like it was triggered by one of the BUG_ONs in remove_session_caps. but I don't see any BUG_ON kernel message, so I'm confused.

Zheng Yan wrote:

I need to know which line caused the crash. looks like it was triggered by one of the BUG_ONs in remove_session_caps. but I don't see any BUG_ON kernel message, so I'm confused.

yeah strangely there isn't one. remove_session_caps+0x33/0x140 also isn't an exact match.. it'd be 0x19f93, but

static int remove_session_caps_cb(struct inode *inode, struct ceph_cap *cap,
                                  void *arg)
{
   19f74:       48 89 5d d8             mov    %rbx,-0x28(%rbp)
   19f78:       4c 89 65 e0             mov    %r12,-0x20(%rbp)
   19f7c:       48 89 fb                mov    %rdi,%rbx
   19f7f:       4c 89 6d e8             mov    %r13,-0x18(%rbp)
   19f83:       4c 89 75 f0             mov    %r14,-0x10(%rbp)
   19f87:       49 89 f4                mov    %rsi,%r12
   19f8a:       4c 89 7d f8             mov    %r15,-0x8(%rbp)
        struct ceph_inode_info *ci = ceph_inode(inode);
        int drop = 0;

        dout("removing cap %p, ci is %p, inode is %p\n",
   19f8e:       0f 85 f9 01 00 00       jne    1a18d <remove_session_caps_cb+0x22d>
        raw_spin_lock_init(&(_lock)->rlock);            \
} while (0)

static inline void spin_lock(spinlock_t *lock)
{
        raw_spin_lock(&lock->rlock);
   19f94:       4c 8d ab b0 fb ff ff    lea    -0x450(%rbx),%r13
   19f9b:       4c 89 ef                mov    %r13,%rdi
   19f9e:       e8 00 00 00 00          callq  19fa3 <remove_session_caps_cb+0x43>
                        19f9f: R_X86_64_PC32    _raw_spin_lock-0x4
             cap, ci, &ci->vfs_inode);
        spin_lock(&ci->i_ceph_lock);

registers:

[0]kdb> rd
ax: 0000000000000000  bx: ffff88022310f800  cx: 0000000000003332
dx: ffffffffa079bcf4  si: ffffffffa079bcf4  di: ffff88022310fe20
bp: ffff88020da1bb08  sp: ffff88020da1bae8  r8: 0000000000000002
r9: 0000000000000001  r10: 0000000000000000  r11: 0000000000000000
r12: ffff88022310f800  r13: ffff88017a887000  r14: ffff88020bd41560
r15: 0000000000000003  ip: ffffffffa079cde4  flags: 00010202  cs: 00000010
ss: 00000018  ds: 00000018  es: 00000018  fs: 00000018  gs: 00000018