Project

General

Profile

Bug #4553

kclient: lockdep report, crash involving ceph fs and libceph

Added by Alex Elder about 11 years ago. Updated over 9 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

I really don't know what to assign this to, but it looks like
a big complex potential deadlock so I thought I should grab the
kernel output so someone else can take a look at it.

I'm testing using a kernel with some recent changes I've made,
but right now I don't think the following is related to that.

Here's the kernel being tested:
review/wip-4392-2 1655ae7 libceph: verify requests queued in order

It is based on the current testing branch:
8383641 libceph: wrap auth methods in a mutex

Here are the newer commits in the kernel being tested:
1655ae7 libceph: verify requests queued in order
532902a libceph: send queued requests when starting new one
b1e0294 libceph: keep request lists in tid order
317b8e1 libceph: requeue only sent requests when kicking
349cd3a libceph: no more kick_requests() race
76752dc libceph: slightly defer registering osd request
5128453 libceph: initialize data fields on last msg put
...

Here's the current state of the ceph "next" branch:
70e0ee8 rgw: bucket index ops on system buckets shouldn't do anything

Here's the state of teuthology I'm using:
c50b143 thrashosds: add test_backfill_full

OK, here's the yaml file:

interactive-on-error:
roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2]
- [mon.b, mds.a, osd.3, osd.4, osd.5]
- [client.0]
kernel:
  mon:
    branch: review_wip-4392-2
    kdb: true
  client:
    branch: review_wip-4392-2
    kdb: true
overrides:
  ceph:
    conf:
      osd:
        osd op complaint time: 120
    fs: btrfs
    log-whitelist:
    - clocks not synchronized
    - old request
tasks:
- install:
- ceph:
    branch: next
    conf:
        global:
            ms inject socket failures: 5000
        mon:
            debug mon: 10
            debug paxos: 
        osd:
            debug ms: 1
            debug journal: 20
            osd debug op order: true
- kclient:
- workunit:
    clients:
      all:
        - misc/trivial_sync.sh
        - kernel_untar_build.sh
        - suites/blogbench.sh
        - suites/dbench.sh
        - suites/tiobench.sh
        - suites/fsstress.sh
        - suites/ffsb.sh

The test that had just started was "suites/ffsb.sh". Here is
the end of the teuthology test output:

INFO:teuthology.task.workunit.client.0.out:make[1]: Leaving directory `/tmp/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2'
INFO:teuthology.task.workunit.client.0.out:FFSB version 6.0-RC2 started
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:benchmark time = 300
INFO:teuthology.task.workunit.client.0.out:ThreadGroup 0
INFO:teuthology.task.workunit.client.0.out:================
INFO:teuthology.task.workunit.client.0.out:     num_threads      = 32
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:     read_random      = off
INFO:teuthology.task.workunit.client.0.out:     read_size        = 0    (0B)
INFO:teuthology.task.workunit.client.0.out:     read_blocksize   = 0    (0B)
INFO:teuthology.task.workunit.client.0.out:     read_skip        = off
INFO:teuthology.task.workunit.client.0.out:     read_skipsize    = 0    (0B)
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:     write_random     = on
INFO:teuthology.task.workunit.client.0.out:     write_size       = 5242880    (5MB)
INFO:teuthology.task.workunit.client.0.out:     fsync_file       = 0
INFO:teuthology.task.workunit.client.0.out:     write_blocksize  = 4096    (4KB)
INFO:teuthology.task.workunit.client.0.out:     wait time        = 0
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:     op weights
INFO:teuthology.task.workunit.client.0.out:                     read = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                  readall = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                    write = 1 (100.00%)
INFO:teuthology.task.workunit.client.0.out:                   create = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                   append = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                   delete = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                   metaop = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                createdir = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                     stat = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:                 writeall = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:           writeall_fsync = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:               open_close = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:              write_fsync = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:             create_fsync = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:             append_fsync = 0 (0.00%)
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:FileSystem .
INFO:teuthology.task.workunit.client.0.out:==========
INFO:teuthology.task.workunit.client.0.out:     num_dirs         = 0
INFO:teuthology.task.workunit.client.0.out:     starting files   = 128
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:     min file size    = 104857600    (100MB)
INFO:teuthology.task.workunit.client.0.out:     max file size    = 104857600    (100MB)
INFO:teuthology.task.workunit.client.0.out:     directio         = off
INFO:teuthology.task.workunit.client.0.out:     alignedio        = on
INFO:teuthology.task.workunit.client.0.out:     bufferedio       = off
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:     aging is off
INFO:teuthology.task.workunit.client.0.out:     current utilization = 0.13%
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:checking existing fs: .
INFO:teuthology.task.workunit.client.0.out:recreating new fileset
INFO:teuthology.task.workunit.client.0.err:opendir: No such file or directory
INFO:teuthology.task.workunit.client.0.out:fs setup took 189 secs
INFO:teuthology.task.workunit.client.0.out:Syncing()...10 sec
INFO:teuthology.task.workunit.client.0.out:Starting Actual Benchmark At: Tue Mar 26 08:59:58 2013
INFO:teuthology.task.workunit.client.0.out:

The test now seems to be hung, which may be related (I think so).
It also eventually led to a crash (which may have been due to
other things, I'm just not sure yet).

Here's information captured from the console, describing the
lockdep issues.

[ 4559.235679] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4559.513882] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4559.574219] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[ 4559.664004] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 4559.772118] 
[ 4559.773638] =========================================================
[ 4559.780540] [ INFO: possible irq lock inversion dependency detected ]
[ 4559.787021] 3.8.0-ceph-00226-g1655ae7 #1 Not tainted
[ 4559.792001] ---------------------------------------------------------
[ 4559.798452] kswapd0/38 just changed the state of lock:
[ 4559.803608]  (&osdc->map_sem){++++.-}, at: [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4559.813556] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[ 4559.820526]  (&monc->mutex){+.+.+.}

and interrupts could create inverse lock ordering between them.

[ 4559.830128] 
[ 4559.830128] other info that might help us debug this:
[ 4559.836677]  Possible interrupt unsafe locking scenario:
[ 4559.836677] 
[ 4559.843487]        CPU0                    CPU1
[ 4559.848029]        ----                    ----
[ 4559.852572]   lock(&monc->mutex);
[ 4559.855948]                                local_irq_disable();
[ 4559.861878]                                lock(&osdc->map_sem);
[ 4559.867932]                                lock(&monc->mutex);
[ 4559.873816]   <Interrupt>
[ 4559.876451]     lock(&osdc->map_sem);
[ 4559.880176] 
[ 4559.880176]  *** DEADLOCK ***
[ 4559.880176] 
[ 4559.886133] no locks held by kswapd0/38.
[ 4559.890073] 
[ 4559.890073] the shortest dependencies between 2nd lock and 1st lock:
[ 4559.899088]  -> (&monc->mutex){+.+.+.} ops: 181682 {
[ 4559.904181]     HARDIRQ-ON-W at:
[ 4559.907442]                       [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10
[ 4559.915060]                       [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4559.922321]                       [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[ 4559.930023]                       [<ffffffffa05e5f36>] ceph_monc_open_session+0x26/0x50 [libceph]
[ 4559.938954]                       [<ffffffffa05de26a>] __ceph_open_session+0x3a/0x250 [libceph]
[ 4559.947728]                       [<ffffffffa06241ed>] ceph_mount+0x37d/0x840 [ceph]
[ 4559.955522]                       [<ffffffff81187ba3>] mount_fs+0x43/0x1b0
[ 4559.962429]                       [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120
[ 4559.969868]                       [<ffffffff811a5284>] do_mount+0x1f4/0xa20
[ 4559.976881]                       [<ffffffff811a5b40>] sys_mount+0x90/0xe0
[ 4559.983787]                       [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4559.991657]     SOFTIRQ-ON-W at:
[ 4559.994925]                       [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10
[ 4560.002534]                       [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.009803]                       [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[ 4560.017503]                       [<ffffffffa05e5f36>] ceph_monc_open_session+0x26/0x50 [libceph]
[ 4560.026421]                       [<ffffffffa05de26a>] __ceph_open_session+0x3a/0x250 [libceph]
[ 4560.035167]                       [<ffffffffa06241ed>] ceph_mount+0x37d/0x840 [ceph]
[ 4560.042961]                       [<ffffffff81187ba3>] mount_fs+0x43/0x1b0
[ 4560.049870]                       [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120
[ 4560.057308]                       [<ffffffff811a5284>] do_mount+0x1f4/0xa20
[ 4560.064301]                       [<ffffffff811a5b40>] sys_mount+0x90/0xe0
[ 4560.071208]                       [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4560.079081]     RECLAIM_FS-ON-W at:
[ 4560.082606]                          [<ffffffff810b75e6>] mark_held_locks+0x86/0x140
[ 4560.090389]                          [<ffffffff810b7d41>] lockdep_trace_alloc+0x71/0xd0
[ 4560.098437]                          [<ffffffff8117b023>] kmem_cache_alloc_trace+0x33/0x150
[ 4560.106831]                          [<ffffffff810705b2>] __request_module+0x112/0x280
[ 4560.114791]                          [<ffffffff812feb3c>] crypto_larval_lookup+0x8c/0x180
[ 4560.123020]                          [<ffffffff812fec53>] crypto_alg_mod_lookup+0x23/0x90
[ 4560.131241]                          [<ffffffff812fed31>] crypto_alloc_base+0x41/0xb0
[ 4560.139114]                          [<ffffffffa05f283f>] ceph_crypto_alloc_cipher+0x1f/0x30 [libceph]
[ 4560.148470]                          [<ffffffffa05f28ad>] ceph_aes_encrypt2+0x5d/0x2c0 [libceph]
[ 4560.157305]                          [<ffffffffa05f3961>] ceph_encrypt2+0xb1/0xc0 [libceph]
[ 4560.165706]                          [<ffffffffa05f3e3f>] ceph_x_encrypt+0x4f/0x70 [libceph]
[ 4560.174196]                          [<ffffffffa05f4e7f>] ceph_x_build_request+0xdf/0x2f0 [libceph]
[ 4560.183290]                          [<ffffffffa05f1dbe>] ceph_build_auth_request+0x4e/0xb0 [libceph]
[ 4560.192558]                          [<ffffffffa05f2440>] ceph_handle_auth_reply+0x250/0x2e0 [libceph]
[ 4560.201912]                          [<ffffffffa05e7162>] dispatch+0x192/0x7d0 [libceph]
[ 4560.210053]                          [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph]
[ 4560.218368]                          [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4560.226328]                          [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4560.234028]                          [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.241014]                          [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.248540]     INITIAL USE at:
[ 4560.251721]                      [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10
[ 4560.259245]                      [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.266422]                      [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[ 4560.274032]                      [<ffffffffa05e5f36>] ceph_monc_open_session+0x26/0x50 [libceph]
[ 4560.282871]                      [<ffffffffa05de26a>] __ceph_open_session+0x3a/0x250 [libceph]
[ 4560.291522]                      [<ffffffffa06241ed>] ceph_mount+0x37d/0x840 [ceph]
[ 4560.299248]                      [<ffffffff81187ba3>] mount_fs+0x43/0x1b0
[ 4560.306069]                      [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120
[ 4560.313420]                      [<ffffffff811a5284>] do_mount+0x1f4/0xa20
[ 4560.320319]                      [<ffffffff811a5b40>] sys_mount+0x90/0xe0
[ 4560.327139]                      [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4560.334923]   }
[ 4560.336696]   ... key      at: [<ffffffffa06060b8>] __key.28897+0x0/0xffffffffffff03fb [libceph]
[ 4560.345519]   ... acquired at:
[ 4560.348587]    [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.354196]    [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[ 4560.360236]    [<ffffffffa05e784d>] ceph_monc_got_osdmap+0x2d/0x60 [libceph]
[ 4560.367318]    [<ffffffffa05eb6f7>] ceph_osdc_handle_map+0x287/0x580 [libceph]
[ 4560.374583]    [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph]
[ 4560.380797]    [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph]
[ 4560.387185]    [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4560.393223]    [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4560.399002]    [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.404089]    [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.409697] 
[ 4560.411208] -> (&osdc->map_sem){++++.-} ops: 707694 {
[ 4560.416388]    HARDIRQ-ON-W at:
[ 4560.419563]                     [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10
[ 4560.427002]                     [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.434083]                     [<ffffffff8165bd46>] down_write+0x36/0x59
[ 4560.440899]                     [<ffffffffa05eb555>] ceph_osdc_handle_map+0xe5/0x580 [libceph]
[ 4560.449561]                     [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph]
[ 4560.457270]                     [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph]
[ 4560.465152]                     [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4560.472679]                     [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4560.479949]                     [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.486509]                     [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.493589]    HARDIRQ-ON-R at:
[ 4560.496766]                     [<ffffffff810b4f6f>] __lock_acquire+0x52f/0x1d10
[ 4560.504204]                     [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.511285]                     [<ffffffff8165bcf9>] down_read+0x39/0x50
[ 4560.518024]                     [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4560.526946]                     [<ffffffffa062f508>] ceph_writepages_start+0x8d8/0xf50 [ceph]
[ 4560.535520]                     [<ffffffff8113c7e3>] do_writepages+0x23/0x40
[ 4560.542601]                     [<ffffffff811ac95b>] __writeback_single_inode+0x3b/0x180
[ 4560.550731]                     [<ffffffff811aedc0>] writeback_sb_inodes+0x280/0x3e0
[ 4560.558519]                     [<ffffffff811aefbe>] __writeback_inodes_wb+0x9e/0xd0
[ 4560.566306]                     [<ffffffff811af1eb>] wb_writeback+0x1fb/0x320
[ 4560.573474]                     [<ffffffff811b08a9>] wb_do_writeback+0xb9/0x1d0
[ 4560.580826]                     [<ffffffff811b0a53>] bdi_writeback_thread+0x93/0x270
[ 4560.588614]                     [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.595176]                     [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.602253]    SOFTIRQ-ON-W at:
[ 4560.605449]                     [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10
[ 4560.612885]                     [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.619959]                     [<ffffffff8165bd46>] down_write+0x36/0x59
[ 4560.626776]                     [<ffffffffa05eb555>] ceph_osdc_handle_map+0xe5/0x580 [libceph]
[ 4560.635438]                     [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph]
[ 4560.643146]                     [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph]
[ 4560.651030]                     [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4560.658556]                     [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4560.665823]                     [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.672378]                     [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.679457]    SOFTIRQ-ON-R at:
[ 4560.682635]                     [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10
[ 4560.690071]                     [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.697150]                     [<ffffffff8165bcf9>] down_read+0x39/0x50
[ 4560.705030]                     [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4560.713946]                     [<ffffffffa062f508>] ceph_writepages_start+0x8d8/0xf50 [ceph]
[ 4560.722522]                     [<ffffffff8113c7e3>] do_writepages+0x23/0x40
[ 4560.729617]                     [<ffffffff811ac95b>] __writeback_single_inode+0x3b/0x180
[ 4560.737750]                     [<ffffffff811aedc0>] writeback_sb_inodes+0x280/0x3e0
[ 4560.745538]                     [<ffffffff811aefbe>] __writeback_inodes_wb+0x9e/0xd0
[ 4560.753325]                     [<ffffffff811af1eb>] wb_writeback+0x1fb/0x320
[ 4560.760492]                     [<ffffffff811b08a9>] wb_do_writeback+0xb9/0x1d0
[ 4560.767844]                     [<ffffffff811b0a53>] bdi_writeback_thread+0x93/0x270
[ 4560.775632]                     [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.782194]                     [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.789272]    IN-RECLAIM_FS-R at:
[ 4560.792710]                        [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10
[ 4560.800406]                        [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.807760]                        [<ffffffff8165bcf9>] down_read+0x39/0x50
[ 4560.814746]                        [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4560.823928]                        [<ffffffffa05eb20f>] ceph_osdc_writepages+0xdf/0x180 [libceph]
[ 4560.832848]                        [<ffffffffa063025a>] writepage_nounlock+0x22a/0x410 [ceph]
[ 4560.841422]                        [<ffffffffa0630bee>] ceph_writepage+0x3e/0x70 [ceph]
[ 4560.849477]                        [<ffffffff81143164>] shrink_page_list+0x554/0x900
[ 4560.857265]                        [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[ 4560.865400]                        [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4560.872925]                        [<ffffffff811452e3>] kswapd+0x5d3/0xa90
[ 4560.879824]                        [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.886660]                        [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.894014]    INITIAL USE at:
[ 4560.897109]                    [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10
[ 4560.904458]                    [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4560.911452]                    [<ffffffff8165bd46>] down_write+0x36/0x59
[ 4560.918184]                    [<ffffffffa05eb555>] ceph_osdc_handle_map+0xe5/0x580 [libceph]
[ 4560.926761]                    [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph]
[ 4560.934382]                    [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph]
[ 4560.942178]                    [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4560.949635]                    [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4560.956812]                    [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4560.963285]                    [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4560.970276]  }
[ 4560.971958]  ... key      at: [<ffffffffa06060e0>] __key.29826+0x0/0xffffffffffff03d3 [libceph]
[ 4560.980704]  ... acquired at:
[ 4560.983687]    [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150
[ 4560.990076]    [<ffffffff810b48e8>] mark_lock+0x198/0x2f0
[ 4560.995509]    [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10
[ 4561.001462]    [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4561.007069]    [<ffffffff8165bcf9>] down_read+0x39/0x50
[ 4561.012329]    [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4561.019771]    [<ffffffffa05eb20f>] ceph_osdc_writepages+0xdf/0x180 [libceph]
[ 4561.026939]    [<ffffffffa063025a>] writepage_nounlock+0x22a/0x410 [ceph]
[ 4561.033756]    [<ffffffffa0630bee>] ceph_writepage+0x3e/0x70 [ceph]
[ 4561.040056]    [<ffffffff81143164>] shrink_page_list+0x554/0x900
[ 4561.046095]    [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[ 4561.052482]    [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4561.058261]    [<ffffffff811452e3>] kswapd+0x5d3/0xa90
[ 4561.063434]    [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4561.068522]    [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4561.074130] 
[ 4561.075665] 
[ 4561.075665] stack backtrace:
[ 4561.080046] Pid: 38, comm: kswapd0 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4561.087021] Call Trace:
[ 4561.089488]  [<ffffffff810b3dd8>] print_irq_inversion_bug.part.37+0x1e8/0x1f0
[ 4561.096635]  [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150
[ 4561.102830]  [<ffffffff810b48e8>] mark_lock+0x198/0x2f0
[ 4561.108072]  [<ffffffff810b3de0>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0
[ 4561.115401]  [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10
[ 4561.121164]  [<ffffffff81338a61>] ? vsnprintf+0x461/0x640
[ 4561.126585]  [<ffffffffa05e8ab9>] ? ceph_osdc_build_request+0x119/0x650 [libceph]
[ 4561.134095]  [<ffffffffa05e95a5>] ? ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4561.141515]  [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[ 4561.146928]  [<ffffffffa05e95a5>] ? ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4561.154349]  [<ffffffff8165bcf9>] down_read+0x39/0x50
[ 4561.159422]  [<ffffffffa05e95a5>] ? ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4561.166855]  [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph]
[ 4561.174108]  [<ffffffffa05eb20f>] ceph_osdc_writepages+0xdf/0x180 [libceph]
[ 4561.181085]  [<ffffffffa063025a>] writepage_nounlock+0x22a/0x410 [ceph]
[ 4561.187711]  [<ffffffffa0630bee>] ceph_writepage+0x3e/0x70 [ceph]
[ 4561.193815]  [<ffffffff81143164>] shrink_page_list+0x554/0x900
[ 4561.199665]  [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[ 4561.205862]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4561.211883]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4561.217472]  [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300
[ 4561.223053]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4561.229857]  [<ffffffff811452e3>] kswapd+0x5d3/0xa90
[ 4561.234838]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4561.241113]  [<ffffffff81144d10>] ? try_to_free_pages+0x1a0/0x1a0
[ 4561.247221]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4561.252109]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4561.258298]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4561.263714]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4561.270132] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4561.305432] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4561.341964] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4561.415362] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4561.461996] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4562.595792] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4562.695317] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 4562.782819] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)

And finally, here's some subsequent console output, leading
ultimately to a crash. The system (or maybe just my connection)
seems to be locked up solid so I can't get any more. I think
I will be power cycling it to get the machine back.

[ 4575.988573] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4576.518337] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4576.584345] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 4577.111907] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4577.391608] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4577.699085] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 4579.518966] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4603.896963] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229053 c=229052 q=482)
[ 4603.906376] Pid: 24447, comm: kworker/0:4 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4603.913972] Call Trace:
[ 4603.916437]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4603.922502]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4603.926917] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2103 jiffies, g=229053, c=229052, q=488)
[ 4603.926918] Task dump for CPU 0:
[ 4603.926920] kworker/0:4     R  running task        0 24447      2 0x00000000
[ 4603.926923]  ffff8804152fb308 ffffffff810b783d ffff8804152fb328 0000000000000282
[ 4603.926924]  ffff88043ff6a438 0000000000000282 ffff8804152fb368 ffffffff81085343
[ 4603.926926]  0000000000000030 ffffea00078ba0c0 ffff8804152fb520 ffffea00078ba0e0
[ 4603.926927] Call Trace:
[ 4603.926932]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4603.926935]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4603.926939]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4603.926941]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4603.926944]  [<ffffffff81339c9a>] ? delay_tsc+0x4a/0x80
[ 4603.926948]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4603.926950]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4603.926952]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4603.926953]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4603.926955]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4603.926957]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4603.926960]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4603.926964]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4603.926966]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4603.926967]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4603.926969]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4603.926973]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4603.926976]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4603.926979]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4603.926982]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4603.926983]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4603.926985]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4603.926989]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4603.926997]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4603.926999]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4603.927008]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4603.927009]  [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330
[ 4603.927011]  [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330
[ 4603.927013]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4603.927014]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4603.927016]  [<ffffffff810f7a7f>] ? rcu_irq_exit+0x7f/0xd0
[ 4603.927018]  [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13
[ 4603.927023]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4603.927024]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4603.927026]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4603.927028]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4603.927030]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4603.927032]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4603.927034]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4604.207021]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4604.213049]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4604.219074]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4604.225527]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4604.231237]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4604.236741]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4604.242686]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4604.249281]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4604.255131]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4604.261333]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4604.267704]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4604.273723]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4604.279399]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4604.285079]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4604.290233]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4604.295044]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4604.300778]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4604.307625]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4604.313824]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4604.320027]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4604.326222]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4604.332073]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4604.337662]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4604.343424]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4604.349759]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4604.355955]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4604.361801]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4604.368182]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4604.374204]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4604.380225]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4604.385641]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4604.390979]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4604.396399]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4604.401729]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4604.407843]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4604.413868]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4604.419377]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4604.425490]  [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330
[ 4604.431181]  [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330
[ 4604.436894]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4604.442744]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4604.448766]  [<ffffffff810f7a7f>] ? rcu_irq_exit+0x7f/0xd0
[ 4604.454271]  [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13
[ 4604.460385]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4604.466974]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4604.472561]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4604.478413]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4604.483308]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4604.489505]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4604.494921]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4613.010446] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4621.954715] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4621.961990] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4622.066824] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4622.235849] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4622.333686] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 4622.696381] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4622.738472] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[ 4624.941354] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4625.140097] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4625.178203] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[ 4625.296607] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4625.427142] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4625.607493] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4625.763427] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[ 4625.808217] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4627.594601] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 4628.153725] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[ 4628.338422] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 4628.380127] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 4628.434586] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 4649.704822] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229113 c=229112 q=598)
[ 4649.714167] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4649.721763] Call Trace:
[ 4649.724230]  <IRQ> [ 4649.724837] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2102 jiffies, g=229113, c=229112, q=598)
[ 4649.724842] Task dump for CPU 0:
[ 4649.724844] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4649.724846]  ffffffff81072b3c 0000000000000046 0000000000000001 0000000000000000
[ 4649.724848]  ffff88043fc1cd35 ffffffffa05e1760 ffff880090289fd8 ffff88043fc0e440
[ 4649.724850]  ffffffffa0605c80 0000000000000000 0000000000000000 ffffffffa05f8d0e
[ 4649.724852] Call Trace:
[ 4649.724853]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4649.724860]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4649.724871]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4649.724873]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4649.724874]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4649.724877]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4649.724879]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4649.724884]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4649.820216]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4649.825874]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4649.832076]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4649.838095]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4649.844121]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4649.850593]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4649.856269]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4649.861771]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4649.867709]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4649.874261]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4649.880117]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4649.886379]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4649.892750]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4649.898774]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4649.904501]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4649.910178]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4649.915376]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4649.920187]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4649.925869]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4649.932708]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4649.938907]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4649.945101]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4649.951296]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4649.957144]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4649.962737]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4649.968499]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4649.974782]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4649.980975]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4649.986826]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4649.993196]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4649.999235]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4650.005300]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4650.010717]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4650.016071]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4650.021489]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4650.026847]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4650.032960]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4650.038996]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4650.044503]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4650.050616]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4650.056290]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4650.061969]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4650.068086]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4650.073935]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4650.079973]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4650.086572]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4650.092166]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4650.099171]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4650.104071]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4650.110272]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4650.115748]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4671.091205] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229114 c=229113 q=3151)
[ 4671.100640] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4671.108296] Call Trace:
[ 4671.110763]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4671.116827]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4671.123027]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4671.129048]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4671.131151] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2104 jiffies, g=229114, c=229113, q=3152)
[ 4671.131153] Task dump for CPU 0:
[ 4671.131158] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4671.131165]  ffffffff81072b3c 0000000000000046 0000000000000001 0000000000000000
[ 4671.131166]  ffff88043fc1cd35 ffffffffa05e1760 ffff880090289fd8 ffff88043fc0e440
[ 4671.131168]  ffffffffa0605c80 0000000000000000 0000000000000000 ffffffffa05f8d0e
[ 4671.131169] Call Trace:
[ 4671.131173]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4671.131184]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4671.131187]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4671.131188]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4671.131190]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4671.131192]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4671.131195]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4671.131197]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4671.228529]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4671.234986]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4671.240666]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4671.246203]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4671.252164]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4671.258708]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4671.264557]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4671.270752]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4671.277121]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4671.283205]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4671.288878]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4671.294558]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4671.299712]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4671.304549]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4671.310223]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4671.317085]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4671.323328]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4671.329524]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4671.335719]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4671.341594]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4671.347182]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4671.353025]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4671.359307]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4671.365529]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4671.371438]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4671.377809]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4671.383846]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4671.389871]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4671.395301]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4671.400630]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4671.406057]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4671.411466]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4671.417579]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4671.423668]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4671.429177]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4671.435325]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4671.441066]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4671.446745]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4671.452854]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4671.458703]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4671.464759]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4671.471365]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4671.476951]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4671.482813]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4671.487706]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4671.493933]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4671.499357]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4676.392896] INFO: task ffsb:26654 blocked for more than 120 seconds.
[ 4676.399268] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4676.407141] ffsb            D ffffffff81130150     0 26654  24977 0x00000000
[ 4676.414254]  ffff880413d0b938 0000000000000046 ffffffff8165e05f 0000000000000002
[ 4676.421767]  ffff88041f77bf20 ffff880413d0bfd8 ffff880413d0bfd8 ffff880413d0bfd8
[ 4676.429288]  ffff88042d585eb0 ffff88041f77bf20 ffff880413d0b938 ffff88043fc54ed8
[ 4676.436807] Call Trace:
[ 4676.439276]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4676.446090]  [<ffffffff81130150>] ? __lock_page+0x70/0x70
[ 4676.451508]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4676.456494]  [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0
[ 4676.461734]  [<ffffffff8113015e>] sleep_on_page+0xe/0x20
[ 4676.467072]  [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0
[ 4676.472928]  [<ffffffff81130147>] __lock_page+0x67/0x70
[ 4676.478175]  [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40
[ 4676.484728]  [<ffffffff81130b87>] find_lock_page+0x67/0x80
[ 4676.490229]  [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0
[ 4676.496874]  [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph]
[ 4676.503163]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4676.509189]  [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290
[ 4676.516000]  [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0
[ 4676.522549]  [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph]
[ 4676.528843]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4676.535395]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4676.541336]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4676.546756]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4676.551916]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4676.556993]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4676.563022] INFO: lockdep is turned off.
[ 4676.566959] INFO: task ffsb:26656 blocked for more than 120 seconds.
[ 4676.573326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4676.581181] ffsb            D ffffffff81130150     0 26656  24977 0x00000000
[ 4676.588293]  ffff8804144ff938 0000000000000046 ffffffff8165e05f 0000000000000002
[ 4676.595814]  ffff880413fe0000 ffff8804144fffd8 ffff8804144fffd8 ffff8804144fffd8
[ 4676.603335]  ffff88042d540000 ffff880413fe0000 ffff8804144ff938 ffff88043fcd4ed8
[ 4676.610837] Call Trace:
[ 4676.613317]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4676.620123]  [<ffffffff81130150>] ? __lock_page+0x70/0x70
[ 4676.625545]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4676.630527]  [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0
[ 4676.635778]  [<ffffffff8113015e>] sleep_on_page+0xe/0x20
[ 4676.641112]  [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0
[ 4676.646969]  [<ffffffff81130147>] __lock_page+0x67/0x70
[ 4676.652216]  [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40
[ 4676.658765]  [<ffffffff81130b87>] find_lock_page+0x67/0x80
[ 4676.664272]  [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0
[ 4676.670914]  [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph]
[ 4676.677205]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4676.683232]  [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290
[ 4676.690039]  [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0
[ 4676.696611]  [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph]
[ 4676.702908]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4676.709455]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4676.715398]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4676.720814]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4676.725975]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4676.731044]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4676.737070] INFO: lockdep is turned off.
[ 4676.741013] INFO: task ffsb:26659 blocked for more than 120 seconds.
[ 4676.747386] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4676.755249] ffsb            D 00000000ffffffff     0 26659  24977 0x00000000
[ 4676.762353]  ffff880412cf7bb8 0000000000000046 0000000000000007 0000000000000006
[ 4676.769855]  ffff880413088000 ffff880412cf7fd8 ffff880412cf7fd8 ffff880412cf7fd8
[ 4676.777365]  ffff8804150c5eb0 ffff880413088000 ffffffff8165ac75 ffff8804289d74c8
[ 4676.784883] Call Trace:
[ 4676.787352]  [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320
[ 4676.793465]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4676.798450]  [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10
[ 4676.804827]  [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320
[ 4676.810771]  [<ffffffffa062cec0>] ? ceph_aio_write+0x910/0xaa0 [ceph]
[ 4676.817241]  [<ffffffffa062cec0>] ceph_aio_write+0x910/0xaa0 [ceph]
[ 4676.823535]  [<ffffffff81090bb4>] ? select_task_rq_fair+0x54/0x8a0
[ 4676.829730]  [<ffffffff81090bb4>] ? select_task_rq_fair+0x54/0x8a0
[ 4676.835937]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4676.841871]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4676.847296]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4676.852483]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4676.857553]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4676.863596] INFO: lockdep is turned off.
[ 4676.867540] INFO: task ffsb:26663 blocked for more than 120 seconds.
[ 4676.873932] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4676.881786] ffsb            D 00000000ffffffff     0 26663  24977 0x00000000
[ 4676.888898]  ffff880414735bb8 0000000000000046 0000000000000007 0000000000000006
[ 4676.896413]  ffff88041f620000 ffff880414735fd8 ffff880414735fd8 ffff880414735fd8
[ 4676.903933]  ffff88042d540000 ffff88041f620000 ffffffff8165ac75 ffff8804289d74c8
[ 4676.912616] Call Trace:
[ 4676.915089]  [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320
[ 4676.921202]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4676.926196]  [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10
[ 4676.932570]  [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320
[ 4676.938517]  [<ffffffffa062cec0>] ? ceph_aio_write+0x910/0xaa0 [ceph]
[ 4676.944989]  [<ffffffffa062cec0>] ceph_aio_write+0x910/0xaa0 [ceph]
[ 4676.951275]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4676.957225]  [<ffffffff8108ca5e>] ? try_to_wake_up+0x2e/0x2f0
[ 4676.963002]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4676.968420]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4676.973607]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4676.978676]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4676.984704] INFO: lockdep is turned off.
[ 4676.988646] INFO: task ffsb:26666 blocked for more than 120 seconds.
[ 4676.995035] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4677.002900] ffsb            D ffffffff81130150     0 26666  24977 0x00000000
[ 4677.010008]  ffff880413da3938 0000000000000046 ffffffff8165e05f 0000000000000002
[ 4677.017530]  ffff880413c78000 ffff880413da3fd8 ffff880413da3fd8 ffff880413da3fd8
[ 4677.025049]  ffff88042d533f20 ffff880413c78000 ffff880413da3938 ffff88043fc54ed8
[ 4677.032569] Call Trace:
[ 4677.035038]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4677.041855]  [<ffffffff81130150>] ? __lock_page+0x70/0x70
[ 4677.047268]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4677.052258]  [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0
[ 4677.057504]  [<ffffffff8113015e>] sleep_on_page+0xe/0x20
[ 4677.062851]  [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0
[ 4677.068703]  [<ffffffff81130147>] __lock_page+0x67/0x70
[ 4677.073957]  [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40
[ 4677.080498]  [<ffffffff81130b87>] find_lock_page+0x67/0x80
[ 4677.086015]  [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0
[ 4677.092662]  [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph]
[ 4677.098946]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4677.104980]  [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290
[ 4677.111819]  [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0
[ 4677.118370]  [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph]
[ 4677.124681]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4677.131231]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4677.137179]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4677.142603]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4677.147763]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4677.152845]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4677.158864] INFO: lockdep is turned off.
[ 4677.162814] INFO: task ffsb:26670 blocked for more than 120 seconds.
[ 4677.169179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4677.177045] ffsb            D ffffffff81130150     0 26670  24977 0x00000000
[ 4677.184157]  ffff880412dc5938 0000000000000046 ffffffff8165e05f 0000000000000002
[ 4677.191671]  ffff88042d0b9f90 ffff880412dc5fd8 ffff880412dc5fd8 ffff880412dc5fd8
[ 4677.199183]  ffff88042d585eb0 ffff88042d0b9f90 ffff880412dc5938 ffff88043fc54ed8
[ 4677.206694] Call Trace:
[ 4677.209163]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4677.215981]  [<ffffffff81130150>] ? __lock_page+0x70/0x70
[ 4677.221403]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4677.226393]  [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0
[ 4677.231646]  [<ffffffff8113015e>] sleep_on_page+0xe/0x20
[ 4677.236977]  [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0
[ 4677.242839]  [<ffffffff81130147>] __lock_page+0x67/0x70
[ 4677.248087]  [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40
[ 4677.254642]  [<ffffffff81130b87>] find_lock_page+0x67/0x80
[ 4677.260150]  [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0
[ 4677.266798]  [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph]
[ 4677.273086]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4677.279110]  [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290
[ 4677.285922]  [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0
[ 4677.292486]  [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph]
[ 4677.298773]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4677.305325]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4677.311266]  [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13
[ 4677.317384]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4677.322803]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4677.327963]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4677.333044]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4677.339064] INFO: lockdep is turned off.
[ 4677.343016] INFO: task ffsb:26673 blocked for more than 120 seconds.
[ 4677.349387] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4677.357253] ffsb            D 00000000ffffffff     0 26673  24977 0x00000000
[ 4677.364358]  ffff880414559e28 0000000000000046 0000000000000007 0000000000000006
[ 4677.371872]  ffff880414511f90 ffff880414559fd8 ffff880414559fd8 ffff880414559fd8
[ 4677.379374]  ffff880415211f90 ffff880414511f90 ffffffff8165ac75 ffff880421c9dce0
[ 4677.386894] Call Trace:
[ 4677.389363]  [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320
[ 4677.395488]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4677.400475]  [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10
[ 4677.406857]  [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320
[ 4677.412811]  [<ffffffffa062b605>] ? ceph_llseek+0x45/0x160 [ceph]
[ 4677.418919]  [<ffffffff811a0f7c>] ? fget_light+0x3c/0x130
[ 4677.424351]  [<ffffffffa062b605>] ceph_llseek+0x45/0x160 [ceph]
[ 4677.430288]  [<ffffffff811836d3>] vfs_llseek+0x33/0x40
[ 4677.435451]  [<ffffffff81183c5e>] sys_lseek+0x7e/0x90
[ 4677.440523]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4677.446551] INFO: lockdep is turned off.
[ 4677.450493] INFO: task ffsb:26674 blocked for more than 120 seconds.
[ 4677.456866] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4677.464729] ffsb            D ffffffff81130150     0 26674  24977 0x00000000
[ 4677.471837]  ffff880413c9f938 0000000000000046 ffffffff8165e05f 0000000000000002
[ 4677.479351]  ffff880412c83f20 ffff880413c9ffd8 ffff880413c9ffd8 ffff880413c9ffd8
[ 4677.486870]  ffff88042d073f20 ffff880412c83f20 ffff880413c9f938 ffff88043fc14ed8
[ 4677.494381] Call Trace:
[ 4677.496851]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4677.503668]  [<ffffffff81130150>] ? __lock_page+0x70/0x70
[ 4677.509089]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4677.514080]  [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0
[ 4677.519326]  [<ffffffff8113015e>] sleep_on_page+0xe/0x20
[ 4677.524666]  [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0
[ 4677.530517]  [<ffffffff81130147>] __lock_page+0x67/0x70
[ 4677.535769]  [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40
[ 4677.542317]  [<ffffffff81130b87>] find_lock_page+0x67/0x80
[ 4677.547819]  [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0
[ 4677.554469]  [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph]
[ 4677.560751]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4677.566784]  [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290
[ 4677.573608]  [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0
[ 4677.580160]  [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph]
[ 4677.586452]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4677.592998]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4677.598936]  [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13
[ 4677.605053]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4677.610467]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4677.615632]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4677.620704]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4677.626734] INFO: lockdep is turned off.
[ 4677.630675] INFO: task ffsb:26676 blocked for more than 120 seconds.
[ 4677.637056] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4677.644921] ffsb            D ffffffff81130150     0 26676  24977 0x00000000
[ 4677.652040]  ffff880412cf3938 0000000000000046 ffffffff8165e05f 0000000000000002
[ 4677.659551]  ffff880413fd8000 ffff880412cf3fd8 ffff880412cf3fd8 ffff880412cf3fd8
[ 4677.667061]  ffff88042d585eb0 ffff880413fd8000 ffff880412cf3938 ffff88043fc54ed8
[ 4677.674573] Call Trace:
[ 4677.677041]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[ 4677.683859]  [<ffffffff81130150>] ? __lock_page+0x70/0x70
[ 4677.689281]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4677.694284]  [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0
[ 4677.699524]  [<ffffffff8113015e>] sleep_on_page+0xe/0x20
[ 4677.704874]  [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0
[ 4677.710744]  [<ffffffff81130147>] __lock_page+0x67/0x70
[ 4677.715983]  [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40
[ 4677.722537]  [<ffffffff81130b87>] find_lock_page+0x67/0x80
[ 4677.728036]  [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0
[ 4677.734684]  [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph]
[ 4677.740973]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4677.746998]  [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290
[ 4677.753811]  [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0
[ 4677.760365]  [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph]
[ 4677.766687]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4677.773238]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4677.779178]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4677.784601]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4677.789758]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4677.794838]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4677.800865] INFO: lockdep is turned off.
[ 4677.804802] INFO: task ffsb:26678 blocked for more than 120 seconds.
[ 4677.811175] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4677.819032] ffsb            D 00000000ffffffff     0 26678  24977 0x00000000
[ 4677.826162]  ffff880413dadbb8 0000000000000046 0000000000000007 0000000000000006
[ 4677.833676]  ffff88042d5c3f20 ffff880413dadfd8 ffff880413dadfd8 ffff880413dadfd8
[ 4677.841213]  ffff88042d535eb0 ffff88042d5c3f20 ffffffff8165ac75 ffff8804289d74c8
[ 4677.849785] Call Trace:
[ 4677.852266]  [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320
[ 4677.858377]  [<ffffffff8165cb69>] schedule+0x29/0x70
[ 4677.863370]  [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10
[ 4677.869741]  [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320
[ 4677.875694]  [<ffffffffa062cec0>] ? ceph_aio_write+0x910/0xaa0 [ceph]
[ 4677.882187]  [<ffffffffa062cec0>] ceph_aio_write+0x910/0xaa0 [ceph]
[ 4677.888477]  [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310
[ 4677.895025]  [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10
[ 4677.900968]  [<ffffffff8108ca5e>] ? try_to_wake_up+0x2e/0x2f0
[ 4677.906732]  [<ffffffff81183843>] do_sync_write+0xa3/0xe0
[ 4677.912161]  [<ffffffff81183ef3>] vfs_write+0xb3/0x180
[ 4677.917312]  [<ffffffff81184232>] sys_write+0x52/0xa0
[ 4677.922394]  [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[ 4677.928416] INFO: lockdep is turned off.
[ 4710.039798] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229118 c=229117 q=863)
[ 4710.049151] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4710.056745] Call Trace:
[ 4710.059212]  <IRQ> [ 4710.059907] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229118, c=229117, q=863)
[ 4710.059911] Task dump for CPU 0:
[ 4710.059912] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4710.059915]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4710.059917]  ffff88043ff72238 0000000000000282 ffff880090289368 ffffffff81085343
[ 4710.059919]  0000000000000002 ffffea000f816140 ffff880090289520 ffffea000f816160
[ 4710.059921] Call Trace:
[ 4710.059922]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4710.059927]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4710.059931]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4710.059935]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4710.059937]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4710.059938]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4710.059942]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4710.059944]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4710.059946]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4710.059948]  [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50
[ 4710.059950]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4710.059952]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4710.059955]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4710.059959]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4710.059961]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4710.059963]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4710.059965]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4710.059968]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4710.059971]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4710.059974]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4710.059977]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4710.059979]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4710.059981]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4710.059990]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4710.059993]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4710.059995]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4710.060003]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4710.060006]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4710.060008]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4710.060010]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4710.060012]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4710.060013]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4710.060018]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4710.060020]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4710.060021]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4710.060024]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4710.060025]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4710.060028]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4710.335519]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4710.341177]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4710.347381]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4710.353409]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4710.359486]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4710.365939]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4710.371623]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4710.377128]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4710.383079]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4710.389624]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4710.395473]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4710.401669]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4710.408039]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4710.414067]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4710.419742]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4710.4254

History

#1 Updated by Ian Colle about 11 years ago

  • Priority changed from Normal to High

#2 Updated by Alex Elder about 11 years ago

I just got (I think) the same lockdep report, running only
the "suites/ffsb.sh" test using the ceph-client "testing" branch.

[  372.042693] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[  372.116802] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[  372.228738] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[  372.329688] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[  372.532394] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[  372.604703] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[  372.846913] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[  373.003111] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[  373.121406] 
[  373.122956] =========================================================
[  373.129436] [ INFO: possible irq lock inversion dependency detected ]
[  373.135917] 3.8.0-ceph-00219-g8383641 #1 Not tainted
[  373.140926] ---------------------------------------------------------
[  373.147409] kswapd0/38 just changed the state of lock:
[  373.152588]  (&osdc->request_mutex){+.+.-.}, at: [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph]
[  373.163198] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[  373.170201]  (&con->mutex){+.+.+.}

and interrupts could create inverse lock ordering between them.

[  373.179817] 
[  373.179817] other info that might help us debug this:
[  373.186402]  Possible interrupt unsafe locking scenario:
[  373.186402] 
[  373.193245]        CPU0                    CPU1
[  373.197814]        ----                    ----
[  373.202383]   lock(&con->mutex);
[  373.205759]                                local_irq_disable();
[  373.211723]                                lock(&osdc->request_mutex);
[  373.218393]                                lock(&con->mutex);
[  373.224286]   <Interrupt>
[  373.226955]     lock(&osdc->request_mutex);
[  373.231321] 
[  373.231321]  *** DEADLOCK ***
[  373.231321] 
[  373.237311] no locks held by kswapd0/38.
[  373.241279] 
[  373.241279] the shortest dependencies between 2nd lock and 1st lock:
[  373.249171]  -> (&con->mutex){+.+.+.} ops: 4935891 {
[  373.254495]     HARDIRQ-ON-W at:
[  373.257825]                       [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10
[  373.265521]                       [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.272849]                       [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.280613]                       [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph]
[  373.288973]                       [<ffffffffa05c7d5b>] __open_session+0xcb/0x160 [libceph]
[  373.297341]                       [<ffffffffa05c7efe>] ceph_monc_open_session+0x2e/0x50 [libceph]
[  373.306315]                       [<ffffffffa05c026a>] __ceph_open_session+0x3a/0x250 [libceph]
[  373.315115]                       [<ffffffffa06061ed>] ceph_mount+0x37d/0x840 [ceph]
[  373.324015]                       [<ffffffff81187ba3>] mount_fs+0x43/0x1b0
[  373.330984]                       [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120
[  373.338484]                       [<ffffffff811a5284>] do_mount+0x1f4/0xa20
[  373.345539]                       [<ffffffff811a5b40>] sys_mount+0x90/0xe0
[  373.352513]                       [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[  373.360445]     SOFTIRQ-ON-W at:
[  373.363772]                       [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10
[  373.371451]                       [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.378779]                       [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.386544]                       [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph]
[  373.394824]                       [<ffffffffa05c7d5b>] __open_session+0xcb/0x160 [libceph]
[  373.403192]                       [<ffffffffa05c7efe>] ceph_monc_open_session+0x2e/0x50 [libceph]
[  373.412167]                       [<ffffffffa05c026a>] __ceph_open_session+0x3a/0x250 [libceph]
[  373.420967]                       [<ffffffffa06061ed>] ceph_mount+0x37d/0x840 [ceph]
[  373.428831]                       [<ffffffff81187ba3>] mount_fs+0x43/0x1b0
[  373.435799]                       [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120
[  373.443305]                       [<ffffffff811a5284>] do_mount+0x1f4/0xa20
[  373.450361]                       [<ffffffff811a5b40>] sys_mount+0x90/0xe0
[  373.457327]                       [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[  373.465260]     RECLAIM_FS-ON-W at:
[  373.468846]                          [<ffffffff810b75e6>] mark_held_locks+0x86/0x140
[  373.476699]                          [<ffffffff810b7d41>] lockdep_trace_alloc+0x71/0xd0
[  373.484859]                          [<ffffffff8117c39f>] kmem_cache_alloc+0x2f/0x150
[  373.492793]                          [<ffffffff81539e5d>] sock_alloc_inode+0x1d/0xd0
[  373.500641]                          [<ffffffff8119de66>] alloc_inode+0x26/0xa0
[  373.508055]                          [<ffffffff8119fb4a>] new_inode_pseudo+0x1a/0x70
[  373.515907]                          [<ffffffff8153a28e>] sock_alloc+0x1e/0x80
[  373.523243]                          [<ffffffff8153b995>] __sock_create+0x95/0x260
[  373.530917]                          [<ffffffff8153bb84>] sock_create_kern+0x24/0x30
[  373.538767]                          [<ffffffffa05c4743>] con_work+0x1023/0x2e80 [libceph]
[  373.547178]                          [<ffffffff81072ba9>] process_one_work+0x199/0x510
[  373.555252]                          [<ffffffff81074895>] worker_thread+0x165/0x3f0
[  373.563012]                          [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  373.570068]                          [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  373.577653]     INITIAL USE at:
[  373.580938]                      [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10
[  373.588532]                      [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.595759]                      [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.603440]                      [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph]
[  373.611643]                      [<ffffffffa05c7d5b>] __open_session+0xcb/0x160 [libceph]
[  373.619926]                      [<ffffffffa05c7efe>] ceph_monc_open_session+0x2e/0x50 [libceph]
[  373.628812]                      [<ffffffffa05c026a>] __ceph_open_session+0x3a/0x250 [libceph]
[  373.637526]                      [<ffffffffa06061ed>] ceph_mount+0x37d/0x840 [ceph]
[  373.645285]                      [<ffffffff81187ba3>] mount_fs+0x43/0x1b0
[  373.652168]                      [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120
[  373.659581]                      [<ffffffff811a5284>] do_mount+0x1f4/0xa20
[  373.666550]                      [<ffffffff811a5b40>] sys_mount+0x90/0xe0
[  373.673428]                      [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b
[  373.681283]   }
[  373.683082]   ... key      at: [<ffffffffa05e7b38>] __key.49261+0x0/0xffffffffffff082b [libceph]
[  373.691971]   ... acquired at:
[  373.695069]    [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.700729]    [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.706829]    [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph]
[  373.713450]    [<ffffffffa05c9ff6>] __map_request+0x446/0x640 [libceph]
[  373.720172]    [<ffffffffa05cb525>] ceph_osdc_start_request+0xd5/0x160 [libceph]
[  373.727675]    [<ffffffffa0611508>] ceph_writepages_start+0x8d8/0xf50 [ceph]
[  373.734823]    [<ffffffff8113c7e3>] do_writepages+0x23/0x40
[  373.740497]    [<ffffffff811ac95b>] __writeback_single_inode+0x3b/0x180
[  373.747198]    [<ffffffff811aedc0>] writeback_sb_inodes+0x280/0x3e0
[  373.753582]    [<ffffffff811aefbe>] __writeback_inodes_wb+0x9e/0xd0
[  373.759942]    [<ffffffff811af1eb>] wb_writeback+0x1fb/0x320
[  373.765695]    [<ffffffff811b0920>] wb_do_writeback+0x130/0x1d0
[  373.771709]    [<ffffffff811b0a53>] bdi_writeback_thread+0x93/0x270
[  373.778072]    [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  373.783278]    [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  373.788939] 
[  373.790483] -> (&osdc->request_mutex){+.+.-.} ops: 1044630 {
[  373.796434]    HARDIRQ-ON-W at:
[  373.799679]                     [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10
[  373.807183]                     [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.814335]                     [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.821929]                     [<ffffffffa05cc56f>] kick_requests+0x3f/0x3e0 [libceph]
[  373.830038]                     [<ffffffffa05cd7b4>] ceph_osdc_handle_map+0x494/0x580 [libceph]
[  373.838839]                     [<ffffffffa05c90b0>] dispatch+0x120/0x7d0 [libceph]
[  373.846599]                     [<ffffffffa05c5118>] con_work+0x19f8/0x2e80 [libceph]
[  373.854569]                     [<ffffffff81072ba9>] process_one_work+0x199/0x510
[  373.862157]                     [<ffffffff81074895>] worker_thread+0x165/0x3f0
[  373.869485]                     [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  373.876143]                     [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  373.883299]    SOFTIRQ-ON-W at:
[  373.886529]                     [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10
[  373.894077]                     [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.901219]                     [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.908805]                     [<ffffffffa05cc56f>] kick_requests+0x3f/0x3e0 [libceph]
[  373.916915]                     [<ffffffffa05cd7b4>] ceph_osdc_handle_map+0x494/0x580 [libceph]
[  373.925743]                     [<ffffffffa05c90b0>] dispatch+0x120/0x7d0 [libceph]
[  373.933511]                     [<ffffffffa05c5118>] con_work+0x19f8/0x2e80 [libceph]
[  373.941444]                     [<ffffffff81072ba9>] process_one_work+0x199/0x510
[  373.949035]                     [<ffffffff81074895>] worker_thread+0x165/0x3f0
[  373.956363]                     [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  373.963010]                     [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  373.970149]    IN-RECLAIM_FS-W at:
[  373.973649]                        [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10
[  373.981413]                        [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  373.988828]                        [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  373.996678]                        [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph]
[  374.005912]                        [<ffffffffa05cd0bf>] ceph_osdc_writepages+0xdf/0x180 [libceph]
[  374.014885]                        [<ffffffffa061225a>] writepage_nounlock+0x22a/0x410 [ceph]
[  374.023513]                        [<ffffffffa0612bee>] ceph_writepage+0x3e/0x70 [ceph]
[  374.031619]                        [<ffffffff81143164>] shrink_page_list+0x554/0x900
[  374.039468]                        [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[  374.047660]                        [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[  374.055284]                        [<ffffffff811452e3>] kswapd+0x5d3/0xa90
[  374.062252]                        [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  374.069132]                        [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  374.076544]    INITIAL USE at:
[  374.079711]                    [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10
[  374.087127]                    [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  374.094182]                    [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  374.101683]                    [<ffffffffa05cc56f>] kick_requests+0x3f/0x3e0 [libceph]
[  374.109721]                    [<ffffffffa05cd7b4>] ceph_osdc_handle_map+0x494/0x580 [libceph]
[  374.118437]                    [<ffffffffa05c90b0>] dispatch+0x120/0x7d0 [libceph]
[  374.127242]                    [<ffffffffa05c5118>] con_work+0x19f8/0x2e80 [libceph]
[  374.135091]                    [<ffffffff81072ba9>] process_one_work+0x199/0x510
[  374.142593]                    [<ffffffff81074895>] worker_thread+0x165/0x3f0
[  374.149899]                    [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  374.156432]                    [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  374.163484]  }
[  374.165194]  ... key      at: [<ffffffffa05e7fa8>] __key.29809+0x0/0xffffffffffff03bb [libceph]
[  374.173999]  ... acquired at:
[  374.177017]    [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150
[  374.183466]    [<ffffffff810b48e8>] mark_lock+0x198/0x2f0
[  374.188968]    [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10
[  374.194982]    [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  374.200641]    [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  374.206741]    [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph]
[  374.214243]    [<ffffffffa05cd0bf>] ceph_osdc_writepages+0xdf/0x180 [libceph]
[  374.221532]    [<ffffffffa061225a>] writepage_nounlock+0x22a/0x410 [ceph]
[  374.228420]    [<ffffffffa0612bee>] ceph_writepage+0x3e/0x70 [ceph]
[  374.234830]    [<ffffffff81143164>] shrink_page_list+0x554/0x900
[  374.240948]    [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[  374.247402]    [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[  374.253242]    [<ffffffff811452e3>] kswapd+0x5d3/0xa90
[  374.258469]    [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  374.263626]    [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  374.269341] 
[  374.270881] 
[  374.270881] stack backtrace:
[  374.275297] Pid: 38, comm: kswapd0 Not tainted 3.8.0-ceph-00219-g8383641 #1
[  374.282296] Call Trace:
[  374.284790]  [<ffffffff810b3dd8>] print_irq_inversion_bug.part.37+0x1e8/0x1f0
[  374.292007]  [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150
[  374.298228]  [<ffffffff810b48e8>] mark_lock+0x198/0x2f0
[  374.303494]  [<ffffffff810b3de0>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0
[  374.310859]  [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10
[  374.316651]  [<ffffffff81338a61>] ? vsnprintf+0x461/0x640
[  374.322137]  [<ffffffffa05cb4b5>] ? ceph_osdc_start_request+0x65/0x160 [libceph]
[  374.329583]  [<ffffffff810b6d22>] lock_acquire+0xa2/0x140
[  374.335030]  [<ffffffffa05cb4b5>] ? ceph_osdc_start_request+0x65/0x160 [libceph]
[  374.342479]  [<ffffffffa05ca9a9>] ? ceph_osdc_build_request+0x119/0x650 [libceph]
[  374.350058]  [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320
[  374.355935]  [<ffffffffa05cb4b5>] ? ceph_osdc_start_request+0x65/0x160 [libceph]
[  374.363396]  [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph]
[  374.370673]  [<ffffffffa05cd0bf>] ceph_osdc_writepages+0xdf/0x180 [libceph]
[  374.377675]  [<ffffffffa061225a>] writepage_nounlock+0x22a/0x410 [ceph]
[  374.384364]  [<ffffffffa0612bee>] ceph_writepage+0x3e/0x70 [ceph]
[  374.390563]  [<ffffffff81143164>] shrink_page_list+0x554/0x900
[  374.396440]  [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[  374.402661]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[  374.408709]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[  374.414332]  [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300
[  374.419993]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[  374.426822]  [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80
[  374.433674]  [<ffffffff811452e3>] kswapd+0x5d3/0xa90
[  374.438681]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[  374.444990]  [<ffffffff81144d10>] ? try_to_free_pages+0x1a0/0x1a0
[  374.451126]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[  374.456049]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[  374.462271]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[  374.467713]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[  374.527388] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[  374.579652] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[  374.620391] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[  374.655635] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[  374.821508] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[  375.142073] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN)
[  375.209835] libceph: get_reply front 119 > preallocated 95
[  375.219197] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[  375.282686] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)

#3 Updated by Alex Elder about 11 years ago

I just noticed the end of the kernel messages, leading up
to the hard crash, was not captured in what I initially
added. Below is the missing part.

[ 4710.419742]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4710.425422]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4710.430591]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4710.435404]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4710.441078]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4710.447924]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4710.454124]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4710.460354]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4710.466549]  [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50
[ 4710.473013]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4710.478602]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4710.484392]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4710.490674]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4710.496868]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4710.502717]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4710.509115]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4710.515145]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4710.521209]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4710.526626]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4710.531971]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4710.537388]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4710.542721]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4710.548835]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4710.554862]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4710.560371]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4710.566484]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4710.572166]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4710.577843]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4710.583978]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4710.589825]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4710.595858]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4710.602401]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4710.607989]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4710.613848]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4710.618744]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4710.624977]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4710.630445]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4731.605879] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229119 c=229118 q=3269)
[ 4731.615303] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4731.622942] Call Trace:
[ 4731.625408]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4731.631480]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4731.637700]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4731.643722]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4731.649763]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4731.656216]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4731.661891]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4731.665841] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2106 jiffies, g=229119, c=229118, q=3269)
[ 4731.665843] Task dump for CPU 0:
[ 4731.665851] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4731.665854]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4731.665855]  ffff88043ff6ec38 0000000000000282 ffff880090289368 ffffffff81085343
[ 4731.665857]  ffffea000abfcdc0 ffffea000f4a8ec0 ffff880090289520 ffffea000f4a8ee0
[ 4731.665857] Call Trace:
[ 4731.665862]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4731.665866]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4731.665870]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4731.665872]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4731.665874]  [<ffffffff81339c9a>] ? delay_tsc+0x4a/0x80
[ 4731.665878]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4731.665880]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4731.665882]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4731.665884]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4731.665885]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4731.665887]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4731.665890]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4731.665894]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4731.665895]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4731.665897]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4731.665899]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4731.665903]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4731.665906]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4731.665909]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4731.665912]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4731.665913]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4731.665915]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4731.665919]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4731.665927]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4731.665929]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4731.665937]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4731.665940]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4731.665942]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4731.665943]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4731.665945]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4731.665947]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4731.665951]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4731.665953]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4731.665955]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4731.665956]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4731.665959]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4731.665962]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4731.665964]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4731.940584]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4731.946523]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4731.953072]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4731.958924]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4731.965123]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4731.971491]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4731.977514]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4731.983186]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4731.988867]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4731.994021]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4731.998841]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4732.004514]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4732.011403]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4732.017603]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4732.023799]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4732.029994]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4732.035858]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4732.041450]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4732.047230]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4732.053513]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4732.059717]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4732.065564]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4732.071936]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4732.077957]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4732.083980]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4732.089395]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4732.094725]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4732.100169]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4732.105499]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4732.111612]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4732.117638]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4732.123149]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4732.129262]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4732.134936]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4732.140613]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4732.146724]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4732.152572]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4732.158617]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4732.165165]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4732.170758]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4732.176611]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4732.181506]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4732.187707]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4732.193129]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4753.171915] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229120 c=229119 q=3487)
[ 4753.181334] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4753.188979] Call Trace:
[ 4753.191440]  <IRQ> [ 4753.191931] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229120, c=229119, q=3487)
[ 4753.191939] Task dump for CPU 0:
[ 4753.191944] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4753.191946]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4753.191948]  ffff88043ff85d38 0000000000000282 ffff880090289368 ffffffff81085343
[ 4753.191950]  ffffea0004fcc780 ffffea000f5df280 ffff880090289520 ffffea000f5df2a0
[ 4753.191952] Call Trace:
[ 4753.191953]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4753.191958]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4753.191963]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4753.191966]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4753.191968]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4753.191973]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4753.191977]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4753.191979]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4753.191980]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4753.191982]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4753.191983]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4753.191985]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4753.191988]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4753.191992]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4753.191994]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4753.191996]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4753.191998]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4753.192001]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4753.192004]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4753.192007]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4753.192010]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4753.192011]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4753.192013]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4753.192023]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4753.192025]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4753.192027]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4753.192036]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4753.192038]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4753.192040]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4753.192042]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4753.192043]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4753.192045]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4753.192049]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4753.192051]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4753.192052]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4753.192054]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4753.192056]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4753.192059]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4753.467433]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4753.473136]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4753.479326]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4753.485400]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4753.491459]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4753.497913]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4753.503613]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4753.509107]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4753.515035]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4753.521586]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4753.527429]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4753.533650]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4753.540019]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4753.546083]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4753.551802]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4753.557472]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4753.562625]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4753.567428]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4753.573156]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4753.579984]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4753.586172]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4753.592378]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4753.598572]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4753.604412]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4753.609993]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4753.615774]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4753.622049]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4753.628243]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4753.634102]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4753.640473]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4753.646493]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4753.652507]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4753.657923]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4753.663271]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4753.668679]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4753.674035]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4753.680137]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4753.686183]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4753.691681]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4753.697789]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4753.703464]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4753.709131]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4753.715267]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4753.721112]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4753.727134]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4753.733681]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4753.739270]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4753.745137]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4753.750024]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4753.756248]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4753.761656]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4778.731618] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229123 c=229122 q=2731)
[ 4778.741038] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4778.748641] Call Trace:
[ 4778.751102]  <IRQ> [ 4778.751647] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229123, c=229122, q=2731)
[ 4778.751656] Task dump for CPU 0:
[ 4778.751657] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4778.751662]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4778.751668]  ffff88043ff7a038 0000000000000282 ffff880090289368 ffffffff81085343
[ 4778.751670]  0000000000000024 ffffea000f6c21c0 ffff880090289520 ffffea000f6c21e0
[ 4778.751672] Call Trace:
[ 4778.751673]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4778.751679]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4778.751683]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4778.751686]  [<ffffffff81143ad4>] ? shrink_inactive_list+0x224/0x400
[ 4778.751688]  [<ffffffff81143ad4>] ? shrink_inactive_list+0x224/0x400
[ 4778.751690]  [<ffffffff81143745>] ? putback_inactive_pages+0x115/0x280
[ 4778.751692]  [<ffffffff81143b9a>] ? shrink_inactive_list+0x2ea/0x400
[ 4778.751694]  [<ffffffff81143b4a>] ? shrink_inactive_list+0x29a/0x400
[ 4778.751695]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4778.751697]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4778.751699]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4778.751702]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4778.751706]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4778.751708]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4778.751710]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4778.751712]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4778.751715]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4778.751718]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4778.751722]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4778.751724]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4778.751725]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4778.751728]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4778.751737]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4778.751740]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4778.751741]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4778.751750]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4778.751753]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4778.751754]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4778.751756]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4778.751758]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4778.751759]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4778.751764]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4778.751766]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4778.751767]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4778.751769]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4778.751771]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4778.751774]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4779.022525]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4779.028168]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4779.034358]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4779.040380]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4779.046421]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4779.052877]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4779.058550]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4779.064046]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4779.069981]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4779.076542]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4779.082391]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4779.088586]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4779.094956]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4779.100977]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4779.106653]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4779.112331]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4779.117485]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4779.122307]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4779.127973]  <EOI>  [<ffffffff81143745>] ? putback_inactive_pages+0x115/0x280
[ 4779.135147]  [<ffffffff81143b9a>] ? shrink_inactive_list+0x2ea/0x400
[ 4779.141509]  [<ffffffff81143b4a>] ? shrink_inactive_list+0x29a/0x400
[ 4779.147877]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4779.153736]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4779.159324]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4779.165080]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4779.171362]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4779.177557]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4779.183424]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4779.189792]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4779.195832]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4779.201855]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4779.207269]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4779.212590]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4779.217998]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4779.223321]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4779.229425]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4779.235446]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4779.241002]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4779.247109]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4779.252802]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4779.258470]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4779.264571]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4779.270411]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4779.276428]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4779.283013]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4779.288599]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4779.294467]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4779.299354]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4779.305579]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4779.310994]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4800.287722] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229124 c=229123 q=1159)
[ 4800.297147] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4800.304775] Call Trace:
[ 4800.307234]  <IRQ> [ 4800.307703] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229124, c=229123, q=1159)
[ 4800.307712] Task dump for CPU 0:
[ 4800.307714] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4800.307719]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4800.307726]  ffff88043ff6ec38 0000000000000282 ffff880090289368 ffffffff81085343
[ 4800.307727]  0000000000000030 ffffea000c6d0ec0 ffff880090289520 ffffea000c6d0ee0
[ 4800.307729] Call Trace:
[ 4800.307731]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4800.307736]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4800.307740]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4800.307744]  [<ffffffff81143ad4>] ? shrink_inactive_list+0x224/0x400
[ 4800.307746]  [<ffffffff8113e4d7>] ? lru_add_drain_cpu+0x27/0x130
[ 4800.307748]  [<ffffffff8113e676>] ? lru_add_drain+0x16/0x20
[ 4800.307749]  [<ffffffff8114396a>] ? shrink_inactive_list+0xba/0x400
[ 4800.307754]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4800.307755]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4800.307757]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4800.307760]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4800.307764]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4800.307766]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4800.307767]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4800.307770]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4800.307772]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4800.307776]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4800.307779]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4800.307781]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4800.307783]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4800.307785]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4800.307794]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4800.307796]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4800.307798]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4800.307807]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4800.307810]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4800.307811]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4800.307813]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4800.307815]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4800.307816]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4800.307821]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4800.307822]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4800.307824]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4800.307826]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4800.307828]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4800.307831]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4800.569834]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4800.575474]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4800.581717]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4800.587740]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4800.593765]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4800.600216]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4800.605884]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4800.611379]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4800.617333]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4800.623875]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4800.629769]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4800.635964]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4800.642334]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4800.648413]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4800.654081]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4800.659778]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4800.664931]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4800.669734]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4800.675402]  <EOI>  [<ffffffff8113e4d7>] ? lru_add_drain_cpu+0x27/0x130
[ 4800.682055]  [<ffffffff8113e676>] lru_add_drain+0x16/0x20
[ 4800.687481]  [<ffffffff8114396a>] shrink_inactive_list+0xba/0x400
[ 4800.693584]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4800.699484]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4800.705066]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4800.710830]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4800.717111]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4800.723308]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4800.729157]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4800.735525]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4800.741555]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4800.747588]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4800.753003]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4800.758324]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4800.763732]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4800.769081]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4800.775193]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4800.781264]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4800.786760]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4800.792867]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4800.798536]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4800.804210]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4800.810321]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4800.816162]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4800.822214]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4800.828763]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4800.834351]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4800.840192]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4800.845087]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4800.851277]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4800.856693]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4821.863718] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229126 c=229125 q=2904)
[ 4821.874318] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4821.881910] Call Trace:
[ 4821.883727] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229126, c=229125, q=2904)
[ 4821.883728] Task dump for CPU 0:
[ 4821.883733] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4821.883738]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4821.883743]  ffff88043ff72238 0000000000000282 ffff880090289368 ffffffff81085343
[ 4821.883745]  0000000000000030 ffffea000c6e2140 ffff880090289520 ffffea000c6e2160
[ 4821.883745] Call Trace:
[ 4821.883752]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4821.883756]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4821.883760]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4821.883761]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4821.883763]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4821.883767]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4821.883772]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4821.883774]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4821.883775]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4821.883777]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4821.883779]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4821.883781]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4821.883785]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4821.883787]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4821.883789]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4821.883791]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4821.883794]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4821.883796]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4821.883799]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4821.883801]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4821.883802]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4821.883804]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4821.883807]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4821.883814]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4821.883816]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4821.883824]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4821.883826]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4821.883828]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4821.883829]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4821.883831]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4821.883832]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4821.883837]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4821.883838]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4821.883840]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4821.883841]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4821.883843]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4821.883847]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4821.883848]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4822.157139]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4822.163194]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4822.169384]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4822.175466]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4822.181489]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4822.187945]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4822.193619]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4822.199112]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4822.205058]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4822.211593]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4822.217444]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4822.223636]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4822.229998]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4822.236063]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4822.241738]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4822.247409]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4822.252562]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4822.257365]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4822.263031]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4822.269858]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4822.276082]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4822.282270]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4822.288464]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4822.294306]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4822.299887]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4822.305695]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4822.311977]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4822.318172]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4822.324021]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4822.330392]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4822.336422]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4822.342444]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4822.347877]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4822.353196]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4822.358605]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4822.363928]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4822.370030]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4822.376096]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4822.381601]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4822.387725]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4822.393400]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4822.399068]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4822.405185]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4822.411026]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4822.417103]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4822.423645]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4822.429231]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4822.435073]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4822.439962]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4822.446159]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4822.451574]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4843.429725] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229127 c=229126 q=1032)
[ 4843.439145] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4843.446749] Call Trace:
[ 4843.449208]  <IRQ> [ 4843.449782] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229127, c=229126, q=1032)
[ 4843.449790] Task dump for CPU 0:
[ 4843.449792] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4843.449797]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4843.449803]  ffff8800902893a8 0000000000000282 0000000000000030 0000000000000000
[ 4843.449805]  ffff880090289540 ffff880090289538 0000000000000000 0000000000000000
[ 4843.449807] Call Trace:
[ 4843.449809]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4843.449814]  [<ffffffff8165c8e0>] ? _cond_resched+0x20/0x40
[ 4843.449818]  [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900
[ 4843.449822]  [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0
[ 4843.449824]  [<ffffffff81143ac7>] ? shrink_inactive_list+0x217/0x400
[ 4843.449826]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4843.449828]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4843.449830]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4843.449833]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4843.449836]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4843.449838]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4843.449840]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4843.449843]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4843.449845]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4843.449848]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4843.449852]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4843.449855]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4843.449856]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4843.449859]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4843.449866]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4843.449869]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4843.449870]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4843.449879]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4843.449881]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4843.449883]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4843.449885]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4843.449887]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4843.449888]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4843.449893]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4843.449895]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4843.449896]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4843.449898]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4843.449900]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4843.449903]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4843.700929]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4843.706572]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4843.712762]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4843.718784]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4843.724817]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4843.731270]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4843.736938]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4843.742449]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4843.748377]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4843.754946]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4843.760797]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4843.766990]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4843.773352]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4843.779374]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4843.785049]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4843.790728]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4843.795881]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4843.800736]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4843.806402]  <EOI>  [<ffffffff8165c8e0>] ? _cond_resched+0x20/0x40
[ 4843.812683]  [<ffffffff81142d07>] shrink_page_list+0xf7/0x900
[ 4843.818441]  [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0
[ 4843.825148]  [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[ 4843.831360]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4843.837219]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4843.842807]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4843.849796]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4843.856075]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4843.862270]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4843.868118]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4843.874487]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4843.880510]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4843.886532]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4843.891957]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4843.897286]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4843.902746]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4843.908068]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4843.914181]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4843.920200]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4843.925697]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4843.931805]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4843.937480]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4843.943175]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4843.949285]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4843.955133]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4843.961159]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4843.967697]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4843.973278]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4843.979119]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4843.984008]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4843.990239]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4843.995654]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4872.174500] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229131 c=229130 q=802)
[ 4872.183835] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4872.191473] Call Trace:
[ 4872.193934]  <IRQ> [ 4872.194486] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229131, c=229130, q=802)
[ 4872.194494] Task dump for CPU 0:
[ 4872.194496] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4872.194501]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4872.194507]  ffff88043ffadc38 0000000000000282 ffff8800902894b8 ffffffff81085343
[ 4872.194509]  0000000000000030 57ffacd1164bbe40 a8003d2ef9000000 f200000000000000
[ 4872.194511] Call Trace:
[ 4872.194513]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4872.194518]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4872.194522]  [<ffffffff81142d4c>] ? shrink_page_list+0x13c/0x900
[ 4872.194526]  [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900
[ 4872.194528]  [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0
[ 4872.194530]  [<ffffffff81143ac7>] ? shrink_inactive_list+0x217/0x400
[ 4872.194531]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4872.194533]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4872.194535]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4872.194538]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4872.194542]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4872.194544]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4872.194546]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4872.194548]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4872.194551]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4872.194554]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4872.194558]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4872.194560]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4872.194562]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4872.194564]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4872.194573]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4872.194576]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4872.194577]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4872.194586]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4872.194588]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4872.194590]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4872.194592]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4872.194594]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4872.194595]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4872.194600]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4872.194602]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4872.194603]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4872.194605]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4872.194607]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4872.194611]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4872.451123]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4872.456791]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4872.462981]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4872.469030]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4872.475053]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4872.481508]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4872.487182]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4872.492677]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4872.498631]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4872.505191]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4872.511042]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4872.517236]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4872.523649]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4872.529704]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4872.535398]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4872.541067]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4872.546223]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4872.551034]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4872.556717]  <EOI>  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4872.562591]  [<ffffffff81142d4c>] ? shrink_page_list+0x13c/0x900
[ 4872.568608]  [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900
[ 4872.574537]  [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0
[ 4872.581251]  [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[ 4872.587490]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4872.593339]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4872.598919]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4872.604676]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4872.610957]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4872.617153]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4872.623001]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4872.629382]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4872.635419]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4872.641441]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4872.646849]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4872.652170]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4872.657604]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4872.662927]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4872.669072]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4872.675154]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4872.680659]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4872.686809]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4872.692484]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4872.698152]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4872.704255]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4872.710102]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4872.716128]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4872.722668]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4872.728273]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4872.734115]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4872.739004]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4872.745245]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4872.750660]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4893.730552] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229132 c=229131 q=1229)
[ 4893.739986] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4893.747580] Call Trace:
[ 4893.750042]  <IRQ> [ 4893.750534] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229132, c=229131, q=1229)
[ 4893.750543] Task dump for CPU 0:
[ 4893.750545] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4893.750549]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4893.750556]  ffff88043ff6f538 0000000000000282 ffff880090289368 ffffffff81085343
[ 4893.750558]  ffffffff8113a207 ffffea0010856c80 ffff880090289520 ffffea0010856ca0
[ 4893.750560] Call Trace:
[ 4893.750561]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4893.750567]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4893.750570]  [<ffffffff8113a207>] ? free_hot_cold_page_list+0x27/0xc0
[ 4893.750573]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4893.750576]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4893.750578]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4893.750580]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4893.750584]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4893.750586]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4893.750588]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4893.750590]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4893.750591]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4893.750593]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4893.750596]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4893.750600]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4893.750602]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4893.750604]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4893.750606]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4893.750608]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4893.750612]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4893.750615]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4893.750618]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4893.750619]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4893.750622]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4893.750631]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4893.750634]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4893.750635]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4893.750644]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4893.750646]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4893.750648]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4893.750650]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4893.750652]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4893.750653]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4893.750658]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4893.750659]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4893.750661]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4893.750663]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4893.750665]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4893.750668]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4894.033598]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4894.039240]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4894.045481]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4894.051503]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4894.057520]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4894.063974]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4894.069649]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4894.075177]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4894.081105]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4894.087640]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4894.093500]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4894.099692]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4894.106081]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4894.112100]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4894.117767]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4894.123437]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4894.128593]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4894.133438]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4894.139105]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4894.145967]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4894.152199]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4894.158387]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4894.164583]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4894.170431]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4894.176012]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4894.181802]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4894.188083]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4894.194295]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4894.200136]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4894.206499]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4894.212520]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4894.218542]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4894.223958]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4894.229279]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4894.234687]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4894.240063]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4894.246172]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4894.252210]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4894.257708]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4894.263841]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4894.269516]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4894.275202]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4894.281310]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4894.287151]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4894.293176]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4894.299722]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4894.305305]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4894.311162]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4894.316094]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4894.322282]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4894.327690]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4915.316514] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229135 c=229134 q=511)
[ 4915.325888] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4915.333534] Call Trace:
[ 4915.335993]  <IRQ> [ 4915.336541] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229135, c=229134, q=511)
[ 4915.336549] Task dump for CPU 0:
[ 4915.336551] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4915.336556]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4915.336562]  ffff88043ffa2838 0000000000000282 ffff880090289368 ffffffff81085343
[ 4915.336564]  ffffea0010335540 ffffea0006785fc0 ffff880090289520 ffffea0006785fe0
[ 4915.336566] Call Trace:
[ 4915.336567]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4915.336573]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4915.336577]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4915.336581]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4915.336582]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4915.336584]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4915.336588]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4915.336590]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4915.336592]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4915.336594]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4915.336595]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4915.336597]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4915.336600]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4915.336604]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4915.336606]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4915.336608]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4915.336610]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4915.336613]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4915.336616]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4915.336620]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4915.336622]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4915.336624]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4915.336626]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4915.336635]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4915.336638]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4915.336640]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4915.336648]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4915.336651]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4915.336653]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4915.336654]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4915.336656]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4915.336658]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4915.336662]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4915.336664]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4915.336665]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4915.336668]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4915.336669]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4915.336673]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 4915.611745]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4915.617420]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4915.623610]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4915.629677]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4915.635692]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4915.642163]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4915.647840]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4915.653342]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4915.659280]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4915.665823]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4915.671689]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4915.677883]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4915.684244]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4915.690266]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4915.696010]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4915.701681]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4915.706859]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4915.711662]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4915.717365]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4915.724193]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4915.730381]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4915.736568]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4915.742762]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4915.748622]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4915.754210]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4915.759983]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4915.766267]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4915.772462]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4915.778311]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4915.784679]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4915.790719]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4915.796743]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4915.802156]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4915.807488]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4915.812894]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4915.818244]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4915.824354]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4915.830392]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4915.835890]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4915.841996]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4915.847681]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4915.853348]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4915.859475]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4915.865315]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4915.871402]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4915.877976]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4915.883557]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4915.889432]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4915.894319]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4915.900509]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4915.905925]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4948.154842] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229140 c=229139 q=419)
[ 4948.164177] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4948.171814] Call Trace:
[ 4948.174273]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4948.180363]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4948.186587]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4948.192610]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4948.194833] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2104 jiffies, g=229140, c=229139, q=419)
[ 4948.194835] Task dump for CPU 0:
[ 4948.194839] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4948.194844]  ffff880090289308 ffff88042d073f20 ffff880090289328 0000000000000282
[ 4948.194850]  ffff88043ffda338 0000000000000282 ffff880090289368 ffffffff81085343
[ 4948.194852]  0000000000000030 ffffea0002608100 ffff880090289520 ffffea0002608120
[ 4948.194852] Call Trace:
[ 4948.194858]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4948.194862]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4948.194864]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4948.194866]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4948.194870]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4948.194871]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4948.194873]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4948.194875]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4948.194878]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4948.194880]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4948.194882]  [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300
[ 4948.194884]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4948.194886]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4948.194888]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4948.194891]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4948.194894]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4948.194897]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4948.194900]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4948.194901]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4948.194903]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4948.194907]  [<ffffffff813060f8>] ? cry

#4 Updated by Alex Elder about 11 years ago

Let's see if this one doesn't exceed the maximum size
something added.

[ 4948.194903]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4948.194907]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4948.194913]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4948.194915]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4948.194924]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4948.194926]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4948.194927]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4948.194929]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4948.194932]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4948.194933]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4948.194938]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4948.194940]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4948.194941]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4948.194944]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4948.194946]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4948.194948]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4948.194950]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4948.460699]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4948.467178]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4948.472853]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4948.478374]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4948.484392]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4948.490930]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4948.496782]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4948.502975]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4948.509364]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4948.515385]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4948.521060]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4948.526746]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4948.531902]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4948.536756]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4948.542423]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4948.549250]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4948.555483]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4948.561678]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4948.567890]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4948.573914]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4948.579511]  [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300
[ 4948.585092]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4948.591279]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4948.597146]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4948.603515]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4948.609556]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4948.615586]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4948.620993]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4948.626324]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4948.631732]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4948.637080]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4948.643182]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4948.649229]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4948.654734]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4948.660842]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4948.666516]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4948.672186]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4948.678346]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4948.684238]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4948.690255]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4948.696812]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4948.702393]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4948.708268]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4948.713157]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4948.719422]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4948.724889]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4969.700894] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229141 c=229140 q=2998)
[ 4969.710323] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4969.717971] Call Trace:
[ 4969.720430]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4969.726502]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4969.732735]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4969.738758]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4969.744799]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4969.751251]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4969.756920]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4969.762422]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4969.768349]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4969.770817] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2107 jiffies, g=229141, c=229140, q=2998)
[ 4969.770818] Task dump for CPU 0:
[ 4969.770823] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4969.770828]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4969.770834]  ffff88043ffaf738 0000000000000282 ffff880090289368 ffffffff81085343
[ 4969.770836]  0000000000000030 ffffea00078b8c00 ffff880090289520 ffffea00078b8c20
[ 4969.770836] Call Trace:
[ 4969.770840]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4969.770844]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4969.770848]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4969.770850]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4969.770852]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4969.770856]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4969.770858]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4969.770859]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4969.770861]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4969.770863]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4969.770865]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4969.770867]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4969.770871]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4969.770873]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4969.770875]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4969.770877]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4969.770880]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4969.770883]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4969.770886]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4969.770889]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4969.770891]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4969.770893]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4969.770896]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4969.770905]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4969.770906]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4969.770915]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4969.770917]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4969.770919]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4969.770920]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4969.770922]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4969.770924]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4969.770928]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4969.770930]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4969.770931]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4969.770933]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4969.770935]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4969.770938]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4969.770940]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4970.047922]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4970.053789]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4970.059984]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4970.066370]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4970.072393]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4970.078067]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4970.083736]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4970.088882]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4970.093728]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4970.099395]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4970.106249]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4970.112481]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4970.118668]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4970.124916]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4970.130836]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4970.136425]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4970.142205]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4970.148487]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4970.154682]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4970.160583]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4970.166944]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4970.172992]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4970.179005]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4970.184448]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4970.189769]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4970.195212]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4970.200537]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4970.206646]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4970.212667]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4970.218172]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4970.224295]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4970.229963]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4970.236806]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4970.242968]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4970.248809]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4970.254860]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4970.261401]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4970.266989]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4970.272841]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4970.277735]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4970.283943]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4970.289358]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4991.266987] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229142 c=229141 q=1032)
[ 4991.276444] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 4991.284037] Call Trace:
[ 4991.286498]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 4991.292553]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 4991.298766]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 4991.304781]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 4991.310832]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 4991.317286]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 4991.322962]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 4991.328464]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 4991.334391]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 4991.336861] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2107 jiffies, g=229142, c=229141, q=1034)
[ 4991.336863] Task dump for CPU 0:
[ 4991.336868] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 4991.336874]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 4991.336876]  ffff88043ffa2838 0000000000000282 ffff880090289368 ffffffff81085343
[ 4991.336878]  0000000000000024 ffffea000c6d9fc0 ffff880090289520 ffffea000c6d9fe0
[ 4991.336878] Call Trace:
[ 4991.336882]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4991.336886]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 4991.336890]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 4991.336891]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4991.336893]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 4991.336897]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4991.336899]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4991.336901]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4991.336903]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 4991.336904]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4991.336906]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 4991.336909]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4991.336912]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4991.336914]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 4991.336916]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 4991.336918]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 4991.336921]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 4991.336925]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 4991.336928]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 4991.336931]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 4991.336932]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4991.336934]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 4991.336938]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4991.336946]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4991.336948]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 4991.336956]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 4991.336958]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4991.336960]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4991.336961]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4991.336963]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 4991.336965]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4991.336969]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4991.336971]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 4991.336972]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4991.336974]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 4991.336976]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4991.336979]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 4991.336981]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4991.613772]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 4991.619649]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 4991.625844]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 4991.632265]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 4991.638286]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 4991.643954]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 4991.649623]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 4991.654777]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 4991.659588]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 4991.665257]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4991.672108]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 4991.678369]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 4991.684564]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 4991.690758]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 4991.696600]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 4991.702188]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 4991.707953]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 4991.714234]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 4991.720448]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 4991.726297]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 4991.732667]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 4991.738689]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 4991.744711]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 4991.750127]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 4991.755447]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 4991.760900]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 4991.766240]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 4991.772349]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 4991.778379]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 4991.783877]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 4991.789992]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 4991.795667]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 4991.801353]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 4991.807462]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 4991.813302]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 4991.819319]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 4991.825859]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 4991.831448]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 4991.837315]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 4991.842212]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 4991.848426]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 4991.853842]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5012.832939] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229143 c=229142 q=3496)
[ 5012.842373] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 5012.849968] Call Trace:
[ 5012.852429]  <IRQ> [ 5012.852967] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229143, c=229142, q=3496)
[ 5012.852976] Task dump for CPU 0:
[ 5012.852978] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 5012.852983]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 5012.852989]  ffff88043ff6ec38 0000000000000282 ffff880090289368 ffffffff81085343
[ 5012.852991]  0200000000040818 ffffea000e8baec0 ffff880090289520 ffffea000e8baee0
[ 5012.852993] Call Trace:
[ 5012.852994]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5012.852999]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5012.853003]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 5012.853007]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5012.853009]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5012.853011]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5012.853017]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5012.853019]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5012.853021]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 5012.853022]  [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50
[ 5012.853025]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 5012.853026]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5012.853029]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5012.853033]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 5012.853035]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 5012.853037]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 5012.853039]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 5012.853041]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 5012.853044]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 5012.853048]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 5012.853050]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5012.853052]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 5012.853054]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5012.853063]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5012.853065]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 5012.853067]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 5012.853075]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5012.853078]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5012.853080]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5012.853082]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 5012.853083]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5012.853085]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5012.853089]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 5012.853091]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5012.853092]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 5012.853094]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5012.853096]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 5012.853099]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 5013.129138]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 5013.134815]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 5013.141007]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 5013.147047]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 5013.153068]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 5013.159523]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 5013.165198]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 5013.170693]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 5013.176657]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 5013.183198]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 5013.190217]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 5013.196482]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 5013.202887]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 5013.208906]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 5013.214574]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 5013.220251]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 5013.225407]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 5013.230208]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 5013.235876]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5013.242737]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5013.248927]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5013.255140]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 5013.261335]  [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50
[ 5013.267782]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 5013.273380]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5013.279142]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5013.285443]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 5013.291638]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 5013.297505]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 5013.303875]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 5013.309896]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 5013.315918]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 5013.321334]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 5013.326682]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5013.332090]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 5013.337437]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5013.343541]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5013.349562]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 5013.355076]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 5013.361181]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5013.366868]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5013.372542]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5013.378644]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 5013.384510]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5013.390526]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5013.397067]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 5013.402655]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5013.408496]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 5013.413419]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5013.419609]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 5013.425049]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5034.508803] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229147 c=229146 q=525)
[ 5034.518137] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 5034.525738] Call Trace:
[ 5034.528199]  <IRQ> [ 5034.528765] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229147, c=229146, q=525)
[ 5034.528769] Task dump for CPU 0:
[ 5034.528770] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 5034.528773]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 5034.528778]  ffff88043ff7a038 0000000000000282 ffff880090289368 ffffffff81085343
[ 5034.528780]  0000000000000030 ffffea0005b3c1c0 ffff880090289520 ffffea0005b3c1e0
[ 5034.528782] Call Trace:
[ 5034.528783]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5034.528788]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5034.528792]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 5034.528795]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5034.528797]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5034.528799]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5034.528803]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5034.528805]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5034.528807]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 5034.528808]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5034.528810]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 5034.528812]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5034.528819]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5034.528825]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 5034.528829]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 5034.528834]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 5034.528838]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 5034.528843]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 5034.528848]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 5034.528856]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 5034.528858]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5034.528860]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 5034.528862]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5034.528871]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5034.528874]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 5034.528876]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 5034.528884]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5034.528887]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5034.528888]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5034.528890]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 5034.528892]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5034.528894]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5034.528899]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 5034.528900]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5034.528902]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 5034.528904]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5034.528906]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 5034.528909]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 5034.804209]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 5034.809895]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 5034.816085]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 5034.822135]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 5034.828158]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 5034.834628]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 5034.840304]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 5034.845799]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 5034.851761]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 5034.858372]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 5034.864223]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 5034.870443]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 5034.876815]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 5034.882843]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 5034.888579]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 5034.894249]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 5034.899438]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 5034.904241]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 5034.909942]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5034.916770]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5034.923035]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5034.929310]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 5034.935504]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5034.941363]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 5034.946951]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5034.952733]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5034.959067]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 5034.965262]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 5034.971112]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 5034.977474]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 5034.983496]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 5034.989525]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 5034.994934]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 5035.000324]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5035.005732]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 5035.011054]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5035.017157]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5035.023204]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 5035.028709]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 5035.034817]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5035.040491]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5035.046160]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5035.052285]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 5035.058126]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5035.064143]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5035.070701]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 5035.076289]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5035.082166]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 5035.087062]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5035.093293]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 5035.098778]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5056.074880] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229148 c=229147 q=2975)
[ 5056.084318] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 5056.091911] Call Trace:
[ 5056.094370]  <IRQ> [ 5056.094801] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229148, c=229147, q=2975)
[ 5056.094805] Task dump for CPU 0:
[ 5056.094806] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 5056.094809]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 5056.094811]  ffff88043ff98638 0000000000000282 ffff880090289368 ffffffff81085343
[ 5056.094813]  0000000000000030 ffffea000f77e840 ffff880090289520 ffffea000f77e860
[ 5056.094814] Call Trace:
[ 5056.094816]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5056.094821]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5056.094824]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 5056.094828]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5056.094830]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5056.094832]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5056.094836]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5056.094837]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5056.094839]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 5056.094841]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5056.094842]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 5056.094844]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5056.094847]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5056.094850]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 5056.094852]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 5056.094854]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 5056.094856]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 5056.094859]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 5056.094862]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 5056.094866]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 5056.094868]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5056.094870]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 5056.094872]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5056.094881]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5056.094883]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 5056.094885]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 5056.094893]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5056.094896]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5056.094898]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5056.094900]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 5056.094901]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5056.094903]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5056.094907]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 5056.094909]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5056.094910]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 5056.094913]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5056.094914]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 5056.094917]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 5056.371644]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 5056.377314]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 5056.383503]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 5056.389552]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 5056.395585]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 5056.402038]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 5056.407730]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 5056.413224]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 5056.419152]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 5056.425686]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 5056.431529]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 5056.437740]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 5056.444109]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 5056.450174]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 5056.455850]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 5056.461519]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 5056.466673]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 5056.471476]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 5056.477187]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5056.484015]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5056.490202]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5056.496390]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 5056.502586]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5056.508434]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 5056.514016]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5056.519804]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5056.526164]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 5056.532352]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 5056.538200]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 5056.544563]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 5056.550585]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 5056.556606]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 5056.562022]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 5056.567361]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5056.572769]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 5056.578152]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5056.584332]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5056.590354]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 5056.595859]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 5056.601966]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5056.607649]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5056.613316]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5056.619444]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 5056.625310]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5056.631326]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5056.637895]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 5056.643481]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5056.649324]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 5056.654219]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5056.660408]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 5056.665824]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5077.660843] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229152 c=229151 q=602)
[ 5077.670183] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 5077.677777] Call Trace:
[ 5077.680236]  <IRQ> [ 5077.680901] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229152, c=229151, q=602)
[ 5077.680910] Task dump for CPU 0:
[ 5077.680912] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 5077.680919]  ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282
[ 5077.680921]  ffff88043ff98638 0000000000000282 ffff880090289368 ffffffff81085343
[ 5077.680923]  0000000000000030 ffffea0010ad8840 ffff880090289520 ffffea0010ad8860
[ 5077.680925] Call Trace:
[ 5077.680927]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5077.680932]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5077.680936]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 5077.680940]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5077.680942]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5077.680944]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5077.680948]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5077.680950]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5077.680952]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 5077.680954]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5077.680955]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 5077.680957]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5077.680960]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5077.680964]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 5077.680966]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 5077.680968]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 5077.680970]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 5077.680973]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 5077.680976]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 5077.680980]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 5077.680982]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5077.680984]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 5077.680986]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5077.680995]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5077.680997]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 5077.681002]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 5077.681010]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5077.681012]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5077.681014]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5077.681016]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 5077.681017]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5077.681019]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5077.681023]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 5077.681025]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5077.681026]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 5077.681028]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5077.681030]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 5077.681032]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 5077.956203]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 5077.961847]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 5077.968037]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 5077.974086]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 5077.980109]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 5077.986579]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 5077.992263]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 5077.997757]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 5078.003721]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 5078.010262]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 5078.016105]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 5078.022300]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 5078.028669]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 5078.034689]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 5078.040364]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 5078.046035]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 5078.051216]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 5078.056018]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 5078.061685]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5078.068512]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5078.074711]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5078.080907]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 5078.087102]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5078.092976]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 5078.098557]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5078.104312]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5078.110586]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 5078.116781]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 5078.122665]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 5078.129026]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 5078.135049]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 5078.141070]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 5078.146478]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 5078.151807]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5078.157215]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 5078.162573]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5078.168675]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5078.174722]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 5078.180236]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 5078.186344]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5078.192037]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5078.197746]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5078.203892]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 5078.209740]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5078.215818]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5078.222418]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 5078.228007]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5078.233881]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 5078.238770]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5078.244986]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 5078.250469]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5122.290598] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229160 c=229159 q=6525)
[ 5122.301268] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 5122.308860] Call Trace:
[ 5122.310589] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229160, c=229159, q=6525)
[ 5122.310591] Task dump for CPU 0:
[ 5122.310596] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 5122.310600]  ffff880090289288 000000018027001a 0000000000000000 ffffffff811b699b
[ 5122.310606]  ffff880400000000 0000000000000000 ffff88043fffbe68 ffff88042d073f20
[ 5122.310607]  0000000000000000 ffff88043ff6ec38 0000000000000000 ffffffff81085322
[ 5122.310608] Call Trace:
[ 5122.310614]  [<ffffffff811b699b>] ? try_to_free_buffers+0x4b/0xc0
[ 5122.310619]  [<ffffffff81085322>] ? __wake_up+0x32/0x70
[ 5122.310620]  [<ffffffff81085322>] ? __wake_up+0x32/0x70
[ 5122.310623]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5122.310624]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5122.310628]  [<ffffffff81142d42>] ? shrink_page_list+0x132/0x900
[ 5122.310630]  [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900
[ 5122.310632]  [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0
[ 5122.310634]  [<ffffffff81143ac7>] ? shrink_inactive_list+0x217/0x400
[ 5122.310635]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5122.310637]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 5122.310640]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5122.310643]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5122.310645]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 5122.310647]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 5122.310649]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 5122.310652]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 5122.310656]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 5122.310659]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 5122.310661]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 5122.310663]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5122.310665]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 5122.310669]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5122.310677]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5122.310679]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 5122.310687]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 5122.310690]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5122.310692]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5122.310693]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5122.310695]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 5122.310697]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5122.310701]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5122.310703]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 5122.310704]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5122.310706]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 5122.310708]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5122.310711]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 5122.310713]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5122.582273]  <IRQ>  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 5122.588326]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 5122.594516]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 5122.600531]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 5122.606555]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 5122.613010]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 5122.618685]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 5122.624196]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 5122.630128]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 5122.636666]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 5122.642525]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 5122.648719]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 5122.655089]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 5122.661111]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 5122.666786]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 5122.672473]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 5122.677627]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 5122.682483]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 5122.688149]  <EOI>  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5122.694022]  [<ffffffff81142d42>] ? shrink_page_list+0x132/0x900
[ 5122.700092]  [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900
[ 5122.706027]  [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0
[ 5122.712763]  [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400
[ 5122.718955]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5122.724823]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 5122.730419]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5122.736174]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5122.742458]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 5122.748652]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 5122.754494]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 5122.760864]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 5122.766886]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 5122.772950]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 5122.778359]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 5122.783722]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5122.789130]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 5122.794452]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5122.800555]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5122.806576]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 5122.812089]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 5122.818197]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5122.823915]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5122.829585]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5122.835695]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 5122.841542]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5122.847559]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5122.854101]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 5122.859687]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5122.865531]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 5122.870425]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5122.876615]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 5122.882057]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5143.856635] INFO: rcu_sched self-detected stall on CPU { 0}  (t=2100 jiffies g=229161 c=229160 q=9091)
[ 5143.866081] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1
[ 5143.873690] Call Trace:
[ 5143.876151]  <IRQ> [ 5143.876621] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2102 jiffies, g=229161, c=229160, q=9091)
[ 5143.876629] Task dump for CPU 0:
[ 5143.876631] kworker/0:2     R  running task        0 26683      2 0x00000000
[ 5143.876636]  ffff880090289288 0000000180270021 0000000000000000 ffffffff811b699b
[ 5143.876642]  ffff880400000000 0000000000000000 ffff88043fffbe68 ffff88042d073f20
[ 5143.876643]  0000000000000000 ffff88043ff98638 0000000000000000 ffffffff81085322
[ 5143.876645] Call Trace:
[ 5143.876647]  [<ffffffff811b699b>] ? try_to_free_buffers+0x4b/0xc0
[ 5143.876652]  [<ffffffff81085322>] ? __wake_up+0x32/0x70
[ 5143.876657]  [<ffffffff81085322>] ? __wake_up+0x32/0x70
[ 5143.876659]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5143.876661]  [<ffffffff81085343>] ? __wake_up+0x53/0x70
[ 5143.876663]  [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900
[ 5143.876666]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5143.876668]  [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400
[ 5143.876670]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5143.876674]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5143.876675]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5143.876677]  [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400
[ 5143.876679]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5143.876680]  [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470
[ 5143.876682]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5143.876685]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5143.876689]  [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0
[ 5143.876691]  [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0
[ 5143.876693]  [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0
[ 5143.876695]  [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170
[ 5143.876698]  [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130
[ 5143.876701]  [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30
[ 5143.876704]  [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0
[ 5143.876707]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5143.876708]  [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0
[ 5143.876711]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5143.876720]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5143.876723]  [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60
[ 5143.876724]  [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph]
[ 5143.876733]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5143.876736]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5143.876737]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5143.876739]  [<ffffffff81072ba9>] ? process_one_work+0x199/0x510
[ 5143.876741]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5143.876742]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5143.876747]  [<ffffffff81074895>] ? worker_thread+0x165/0x3f0
[ 5143.876749]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5143.876751]  [<ffffffff8107a4ba>] ? kthread+0xea/0xf0
[ 5143.876753]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5143.876755]  [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0
[ 5143.876758]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0

[ 5144.168724]  [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0
[ 5144.174365]  [<ffffffff8108f653>] ? account_system_time+0xb3/0x210
[ 5144.180611]  [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160
[ 5144.186691]  [<ffffffff81066638>] update_process_times+0x48/0x90
[ 5144.192715]  [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50
[ 5144.199187]  [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80
[ 5144.204860]  [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0
[ 5144.210373]  [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0
[ 5144.216300]  [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0
[ 5144.222835]  [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250
[ 5144.228704]  [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40
[ 5144.234896]  [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230
[ 5144.241284]  [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0
[ 5144.247306]  [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70
[ 5144.252981]  [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110
[ 5144.258649]  [<ffffffff81016212>] handle_irq+0x22/0x40
[ 5144.263804]  [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0
[ 5144.269806]  [<ffffffff8165e732>] common_interrupt+0x72/0x72
[ 5144.275478]  <EOI>  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5144.282303]  [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40
[ 5144.288528]  [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40
[ 5144.294725]  [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400
[ 5144.300927]  [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10
[ 5144.306769]  [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470
[ 5144.312358]  [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150
[ 5144.318129]  [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60
[ 5144.324403]  [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0
[ 5144.330624]  [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0
[ 5144.336464]  [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0
[ 5144.342828]  [<ffffffff8117110a>] alloc_pages_current+0xba/0x170
[ 5144.348857]  [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130
[ 5144.354879]  [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30
[ 5144.360313]  [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0
[ 5144.365634]  [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0
[ 5144.371084]  [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0
[ 5144.376417]  [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30
[ 5144.382528]  [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c]
[ 5144.388549]  [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60
[ 5144.394054]  [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph]
[ 5144.400169]  [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0
[ 5144.405852]  [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0
[ 5144.411522]  [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110
[ 5144.417632]  [<ffffffff81072ba9>] process_one_work+0x199/0x510
[ 5144.423480]  [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510
[ 5144.429515]  [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph]
[ 5144.436054]  [<ffffffff81074895>] worker_thread+0x165/0x3f0
[ 5144.441643]  [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0
[ 5144.447492]  [<ffffffff8107a4ba>] kthread+0xea/0xf0
[ 5144.452379]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5144.458638]  [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0
[ 5144.464046]  [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0
[ 5146.035958] libceph: mds0 10.214.132.118:6809 socket closed (con state OPEN)
[ 5146.077716] ceph: mds0 hung
[ 5146.080535] ceph: mds0 caps stale
[ 5146.083961] libceph: mon1 10.214.132.118:6789 socket closed (con state OPEN)
[ 5146.091143] libceph: mon1 10.214.132.118:6789 session lost, hunting for new mon
[ 5146.120629] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)
[ 5146.184423] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN)
[ 5146.191666] libceph: mds0 10.214.132.118:6809 connection reset
[ 5146.197690] libceph: reset on mds0
[ 5146.201208] ceph: mds0 closed our session
[ 5146.205344] ceph: mds0 reconnect start
[ 5146.273394] libceph: mon1 10.214.132.118:6789 session established
[ 5146.279819] ceph: mds0 reconnect denied
[ 5146.284105] ceph:  dropping dirty+flushing Fw state for ffff880421c9b430 1099511787263
[ 5146.292254] ceph:  dropping dirty data for ffff880421c9b430 1099511787263
[ 5146.299249] ceph:  dropping dirty Fw state for ffff880420e8f3f0 1099511787213
[ 5146.306606] ceph:  dropping dirty data for ffff880420e8f3f0 1099511787213
[ 5146.313569] ceph:  dropping dirty Fw state for ffff880420e8ebf8 1099511787217
[ 5146.320818] ceph:  dropping dirty data for ffff880420e8ebf8 1099511787217
[ 5146.327727] ceph:  dropping dirty+flushing Fw state for ffff880423455c08 1099511787205
[ 5146.335736] ceph:  dropping dirty data for ffff880423455c08 1099511787205
[ 5146.342616] ceph:  dropping dirty Fw state for ffff88042504fbe8 1099511787245
[ 5146.349833] ceph:  dropping dirty data for ffff88042504fbe8 1099511787245
[ 5146.356680] ceph:  dropping dirty Fw state for ffff880425049c48 1099511787254
[ 5146.363893] ceph:  dropping dirty data for ffff880425049c48 1099511787254
[ 5146.405995] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 5146.464283] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN)
[ 5146.508710] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
[ 5147.386139] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
[ 5147.443783] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN)

Entering kdb (current=0xffff880412c83f20, pid 26674) on processor 1 Oops: (null)
due to oops @ 0xffffffffa06308a0
CPU 1 dPid: 26674, comm: ffsb Not tainted 3.8.0-ceph-00226-g1655ae7 #1 Supermicro X8SIL/X8SIL
dRIP: 0010:[<ffffffffa06308a0>]  [<ffffffffa06308a0>] ceph_update_writeable_page+0x460/0x4e0 [ceph]
dRSP: 0018:ffff880413c9fa08  EFLAGS: 00010246
dRAX: 0200000000000001 RBX: ffffea000a892280 RCX: ffffea000a892280
dRDX: 0000000000001000 RSI: 000000000200a000 RDI: ffff880426410b40
dRBP: ffff880413c9fa98 R08: 0000000000000002 R09: 0000000000000001
dR10: 0000000000000000 R11: 0000000000000000 R12: ffff880423456bf8
dR13: ffff880413c9fb98 R14: ffff88042c04c150 R15: 0000000000001000
dFS:  00007f39df7ee700(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
dCS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
dCR2: 00007f5b7631d6a0 CR3: 000000042c678000 CR4: 00000000000007e0
dDR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
dDR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
dProcess ffsb (pid: 26674, threadinfo ffff880413c9e000, task ffff880412c83f20)
dStack:
 ffffea000a892280 0000100023456dd8 000000000200a000 0000000000001000
 0000000002009000 ffff880426410b40 ffffea000a892280 ffff880423456dd8
 ffff880413c9fa98 ffffffff81130d65 ffff880413c9fb84 ffff88042c04de70
dCall Trace:
more>

At this point the machine was dead and power cycle was required
to get it back again.

#5 Updated by Alex Elder about 11 years ago

I have run this same test (only "suites/ffsb.sh") with the
"testing" branch twice. I got the same lockdep warning,
so the issue is unrelated to my changes.

Neither of these runs resulted in the crash that I saw
originally.

In addition I have run with my "review/wip-4392-2" branch
twice more.

Neither of these two runs led to the crash I originally
saw either.

The crash indicated ceph_update_writeable_page() was
involved at the time of the crash.

Looking at a copy of the ceph kernel object and disassembling
it, it looks to me like this is the line of code where the
problem occurred in ceph_update_writeable_page():
BUG_ON(!ci->i_snap_realm);

I assert therefore that this crash is not related to the changes
I made.

#6 Updated by Alex Elder about 11 years ago

OK, after this I'm done with this problem.

I just wanted to point out that I think there are two
distinct problems reported here, and maybe one should
go off into a separate issue:
- lockdep warnings
- crash, likely due to an assertion failure (null snap realm pointer)

#7 Updated by Sage Weil almost 11 years ago

  • Project changed from Ceph to Linux kernel client

#8 Updated by Sage Weil over 9 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF