Bug #4553
kclient: lockdep report, crash involving ceph fs and libceph
0%
Description
I really don't know what to assign this to, but it looks like
a big complex potential deadlock so I thought I should grab the
kernel output so someone else can take a look at it.
I'm testing using a kernel with some recent changes I've made,
but right now I don't think the following is related to that.
Here's the kernel being tested:
review/wip-4392-2 1655ae7 libceph: verify requests queued in order
It is based on the current testing branch:
8383641 libceph: wrap auth methods in a mutex
Here are the newer commits in the kernel being tested:
1655ae7 libceph: verify requests queued in order
532902a libceph: send queued requests when starting new one
b1e0294 libceph: keep request lists in tid order
317b8e1 libceph: requeue only sent requests when kicking
349cd3a libceph: no more kick_requests() race
76752dc libceph: slightly defer registering osd request
5128453 libceph: initialize data fields on last msg put
...
Here's the current state of the ceph "next" branch:
70e0ee8 rgw: bucket index ops on system buckets shouldn't do anything
Here's the state of teuthology I'm using:
c50b143 thrashosds: add test_backfill_full
OK, here's the yaml file:
interactive-on-error: roles: - [mon.a, mon.c, osd.0, osd.1, osd.2] - [mon.b, mds.a, osd.3, osd.4, osd.5] - [client.0] kernel: mon: branch: review_wip-4392-2 kdb: true client: branch: review_wip-4392-2 kdb: true overrides: ceph: conf: osd: osd op complaint time: 120 fs: btrfs log-whitelist: - clocks not synchronized - old request tasks: - install: - ceph: branch: next conf: global: ms inject socket failures: 5000 mon: debug mon: 10 debug paxos: osd: debug ms: 1 debug journal: 20 osd debug op order: true - kclient: - workunit: clients: all: - misc/trivial_sync.sh - kernel_untar_build.sh - suites/blogbench.sh - suites/dbench.sh - suites/tiobench.sh - suites/fsstress.sh - suites/ffsb.sh
The test that had just started was "suites/ffsb.sh". Here is
the end of the teuthology test output:
INFO:teuthology.task.workunit.client.0.out:make[1]: Leaving directory `/tmp/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' INFO:teuthology.task.workunit.client.0.out:FFSB version 6.0-RC2 started INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out:benchmark time = 300 INFO:teuthology.task.workunit.client.0.out:ThreadGroup 0 INFO:teuthology.task.workunit.client.0.out:================ INFO:teuthology.task.workunit.client.0.out: num_threads = 32 INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: read_random = off INFO:teuthology.task.workunit.client.0.out: read_size = 0 (0B) INFO:teuthology.task.workunit.client.0.out: read_blocksize = 0 (0B) INFO:teuthology.task.workunit.client.0.out: read_skip = off INFO:teuthology.task.workunit.client.0.out: read_skipsize = 0 (0B) INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: write_random = on INFO:teuthology.task.workunit.client.0.out: write_size = 5242880 (5MB) INFO:teuthology.task.workunit.client.0.out: fsync_file = 0 INFO:teuthology.task.workunit.client.0.out: write_blocksize = 4096 (4KB) INFO:teuthology.task.workunit.client.0.out: wait time = 0 INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: op weights INFO:teuthology.task.workunit.client.0.out: read = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: readall = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: write = 1 (100.00%) INFO:teuthology.task.workunit.client.0.out: create = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: append = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: delete = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: metaop = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: createdir = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: stat = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: writeall = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: writeall_fsync = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: open_close = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: write_fsync = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: create_fsync = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: append_fsync = 0 (0.00%) INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out:FileSystem . INFO:teuthology.task.workunit.client.0.out:========== INFO:teuthology.task.workunit.client.0.out: num_dirs = 0 INFO:teuthology.task.workunit.client.0.out: starting files = 128 INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: min file size = 104857600 (100MB) INFO:teuthology.task.workunit.client.0.out: max file size = 104857600 (100MB) INFO:teuthology.task.workunit.client.0.out: directio = off INFO:teuthology.task.workunit.client.0.out: alignedio = on INFO:teuthology.task.workunit.client.0.out: bufferedio = off INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: aging is off INFO:teuthology.task.workunit.client.0.out: current utilization = 0.13% INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out:checking existing fs: . INFO:teuthology.task.workunit.client.0.out:recreating new fileset INFO:teuthology.task.workunit.client.0.err:opendir: No such file or directory INFO:teuthology.task.workunit.client.0.out:fs setup took 189 secs INFO:teuthology.task.workunit.client.0.out:Syncing()...10 sec INFO:teuthology.task.workunit.client.0.out:Starting Actual Benchmark At: Tue Mar 26 08:59:58 2013 INFO:teuthology.task.workunit.client.0.out:
The test now seems to be hung, which may be related (I think so).
It also eventually led to a crash (which may have been due to
other things, I'm just not sure yet).
Here's information captured from the console, describing the
lockdep issues.
[ 4559.235679] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4559.513882] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4559.574219] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 4559.664004] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 4559.772118] [ 4559.773638] ========================================================= [ 4559.780540] [ INFO: possible irq lock inversion dependency detected ] [ 4559.787021] 3.8.0-ceph-00226-g1655ae7 #1 Not tainted [ 4559.792001] --------------------------------------------------------- [ 4559.798452] kswapd0/38 just changed the state of lock: [ 4559.803608] (&osdc->map_sem){++++.-}, at: [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph] [ 4559.813556] but this lock took another, RECLAIM_FS-unsafe lock in the past: [ 4559.820526] (&monc->mutex){+.+.+.} and interrupts could create inverse lock ordering between them. [ 4559.830128] [ 4559.830128] other info that might help us debug this: [ 4559.836677] Possible interrupt unsafe locking scenario: [ 4559.836677] [ 4559.843487] CPU0 CPU1 [ 4559.848029] ---- ---- [ 4559.852572] lock(&monc->mutex); [ 4559.855948] local_irq_disable(); [ 4559.861878] lock(&osdc->map_sem); [ 4559.867932] lock(&monc->mutex); [ 4559.873816] <Interrupt> [ 4559.876451] lock(&osdc->map_sem); [ 4559.880176] [ 4559.880176] *** DEADLOCK *** [ 4559.880176] [ 4559.886133] no locks held by kswapd0/38. [ 4559.890073] [ 4559.890073] the shortest dependencies between 2nd lock and 1st lock: [ 4559.899088] -> (&monc->mutex){+.+.+.} ops: 181682 { [ 4559.904181] HARDIRQ-ON-W at: [ 4559.907442] [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10 [ 4559.915060] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4559.922321] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 4559.930023] [<ffffffffa05e5f36>] ceph_monc_open_session+0x26/0x50 [libceph] [ 4559.938954] [<ffffffffa05de26a>] __ceph_open_session+0x3a/0x250 [libceph] [ 4559.947728] [<ffffffffa06241ed>] ceph_mount+0x37d/0x840 [ceph] [ 4559.955522] [<ffffffff81187ba3>] mount_fs+0x43/0x1b0 [ 4559.962429] [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120 [ 4559.969868] [<ffffffff811a5284>] do_mount+0x1f4/0xa20 [ 4559.976881] [<ffffffff811a5b40>] sys_mount+0x90/0xe0 [ 4559.983787] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4559.991657] SOFTIRQ-ON-W at: [ 4559.994925] [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10 [ 4560.002534] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.009803] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 4560.017503] [<ffffffffa05e5f36>] ceph_monc_open_session+0x26/0x50 [libceph] [ 4560.026421] [<ffffffffa05de26a>] __ceph_open_session+0x3a/0x250 [libceph] [ 4560.035167] [<ffffffffa06241ed>] ceph_mount+0x37d/0x840 [ceph] [ 4560.042961] [<ffffffff81187ba3>] mount_fs+0x43/0x1b0 [ 4560.049870] [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120 [ 4560.057308] [<ffffffff811a5284>] do_mount+0x1f4/0xa20 [ 4560.064301] [<ffffffff811a5b40>] sys_mount+0x90/0xe0 [ 4560.071208] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4560.079081] RECLAIM_FS-ON-W at: [ 4560.082606] [<ffffffff810b75e6>] mark_held_locks+0x86/0x140 [ 4560.090389] [<ffffffff810b7d41>] lockdep_trace_alloc+0x71/0xd0 [ 4560.098437] [<ffffffff8117b023>] kmem_cache_alloc_trace+0x33/0x150 [ 4560.106831] [<ffffffff810705b2>] __request_module+0x112/0x280 [ 4560.114791] [<ffffffff812feb3c>] crypto_larval_lookup+0x8c/0x180 [ 4560.123020] [<ffffffff812fec53>] crypto_alg_mod_lookup+0x23/0x90 [ 4560.131241] [<ffffffff812fed31>] crypto_alloc_base+0x41/0xb0 [ 4560.139114] [<ffffffffa05f283f>] ceph_crypto_alloc_cipher+0x1f/0x30 [libceph] [ 4560.148470] [<ffffffffa05f28ad>] ceph_aes_encrypt2+0x5d/0x2c0 [libceph] [ 4560.157305] [<ffffffffa05f3961>] ceph_encrypt2+0xb1/0xc0 [libceph] [ 4560.165706] [<ffffffffa05f3e3f>] ceph_x_encrypt+0x4f/0x70 [libceph] [ 4560.174196] [<ffffffffa05f4e7f>] ceph_x_build_request+0xdf/0x2f0 [libceph] [ 4560.183290] [<ffffffffa05f1dbe>] ceph_build_auth_request+0x4e/0xb0 [libceph] [ 4560.192558] [<ffffffffa05f2440>] ceph_handle_auth_reply+0x250/0x2e0 [libceph] [ 4560.201912] [<ffffffffa05e7162>] dispatch+0x192/0x7d0 [libceph] [ 4560.210053] [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph] [ 4560.218368] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4560.226328] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4560.234028] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.241014] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.248540] INITIAL USE at: [ 4560.251721] [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10 [ 4560.259245] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.266422] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 4560.274032] [<ffffffffa05e5f36>] ceph_monc_open_session+0x26/0x50 [libceph] [ 4560.282871] [<ffffffffa05de26a>] __ceph_open_session+0x3a/0x250 [libceph] [ 4560.291522] [<ffffffffa06241ed>] ceph_mount+0x37d/0x840 [ceph] [ 4560.299248] [<ffffffff81187ba3>] mount_fs+0x43/0x1b0 [ 4560.306069] [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120 [ 4560.313420] [<ffffffff811a5284>] do_mount+0x1f4/0xa20 [ 4560.320319] [<ffffffff811a5b40>] sys_mount+0x90/0xe0 [ 4560.327139] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4560.334923] } [ 4560.336696] ... key at: [<ffffffffa06060b8>] __key.28897+0x0/0xffffffffffff03fb [libceph] [ 4560.345519] ... acquired at: [ 4560.348587] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.354196] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 4560.360236] [<ffffffffa05e784d>] ceph_monc_got_osdmap+0x2d/0x60 [libceph] [ 4560.367318] [<ffffffffa05eb6f7>] ceph_osdc_handle_map+0x287/0x580 [libceph] [ 4560.374583] [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph] [ 4560.380797] [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph] [ 4560.387185] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4560.393223] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4560.399002] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.404089] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.409697] [ 4560.411208] -> (&osdc->map_sem){++++.-} ops: 707694 { [ 4560.416388] HARDIRQ-ON-W at: [ 4560.419563] [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10 [ 4560.427002] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.434083] [<ffffffff8165bd46>] down_write+0x36/0x59 [ 4560.440899] [<ffffffffa05eb555>] ceph_osdc_handle_map+0xe5/0x580 [libceph] [ 4560.449561] [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph] [ 4560.457270] [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph] [ 4560.465152] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4560.472679] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4560.479949] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.486509] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.493589] HARDIRQ-ON-R at: [ 4560.496766] [<ffffffff810b4f6f>] __lock_acquire+0x52f/0x1d10 [ 4560.504204] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.511285] [<ffffffff8165bcf9>] down_read+0x39/0x50 [ 4560.518024] [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph] [ 4560.526946] [<ffffffffa062f508>] ceph_writepages_start+0x8d8/0xf50 [ceph] [ 4560.535520] [<ffffffff8113c7e3>] do_writepages+0x23/0x40 [ 4560.542601] [<ffffffff811ac95b>] __writeback_single_inode+0x3b/0x180 [ 4560.550731] [<ffffffff811aedc0>] writeback_sb_inodes+0x280/0x3e0 [ 4560.558519] [<ffffffff811aefbe>] __writeback_inodes_wb+0x9e/0xd0 [ 4560.566306] [<ffffffff811af1eb>] wb_writeback+0x1fb/0x320 [ 4560.573474] [<ffffffff811b08a9>] wb_do_writeback+0xb9/0x1d0 [ 4560.580826] [<ffffffff811b0a53>] bdi_writeback_thread+0x93/0x270 [ 4560.588614] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.595176] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.602253] SOFTIRQ-ON-W at: [ 4560.605449] [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10 [ 4560.612885] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.619959] [<ffffffff8165bd46>] down_write+0x36/0x59 [ 4560.626776] [<ffffffffa05eb555>] ceph_osdc_handle_map+0xe5/0x580 [libceph] [ 4560.635438] [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph] [ 4560.643146] [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph] [ 4560.651030] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4560.658556] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4560.665823] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.672378] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.679457] SOFTIRQ-ON-R at: [ 4560.682635] [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10 [ 4560.690071] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.697150] [<ffffffff8165bcf9>] down_read+0x39/0x50 [ 4560.705030] [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph] [ 4560.713946] [<ffffffffa062f508>] ceph_writepages_start+0x8d8/0xf50 [ceph] [ 4560.722522] [<ffffffff8113c7e3>] do_writepages+0x23/0x40 [ 4560.729617] [<ffffffff811ac95b>] __writeback_single_inode+0x3b/0x180 [ 4560.737750] [<ffffffff811aedc0>] writeback_sb_inodes+0x280/0x3e0 [ 4560.745538] [<ffffffff811aefbe>] __writeback_inodes_wb+0x9e/0xd0 [ 4560.753325] [<ffffffff811af1eb>] wb_writeback+0x1fb/0x320 [ 4560.760492] [<ffffffff811b08a9>] wb_do_writeback+0xb9/0x1d0 [ 4560.767844] [<ffffffff811b0a53>] bdi_writeback_thread+0x93/0x270 [ 4560.775632] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.782194] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.789272] IN-RECLAIM_FS-R at: [ 4560.792710] [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10 [ 4560.800406] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.807760] [<ffffffff8165bcf9>] down_read+0x39/0x50 [ 4560.814746] [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph] [ 4560.823928] [<ffffffffa05eb20f>] ceph_osdc_writepages+0xdf/0x180 [libceph] [ 4560.832848] [<ffffffffa063025a>] writepage_nounlock+0x22a/0x410 [ceph] [ 4560.841422] [<ffffffffa0630bee>] ceph_writepage+0x3e/0x70 [ceph] [ 4560.849477] [<ffffffff81143164>] shrink_page_list+0x554/0x900 [ 4560.857265] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 4560.865400] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4560.872925] [<ffffffff811452e3>] kswapd+0x5d3/0xa90 [ 4560.879824] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.886660] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.894014] INITIAL USE at: [ 4560.897109] [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10 [ 4560.904458] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4560.911452] [<ffffffff8165bd46>] down_write+0x36/0x59 [ 4560.918184] [<ffffffffa05eb555>] ceph_osdc_handle_map+0xe5/0x580 [libceph] [ 4560.926761] [<ffffffffa05e70f0>] dispatch+0x120/0x7d0 [libceph] [ 4560.934382] [<ffffffffa05e3158>] con_work+0x19f8/0x2e80 [libceph] [ 4560.942178] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4560.949635] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4560.956812] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4560.963285] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4560.970276] } [ 4560.971958] ... key at: [<ffffffffa06060e0>] __key.29826+0x0/0xffffffffffff03d3 [libceph] [ 4560.980704] ... acquired at: [ 4560.983687] [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150 [ 4560.990076] [<ffffffff810b48e8>] mark_lock+0x198/0x2f0 [ 4560.995509] [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10 [ 4561.001462] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4561.007069] [<ffffffff8165bcf9>] down_read+0x39/0x50 [ 4561.012329] [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph] [ 4561.019771] [<ffffffffa05eb20f>] ceph_osdc_writepages+0xdf/0x180 [libceph] [ 4561.026939] [<ffffffffa063025a>] writepage_nounlock+0x22a/0x410 [ceph] [ 4561.033756] [<ffffffffa0630bee>] ceph_writepage+0x3e/0x70 [ceph] [ 4561.040056] [<ffffffff81143164>] shrink_page_list+0x554/0x900 [ 4561.046095] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 4561.052482] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4561.058261] [<ffffffff811452e3>] kswapd+0x5d3/0xa90 [ 4561.063434] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4561.068522] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4561.074130] [ 4561.075665] [ 4561.075665] stack backtrace: [ 4561.080046] Pid: 38, comm: kswapd0 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4561.087021] Call Trace: [ 4561.089488] [<ffffffff810b3dd8>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [ 4561.096635] [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150 [ 4561.102830] [<ffffffff810b48e8>] mark_lock+0x198/0x2f0 [ 4561.108072] [<ffffffff810b3de0>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [ 4561.115401] [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10 [ 4561.121164] [<ffffffff81338a61>] ? vsnprintf+0x461/0x640 [ 4561.126585] [<ffffffffa05e8ab9>] ? ceph_osdc_build_request+0x119/0x650 [libceph] [ 4561.134095] [<ffffffffa05e95a5>] ? ceph_osdc_start_request+0x65/0x170 [libceph] [ 4561.141515] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 4561.146928] [<ffffffffa05e95a5>] ? ceph_osdc_start_request+0x65/0x170 [libceph] [ 4561.154349] [<ffffffff8165bcf9>] down_read+0x39/0x50 [ 4561.159422] [<ffffffffa05e95a5>] ? ceph_osdc_start_request+0x65/0x170 [libceph] [ 4561.166855] [<ffffffffa05e95a5>] ceph_osdc_start_request+0x65/0x170 [libceph] [ 4561.174108] [<ffffffffa05eb20f>] ceph_osdc_writepages+0xdf/0x180 [libceph] [ 4561.181085] [<ffffffffa063025a>] writepage_nounlock+0x22a/0x410 [ceph] [ 4561.187711] [<ffffffffa0630bee>] ceph_writepage+0x3e/0x70 [ceph] [ 4561.193815] [<ffffffff81143164>] shrink_page_list+0x554/0x900 [ 4561.199665] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 4561.205862] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4561.211883] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4561.217472] [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300 [ 4561.223053] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4561.229857] [<ffffffff811452e3>] kswapd+0x5d3/0xa90 [ 4561.234838] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4561.241113] [<ffffffff81144d10>] ? try_to_free_pages+0x1a0/0x1a0 [ 4561.247221] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4561.252109] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4561.258298] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4561.263714] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4561.270132] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4561.305432] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4561.341964] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4561.415362] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4561.461996] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4562.595792] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4562.695317] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 4562.782819] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN)
And finally, here's some subsequent console output, leading
ultimately to a crash. The system (or maybe just my connection)
seems to be locked up solid so I can't get any more. I think
I will be power cycling it to get the machine back.
[ 4575.988573] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4576.518337] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4576.584345] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 4577.111907] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4577.391608] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4577.699085] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 4579.518966] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4603.896963] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229053 c=229052 q=482) [ 4603.906376] Pid: 24447, comm: kworker/0:4 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4603.913972] Call Trace: [ 4603.916437] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4603.922502] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4603.926917] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2103 jiffies, g=229053, c=229052, q=488) [ 4603.926918] Task dump for CPU 0: [ 4603.926920] kworker/0:4 R running task 0 24447 2 0x00000000 [ 4603.926923] ffff8804152fb308 ffffffff810b783d ffff8804152fb328 0000000000000282 [ 4603.926924] ffff88043ff6a438 0000000000000282 ffff8804152fb368 ffffffff81085343 [ 4603.926926] 0000000000000030 ffffea00078ba0c0 ffff8804152fb520 ffffea00078ba0e0 [ 4603.926927] Call Trace: [ 4603.926932] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4603.926935] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4603.926939] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4603.926941] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4603.926944] [<ffffffff81339c9a>] ? delay_tsc+0x4a/0x80 [ 4603.926948] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4603.926950] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4603.926952] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4603.926953] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4603.926955] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4603.926957] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4603.926960] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4603.926964] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4603.926966] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4603.926967] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4603.926969] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4603.926973] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4603.926976] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4603.926979] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4603.926982] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4603.926983] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4603.926985] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4603.926989] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4603.926997] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4603.926999] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4603.927008] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4603.927009] [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330 [ 4603.927011] [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330 [ 4603.927013] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4603.927014] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4603.927016] [<ffffffff810f7a7f>] ? rcu_irq_exit+0x7f/0xd0 [ 4603.927018] [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13 [ 4603.927023] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4603.927024] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4603.927026] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4603.927028] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4603.927030] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4603.927032] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4603.927034] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4604.207021] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4604.213049] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4604.219074] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4604.225527] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4604.231237] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4604.236741] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4604.242686] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4604.249281] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4604.255131] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4604.261333] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4604.267704] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4604.273723] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4604.279399] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4604.285079] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4604.290233] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4604.295044] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4604.300778] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4604.307625] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4604.313824] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4604.320027] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4604.326222] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4604.332073] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4604.337662] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4604.343424] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4604.349759] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4604.355955] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4604.361801] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4604.368182] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4604.374204] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4604.380225] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4604.385641] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4604.390979] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4604.396399] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4604.401729] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4604.407843] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4604.413868] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4604.419377] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4604.425490] [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330 [ 4604.431181] [<ffffffff8109651b>] ? idle_balance+0x1fb/0x330 [ 4604.436894] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4604.442744] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4604.448766] [<ffffffff810f7a7f>] ? rcu_irq_exit+0x7f/0xd0 [ 4604.454271] [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13 [ 4604.460385] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4604.466974] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4604.472561] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4604.478413] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4604.483308] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4604.489505] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4604.494921] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4613.010446] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4621.954715] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4621.961990] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4622.066824] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4622.235849] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4622.333686] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 4622.696381] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4622.738472] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 4624.941354] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4625.140097] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4625.178203] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 4625.296607] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4625.427142] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4625.607493] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4625.763427] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 4625.808217] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4627.594601] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 4628.153725] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 4628.338422] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 4628.380127] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 4628.434586] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 4649.704822] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229113 c=229112 q=598) [ 4649.714167] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4649.721763] Call Trace: [ 4649.724230] <IRQ> [ 4649.724837] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2102 jiffies, g=229113, c=229112, q=598) [ 4649.724842] Task dump for CPU 0: [ 4649.724844] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4649.724846] ffffffff81072b3c 0000000000000046 0000000000000001 0000000000000000 [ 4649.724848] ffff88043fc1cd35 ffffffffa05e1760 ffff880090289fd8 ffff88043fc0e440 [ 4649.724850] ffffffffa0605c80 0000000000000000 0000000000000000 ffffffffa05f8d0e [ 4649.724852] Call Trace: [ 4649.724853] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4649.724860] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4649.724871] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4649.724873] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4649.724874] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4649.724877] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4649.724879] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4649.724884] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4649.820216] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4649.825874] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4649.832076] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4649.838095] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4649.844121] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4649.850593] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4649.856269] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4649.861771] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4649.867709] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4649.874261] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4649.880117] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4649.886379] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4649.892750] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4649.898774] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4649.904501] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4649.910178] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4649.915376] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4649.920187] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4649.925869] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4649.932708] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4649.938907] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4649.945101] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4649.951296] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4649.957144] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4649.962737] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4649.968499] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4649.974782] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4649.980975] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4649.986826] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4649.993196] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4649.999235] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4650.005300] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4650.010717] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4650.016071] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4650.021489] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4650.026847] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4650.032960] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4650.038996] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4650.044503] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4650.050616] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4650.056290] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4650.061969] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4650.068086] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4650.073935] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4650.079973] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4650.086572] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4650.092166] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4650.099171] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4650.104071] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4650.110272] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4650.115748] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4671.091205] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229114 c=229113 q=3151) [ 4671.100640] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4671.108296] Call Trace: [ 4671.110763] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4671.116827] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4671.123027] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4671.129048] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4671.131151] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2104 jiffies, g=229114, c=229113, q=3152) [ 4671.131153] Task dump for CPU 0: [ 4671.131158] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4671.131165] ffffffff81072b3c 0000000000000046 0000000000000001 0000000000000000 [ 4671.131166] ffff88043fc1cd35 ffffffffa05e1760 ffff880090289fd8 ffff88043fc0e440 [ 4671.131168] ffffffffa0605c80 0000000000000000 0000000000000000 ffffffffa05f8d0e [ 4671.131169] Call Trace: [ 4671.131173] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4671.131184] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4671.131187] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4671.131188] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4671.131190] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4671.131192] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4671.131195] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4671.131197] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4671.228529] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4671.234986] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4671.240666] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4671.246203] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4671.252164] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4671.258708] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4671.264557] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4671.270752] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4671.277121] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4671.283205] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4671.288878] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4671.294558] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4671.299712] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4671.304549] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4671.310223] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4671.317085] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4671.323328] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4671.329524] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4671.335719] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4671.341594] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4671.347182] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4671.353025] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4671.359307] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4671.365529] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4671.371438] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4671.377809] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4671.383846] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4671.389871] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4671.395301] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4671.400630] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4671.406057] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4671.411466] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4671.417579] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4671.423668] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4671.429177] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4671.435325] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4671.441066] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4671.446745] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4671.452854] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4671.458703] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4671.464759] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4671.471365] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4671.476951] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4671.482813] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4671.487706] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4671.493933] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4671.499357] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4676.392896] INFO: task ffsb:26654 blocked for more than 120 seconds. [ 4676.399268] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4676.407141] ffsb D ffffffff81130150 0 26654 24977 0x00000000 [ 4676.414254] ffff880413d0b938 0000000000000046 ffffffff8165e05f 0000000000000002 [ 4676.421767] ffff88041f77bf20 ffff880413d0bfd8 ffff880413d0bfd8 ffff880413d0bfd8 [ 4676.429288] ffff88042d585eb0 ffff88041f77bf20 ffff880413d0b938 ffff88043fc54ed8 [ 4676.436807] Call Trace: [ 4676.439276] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4676.446090] [<ffffffff81130150>] ? __lock_page+0x70/0x70 [ 4676.451508] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4676.456494] [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0 [ 4676.461734] [<ffffffff8113015e>] sleep_on_page+0xe/0x20 [ 4676.467072] [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0 [ 4676.472928] [<ffffffff81130147>] __lock_page+0x67/0x70 [ 4676.478175] [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40 [ 4676.484728] [<ffffffff81130b87>] find_lock_page+0x67/0x80 [ 4676.490229] [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0 [ 4676.496874] [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph] [ 4676.503163] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4676.509189] [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290 [ 4676.516000] [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0 [ 4676.522549] [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph] [ 4676.528843] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4676.535395] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4676.541336] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4676.546756] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4676.551916] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4676.556993] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4676.563022] INFO: lockdep is turned off. [ 4676.566959] INFO: task ffsb:26656 blocked for more than 120 seconds. [ 4676.573326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4676.581181] ffsb D ffffffff81130150 0 26656 24977 0x00000000 [ 4676.588293] ffff8804144ff938 0000000000000046 ffffffff8165e05f 0000000000000002 [ 4676.595814] ffff880413fe0000 ffff8804144fffd8 ffff8804144fffd8 ffff8804144fffd8 [ 4676.603335] ffff88042d540000 ffff880413fe0000 ffff8804144ff938 ffff88043fcd4ed8 [ 4676.610837] Call Trace: [ 4676.613317] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4676.620123] [<ffffffff81130150>] ? __lock_page+0x70/0x70 [ 4676.625545] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4676.630527] [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0 [ 4676.635778] [<ffffffff8113015e>] sleep_on_page+0xe/0x20 [ 4676.641112] [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0 [ 4676.646969] [<ffffffff81130147>] __lock_page+0x67/0x70 [ 4676.652216] [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40 [ 4676.658765] [<ffffffff81130b87>] find_lock_page+0x67/0x80 [ 4676.664272] [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0 [ 4676.670914] [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph] [ 4676.677205] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4676.683232] [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290 [ 4676.690039] [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0 [ 4676.696611] [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph] [ 4676.702908] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4676.709455] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4676.715398] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4676.720814] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4676.725975] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4676.731044] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4676.737070] INFO: lockdep is turned off. [ 4676.741013] INFO: task ffsb:26659 blocked for more than 120 seconds. [ 4676.747386] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4676.755249] ffsb D 00000000ffffffff 0 26659 24977 0x00000000 [ 4676.762353] ffff880412cf7bb8 0000000000000046 0000000000000007 0000000000000006 [ 4676.769855] ffff880413088000 ffff880412cf7fd8 ffff880412cf7fd8 ffff880412cf7fd8 [ 4676.777365] ffff8804150c5eb0 ffff880413088000 ffffffff8165ac75 ffff8804289d74c8 [ 4676.784883] Call Trace: [ 4676.787352] [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320 [ 4676.793465] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4676.798450] [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10 [ 4676.804827] [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320 [ 4676.810771] [<ffffffffa062cec0>] ? ceph_aio_write+0x910/0xaa0 [ceph] [ 4676.817241] [<ffffffffa062cec0>] ceph_aio_write+0x910/0xaa0 [ceph] [ 4676.823535] [<ffffffff81090bb4>] ? select_task_rq_fair+0x54/0x8a0 [ 4676.829730] [<ffffffff81090bb4>] ? select_task_rq_fair+0x54/0x8a0 [ 4676.835937] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4676.841871] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4676.847296] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4676.852483] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4676.857553] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4676.863596] INFO: lockdep is turned off. [ 4676.867540] INFO: task ffsb:26663 blocked for more than 120 seconds. [ 4676.873932] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4676.881786] ffsb D 00000000ffffffff 0 26663 24977 0x00000000 [ 4676.888898] ffff880414735bb8 0000000000000046 0000000000000007 0000000000000006 [ 4676.896413] ffff88041f620000 ffff880414735fd8 ffff880414735fd8 ffff880414735fd8 [ 4676.903933] ffff88042d540000 ffff88041f620000 ffffffff8165ac75 ffff8804289d74c8 [ 4676.912616] Call Trace: [ 4676.915089] [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320 [ 4676.921202] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4676.926196] [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10 [ 4676.932570] [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320 [ 4676.938517] [<ffffffffa062cec0>] ? ceph_aio_write+0x910/0xaa0 [ceph] [ 4676.944989] [<ffffffffa062cec0>] ceph_aio_write+0x910/0xaa0 [ceph] [ 4676.951275] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4676.957225] [<ffffffff8108ca5e>] ? try_to_wake_up+0x2e/0x2f0 [ 4676.963002] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4676.968420] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4676.973607] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4676.978676] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4676.984704] INFO: lockdep is turned off. [ 4676.988646] INFO: task ffsb:26666 blocked for more than 120 seconds. [ 4676.995035] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4677.002900] ffsb D ffffffff81130150 0 26666 24977 0x00000000 [ 4677.010008] ffff880413da3938 0000000000000046 ffffffff8165e05f 0000000000000002 [ 4677.017530] ffff880413c78000 ffff880413da3fd8 ffff880413da3fd8 ffff880413da3fd8 [ 4677.025049] ffff88042d533f20 ffff880413c78000 ffff880413da3938 ffff88043fc54ed8 [ 4677.032569] Call Trace: [ 4677.035038] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4677.041855] [<ffffffff81130150>] ? __lock_page+0x70/0x70 [ 4677.047268] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4677.052258] [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0 [ 4677.057504] [<ffffffff8113015e>] sleep_on_page+0xe/0x20 [ 4677.062851] [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0 [ 4677.068703] [<ffffffff81130147>] __lock_page+0x67/0x70 [ 4677.073957] [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40 [ 4677.080498] [<ffffffff81130b87>] find_lock_page+0x67/0x80 [ 4677.086015] [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0 [ 4677.092662] [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph] [ 4677.098946] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4677.104980] [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290 [ 4677.111819] [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0 [ 4677.118370] [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph] [ 4677.124681] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4677.131231] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4677.137179] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4677.142603] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4677.147763] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4677.152845] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4677.158864] INFO: lockdep is turned off. [ 4677.162814] INFO: task ffsb:26670 blocked for more than 120 seconds. [ 4677.169179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4677.177045] ffsb D ffffffff81130150 0 26670 24977 0x00000000 [ 4677.184157] ffff880412dc5938 0000000000000046 ffffffff8165e05f 0000000000000002 [ 4677.191671] ffff88042d0b9f90 ffff880412dc5fd8 ffff880412dc5fd8 ffff880412dc5fd8 [ 4677.199183] ffff88042d585eb0 ffff88042d0b9f90 ffff880412dc5938 ffff88043fc54ed8 [ 4677.206694] Call Trace: [ 4677.209163] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4677.215981] [<ffffffff81130150>] ? __lock_page+0x70/0x70 [ 4677.221403] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4677.226393] [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0 [ 4677.231646] [<ffffffff8113015e>] sleep_on_page+0xe/0x20 [ 4677.236977] [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0 [ 4677.242839] [<ffffffff81130147>] __lock_page+0x67/0x70 [ 4677.248087] [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40 [ 4677.254642] [<ffffffff81130b87>] find_lock_page+0x67/0x80 [ 4677.260150] [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0 [ 4677.266798] [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph] [ 4677.273086] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4677.279110] [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290 [ 4677.285922] [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0 [ 4677.292486] [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph] [ 4677.298773] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4677.305325] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4677.311266] [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13 [ 4677.317384] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4677.322803] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4677.327963] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4677.333044] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4677.339064] INFO: lockdep is turned off. [ 4677.343016] INFO: task ffsb:26673 blocked for more than 120 seconds. [ 4677.349387] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4677.357253] ffsb D 00000000ffffffff 0 26673 24977 0x00000000 [ 4677.364358] ffff880414559e28 0000000000000046 0000000000000007 0000000000000006 [ 4677.371872] ffff880414511f90 ffff880414559fd8 ffff880414559fd8 ffff880414559fd8 [ 4677.379374] ffff880415211f90 ffff880414511f90 ffffffff8165ac75 ffff880421c9dce0 [ 4677.386894] Call Trace: [ 4677.389363] [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320 [ 4677.395488] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4677.400475] [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10 [ 4677.406857] [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320 [ 4677.412811] [<ffffffffa062b605>] ? ceph_llseek+0x45/0x160 [ceph] [ 4677.418919] [<ffffffff811a0f7c>] ? fget_light+0x3c/0x130 [ 4677.424351] [<ffffffffa062b605>] ceph_llseek+0x45/0x160 [ceph] [ 4677.430288] [<ffffffff811836d3>] vfs_llseek+0x33/0x40 [ 4677.435451] [<ffffffff81183c5e>] sys_lseek+0x7e/0x90 [ 4677.440523] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4677.446551] INFO: lockdep is turned off. [ 4677.450493] INFO: task ffsb:26674 blocked for more than 120 seconds. [ 4677.456866] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4677.464729] ffsb D ffffffff81130150 0 26674 24977 0x00000000 [ 4677.471837] ffff880413c9f938 0000000000000046 ffffffff8165e05f 0000000000000002 [ 4677.479351] ffff880412c83f20 ffff880413c9ffd8 ffff880413c9ffd8 ffff880413c9ffd8 [ 4677.486870] ffff88042d073f20 ffff880412c83f20 ffff880413c9f938 ffff88043fc14ed8 [ 4677.494381] Call Trace: [ 4677.496851] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4677.503668] [<ffffffff81130150>] ? __lock_page+0x70/0x70 [ 4677.509089] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4677.514080] [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0 [ 4677.519326] [<ffffffff8113015e>] sleep_on_page+0xe/0x20 [ 4677.524666] [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0 [ 4677.530517] [<ffffffff81130147>] __lock_page+0x67/0x70 [ 4677.535769] [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40 [ 4677.542317] [<ffffffff81130b87>] find_lock_page+0x67/0x80 [ 4677.547819] [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0 [ 4677.554469] [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph] [ 4677.560751] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4677.566784] [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290 [ 4677.573608] [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0 [ 4677.580160] [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph] [ 4677.586452] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4677.592998] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4677.598936] [<ffffffff8165e7f3>] ? retint_restore_args+0x13/0x13 [ 4677.605053] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4677.610467] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4677.615632] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4677.620704] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4677.626734] INFO: lockdep is turned off. [ 4677.630675] INFO: task ffsb:26676 blocked for more than 120 seconds. [ 4677.637056] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4677.644921] ffsb D ffffffff81130150 0 26676 24977 0x00000000 [ 4677.652040] ffff880412cf3938 0000000000000046 ffffffff8165e05f 0000000000000002 [ 4677.659551] ffff880413fd8000 ffff880412cf3fd8 ffff880412cf3fd8 ffff880412cf3fd8 [ 4677.667061] ffff88042d585eb0 ffff880413fd8000 ffff880412cf3938 ffff88043fc54ed8 [ 4677.674573] Call Trace: [ 4677.677041] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 4677.683859] [<ffffffff81130150>] ? __lock_page+0x70/0x70 [ 4677.689281] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4677.694284] [<ffffffff8165cc3f>] io_schedule+0x8f/0xd0 [ 4677.699524] [<ffffffff8113015e>] sleep_on_page+0xe/0x20 [ 4677.704874] [<ffffffff8165a6da>] __wait_on_bit_lock+0x5a/0xc0 [ 4677.710744] [<ffffffff81130147>] __lock_page+0x67/0x70 [ 4677.715983] [<ffffffff8107ae50>] ? autoremove_wake_function+0x40/0x40 [ 4677.722537] [<ffffffff81130b87>] find_lock_page+0x67/0x80 [ 4677.728036] [<ffffffff81130d3f>] grab_cache_page_write_begin+0x6f/0xf0 [ 4677.734684] [<ffffffffa063096e>] ceph_write_begin+0x4e/0xd0 [ceph] [ 4677.740973] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4677.746998] [<ffffffff8113064a>] generic_file_buffered_write+0x11a/0x290 [ 4677.753811] [<ffffffff81131df6>] __generic_file_aio_write+0x1b6/0x3b0 [ 4677.760365] [<ffffffffa062cee5>] ceph_aio_write+0x935/0xaa0 [ceph] [ 4677.766687] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4677.773238] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4677.779178] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4677.784601] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4677.789758] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4677.794838] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4677.800865] INFO: lockdep is turned off. [ 4677.804802] INFO: task ffsb:26678 blocked for more than 120 seconds. [ 4677.811175] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4677.819032] ffsb D 00000000ffffffff 0 26678 24977 0x00000000 [ 4677.826162] ffff880413dadbb8 0000000000000046 0000000000000007 0000000000000006 [ 4677.833676] ffff88042d5c3f20 ffff880413dadfd8 ffff880413dadfd8 ffff880413dadfd8 [ 4677.841213] ffff88042d535eb0 ffff88042d5c3f20 ffffffff8165ac75 ffff8804289d74c8 [ 4677.849785] Call Trace: [ 4677.852266] [<ffffffff8165ac75>] ? mutex_lock_nested+0x1e5/0x320 [ 4677.858377] [<ffffffff8165cb69>] schedule+0x29/0x70 [ 4677.863370] [<ffffffff8165ce8e>] schedule_preempt_disabled+0xe/0x10 [ 4677.869741] [<ffffffff8165abe1>] mutex_lock_nested+0x151/0x320 [ 4677.875694] [<ffffffffa062cec0>] ? ceph_aio_write+0x910/0xaa0 [ceph] [ 4677.882187] [<ffffffffa062cec0>] ceph_aio_write+0x910/0xaa0 [ceph] [ 4677.888477] [<ffffffff810b67f0>] ? lock_release_non_nested+0xa0/0x310 [ 4677.895025] [<ffffffff810b4d42>] ? __lock_acquire+0x302/0x1d10 [ 4677.900968] [<ffffffff8108ca5e>] ? try_to_wake_up+0x2e/0x2f0 [ 4677.906732] [<ffffffff81183843>] do_sync_write+0xa3/0xe0 [ 4677.912161] [<ffffffff81183ef3>] vfs_write+0xb3/0x180 [ 4677.917312] [<ffffffff81184232>] sys_write+0x52/0xa0 [ 4677.922394] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 4677.928416] INFO: lockdep is turned off. [ 4710.039798] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229118 c=229117 q=863) [ 4710.049151] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4710.056745] Call Trace: [ 4710.059212] <IRQ> [ 4710.059907] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229118, c=229117, q=863) [ 4710.059911] Task dump for CPU 0: [ 4710.059912] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4710.059915] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4710.059917] ffff88043ff72238 0000000000000282 ffff880090289368 ffffffff81085343 [ 4710.059919] 0000000000000002 ffffea000f816140 ffff880090289520 ffffea000f816160 [ 4710.059921] Call Trace: [ 4710.059922] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4710.059927] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4710.059931] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4710.059935] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4710.059937] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4710.059938] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4710.059942] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4710.059944] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4710.059946] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4710.059948] [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50 [ 4710.059950] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4710.059952] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4710.059955] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4710.059959] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4710.059961] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4710.059963] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4710.059965] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4710.059968] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4710.059971] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4710.059974] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4710.059977] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4710.059979] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4710.059981] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4710.059990] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4710.059993] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4710.059995] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4710.060003] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4710.060006] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4710.060008] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4710.060010] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4710.060012] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4710.060013] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4710.060018] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4710.060020] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4710.060021] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4710.060024] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4710.060025] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4710.060028] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4710.335519] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4710.341177] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4710.347381] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4710.353409] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4710.359486] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4710.365939] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4710.371623] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4710.377128] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4710.383079] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4710.389624] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4710.395473] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4710.401669] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4710.408039] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4710.414067] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4710.419742] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4710.4254
History
#1 Updated by Ian Colle about 11 years ago
- Priority changed from Normal to High
#2 Updated by Alex Elder about 11 years ago
I just got (I think) the same lockdep report, running only
the "suites/ffsb.sh" test using the ceph-client "testing" branch.
[ 372.042693] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 372.116802] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 372.228738] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 372.329688] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 372.532394] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 372.604703] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 372.846913] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 373.003111] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 373.121406] [ 373.122956] ========================================================= [ 373.129436] [ INFO: possible irq lock inversion dependency detected ] [ 373.135917] 3.8.0-ceph-00219-g8383641 #1 Not tainted [ 373.140926] --------------------------------------------------------- [ 373.147409] kswapd0/38 just changed the state of lock: [ 373.152588] (&osdc->request_mutex){+.+.-.}, at: [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph] [ 373.163198] but this lock took another, RECLAIM_FS-unsafe lock in the past: [ 373.170201] (&con->mutex){+.+.+.} and interrupts could create inverse lock ordering between them. [ 373.179817] [ 373.179817] other info that might help us debug this: [ 373.186402] Possible interrupt unsafe locking scenario: [ 373.186402] [ 373.193245] CPU0 CPU1 [ 373.197814] ---- ---- [ 373.202383] lock(&con->mutex); [ 373.205759] local_irq_disable(); [ 373.211723] lock(&osdc->request_mutex); [ 373.218393] lock(&con->mutex); [ 373.224286] <Interrupt> [ 373.226955] lock(&osdc->request_mutex); [ 373.231321] [ 373.231321] *** DEADLOCK *** [ 373.231321] [ 373.237311] no locks held by kswapd0/38. [ 373.241279] [ 373.241279] the shortest dependencies between 2nd lock and 1st lock: [ 373.249171] -> (&con->mutex){+.+.+.} ops: 4935891 { [ 373.254495] HARDIRQ-ON-W at: [ 373.257825] [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10 [ 373.265521] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.272849] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.280613] [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph] [ 373.288973] [<ffffffffa05c7d5b>] __open_session+0xcb/0x160 [libceph] [ 373.297341] [<ffffffffa05c7efe>] ceph_monc_open_session+0x2e/0x50 [libceph] [ 373.306315] [<ffffffffa05c026a>] __ceph_open_session+0x3a/0x250 [libceph] [ 373.315115] [<ffffffffa06061ed>] ceph_mount+0x37d/0x840 [ceph] [ 373.324015] [<ffffffff81187ba3>] mount_fs+0x43/0x1b0 [ 373.330984] [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120 [ 373.338484] [<ffffffff811a5284>] do_mount+0x1f4/0xa20 [ 373.345539] [<ffffffff811a5b40>] sys_mount+0x90/0xe0 [ 373.352513] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 373.360445] SOFTIRQ-ON-W at: [ 373.363772] [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10 [ 373.371451] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.378779] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.386544] [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph] [ 373.394824] [<ffffffffa05c7d5b>] __open_session+0xcb/0x160 [libceph] [ 373.403192] [<ffffffffa05c7efe>] ceph_monc_open_session+0x2e/0x50 [libceph] [ 373.412167] [<ffffffffa05c026a>] __ceph_open_session+0x3a/0x250 [libceph] [ 373.420967] [<ffffffffa06061ed>] ceph_mount+0x37d/0x840 [ceph] [ 373.428831] [<ffffffff81187ba3>] mount_fs+0x43/0x1b0 [ 373.435799] [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120 [ 373.443305] [<ffffffff811a5284>] do_mount+0x1f4/0xa20 [ 373.450361] [<ffffffff811a5b40>] sys_mount+0x90/0xe0 [ 373.457327] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 373.465260] RECLAIM_FS-ON-W at: [ 373.468846] [<ffffffff810b75e6>] mark_held_locks+0x86/0x140 [ 373.476699] [<ffffffff810b7d41>] lockdep_trace_alloc+0x71/0xd0 [ 373.484859] [<ffffffff8117c39f>] kmem_cache_alloc+0x2f/0x150 [ 373.492793] [<ffffffff81539e5d>] sock_alloc_inode+0x1d/0xd0 [ 373.500641] [<ffffffff8119de66>] alloc_inode+0x26/0xa0 [ 373.508055] [<ffffffff8119fb4a>] new_inode_pseudo+0x1a/0x70 [ 373.515907] [<ffffffff8153a28e>] sock_alloc+0x1e/0x80 [ 373.523243] [<ffffffff8153b995>] __sock_create+0x95/0x260 [ 373.530917] [<ffffffff8153bb84>] sock_create_kern+0x24/0x30 [ 373.538767] [<ffffffffa05c4743>] con_work+0x1023/0x2e80 [libceph] [ 373.547178] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 373.555252] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 373.563012] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 373.570068] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 373.577653] INITIAL USE at: [ 373.580938] [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10 [ 373.588532] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.595759] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.603440] [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph] [ 373.611643] [<ffffffffa05c7d5b>] __open_session+0xcb/0x160 [libceph] [ 373.619926] [<ffffffffa05c7efe>] ceph_monc_open_session+0x2e/0x50 [libceph] [ 373.628812] [<ffffffffa05c026a>] __ceph_open_session+0x3a/0x250 [libceph] [ 373.637526] [<ffffffffa06061ed>] ceph_mount+0x37d/0x840 [ceph] [ 373.645285] [<ffffffff81187ba3>] mount_fs+0x43/0x1b0 [ 373.652168] [<ffffffff811a2fc6>] vfs_kern_mount+0x76/0x120 [ 373.659581] [<ffffffff811a5284>] do_mount+0x1f4/0xa20 [ 373.666550] [<ffffffff811a5b40>] sys_mount+0x90/0xe0 [ 373.673428] [<ffffffff81666e59>] system_call_fastpath+0x16/0x1b [ 373.681283] } [ 373.683082] ... key at: [<ffffffffa05e7b38>] __key.49261+0x0/0xffffffffffff082b [libceph] [ 373.691971] ... acquired at: [ 373.695069] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.700729] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.706829] [<ffffffffa05c2d4e>] ceph_con_open+0x3e/0x1a0 [libceph] [ 373.713450] [<ffffffffa05c9ff6>] __map_request+0x446/0x640 [libceph] [ 373.720172] [<ffffffffa05cb525>] ceph_osdc_start_request+0xd5/0x160 [libceph] [ 373.727675] [<ffffffffa0611508>] ceph_writepages_start+0x8d8/0xf50 [ceph] [ 373.734823] [<ffffffff8113c7e3>] do_writepages+0x23/0x40 [ 373.740497] [<ffffffff811ac95b>] __writeback_single_inode+0x3b/0x180 [ 373.747198] [<ffffffff811aedc0>] writeback_sb_inodes+0x280/0x3e0 [ 373.753582] [<ffffffff811aefbe>] __writeback_inodes_wb+0x9e/0xd0 [ 373.759942] [<ffffffff811af1eb>] wb_writeback+0x1fb/0x320 [ 373.765695] [<ffffffff811b0920>] wb_do_writeback+0x130/0x1d0 [ 373.771709] [<ffffffff811b0a53>] bdi_writeback_thread+0x93/0x270 [ 373.778072] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 373.783278] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 373.788939] [ 373.790483] -> (&osdc->request_mutex){+.+.-.} ops: 1044630 { [ 373.796434] HARDIRQ-ON-W at: [ 373.799679] [<ffffffff810b50bd>] __lock_acquire+0x67d/0x1d10 [ 373.807183] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.814335] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.821929] [<ffffffffa05cc56f>] kick_requests+0x3f/0x3e0 [libceph] [ 373.830038] [<ffffffffa05cd7b4>] ceph_osdc_handle_map+0x494/0x580 [libceph] [ 373.838839] [<ffffffffa05c90b0>] dispatch+0x120/0x7d0 [libceph] [ 373.846599] [<ffffffffa05c5118>] con_work+0x19f8/0x2e80 [libceph] [ 373.854569] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 373.862157] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 373.869485] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 373.876143] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 373.883299] SOFTIRQ-ON-W at: [ 373.886529] [<ffffffff810b50f5>] __lock_acquire+0x6b5/0x1d10 [ 373.894077] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.901219] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.908805] [<ffffffffa05cc56f>] kick_requests+0x3f/0x3e0 [libceph] [ 373.916915] [<ffffffffa05cd7b4>] ceph_osdc_handle_map+0x494/0x580 [libceph] [ 373.925743] [<ffffffffa05c90b0>] dispatch+0x120/0x7d0 [libceph] [ 373.933511] [<ffffffffa05c5118>] con_work+0x19f8/0x2e80 [libceph] [ 373.941444] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 373.949035] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 373.956363] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 373.963010] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 373.970149] IN-RECLAIM_FS-W at: [ 373.973649] [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10 [ 373.981413] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 373.988828] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 373.996678] [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph] [ 374.005912] [<ffffffffa05cd0bf>] ceph_osdc_writepages+0xdf/0x180 [libceph] [ 374.014885] [<ffffffffa061225a>] writepage_nounlock+0x22a/0x410 [ceph] [ 374.023513] [<ffffffffa0612bee>] ceph_writepage+0x3e/0x70 [ceph] [ 374.031619] [<ffffffff81143164>] shrink_page_list+0x554/0x900 [ 374.039468] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 374.047660] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 374.055284] [<ffffffff811452e3>] kswapd+0x5d3/0xa90 [ 374.062252] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 374.069132] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 374.076544] INITIAL USE at: [ 374.079711] [<ffffffff810b4d42>] __lock_acquire+0x302/0x1d10 [ 374.087127] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 374.094182] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 374.101683] [<ffffffffa05cc56f>] kick_requests+0x3f/0x3e0 [libceph] [ 374.109721] [<ffffffffa05cd7b4>] ceph_osdc_handle_map+0x494/0x580 [libceph] [ 374.118437] [<ffffffffa05c90b0>] dispatch+0x120/0x7d0 [libceph] [ 374.127242] [<ffffffffa05c5118>] con_work+0x19f8/0x2e80 [libceph] [ 374.135091] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 374.142593] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 374.149899] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 374.156432] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 374.163484] } [ 374.165194] ... key at: [<ffffffffa05e7fa8>] __key.29809+0x0/0xffffffffffff03bb [libceph] [ 374.173999] ... acquired at: [ 374.177017] [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150 [ 374.183466] [<ffffffff810b48e8>] mark_lock+0x198/0x2f0 [ 374.188968] [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10 [ 374.194982] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 374.200641] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 374.206741] [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph] [ 374.214243] [<ffffffffa05cd0bf>] ceph_osdc_writepages+0xdf/0x180 [libceph] [ 374.221532] [<ffffffffa061225a>] writepage_nounlock+0x22a/0x410 [ceph] [ 374.228420] [<ffffffffa0612bee>] ceph_writepage+0x3e/0x70 [ceph] [ 374.234830] [<ffffffff81143164>] shrink_page_list+0x554/0x900 [ 374.240948] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 374.247402] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 374.253242] [<ffffffff811452e3>] kswapd+0x5d3/0xa90 [ 374.258469] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 374.263626] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 374.269341] [ 374.270881] [ 374.270881] stack backtrace: [ 374.275297] Pid: 38, comm: kswapd0 Not tainted 3.8.0-ceph-00219-g8383641 #1 [ 374.282296] Call Trace: [ 374.284790] [<ffffffff810b3dd8>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [ 374.292007] [<ffffffff810b3eff>] check_usage_forwards+0x11f/0x150 [ 374.298228] [<ffffffff810b48e8>] mark_lock+0x198/0x2f0 [ 374.303494] [<ffffffff810b3de0>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [ 374.310859] [<ffffffff810b4ffd>] __lock_acquire+0x5bd/0x1d10 [ 374.316651] [<ffffffff81338a61>] ? vsnprintf+0x461/0x640 [ 374.322137] [<ffffffffa05cb4b5>] ? ceph_osdc_start_request+0x65/0x160 [libceph] [ 374.329583] [<ffffffff810b6d22>] lock_acquire+0xa2/0x140 [ 374.335030] [<ffffffffa05cb4b5>] ? ceph_osdc_start_request+0x65/0x160 [libceph] [ 374.342479] [<ffffffffa05ca9a9>] ? ceph_osdc_build_request+0x119/0x650 [libceph] [ 374.350058] [<ffffffff8165aadb>] mutex_lock_nested+0x4b/0x320 [ 374.355935] [<ffffffffa05cb4b5>] ? ceph_osdc_start_request+0x65/0x160 [libceph] [ 374.363396] [<ffffffffa05cb4b5>] ceph_osdc_start_request+0x65/0x160 [libceph] [ 374.370673] [<ffffffffa05cd0bf>] ceph_osdc_writepages+0xdf/0x180 [libceph] [ 374.377675] [<ffffffffa061225a>] writepage_nounlock+0x22a/0x410 [ceph] [ 374.384364] [<ffffffffa0612bee>] ceph_writepage+0x3e/0x70 [ceph] [ 374.390563] [<ffffffff81143164>] shrink_page_list+0x554/0x900 [ 374.396440] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 374.402661] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 374.408709] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 374.414332] [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300 [ 374.419993] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 374.426822] [<ffffffff8165e05f>] ? _raw_spin_unlock_irqrestore+0x3f/0x80 [ 374.433674] [<ffffffff811452e3>] kswapd+0x5d3/0xa90 [ 374.438681] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 374.444990] [<ffffffff81144d10>] ? try_to_free_pages+0x1a0/0x1a0 [ 374.451126] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 374.456049] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 374.462271] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 374.467713] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 374.527388] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 374.579652] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 374.620391] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 374.655635] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 374.821508] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 375.142073] libceph: osd3 10.214.132.118:6800 socket closed (con state OPEN) [ 375.209835] libceph: get_reply front 119 > preallocated 95 [ 375.219197] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 375.282686] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN)
#3 Updated by Alex Elder about 11 years ago
I just noticed the end of the kernel messages, leading up
to the hard crash, was not captured in what I initially
added. Below is the missing part.
[ 4710.419742] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4710.425422] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4710.430591] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4710.435404] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4710.441078] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4710.447924] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4710.454124] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4710.460354] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4710.466549] [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50 [ 4710.473013] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4710.478602] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4710.484392] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4710.490674] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4710.496868] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4710.502717] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4710.509115] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4710.515145] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4710.521209] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4710.526626] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4710.531971] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4710.537388] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4710.542721] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4710.548835] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4710.554862] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4710.560371] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4710.566484] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4710.572166] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4710.577843] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4710.583978] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4710.589825] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4710.595858] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4710.602401] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4710.607989] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4710.613848] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4710.618744] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4710.624977] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4710.630445] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4731.605879] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229119 c=229118 q=3269) [ 4731.615303] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4731.622942] Call Trace: [ 4731.625408] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4731.631480] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4731.637700] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4731.643722] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4731.649763] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4731.656216] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4731.661891] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4731.665841] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2106 jiffies, g=229119, c=229118, q=3269) [ 4731.665843] Task dump for CPU 0: [ 4731.665851] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4731.665854] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4731.665855] ffff88043ff6ec38 0000000000000282 ffff880090289368 ffffffff81085343 [ 4731.665857] ffffea000abfcdc0 ffffea000f4a8ec0 ffff880090289520 ffffea000f4a8ee0 [ 4731.665857] Call Trace: [ 4731.665862] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4731.665866] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4731.665870] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4731.665872] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4731.665874] [<ffffffff81339c9a>] ? delay_tsc+0x4a/0x80 [ 4731.665878] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4731.665880] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4731.665882] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4731.665884] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4731.665885] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4731.665887] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4731.665890] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4731.665894] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4731.665895] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4731.665897] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4731.665899] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4731.665903] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4731.665906] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4731.665909] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4731.665912] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4731.665913] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4731.665915] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4731.665919] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4731.665927] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4731.665929] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4731.665937] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4731.665940] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4731.665942] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4731.665943] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4731.665945] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4731.665947] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4731.665951] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4731.665953] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4731.665955] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4731.665956] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4731.665959] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4731.665962] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4731.665964] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4731.940584] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4731.946523] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4731.953072] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4731.958924] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4731.965123] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4731.971491] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4731.977514] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4731.983186] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4731.988867] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4731.994021] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4731.998841] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4732.004514] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4732.011403] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4732.017603] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4732.023799] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4732.029994] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4732.035858] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4732.041450] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4732.047230] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4732.053513] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4732.059717] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4732.065564] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4732.071936] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4732.077957] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4732.083980] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4732.089395] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4732.094725] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4732.100169] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4732.105499] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4732.111612] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4732.117638] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4732.123149] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4732.129262] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4732.134936] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4732.140613] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4732.146724] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4732.152572] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4732.158617] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4732.165165] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4732.170758] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4732.176611] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4732.181506] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4732.187707] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4732.193129] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4753.171915] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229120 c=229119 q=3487) [ 4753.181334] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4753.188979] Call Trace: [ 4753.191440] <IRQ> [ 4753.191931] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229120, c=229119, q=3487) [ 4753.191939] Task dump for CPU 0: [ 4753.191944] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4753.191946] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4753.191948] ffff88043ff85d38 0000000000000282 ffff880090289368 ffffffff81085343 [ 4753.191950] ffffea0004fcc780 ffffea000f5df280 ffff880090289520 ffffea000f5df2a0 [ 4753.191952] Call Trace: [ 4753.191953] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4753.191958] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4753.191963] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4753.191966] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4753.191968] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4753.191973] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4753.191977] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4753.191979] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4753.191980] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4753.191982] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4753.191983] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4753.191985] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4753.191988] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4753.191992] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4753.191994] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4753.191996] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4753.191998] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4753.192001] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4753.192004] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4753.192007] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4753.192010] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4753.192011] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4753.192013] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4753.192023] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4753.192025] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4753.192027] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4753.192036] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4753.192038] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4753.192040] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4753.192042] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4753.192043] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4753.192045] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4753.192049] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4753.192051] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4753.192052] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4753.192054] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4753.192056] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4753.192059] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4753.467433] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4753.473136] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4753.479326] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4753.485400] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4753.491459] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4753.497913] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4753.503613] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4753.509107] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4753.515035] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4753.521586] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4753.527429] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4753.533650] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4753.540019] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4753.546083] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4753.551802] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4753.557472] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4753.562625] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4753.567428] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4753.573156] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4753.579984] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4753.586172] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4753.592378] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4753.598572] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4753.604412] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4753.609993] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4753.615774] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4753.622049] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4753.628243] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4753.634102] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4753.640473] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4753.646493] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4753.652507] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4753.657923] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4753.663271] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4753.668679] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4753.674035] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4753.680137] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4753.686183] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4753.691681] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4753.697789] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4753.703464] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4753.709131] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4753.715267] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4753.721112] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4753.727134] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4753.733681] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4753.739270] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4753.745137] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4753.750024] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4753.756248] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4753.761656] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4778.731618] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229123 c=229122 q=2731) [ 4778.741038] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4778.748641] Call Trace: [ 4778.751102] <IRQ> [ 4778.751647] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229123, c=229122, q=2731) [ 4778.751656] Task dump for CPU 0: [ 4778.751657] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4778.751662] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4778.751668] ffff88043ff7a038 0000000000000282 ffff880090289368 ffffffff81085343 [ 4778.751670] 0000000000000024 ffffea000f6c21c0 ffff880090289520 ffffea000f6c21e0 [ 4778.751672] Call Trace: [ 4778.751673] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4778.751679] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4778.751683] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4778.751686] [<ffffffff81143ad4>] ? shrink_inactive_list+0x224/0x400 [ 4778.751688] [<ffffffff81143ad4>] ? shrink_inactive_list+0x224/0x400 [ 4778.751690] [<ffffffff81143745>] ? putback_inactive_pages+0x115/0x280 [ 4778.751692] [<ffffffff81143b9a>] ? shrink_inactive_list+0x2ea/0x400 [ 4778.751694] [<ffffffff81143b4a>] ? shrink_inactive_list+0x29a/0x400 [ 4778.751695] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4778.751697] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4778.751699] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4778.751702] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4778.751706] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4778.751708] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4778.751710] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4778.751712] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4778.751715] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4778.751718] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4778.751722] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4778.751724] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4778.751725] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4778.751728] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4778.751737] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4778.751740] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4778.751741] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4778.751750] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4778.751753] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4778.751754] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4778.751756] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4778.751758] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4778.751759] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4778.751764] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4778.751766] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4778.751767] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4778.751769] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4778.751771] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4778.751774] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4779.022525] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4779.028168] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4779.034358] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4779.040380] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4779.046421] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4779.052877] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4779.058550] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4779.064046] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4779.069981] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4779.076542] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4779.082391] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4779.088586] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4779.094956] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4779.100977] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4779.106653] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4779.112331] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4779.117485] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4779.122307] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4779.127973] <EOI> [<ffffffff81143745>] ? putback_inactive_pages+0x115/0x280 [ 4779.135147] [<ffffffff81143b9a>] ? shrink_inactive_list+0x2ea/0x400 [ 4779.141509] [<ffffffff81143b4a>] ? shrink_inactive_list+0x29a/0x400 [ 4779.147877] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4779.153736] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4779.159324] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4779.165080] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4779.171362] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4779.177557] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4779.183424] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4779.189792] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4779.195832] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4779.201855] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4779.207269] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4779.212590] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4779.217998] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4779.223321] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4779.229425] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4779.235446] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4779.241002] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4779.247109] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4779.252802] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4779.258470] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4779.264571] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4779.270411] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4779.276428] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4779.283013] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4779.288599] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4779.294467] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4779.299354] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4779.305579] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4779.310994] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4800.287722] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229124 c=229123 q=1159) [ 4800.297147] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4800.304775] Call Trace: [ 4800.307234] <IRQ> [ 4800.307703] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229124, c=229123, q=1159) [ 4800.307712] Task dump for CPU 0: [ 4800.307714] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4800.307719] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4800.307726] ffff88043ff6ec38 0000000000000282 ffff880090289368 ffffffff81085343 [ 4800.307727] 0000000000000030 ffffea000c6d0ec0 ffff880090289520 ffffea000c6d0ee0 [ 4800.307729] Call Trace: [ 4800.307731] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4800.307736] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4800.307740] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4800.307744] [<ffffffff81143ad4>] ? shrink_inactive_list+0x224/0x400 [ 4800.307746] [<ffffffff8113e4d7>] ? lru_add_drain_cpu+0x27/0x130 [ 4800.307748] [<ffffffff8113e676>] ? lru_add_drain+0x16/0x20 [ 4800.307749] [<ffffffff8114396a>] ? shrink_inactive_list+0xba/0x400 [ 4800.307754] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4800.307755] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4800.307757] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4800.307760] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4800.307764] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4800.307766] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4800.307767] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4800.307770] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4800.307772] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4800.307776] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4800.307779] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4800.307781] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4800.307783] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4800.307785] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4800.307794] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4800.307796] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4800.307798] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4800.307807] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4800.307810] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4800.307811] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4800.307813] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4800.307815] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4800.307816] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4800.307821] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4800.307822] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4800.307824] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4800.307826] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4800.307828] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4800.307831] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4800.569834] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4800.575474] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4800.581717] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4800.587740] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4800.593765] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4800.600216] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4800.605884] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4800.611379] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4800.617333] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4800.623875] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4800.629769] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4800.635964] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4800.642334] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4800.648413] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4800.654081] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4800.659778] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4800.664931] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4800.669734] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4800.675402] <EOI> [<ffffffff8113e4d7>] ? lru_add_drain_cpu+0x27/0x130 [ 4800.682055] [<ffffffff8113e676>] lru_add_drain+0x16/0x20 [ 4800.687481] [<ffffffff8114396a>] shrink_inactive_list+0xba/0x400 [ 4800.693584] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4800.699484] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4800.705066] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4800.710830] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4800.717111] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4800.723308] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4800.729157] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4800.735525] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4800.741555] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4800.747588] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4800.753003] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4800.758324] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4800.763732] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4800.769081] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4800.775193] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4800.781264] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4800.786760] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4800.792867] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4800.798536] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4800.804210] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4800.810321] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4800.816162] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4800.822214] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4800.828763] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4800.834351] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4800.840192] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4800.845087] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4800.851277] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4800.856693] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4821.863718] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229126 c=229125 q=2904) [ 4821.874318] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4821.881910] Call Trace: [ 4821.883727] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229126, c=229125, q=2904) [ 4821.883728] Task dump for CPU 0: [ 4821.883733] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4821.883738] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4821.883743] ffff88043ff72238 0000000000000282 ffff880090289368 ffffffff81085343 [ 4821.883745] 0000000000000030 ffffea000c6e2140 ffff880090289520 ffffea000c6e2160 [ 4821.883745] Call Trace: [ 4821.883752] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4821.883756] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4821.883760] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4821.883761] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4821.883763] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4821.883767] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4821.883772] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4821.883774] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4821.883775] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4821.883777] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4821.883779] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4821.883781] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4821.883785] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4821.883787] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4821.883789] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4821.883791] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4821.883794] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4821.883796] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4821.883799] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4821.883801] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4821.883802] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4821.883804] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4821.883807] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4821.883814] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4821.883816] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4821.883824] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4821.883826] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4821.883828] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4821.883829] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4821.883831] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4821.883832] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4821.883837] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4821.883838] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4821.883840] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4821.883841] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4821.883843] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4821.883847] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4821.883848] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4822.157139] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4822.163194] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4822.169384] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4822.175466] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4822.181489] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4822.187945] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4822.193619] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4822.199112] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4822.205058] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4822.211593] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4822.217444] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4822.223636] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4822.229998] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4822.236063] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4822.241738] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4822.247409] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4822.252562] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4822.257365] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4822.263031] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4822.269858] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4822.276082] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4822.282270] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4822.288464] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4822.294306] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4822.299887] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4822.305695] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4822.311977] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4822.318172] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4822.324021] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4822.330392] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4822.336422] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4822.342444] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4822.347877] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4822.353196] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4822.358605] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4822.363928] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4822.370030] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4822.376096] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4822.381601] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4822.387725] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4822.393400] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4822.399068] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4822.405185] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4822.411026] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4822.417103] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4822.423645] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4822.429231] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4822.435073] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4822.439962] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4822.446159] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4822.451574] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4843.429725] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229127 c=229126 q=1032) [ 4843.439145] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4843.446749] Call Trace: [ 4843.449208] <IRQ> [ 4843.449782] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229127, c=229126, q=1032) [ 4843.449790] Task dump for CPU 0: [ 4843.449792] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4843.449797] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4843.449803] ffff8800902893a8 0000000000000282 0000000000000030 0000000000000000 [ 4843.449805] ffff880090289540 ffff880090289538 0000000000000000 0000000000000000 [ 4843.449807] Call Trace: [ 4843.449809] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4843.449814] [<ffffffff8165c8e0>] ? _cond_resched+0x20/0x40 [ 4843.449818] [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900 [ 4843.449822] [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0 [ 4843.449824] [<ffffffff81143ac7>] ? shrink_inactive_list+0x217/0x400 [ 4843.449826] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4843.449828] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4843.449830] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4843.449833] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4843.449836] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4843.449838] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4843.449840] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4843.449843] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4843.449845] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4843.449848] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4843.449852] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4843.449855] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4843.449856] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4843.449859] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4843.449866] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4843.449869] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4843.449870] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4843.449879] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4843.449881] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4843.449883] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4843.449885] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4843.449887] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4843.449888] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4843.449893] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4843.449895] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4843.449896] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4843.449898] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4843.449900] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4843.449903] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4843.700929] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4843.706572] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4843.712762] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4843.718784] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4843.724817] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4843.731270] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4843.736938] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4843.742449] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4843.748377] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4843.754946] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4843.760797] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4843.766990] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4843.773352] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4843.779374] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4843.785049] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4843.790728] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4843.795881] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4843.800736] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4843.806402] <EOI> [<ffffffff8165c8e0>] ? _cond_resched+0x20/0x40 [ 4843.812683] [<ffffffff81142d07>] shrink_page_list+0xf7/0x900 [ 4843.818441] [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0 [ 4843.825148] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 4843.831360] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4843.837219] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4843.842807] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4843.849796] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4843.856075] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4843.862270] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4843.868118] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4843.874487] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4843.880510] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4843.886532] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4843.891957] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4843.897286] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4843.902746] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4843.908068] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4843.914181] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4843.920200] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4843.925697] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4843.931805] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4843.937480] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4843.943175] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4843.949285] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4843.955133] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4843.961159] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4843.967697] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4843.973278] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4843.979119] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4843.984008] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4843.990239] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4843.995654] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4872.174500] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229131 c=229130 q=802) [ 4872.183835] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4872.191473] Call Trace: [ 4872.193934] <IRQ> [ 4872.194486] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229131, c=229130, q=802) [ 4872.194494] Task dump for CPU 0: [ 4872.194496] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4872.194501] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4872.194507] ffff88043ffadc38 0000000000000282 ffff8800902894b8 ffffffff81085343 [ 4872.194509] 0000000000000030 57ffacd1164bbe40 a8003d2ef9000000 f200000000000000 [ 4872.194511] Call Trace: [ 4872.194513] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4872.194518] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4872.194522] [<ffffffff81142d4c>] ? shrink_page_list+0x13c/0x900 [ 4872.194526] [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900 [ 4872.194528] [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0 [ 4872.194530] [<ffffffff81143ac7>] ? shrink_inactive_list+0x217/0x400 [ 4872.194531] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4872.194533] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4872.194535] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4872.194538] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4872.194542] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4872.194544] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4872.194546] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4872.194548] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4872.194551] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4872.194554] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4872.194558] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4872.194560] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4872.194562] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4872.194564] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4872.194573] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4872.194576] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4872.194577] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4872.194586] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4872.194588] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4872.194590] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4872.194592] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4872.194594] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4872.194595] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4872.194600] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4872.194602] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4872.194603] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4872.194605] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4872.194607] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4872.194611] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4872.451123] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4872.456791] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4872.462981] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4872.469030] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4872.475053] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4872.481508] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4872.487182] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4872.492677] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4872.498631] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4872.505191] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4872.511042] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4872.517236] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4872.523649] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4872.529704] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4872.535398] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4872.541067] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4872.546223] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4872.551034] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4872.556717] <EOI> [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4872.562591] [<ffffffff81142d4c>] ? shrink_page_list+0x13c/0x900 [ 4872.568608] [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900 [ 4872.574537] [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0 [ 4872.581251] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 4872.587490] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4872.593339] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4872.598919] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4872.604676] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4872.610957] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4872.617153] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4872.623001] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4872.629382] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4872.635419] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4872.641441] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4872.646849] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4872.652170] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4872.657604] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4872.662927] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4872.669072] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4872.675154] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4872.680659] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4872.686809] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4872.692484] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4872.698152] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4872.704255] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4872.710102] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4872.716128] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4872.722668] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4872.728273] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4872.734115] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4872.739004] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4872.745245] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4872.750660] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4893.730552] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229132 c=229131 q=1229) [ 4893.739986] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4893.747580] Call Trace: [ 4893.750042] <IRQ> [ 4893.750534] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=2102 jiffies, g=229132, c=229131, q=1229) [ 4893.750543] Task dump for CPU 0: [ 4893.750545] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4893.750549] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4893.750556] ffff88043ff6f538 0000000000000282 ffff880090289368 ffffffff81085343 [ 4893.750558] ffffffff8113a207 ffffea0010856c80 ffff880090289520 ffffea0010856ca0 [ 4893.750560] Call Trace: [ 4893.750561] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4893.750567] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4893.750570] [<ffffffff8113a207>] ? free_hot_cold_page_list+0x27/0xc0 [ 4893.750573] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4893.750576] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4893.750578] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4893.750580] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4893.750584] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4893.750586] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4893.750588] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4893.750590] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4893.750591] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4893.750593] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4893.750596] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4893.750600] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4893.750602] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4893.750604] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4893.750606] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4893.750608] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4893.750612] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4893.750615] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4893.750618] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4893.750619] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4893.750622] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4893.750631] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4893.750634] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4893.750635] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4893.750644] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4893.750646] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4893.750648] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4893.750650] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4893.750652] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4893.750653] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4893.750658] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4893.750659] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4893.750661] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4893.750663] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4893.750665] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4893.750668] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4894.033598] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4894.039240] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4894.045481] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4894.051503] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4894.057520] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4894.063974] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4894.069649] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4894.075177] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4894.081105] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4894.087640] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4894.093500] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4894.099692] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4894.106081] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4894.112100] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4894.117767] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4894.123437] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4894.128593] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4894.133438] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4894.139105] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4894.145967] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4894.152199] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4894.158387] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4894.164583] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4894.170431] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4894.176012] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4894.181802] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4894.188083] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4894.194295] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4894.200136] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4894.206499] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4894.212520] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4894.218542] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4894.223958] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4894.229279] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4894.234687] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4894.240063] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4894.246172] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4894.252210] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4894.257708] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4894.263841] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4894.269516] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4894.275202] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4894.281310] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4894.287151] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4894.293176] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4894.299722] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4894.305305] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4894.311162] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4894.316094] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4894.322282] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4894.327690] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4915.316514] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229135 c=229134 q=511) [ 4915.325888] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4915.333534] Call Trace: [ 4915.335993] <IRQ> [ 4915.336541] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229135, c=229134, q=511) [ 4915.336549] Task dump for CPU 0: [ 4915.336551] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4915.336556] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4915.336562] ffff88043ffa2838 0000000000000282 ffff880090289368 ffffffff81085343 [ 4915.336564] ffffea0010335540 ffffea0006785fc0 ffff880090289520 ffffea0006785fe0 [ 4915.336566] Call Trace: [ 4915.336567] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4915.336573] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4915.336577] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4915.336581] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4915.336582] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4915.336584] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4915.336588] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4915.336590] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4915.336592] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4915.336594] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4915.336595] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4915.336597] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4915.336600] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4915.336604] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4915.336606] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4915.336608] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4915.336610] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4915.336613] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4915.336616] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4915.336620] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4915.336622] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4915.336624] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4915.336626] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4915.336635] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4915.336638] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4915.336640] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4915.336648] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4915.336651] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4915.336653] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4915.336654] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4915.336656] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4915.336658] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4915.336662] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4915.336664] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4915.336665] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4915.336668] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4915.336669] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4915.336673] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4915.611745] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4915.617420] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4915.623610] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4915.629677] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4915.635692] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4915.642163] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4915.647840] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4915.653342] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4915.659280] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4915.665823] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4915.671689] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4915.677883] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4915.684244] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4915.690266] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4915.696010] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4915.701681] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4915.706859] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4915.711662] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4915.717365] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4915.724193] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4915.730381] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4915.736568] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4915.742762] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4915.748622] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4915.754210] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4915.759983] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4915.766267] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4915.772462] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4915.778311] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4915.784679] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4915.790719] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4915.796743] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4915.802156] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4915.807488] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4915.812894] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4915.818244] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4915.824354] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4915.830392] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4915.835890] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4915.841996] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4915.847681] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4915.853348] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4915.859475] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4915.865315] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4915.871402] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4915.877976] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4915.883557] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4915.889432] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4915.894319] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4915.900509] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4915.905925] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4948.154842] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229140 c=229139 q=419) [ 4948.164177] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4948.171814] Call Trace: [ 4948.174273] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4948.180363] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4948.186587] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4948.192610] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4948.194833] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2104 jiffies, g=229140, c=229139, q=419) [ 4948.194835] Task dump for CPU 0: [ 4948.194839] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4948.194844] ffff880090289308 ffff88042d073f20 ffff880090289328 0000000000000282 [ 4948.194850] ffff88043ffda338 0000000000000282 ffff880090289368 ffffffff81085343 [ 4948.194852] 0000000000000030 ffffea0002608100 ffff880090289520 ffffea0002608120 [ 4948.194852] Call Trace: [ 4948.194858] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4948.194862] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4948.194864] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4948.194866] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4948.194870] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4948.194871] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4948.194873] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4948.194875] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4948.194878] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4948.194880] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4948.194882] [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300 [ 4948.194884] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4948.194886] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4948.194888] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4948.194891] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4948.194894] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4948.194897] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4948.194900] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4948.194901] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4948.194903] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4948.194907] [<ffffffff813060f8>] ? cry
#4 Updated by Alex Elder about 11 years ago
Let's see if this one doesn't exceed the maximum size
something added.
[ 4948.194903] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4948.194907] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4948.194913] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4948.194915] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4948.194924] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4948.194926] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4948.194927] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4948.194929] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4948.194932] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4948.194933] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4948.194938] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4948.194940] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4948.194941] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4948.194944] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4948.194946] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4948.194948] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4948.194950] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4948.460699] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4948.467178] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4948.472853] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4948.478374] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4948.484392] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4948.490930] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4948.496782] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4948.502975] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4948.509364] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4948.515385] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4948.521060] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4948.526746] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4948.531902] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4948.536756] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4948.542423] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4948.549250] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4948.555483] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4948.561678] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4948.567890] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4948.573914] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4948.579511] [<ffffffff8114227c>] ? shrink_slab+0x1dc/0x300 [ 4948.585092] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4948.591279] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4948.597146] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4948.603515] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4948.609556] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4948.615586] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4948.620993] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4948.626324] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4948.631732] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4948.637080] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4948.643182] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4948.649229] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4948.654734] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4948.660842] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4948.666516] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4948.672186] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4948.678346] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4948.684238] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4948.690255] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4948.696812] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4948.702393] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4948.708268] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4948.713157] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4948.719422] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4948.724889] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4969.700894] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229141 c=229140 q=2998) [ 4969.710323] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4969.717971] Call Trace: [ 4969.720430] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4969.726502] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4969.732735] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4969.738758] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4969.744799] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4969.751251] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4969.756920] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4969.762422] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4969.768349] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4969.770817] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2107 jiffies, g=229141, c=229140, q=2998) [ 4969.770818] Task dump for CPU 0: [ 4969.770823] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4969.770828] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4969.770834] ffff88043ffaf738 0000000000000282 ffff880090289368 ffffffff81085343 [ 4969.770836] 0000000000000030 ffffea00078b8c00 ffff880090289520 ffffea00078b8c20 [ 4969.770836] Call Trace: [ 4969.770840] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4969.770844] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4969.770848] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4969.770850] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4969.770852] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4969.770856] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4969.770858] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4969.770859] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4969.770861] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4969.770863] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4969.770865] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4969.770867] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4969.770871] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4969.770873] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4969.770875] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4969.770877] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4969.770880] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4969.770883] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4969.770886] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4969.770889] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4969.770891] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4969.770893] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4969.770896] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4969.770905] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4969.770906] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4969.770915] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4969.770917] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4969.770919] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4969.770920] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4969.770922] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4969.770924] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4969.770928] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4969.770930] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4969.770931] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4969.770933] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4969.770935] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4969.770938] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4969.770940] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4970.047922] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4970.053789] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4970.059984] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4970.066370] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4970.072393] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4970.078067] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4970.083736] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4970.088882] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4970.093728] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4970.099395] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4970.106249] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4970.112481] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4970.118668] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4970.124916] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4970.130836] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4970.136425] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4970.142205] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4970.148487] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4970.154682] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4970.160583] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4970.166944] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4970.172992] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4970.179005] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4970.184448] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4970.189769] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4970.195212] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4970.200537] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4970.206646] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4970.212667] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4970.218172] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4970.224295] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4970.229963] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4970.236806] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4970.242968] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4970.248809] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4970.254860] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4970.261401] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4970.266989] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4970.272841] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4970.277735] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4970.283943] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4970.289358] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4991.266987] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229142 c=229141 q=1032) [ 4991.276444] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 4991.284037] Call Trace: [ 4991.286498] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 4991.292553] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 4991.298766] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 4991.304781] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 4991.310832] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 4991.317286] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 4991.322962] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 4991.328464] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 4991.334391] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 4991.336861] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2107 jiffies, g=229142, c=229141, q=1034) [ 4991.336863] Task dump for CPU 0: [ 4991.336868] kworker/0:2 R running task 0 26683 2 0x00000000 [ 4991.336874] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 4991.336876] ffff88043ffa2838 0000000000000282 ffff880090289368 ffffffff81085343 [ 4991.336878] 0000000000000024 ffffea000c6d9fc0 ffff880090289520 ffffea000c6d9fe0 [ 4991.336878] Call Trace: [ 4991.336882] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4991.336886] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 4991.336890] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 4991.336891] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4991.336893] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 4991.336897] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4991.336899] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4991.336901] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4991.336903] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 4991.336904] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4991.336906] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 4991.336909] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4991.336912] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4991.336914] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 4991.336916] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 4991.336918] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 4991.336921] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 4991.336925] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 4991.336928] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 4991.336931] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 4991.336932] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4991.336934] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 4991.336938] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4991.336946] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4991.336948] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 4991.336956] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 4991.336958] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4991.336960] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4991.336961] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4991.336963] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 4991.336965] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4991.336969] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4991.336971] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 4991.336972] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4991.336974] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 4991.336976] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4991.336979] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 4991.336981] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4991.613772] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 4991.619649] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 4991.625844] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 4991.632265] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 4991.638286] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 4991.643954] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 4991.649623] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 4991.654777] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 4991.659588] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 4991.665257] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4991.672108] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 4991.678369] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 4991.684564] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 4991.690758] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 4991.696600] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 4991.702188] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 4991.707953] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 4991.714234] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 4991.720448] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 4991.726297] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 4991.732667] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 4991.738689] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 4991.744711] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 4991.750127] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 4991.755447] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 4991.760900] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 4991.766240] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 4991.772349] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 4991.778379] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 4991.783877] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 4991.789992] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 4991.795667] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 4991.801353] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 4991.807462] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 4991.813302] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 4991.819319] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 4991.825859] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 4991.831448] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 4991.837315] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 4991.842212] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 4991.848426] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 4991.853842] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5012.832939] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229143 c=229142 q=3496) [ 5012.842373] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 5012.849968] Call Trace: [ 5012.852429] <IRQ> [ 5012.852967] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229143, c=229142, q=3496) [ 5012.852976] Task dump for CPU 0: [ 5012.852978] kworker/0:2 R running task 0 26683 2 0x00000000 [ 5012.852983] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 5012.852989] ffff88043ff6ec38 0000000000000282 ffff880090289368 ffffffff81085343 [ 5012.852991] 0200000000040818 ffffea000e8baec0 ffff880090289520 ffffea000e8baee0 [ 5012.852993] Call Trace: [ 5012.852994] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5012.852999] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5012.853003] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 5012.853007] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5012.853009] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5012.853011] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5012.853017] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5012.853019] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5012.853021] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 5012.853022] [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50 [ 5012.853025] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 5012.853026] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5012.853029] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5012.853033] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 5012.853035] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 5012.853037] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 5012.853039] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 5012.853041] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 5012.853044] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 5012.853048] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 5012.853050] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5012.853052] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 5012.853054] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5012.853063] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5012.853065] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 5012.853067] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 5012.853075] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5012.853078] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5012.853080] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5012.853082] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 5012.853083] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5012.853085] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5012.853089] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 5012.853091] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5012.853092] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 5012.853094] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5012.853096] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 5012.853099] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5013.129138] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 5013.134815] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 5013.141007] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 5013.147047] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 5013.153068] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 5013.159523] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 5013.165198] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 5013.170693] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 5013.176657] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 5013.183198] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 5013.190217] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 5013.196482] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 5013.202887] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 5013.208906] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 5013.214574] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 5013.220251] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 5013.225407] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 5013.230208] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 5013.235876] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5013.242737] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5013.248927] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5013.255140] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 5013.261335] [<ffffffff8113a4da>] ? global_dirtyable_memory+0x1a/0x50 [ 5013.267782] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 5013.273380] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5013.279142] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5013.285443] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 5013.291638] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 5013.297505] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 5013.303875] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 5013.309896] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 5013.315918] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 5013.321334] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 5013.326682] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5013.332090] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 5013.337437] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5013.343541] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5013.349562] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 5013.355076] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 5013.361181] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5013.366868] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5013.372542] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5013.378644] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 5013.384510] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5013.390526] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5013.397067] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 5013.402655] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5013.408496] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 5013.413419] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5013.419609] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 5013.425049] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5034.508803] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229147 c=229146 q=525) [ 5034.518137] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 5034.525738] Call Trace: [ 5034.528199] <IRQ> [ 5034.528765] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229147, c=229146, q=525) [ 5034.528769] Task dump for CPU 0: [ 5034.528770] kworker/0:2 R running task 0 26683 2 0x00000000 [ 5034.528773] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 5034.528778] ffff88043ff7a038 0000000000000282 ffff880090289368 ffffffff81085343 [ 5034.528780] 0000000000000030 ffffea0005b3c1c0 ffff880090289520 ffffea0005b3c1e0 [ 5034.528782] Call Trace: [ 5034.528783] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5034.528788] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5034.528792] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 5034.528795] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5034.528797] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5034.528799] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5034.528803] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5034.528805] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5034.528807] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 5034.528808] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5034.528810] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 5034.528812] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5034.528819] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5034.528825] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 5034.528829] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 5034.528834] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 5034.528838] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 5034.528843] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 5034.528848] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 5034.528856] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 5034.528858] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5034.528860] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 5034.528862] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5034.528871] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5034.528874] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 5034.528876] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 5034.528884] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5034.528887] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5034.528888] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5034.528890] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 5034.528892] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5034.528894] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5034.528899] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 5034.528900] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5034.528902] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 5034.528904] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5034.528906] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 5034.528909] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5034.804209] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 5034.809895] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 5034.816085] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 5034.822135] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 5034.828158] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 5034.834628] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 5034.840304] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 5034.845799] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 5034.851761] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 5034.858372] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 5034.864223] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 5034.870443] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 5034.876815] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 5034.882843] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 5034.888579] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 5034.894249] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 5034.899438] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 5034.904241] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 5034.909942] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5034.916770] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5034.923035] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5034.929310] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 5034.935504] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5034.941363] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 5034.946951] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5034.952733] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5034.959067] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 5034.965262] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 5034.971112] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 5034.977474] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 5034.983496] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 5034.989525] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 5034.994934] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 5035.000324] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5035.005732] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 5035.011054] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5035.017157] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5035.023204] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 5035.028709] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 5035.034817] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5035.040491] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5035.046160] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5035.052285] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 5035.058126] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5035.064143] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5035.070701] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 5035.076289] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5035.082166] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 5035.087062] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5035.093293] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 5035.098778] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5056.074880] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229148 c=229147 q=2975) [ 5056.084318] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 5056.091911] Call Trace: [ 5056.094370] <IRQ> [ 5056.094801] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229148, c=229147, q=2975) [ 5056.094805] Task dump for CPU 0: [ 5056.094806] kworker/0:2 R running task 0 26683 2 0x00000000 [ 5056.094809] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 5056.094811] ffff88043ff98638 0000000000000282 ffff880090289368 ffffffff81085343 [ 5056.094813] 0000000000000030 ffffea000f77e840 ffff880090289520 ffffea000f77e860 [ 5056.094814] Call Trace: [ 5056.094816] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5056.094821] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5056.094824] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 5056.094828] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5056.094830] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5056.094832] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5056.094836] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5056.094837] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5056.094839] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 5056.094841] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5056.094842] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 5056.094844] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5056.094847] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5056.094850] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 5056.094852] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 5056.094854] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 5056.094856] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 5056.094859] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 5056.094862] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 5056.094866] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 5056.094868] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5056.094870] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 5056.094872] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5056.094881] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5056.094883] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 5056.094885] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 5056.094893] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5056.094896] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5056.094898] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5056.094900] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 5056.094901] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5056.094903] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5056.094907] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 5056.094909] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5056.094910] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 5056.094913] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5056.094914] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 5056.094917] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5056.371644] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 5056.377314] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 5056.383503] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 5056.389552] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 5056.395585] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 5056.402038] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 5056.407730] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 5056.413224] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 5056.419152] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 5056.425686] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 5056.431529] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 5056.437740] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 5056.444109] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 5056.450174] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 5056.455850] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 5056.461519] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 5056.466673] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 5056.471476] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 5056.477187] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5056.484015] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5056.490202] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5056.496390] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 5056.502586] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5056.508434] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 5056.514016] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5056.519804] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5056.526164] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 5056.532352] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 5056.538200] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 5056.544563] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 5056.550585] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 5056.556606] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 5056.562022] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 5056.567361] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5056.572769] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 5056.578152] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5056.584332] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5056.590354] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 5056.595859] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 5056.601966] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5056.607649] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5056.613316] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5056.619444] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 5056.625310] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5056.631326] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5056.637895] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 5056.643481] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5056.649324] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 5056.654219] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5056.660408] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 5056.665824] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5077.660843] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229152 c=229151 q=602) [ 5077.670183] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 5077.677777] Call Trace: [ 5077.680236] <IRQ> [ 5077.680901] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229152, c=229151, q=602) [ 5077.680910] Task dump for CPU 0: [ 5077.680912] kworker/0:2 R running task 0 26683 2 0x00000000 [ 5077.680919] ffff880090289308 ffffffff810b783d ffff880090289328 0000000000000282 [ 5077.680921] ffff88043ff98638 0000000000000282 ffff880090289368 ffffffff81085343 [ 5077.680923] 0000000000000030 ffffea0010ad8840 ffff880090289520 ffffea0010ad8860 [ 5077.680925] Call Trace: [ 5077.680927] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5077.680932] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5077.680936] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 5077.680940] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5077.680942] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5077.680944] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5077.680948] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5077.680950] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5077.680952] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 5077.680954] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5077.680955] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 5077.680957] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5077.680960] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5077.680964] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 5077.680966] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 5077.680968] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 5077.680970] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 5077.680973] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 5077.680976] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 5077.680980] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 5077.680982] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5077.680984] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 5077.680986] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5077.680995] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5077.680997] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 5077.681002] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 5077.681010] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5077.681012] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5077.681014] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5077.681016] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 5077.681017] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5077.681019] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5077.681023] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 5077.681025] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5077.681026] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 5077.681028] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5077.681030] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 5077.681032] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5077.956203] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 5077.961847] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 5077.968037] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 5077.974086] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 5077.980109] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 5077.986579] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 5077.992263] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 5077.997757] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 5078.003721] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 5078.010262] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 5078.016105] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 5078.022300] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 5078.028669] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 5078.034689] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 5078.040364] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 5078.046035] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 5078.051216] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 5078.056018] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 5078.061685] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5078.068512] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5078.074711] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5078.080907] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 5078.087102] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5078.092976] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 5078.098557] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5078.104312] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5078.110586] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 5078.116781] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 5078.122665] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 5078.129026] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 5078.135049] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 5078.141070] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 5078.146478] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 5078.151807] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5078.157215] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 5078.162573] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5078.168675] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5078.174722] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 5078.180236] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 5078.186344] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5078.192037] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5078.197746] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5078.203892] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 5078.209740] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5078.215818] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5078.222418] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 5078.228007] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5078.233881] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 5078.238770] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5078.244986] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 5078.250469] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5122.290598] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229160 c=229159 q=6525) [ 5122.301268] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 5122.308860] Call Trace: [ 5122.310589] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=2102 jiffies, g=229160, c=229159, q=6525) [ 5122.310591] Task dump for CPU 0: [ 5122.310596] kworker/0:2 R running task 0 26683 2 0x00000000 [ 5122.310600] ffff880090289288 000000018027001a 0000000000000000 ffffffff811b699b [ 5122.310606] ffff880400000000 0000000000000000 ffff88043fffbe68 ffff88042d073f20 [ 5122.310607] 0000000000000000 ffff88043ff6ec38 0000000000000000 ffffffff81085322 [ 5122.310608] Call Trace: [ 5122.310614] [<ffffffff811b699b>] ? try_to_free_buffers+0x4b/0xc0 [ 5122.310619] [<ffffffff81085322>] ? __wake_up+0x32/0x70 [ 5122.310620] [<ffffffff81085322>] ? __wake_up+0x32/0x70 [ 5122.310623] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5122.310624] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5122.310628] [<ffffffff81142d42>] ? shrink_page_list+0x132/0x900 [ 5122.310630] [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900 [ 5122.310632] [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0 [ 5122.310634] [<ffffffff81143ac7>] ? shrink_inactive_list+0x217/0x400 [ 5122.310635] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5122.310637] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 5122.310640] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5122.310643] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5122.310645] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 5122.310647] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 5122.310649] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 5122.310652] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 5122.310656] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 5122.310659] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 5122.310661] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 5122.310663] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5122.310665] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 5122.310669] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5122.310677] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5122.310679] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 5122.310687] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 5122.310690] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5122.310692] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5122.310693] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5122.310695] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 5122.310697] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5122.310701] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5122.310703] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 5122.310704] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5122.310706] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 5122.310708] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5122.310711] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 5122.310713] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5122.582273] <IRQ> [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 5122.588326] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 5122.594516] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 5122.600531] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 5122.606555] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 5122.613010] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 5122.618685] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 5122.624196] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 5122.630128] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 5122.636666] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 5122.642525] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 5122.648719] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 5122.655089] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 5122.661111] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 5122.666786] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 5122.672473] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 5122.677627] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 5122.682483] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 5122.688149] <EOI> [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5122.694022] [<ffffffff81142d42>] ? shrink_page_list+0x132/0x900 [ 5122.700092] [<ffffffff81142d07>] ? shrink_page_list+0xf7/0x900 [ 5122.706027] [<ffffffff81142528>] ? isolate_lru_pages.isra.58+0x78/0x1b0 [ 5122.712763] [<ffffffff81143ac7>] shrink_inactive_list+0x217/0x400 [ 5122.718955] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5122.724823] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 5122.730419] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5122.736174] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5122.742458] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 5122.748652] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 5122.754494] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 5122.760864] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 5122.766886] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 5122.772950] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 5122.778359] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 5122.783722] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5122.789130] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 5122.794452] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5122.800555] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5122.806576] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 5122.812089] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 5122.818197] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5122.823915] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5122.829585] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5122.835695] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 5122.841542] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5122.847559] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5122.854101] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 5122.859687] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5122.865531] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 5122.870425] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5122.876615] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 5122.882057] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5143.856635] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=229161 c=229160 q=9091) [ 5143.866081] Pid: 26683, comm: kworker/0:2 Not tainted 3.8.0-ceph-00226-g1655ae7 #1 [ 5143.873690] Call Trace: [ 5143.876151] <IRQ> [ 5143.876621] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=2102 jiffies, g=229161, c=229160, q=9091) [ 5143.876629] Task dump for CPU 0: [ 5143.876631] kworker/0:2 R running task 0 26683 2 0x00000000 [ 5143.876636] ffff880090289288 0000000180270021 0000000000000000 ffffffff811b699b [ 5143.876642] ffff880400000000 0000000000000000 ffff88043fffbe68 ffff88042d073f20 [ 5143.876643] 0000000000000000 ffff88043ff98638 0000000000000000 ffffffff81085322 [ 5143.876645] Call Trace: [ 5143.876647] [<ffffffff811b699b>] ? try_to_free_buffers+0x4b/0xc0 [ 5143.876652] [<ffffffff81085322>] ? __wake_up+0x32/0x70 [ 5143.876657] [<ffffffff81085322>] ? __wake_up+0x32/0x70 [ 5143.876659] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5143.876661] [<ffffffff81085343>] ? __wake_up+0x53/0x70 [ 5143.876663] [<ffffffff81142f20>] ? shrink_page_list+0x310/0x900 [ 5143.876666] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5143.876668] [<ffffffff811439a6>] ? shrink_inactive_list+0xf6/0x400 [ 5143.876670] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5143.876674] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5143.876675] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5143.876677] [<ffffffff81143a3b>] ? shrink_inactive_list+0x18b/0x400 [ 5143.876679] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5143.876680] [<ffffffff811442ed>] ? shrink_lruvec+0x2fd/0x470 [ 5143.876682] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5143.876685] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5143.876689] [<ffffffff81144806>] ? do_try_to_free_pages+0x126/0x3a0 [ 5143.876691] [<ffffffff81144c3f>] ? try_to_free_pages+0xcf/0x1a0 [ 5143.876693] [<ffffffff811393d9>] ? __alloc_pages_nodemask+0x579/0x8a0 [ 5143.876695] [<ffffffff8117110a>] ? alloc_pages_current+0xba/0x170 [ 5143.876698] [<ffffffff8154074e>] ? sk_page_frag_refill+0x7e/0x130 [ 5143.876701] [<ffffffff815980e9>] ? tcp_sendmsg+0x309/0xe30 [ 5143.876704] [<ffffffff815c34b9>] ? inet_sendmsg+0xb9/0xf0 [ 5143.876707] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5143.876708] [<ffffffff8153a912>] ? sock_sendmsg+0xd2/0xf0 [ 5143.876711] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5143.876720] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5143.876723] [<ffffffff8153a970>] ? kernel_sendmsg+0x40/0x60 [ 5143.876724] [<ffffffffa05e23c0>] ? con_work+0xc60/0x2e80 [libceph] [ 5143.876733] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5143.876736] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5143.876737] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5143.876739] [<ffffffff81072ba9>] ? process_one_work+0x199/0x510 [ 5143.876741] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5143.876742] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5143.876747] [<ffffffff81074895>] ? worker_thread+0x165/0x3f0 [ 5143.876749] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5143.876751] [<ffffffff8107a4ba>] ? kthread+0xea/0xf0 [ 5143.876753] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5143.876755] [<ffffffff81666dac>] ? ret_from_fork+0x7c/0xb0 [ 5143.876758] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5144.168724] [<ffffffff810f5d02>] rcu_pending+0x192/0x6f0 [ 5144.174365] [<ffffffff8108f653>] ? account_system_time+0xb3/0x210 [ 5144.180611] [<ffffffff810f845b>] rcu_check_callbacks+0xab/0x160 [ 5144.186691] [<ffffffff81066638>] update_process_times+0x48/0x90 [ 5144.192715] [<ffffffff810afdb6>] tick_sched_handle.isra.10+0x36/0x50 [ 5144.199187] [<ffffffff810afebc>] tick_sched_timer+0x4c/0x80 [ 5144.204860] [<ffffffff8107e9c6>] __run_hrtimer+0x86/0x1f0 [ 5144.210373] [<ffffffff810afe70>] ? tick_nohz_handler+0xa0/0xa0 [ 5144.216300] [<ffffffff810a970c>] ? ktime_get_update_offsets+0x4c/0xd0 [ 5144.222835] [<ffffffff8107f2b7>] hrtimer_interrupt+0xf7/0x250 [ 5144.228704] [<ffffffff81041b46>] hpet_interrupt_handler+0x16/0x40 [ 5144.234896] [<ffffffff810ee7ae>] handle_irq_event_percpu+0x5e/0x230 [ 5144.241284] [<ffffffff8134141d>] ? do_raw_spin_unlock+0x5d/0xb0 [ 5144.247306] [<ffffffff810ee9c8>] handle_irq_event+0x48/0x70 [ 5144.252981] [<ffffffff810f12d7>] handle_edge_irq+0x77/0x110 [ 5144.258649] [<ffffffff81016212>] handle_irq+0x22/0x40 [ 5144.263804] [<ffffffff81668aaa>] do_IRQ+0x5a/0xe0 [ 5144.269806] [<ffffffff8165e732>] common_interrupt+0x72/0x72 [ 5144.275478] <EOI> [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5144.282303] [<ffffffff8165de34>] ? _raw_spin_unlock_irq+0x34/0x40 [ 5144.288528] [<ffffffff8165de30>] ? _raw_spin_unlock_irq+0x30/0x40 [ 5144.294725] [<ffffffff81143a3b>] shrink_inactive_list+0x18b/0x400 [ 5144.300927] [<ffffffff810b783d>] ? trace_hardirqs_on+0xd/0x10 [ 5144.306769] [<ffffffff811442ed>] shrink_lruvec+0x2fd/0x470 [ 5144.312358] [<ffffffff811accd6>] ? bdi_queue_work+0x86/0x150 [ 5144.318129] [<ffffffff8107ae10>] ? __init_waitqueue_head+0x60/0x60 [ 5144.324403] [<ffffffff81144806>] do_try_to_free_pages+0x126/0x3a0 [ 5144.330624] [<ffffffff81144c3f>] try_to_free_pages+0xcf/0x1a0 [ 5144.336464] [<ffffffff811393d9>] __alloc_pages_nodemask+0x579/0x8a0 [ 5144.342828] [<ffffffff8117110a>] alloc_pages_current+0xba/0x170 [ 5144.348857] [<ffffffff8154074e>] sk_page_frag_refill+0x7e/0x130 [ 5144.354879] [<ffffffff815980e9>] tcp_sendmsg+0x309/0xe30 [ 5144.360313] [<ffffffff815c34b9>] inet_sendmsg+0xb9/0xf0 [ 5144.365634] [<ffffffff815c3405>] ? inet_sendmsg+0x5/0xf0 [ 5144.371084] [<ffffffff8153a912>] sock_sendmsg+0xd2/0xf0 [ 5144.376417] [<ffffffff813060f8>] ? crypto_shash_update+0x18/0x30 [ 5144.382528] [<ffffffffa0000056>] ? crc32c+0x56/0x7c [libcrc32c] [ 5144.388549] [<ffffffff8153a970>] kernel_sendmsg+0x40/0x60 [ 5144.394054] [<ffffffffa05e23c0>] con_work+0xc60/0x2e80 [libceph] [ 5144.400169] [<ffffffff8108f000>] ? cpuacct_charge+0xb0/0xc0 [ 5144.405852] [<ffffffff8108ef78>] ? cpuacct_charge+0x28/0xc0 [ 5144.411522] [<ffffffff810861c8>] ? finish_task_switch+0x48/0x110 [ 5144.417632] [<ffffffff81072ba9>] process_one_work+0x199/0x510 [ 5144.423480] [<ffffffff81072b3c>] ? process_one_work+0x12c/0x510 [ 5144.429515] [<ffffffffa05e1760>] ? ceph_con_close+0xd0/0xd0 [libceph] [ 5144.436054] [<ffffffff81074895>] worker_thread+0x165/0x3f0 [ 5144.441643] [<ffffffff81074730>] ? manage_workers+0x2a0/0x2a0 [ 5144.447492] [<ffffffff8107a4ba>] kthread+0xea/0xf0 [ 5144.452379] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5144.458638] [<ffffffff81666dac>] ret_from_fork+0x7c/0xb0 [ 5144.464046] [<ffffffff8107a3d0>] ? flush_kthread_work+0x1a0/0x1a0 [ 5146.035958] libceph: mds0 10.214.132.118:6809 socket closed (con state OPEN) [ 5146.077716] ceph: mds0 hung [ 5146.080535] ceph: mds0 caps stale [ 5146.083961] libceph: mon1 10.214.132.118:6789 socket closed (con state OPEN) [ 5146.091143] libceph: mon1 10.214.132.118:6789 session lost, hunting for new mon [ 5146.120629] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) [ 5146.184423] libceph: osd0 10.214.132.116:6800 socket closed (con state OPEN) [ 5146.191666] libceph: mds0 10.214.132.118:6809 connection reset [ 5146.197690] libceph: reset on mds0 [ 5146.201208] ceph: mds0 closed our session [ 5146.205344] ceph: mds0 reconnect start [ 5146.273394] libceph: mon1 10.214.132.118:6789 session established [ 5146.279819] ceph: mds0 reconnect denied [ 5146.284105] ceph: dropping dirty+flushing Fw state for ffff880421c9b430 1099511787263 [ 5146.292254] ceph: dropping dirty data for ffff880421c9b430 1099511787263 [ 5146.299249] ceph: dropping dirty Fw state for ffff880420e8f3f0 1099511787213 [ 5146.306606] ceph: dropping dirty data for ffff880420e8f3f0 1099511787213 [ 5146.313569] ceph: dropping dirty Fw state for ffff880420e8ebf8 1099511787217 [ 5146.320818] ceph: dropping dirty data for ffff880420e8ebf8 1099511787217 [ 5146.327727] ceph: dropping dirty+flushing Fw state for ffff880423455c08 1099511787205 [ 5146.335736] ceph: dropping dirty data for ffff880423455c08 1099511787205 [ 5146.342616] ceph: dropping dirty Fw state for ffff88042504fbe8 1099511787245 [ 5146.349833] ceph: dropping dirty data for ffff88042504fbe8 1099511787245 [ 5146.356680] ceph: dropping dirty Fw state for ffff880425049c48 1099511787254 [ 5146.363893] ceph: dropping dirty data for ffff880425049c48 1099511787254 [ 5146.405995] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 5146.464283] libceph: osd5 10.214.132.118:6806 socket closed (con state OPEN) [ 5146.508710] libceph: osd2 10.214.132.116:6806 socket closed (con state OPEN) [ 5147.386139] libceph: osd4 10.214.132.118:6803 socket closed (con state OPEN) [ 5147.443783] libceph: osd1 10.214.132.116:6803 socket closed (con state OPEN) Entering kdb (current=0xffff880412c83f20, pid 26674) on processor 1 Oops: (null) due to oops @ 0xffffffffa06308a0 CPU 1 dPid: 26674, comm: ffsb Not tainted 3.8.0-ceph-00226-g1655ae7 #1 Supermicro X8SIL/X8SIL dRIP: 0010:[<ffffffffa06308a0>] [<ffffffffa06308a0>] ceph_update_writeable_page+0x460/0x4e0 [ceph] dRSP: 0018:ffff880413c9fa08 EFLAGS: 00010246 dRAX: 0200000000000001 RBX: ffffea000a892280 RCX: ffffea000a892280 dRDX: 0000000000001000 RSI: 000000000200a000 RDI: ffff880426410b40 dRBP: ffff880413c9fa98 R08: 0000000000000002 R09: 0000000000000001 dR10: 0000000000000000 R11: 0000000000000000 R12: ffff880423456bf8 dR13: ffff880413c9fb98 R14: ffff88042c04c150 R15: 0000000000001000 dFS: 00007f39df7ee700(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000 dCS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b dCR2: 00007f5b7631d6a0 CR3: 000000042c678000 CR4: 00000000000007e0 dDR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 dDR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 dProcess ffsb (pid: 26674, threadinfo ffff880413c9e000, task ffff880412c83f20) dStack: ffffea000a892280 0000100023456dd8 000000000200a000 0000000000001000 0000000002009000 ffff880426410b40 ffffea000a892280 ffff880423456dd8 ffff880413c9fa98 ffffffff81130d65 ffff880413c9fb84 ffff88042c04de70 dCall Trace: more>
At this point the machine was dead and power cycle was required
to get it back again.
#5 Updated by Alex Elder about 11 years ago
I have run this same test (only "suites/ffsb.sh") with the
"testing" branch twice. I got the same lockdep warning,
so the issue is unrelated to my changes.
Neither of these runs resulted in the crash that I saw
originally.
In addition I have run with my "review/wip-4392-2" branch
twice more.
Neither of these two runs led to the crash I originally
saw either.
The crash indicated ceph_update_writeable_page() was
involved at the time of the crash.
Looking at a copy of the ceph kernel object and disassembling
it, it looks to me like this is the line of code where the
problem occurred in ceph_update_writeable_page():
BUG_ON(!ci->i_snap_realm);
I assert therefore that this crash is not related to the changes
I made.
#6 Updated by Alex Elder about 11 years ago
OK, after this I'm done with this problem.
I just wanted to point out that I think there are two
distinct problems reported here, and maybe one should
go off into a separate issue:
- lockdep warnings
- crash, likely due to an assertion failure (null snap realm pointer)
#7 Updated by Sage Weil almost 11 years ago
- Project changed from Ceph to Linux kernel client
#8 Updated by Sage Weil over 9 years ago
- Status changed from New to Can't reproduce