Bug #2590: possible irq lock inversion dependency with con->mutex and osdc->request_mutex - Linux kernel client - Ceph

Actions

Copy link

Bug #2590

open

possible irq lock inversion dependency with con->mutex and osdc->request_mutex

Added by Sage Weil almost 12 years ago. Updated over 4 years ago.

Status:

New

Priority:

Normal

Assignee:

Category:

libceph

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

i thought this was #147, but on closer inspection it's something else;

2012-06-13T13:58:21.449592-07:00 plana65 kernel: [ 3476.880534] =========================================================
2012-06-13T13:58:21.449608-07:00 plana65 kernel: [ 3476.908822] [ INFO: possible irq lock inversion dependency detected ]
2012-06-13T13:58:21.449613-07:00 plana65 kernel: [ 3476.938287] 3.5.0-rc1-ceph-00031-gc9b7406 #1 Not tainted
2012-06-13T13:58:21.449618-07:00 plana65 kernel: [ 3476.967012] ---------------------------------------------------------
2012-06-13T13:58:21.449622-07:00 plana65 kernel: [ 3476.997784] kswapd0/41 just changed the state of lock:
2012-06-13T13:58:21.449630-07:00 plana65 kernel: [ 3477.027662]  (&osdc->request_mutex){+.+.-.}, at: [<ffffffffa028068b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.449634-07:00 plana65 kernel: [ 3477.090018] but this lock took another, RECLAIM_FS-unsafe lock in the past:
2012-06-13T13:58:21.449641-07:00 plana65 kernel: [ 3477.123868]  (&con->mutex){+.+.+.}
2012-06-13T13:58:21.449645-07:00 plana65 kernel: [ 3477.123868] 
2012-06-13T13:58:21.449647-07:00 plana65 kernel: [ 3477.123868] and interrupts could create inverse lock ordering between them.
2012-06-13T13:58:21.449648-07:00 plana65 kernel: [ 3477.123868] 
2012-06-13T13:58:21.449653-07:00 plana65 kernel: [ 3477.238792] 
2012-06-13T13:58:21.449655-07:00 plana65 kernel: [ 3477.238792] other info that might help us debug this:
2012-06-13T13:58:21.449660-07:00 plana65 kernel: [ 3477.295691]  Possible interrupt unsafe locking scenario:
2012-06-13T13:58:21.449661-07:00 plana65 kernel: [ 3477.295691] 
2012-06-13T13:58:21.449665-07:00 plana65 kernel: [ 3477.352788]        CPU0                    CPU1
2012-06-13T13:58:21.449672-07:00 plana65 kernel: [ 3477.382400]        ----                    ----
2012-06-13T13:58:21.449676-07:00 plana65 kernel: [ 3477.411629]   lock(&con->mutex);
2012-06-13T13:58:21.449681-07:00 plana65 kernel: [ 3477.439192]                                local_irq_disable();
2012-06-13T13:58:21.449685-07:00 plana65 kernel: [ 3477.470229]                                lock(&osdc->request_mutex);
2012-06-13T13:58:21.449689-07:00 plana65 kernel: [ 3477.502045]                                lock(&con->mutex);
2012-06-13T13:58:21.449693-07:00 plana65 kernel: [ 3477.533080]   <Interrupt>
2012-06-13T13:58:21.449697-07:00 plana65 kernel: [ 3477.560193]     lock(&osdc->request_mutex);
2012-06-13T13:58:21.449704-07:00 plana65 kernel: [ 3477.589111] 
2012-06-13T13:58:21.449706-07:00 plana65 kernel: [ 3477.589111]  *** DEADLOCK ***
2012-06-13T13:58:21.449707-07:00 plana65 kernel: [ 3477.589111] 
2012-06-13T13:58:21.449712-07:00 plana65 kernel: [ 3477.667924] no locks held by kswapd0/41.
2012-06-13T13:58:21.449716-07:00 plana65 kernel: [ 3477.696274] 
2012-06-13T13:58:21.449718-07:00 plana65 kernel: [ 3477.696274] the shortest dependencies between 2nd lock and 1st lock:
2012-06-13T13:58:21.449722-07:00 plana65 kernel: [ 3477.752495]  -> (&con->mutex){+.+.+.} ops: 385786 {
2012-06-13T13:58:21.449726-07:00 plana65 kernel: [ 3477.781565]     HARDIRQ-ON-W at:
2012-06-13T13:58:21.449732-07:00 plana65 kernel: [ 3477.808189]   [<ffffffff810af31b>] __lock_acquire+0x5ab/0x15d0
2012-06-13T13:58:21.449737-07:00 plana65 kernel: [ 3477.837887]   [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.449741-07:00 plana65 kernel: [ 3477.866661]   [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.449746-07:00 plana65 kernel: [ 3477.895674]   [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.449750-07:00 plana65 kernel: [ 3477.923627]   [<ffffffffa0278f77>] ceph_con_send+0xa7/0x200 [libceph]
2012-06-13T13:58:21.449755-07:00 plana65 kernel: [ 3477.951761]   [<ffffffffa027c33c>] __send_prepared_auth_request+0x6c/0x90 [libceph]
2012-06-13T13:58:21.449760-07:00 plana65 kernel: [ 3478.002188]   [<ffffffffa027c533>] __open_session+0x123/0x190 [libceph]
2012-06-13T13:58:21.449767-07:00 plana65 kernel: [ 3478.031571]   [<ffffffffa027c6af>] ceph_monc_open_session+0x2f/0x50 [libceph]
2012-06-13T13:58:21.449772-07:00 plana65 kernel: [ 3478.061474]   [<ffffffffa0275259>] __ceph_open_session+0x39/0x270 [libceph]
2012-06-13T13:58:21.449777-07:00 plana65 kernel: [ 3478.091973]   [<ffffffffa04902e3>] ceph_mount+0x423/0x800 [ceph]
2012-06-13T13:58:21.449781-07:00 plana65 kernel: [ 3478.121535]   [<ffffffff81184ea3>] mount_fs+0x43/0x1b0
2012-06-13T13:58:21.449786-07:00 plana65 kernel: [ 3478.149905]   [<ffffffff811a1bb6>] vfs_kern_mount+0x76/0x120
2012-06-13T13:58:21.449790-07:00 plana65 kernel: [ 3478.178814]   [<ffffffff811a1ce2>] do_kern_mount+0x52/0x110
2012-06-13T13:58:21.449795-07:00 plana65 kernel: [ 3478.207962]   [<ffffffff811a33c7>] do_mount+0x257/0x820
2012-06-13T13:58:21.449800-07:00 plana65 kernel: [ 3478.237014]   [<ffffffff811a3a20>] sys_mount+0x90/0xe0
2012-06-13T13:58:21.449807-07:00 plana65 kernel: [ 3478.265883]   [<ffffffff816360e9>] system_call_fastpath+0x16/0x1b
2012-06-13T13:58:21.449811-07:00 plana65 kernel: [ 3478.295818]     SOFTIRQ-ON-W at:
2012-06-13T13:58:21.449816-07:00 plana65 kernel: [ 3478.322743]   [<ffffffff810af34d>] __lock_acquire+0x5dd/0x15d0
2012-06-13T13:58:21.449820-07:00 plana65 kernel: [ 3478.353341]   [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.449825-07:00 plana65 kernel: [ 3478.383554]   [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.449829-07:00 plana65 kernel: [ 3478.414625]   [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.449834-07:00 plana65 kernel: [ 3478.445554]   [<ffffffffa0278f77>] ceph_con_send+0xa7/0x200 [libceph]
2012-06-13T13:58:21.449841-07:00 plana65 kernel: [ 3478.477800]   [<ffffffffa027c33c>] __send_prepared_auth_request+0x6c/0x90 [libceph]
2012-06-13T13:58:21.449845-07:00 plana65 kernel: [ 3478.538161]   [<ffffffffa027c533>] __open_session+0x123/0x190 [libceph]
2012-06-13T13:58:21.449850-07:00 plana65 kernel: [ 3478.572943]   [<ffffffffa027c6af>] ceph_monc_open_session+0x2f/0x50 [libceph]
2012-06-13T13:58:21.449855-07:00 plana65 kernel: [ 3478.608380]   [<ffffffffa0275259>] __ceph_open_session+0x39/0x270 [libceph]
2012-06-13T13:58:21.449860-07:00 plana65 kernel: [ 3478.643528]   [<ffffffffa04902e3>] ceph_mount+0x423/0x800 [ceph]
2012-06-13T13:58:21.449864-07:00 plana65 kernel: [ 3478.677702]   [<ffffffff81184ea3>] mount_fs+0x43/0x1b0
2012-06-13T13:58:21.449868-07:00 plana65 kernel: [ 3478.710569]   [<ffffffff811a1bb6>] vfs_kern_mount+0x76/0x120
2012-06-13T13:58:21.449873-07:00 plana65 kernel: [ 3478.744090]   [<ffffffff811a1ce2>] do_kern_mount+0x52/0x110
2012-06-13T13:58:21.449879-07:00 plana65 kernel: [ 3478.777404]   [<ffffffff811a33c7>] do_mount+0x257/0x820
2012-06-13T13:58:21.449884-07:00 plana65 kernel: [ 3478.810031]   [<ffffffff811a3a20>] sys_mount+0x90/0xe0
2012-06-13T13:58:21.449888-07:00 plana65 kernel: [ 3478.842207]   [<ffffffff816360e9>] system_call_fastpath+0x16/0x1b
2012-06-13T13:58:21.449892-07:00 plana65 kernel: [ 3478.875331]     RECLAIM_FS-ON-W at:
2012-06-13T13:58:21.449897-07:00 plana65 kernel: [ 3478.904919]   [<ffffffff810ae68d>] mark_held_locks+0x7d/0x120
2012-06-13T13:58:21.449901-07:00 plana65 kernel: [ 3478.936951]   [<ffffffff810ae7cf>] lockdep_trace_alloc+0x9f/0xd0
2012-06-13T13:58:21.449906-07:00 plana65 kernel: [ 3478.969013]   [<ffffffff81172dca>] kmem_cache_alloc+0x3a/0x140
2012-06-13T13:58:21.449912-07:00 plana65 kernel: [ 3479.000690]   [<ffffffff81510e9d>] sock_alloc_inode+0x1d/0xd0
2012-06-13T13:58:21.449917-07:00 plana65 kernel: [ 3479.031992]   [<ffffffff8119dcd7>] alloc_inode+0x27/0xa0
2012-06-13T13:58:21.449922-07:00 plana65 kernel: [ 3479.062630]   [<ffffffff8119dd6b>] new_inode_pseudo+0x1b/0x70
2012-06-13T13:58:21.449926-07:00 plana65 kernel: [ 3479.093758]   [<ffffffff81511b9e>] sock_alloc+0x1e/0x70
2012-06-13T13:58:21.449931-07:00 plana65 kernel: [ 3479.124348]   [<ffffffff81511ff6>] __sock_create+0x96/0x280
2012-06-13T13:58:21.449935-07:00 plana65 kernel: [ 3479.155398]   [<ffffffff81512204>] sock_create_kern+0x24/0x30
2012-06-13T13:58:21.449940-07:00 plana65 kernel: [ 3479.186838]   [<ffffffffa0278900>] try_write+0xb00/0x10d0 [libceph]
2012-06-13T13:58:21.449945-07:00 plana65 kernel: [ 3479.219008]   [<ffffffffa027a3d0>] con_work+0xc60/0x1c60 [libceph]
2012-06-13T13:58:21.449951-07:00 plana65 kernel: [ 3479.250761]   [<ffffffff8106f276>] process_one_work+0x1b6/0x530
2012-06-13T13:58:21.449956-07:00 plana65 kernel: [ 3479.282098]   [<ffffffff810715b3>] worker_thread+0x173/0x400
2012-06-13T13:58:21.449960-07:00 plana65 kernel: [ 3479.313045]   [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.449965-07:00 plana65 kernel: [ 3479.343009]   [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.449969-07:00 plana65 kernel: [ 3479.374165]     INITIAL USE at:
2012-06-13T13:58:21.449974-07:00 plana65 kernel: [ 3479.402023]   [<ffffffff810aef76>] __lock_acquire+0x206/0x15d0
2012-06-13T13:58:21.449978-07:00 plana65 kernel: [ 3479.433238]   [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.449983-07:00 plana65 kernel: [ 3479.463279]   [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.449989-07:00 plana65 kernel: [ 3479.493484]   [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.449994-07:00 plana65 kernel: [ 3479.523165]   [<ffffffffa0278f77>] ceph_con_send+0xa7/0x200 [libceph]
2012-06-13T13:58:21.449999-07:00 plana65 kernel: [ 3479.553452]   [<ffffffffa027c33c>] __send_prepared_auth_request+0x6c/0x90 [libceph]
2012-06-13T13:58:21.450004-07:00 plana65 kernel: [ 3479.608360]   [<ffffffffa027c533>] __open_session+0x123/0x190 [libceph]
2012-06-13T13:58:21.450008-07:00 plana65 kernel: [ 3479.640165]   [<ffffffffa027c6af>] ceph_monc_open_session+0x2f/0x50 [libceph]
2012-06-13T13:58:21.450013-07:00 plana65 kernel: [ 3479.672912]   [<ffffffffa0275259>] __ceph_open_session+0x39/0x270 [libceph]
2012-06-13T13:58:21.450018-07:00 plana65 kernel: [ 3479.705871]   [<ffffffffa04902e3>] ceph_mount+0x423/0x800 [ceph]
2012-06-13T13:58:21.450024-07:00 plana65 kernel: [ 3479.738104]   [<ffffffff81184ea3>] mount_fs+0x43/0x1b0
2012-06-13T13:58:21.450029-07:00 plana65 kernel: [ 3479.769394]   [<ffffffff811a1bb6>] vfs_kern_mount+0x76/0x120
2012-06-13T13:58:21.450033-07:00 plana65 kernel: [ 3479.800819]   [<ffffffff811a1ce2>] do_kern_mount+0x52/0x110
2012-06-13T13:58:21.450038-07:00 plana65 kernel: [ 3479.831754]   [<ffffffff811a33c7>] do_mount+0x257/0x820
2012-06-13T13:58:21.450042-07:00 plana65 kernel: [ 3479.862151]   [<ffffffff811a3a20>] sys_mount+0x90/0xe0
2012-06-13T13:58:21.450047-07:00 plana65 kernel: [ 3479.892154]   [<ffffffff816360e9>] system_call_fastpath+0x16/0x1b
2012-06-13T13:58:21.450050-07:00 plana65 kernel: [ 3479.923375]   }
2012-06-13T13:58:21.450055-07:00 plana65 kernel: [ 3479.949754]   ... key      at: [<ffffffffa0299f18>] __key.64564+0x0/0xffffffffffff214f [libceph]
2012-06-13T13:58:21.450063-07:00 plana65 kernel: [ 3480.010658]   ... acquired at:
2012-06-13T13:58:21.450068-07:00 plana65 kernel: [ 3480.039794]    [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450072-07:00 plana65 kernel: [ 3480.072304]    [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450077-07:00 plana65 kernel: [ 3480.105496]    [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450081-07:00 plana65 kernel: [ 3480.138432]    [<ffffffffa0278f77>] ceph_con_send+0xa7/0x200 [libceph]
2012-06-13T13:58:21.450086-07:00 plana65 kernel: [ 3480.172090]    [<ffffffffa027f8d5>] __send_request+0xc5/0x120 [libceph]
2012-06-13T13:58:21.450091-07:00 plana65 kernel: [ 3480.206028]    [<ffffffffa0280713>] ceph_osdc_start_request+0x113/0x190 [libceph]
2012-06-13T13:58:21.450098-07:00 plana65 kernel: [ 3480.268348]    [<ffffffffa0497ed9>] ceph_aio_write+0x629/0xab0 [ceph]
2012-06-13T13:58:21.450103-07:00 plana65 kernel: [ 3480.303339]    [<ffffffff811820e2>] do_sync_write+0xe2/0x120
2012-06-13T13:58:21.450108-07:00 plana65 kernel: [ 3480.337223]    [<ffffffff81182678>] vfs_write+0xc8/0x190
2012-06-13T13:58:21.450112-07:00 plana65 kernel: [ 3480.370588]    [<ffffffff81182831>] sys_write+0x51/0x90
2012-06-13T13:58:21.450117-07:00 plana65 kernel: [ 3480.403388]    [<ffffffff816360e9>] system_call_fastpath+0x16/0x1b
2012-06-13T13:58:21.450121-07:00 plana65 kernel: [ 3480.437328] 
2012-06-13T13:58:21.450125-07:00 plana65 kernel: [ 3480.465734] -> (&osdc->request_mutex){+.+.-.} ops: 568458 {
2012-06-13T13:58:21.450129-07:00 plana65 kernel: [ 3480.499110]    HARDIRQ-ON-W at:
2012-06-13T13:58:21.450137-07:00 plana65 kernel: [ 3480.528984]  [<ffffffff810af31b>] __lock_acquire+0x5ab/0x15d0
2012-06-13T13:58:21.450142-07:00 plana65 kernel: [ 3480.561725]  [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450146-07:00 plana65 kernel: [ 3480.593813]  [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450151-07:00 plana65 kernel: [ 3480.626469]  [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450156-07:00 plana65 kernel: [ 3480.658640]  [<ffffffffa02807ce>] kick_requests+0x3e/0x2b0 [libceph]
2012-06-13T13:58:21.450161-07:00 plana65 kernel: [ 3480.691665]  [<ffffffffa0280f48>] ceph_osdc_handle_map+0x508/0x5f0 [libceph]
2012-06-13T13:58:21.450166-07:00 plana65 kernel: [ 3480.725805]  [<ffffffffa027d210>] dispatch+0x120/0x720 [libceph]
2012-06-13T13:58:21.450173-07:00 plana65 kernel: [ 3480.759022]  [<ffffffffa027ae48>] con_work+0x16d8/0x1c60 [libceph]
2012-06-13T13:58:21.450177-07:00 plana65 kernel: [ 3480.792695]  [<ffffffff8106f276>] process_one_work+0x1b6/0x530
2012-06-13T13:58:21.450182-07:00 plana65 kernel: [ 3480.826274]  [<ffffffff810715b3>] worker_thread+0x173/0x400
2012-06-13T13:58:21.450197-07:00 plana65 kernel: [ 3480.859553]  [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.450202-07:00 plana65 kernel: [ 3480.891544]  [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.450206-07:00 plana65 kernel: [ 3480.924382]    SOFTIRQ-ON-W at:
2012-06-13T13:58:21.450211-07:00 plana65 kernel: [ 3480.953526]  [<ffffffff810af34d>] __lock_acquire+0x5dd/0x15d0
2012-06-13T13:58:21.450216-07:00 plana65 kernel: [ 3480.986255]  [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450224-07:00 plana65 kernel: [ 3481.018306]  [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450229-07:00 plana65 kernel: [ 3481.051154]  [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450250-07:00 plana65 kernel: [ 3481.083672]  [<ffffffffa02807ce>] kick_requests+0x3e/0x2b0 [libceph]
2012-06-13T13:58:21.450256-07:00 plana65 kernel: [ 3481.116986]  [<ffffffffa0280f48>] ceph_osdc_handle_map+0x508/0x5f0 [libceph]
2012-06-13T13:58:21.450261-07:00 plana65 kernel: [ 3481.151262]  [<ffffffffa027d210>] dispatch+0x120/0x720 [libceph]
2012-06-13T13:58:21.450266-07:00 plana65 kernel: [ 3481.184306]  [<ffffffffa027ae48>] con_work+0x16d8/0x1c60 [libceph]
2012-06-13T13:58:21.450271-07:00 plana65 kernel: [ 3481.216846]  [<ffffffff8106f276>] process_one_work+0x1b6/0x530
2012-06-13T13:58:21.450278-07:00 plana65 kernel: [ 3481.248561]  [<ffffffff810715b3>] worker_thread+0x173/0x400
2012-06-13T13:58:21.450283-07:00 plana65 kernel: [ 3481.279842]  [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.450290-07:00 plana65 kernel: [ 3481.310238]  [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.450295-07:00 plana65 kernel: [ 3481.342365]    IN-RECLAIM_FS-W at:
2012-06-13T13:58:21.450300-07:00 plana65 kernel: [ 3481.371770]  [<ffffffff810af1a1>] __lock_acquire+0x431/0x15d0
2012-06-13T13:58:21.450304-07:00 plana65 kernel: [ 3481.404035]  [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450309-07:00 plana65 kernel: [ 3481.435683]  [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450314-07:00 plana65 kernel: [ 3481.467955]  [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450322-07:00 plana65 kernel: [ 3481.499647]  [<ffffffffa028068b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450328-07:00 plana65 kernel: [ 3481.559119]  [<ffffffffa0282844>] ceph_osdc_writepages+0xe4/0x190 [libceph]
2012-06-13T13:58:21.450333-07:00 plana65 kernel: [ 3481.593888]  [<ffffffffa049a902>] writepage_nounlock+0x232/0x420 [ceph]
2012-06-13T13:58:21.450337-07:00 plana65 kernel: [ 3481.628328]  [<ffffffffa049ab2e>] ceph_writepage+0x3e/0x70 [ceph]
2012-06-13T13:58:21.450342-07:00 plana65 kernel: [ 3481.662072]  [<ffffffff81139d7f>] shrink_page_list+0x59f/0x900
2012-06-13T13:58:21.450347-07:00 plana65 kernel: [ 3481.695344]  [<ffffffff8113a566>] shrink_inactive_list+0x176/0x440
2012-06-13T13:58:21.450352-07:00 plana65 kernel: [ 3481.728964]  [<ffffffff8113af45>] shrink_lruvec+0x3d5/0x490
2012-06-13T13:58:21.450358-07:00 plana65 kernel: [ 3481.761612]  [<ffffffff8113b066>] shrink_zone+0x66/0xc0
2012-06-13T13:58:21.450363-07:00 plana65 kernel: [ 3481.793646]  [<ffffffff8113c497>] balance_pgdat+0x6c7/0x8b0
2012-06-13T13:58:21.450368-07:00 plana65 kernel: [ 3481.826064]  [<ffffffff8113c86d>] kswapd+0x1ed/0x420
2012-06-13T13:58:21.450376-07:00 plana65 kernel: [ 3481.857665]  [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.450381-07:00 plana65 kernel: [ 3481.888562]  [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.450385-07:00 plana65 kernel: [ 3481.920439]    INITIAL USE at:
2012-06-13T13:58:21.450389-07:00 plana65 kernel: [ 3481.949193]  [<ffffffff810aef76>] __lock_acquire+0x206/0x15d0
2012-06-13T13:58:21.450394-07:00 plana65 kernel: [ 3481.981399]  [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450402-07:00 plana65 kernel: [ 3482.013106]  [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450407-07:00 plana65 kernel: [ 3482.045493]  [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450412-07:00 plana65 kernel: [ 3482.077357]  [<ffffffffa02807ce>] kick_requests+0x3e/0x2b0 [libceph]
2012-06-13T13:58:21.450417-07:00 plana65 kernel: [ 3482.109866]  [<ffffffffa0280f48>] ceph_osdc_handle_map+0x508/0x5f0 [libceph]
2012-06-13T13:58:21.450422-07:00 plana65 kernel: [ 3482.143470]  [<ffffffffa027d210>] dispatch+0x120/0x720 [libceph]
2012-06-13T13:58:21.450426-07:00 plana65 kernel: [ 3482.175919]  [<ffffffffa027ae48>] con_work+0x16d8/0x1c60 [libceph]
2012-06-13T13:58:21.450431-07:00 plana65 kernel: [ 3482.208615]  [<ffffffff8106f276>] process_one_work+0x1b6/0x530
2012-06-13T13:58:21.450437-07:00 plana65 kernel: [ 3482.241118]  [<ffffffff810715b3>] worker_thread+0x173/0x400
2012-06-13T13:58:21.450442-07:00 plana65 kernel: [ 3482.273284]  [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.450449-07:00 plana65 kernel: [ 3482.304104]  [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.450453-07:00 plana65 kernel: [ 3482.335903]  }
2012-06-13T13:58:21.450459-07:00 plana65 kernel: [ 3482.362812]  ... key      at: [<ffffffffa029aba0>] __key.38687+0x0/0xffffffffffff14c7 [libceph]
2012-06-13T13:58:21.450463-07:00 plana65 kernel: [ 3482.424535]  ... acquired at:
2012-06-13T13:58:21.450467-07:00 plana65 kernel: [ 3482.453586]    [<ffffffff810ad550>] check_usage_forwards+0x90/0x100
2012-06-13T13:58:21.450472-07:00 plana65 kernel: [ 3482.486307]    [<ffffffff810ae3b4>] mark_lock+0x1b4/0x410
2012-06-13T13:58:21.450481-07:00 plana65 kernel: [ 3482.517338]    [<ffffffff810af1a1>] __lock_acquire+0x431/0x15d0
2012-06-13T13:58:21.450486-07:00 plana65 kernel: [ 3482.548695]    [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450491-07:00 plana65 kernel: [ 3482.579476]    [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450495-07:00 plana65 kernel: [ 3482.610961]    [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450500-07:00 plana65 kernel: [ 3482.642266]    [<ffffffffa028068b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450505-07:00 plana65 kernel: [ 3482.701023]    [<ffffffffa0282844>] ceph_osdc_writepages+0xe4/0x190 [libceph]
2012-06-13T13:58:21.450510-07:00 plana65 kernel: [ 3482.735866]    [<ffffffffa049a902>] writepage_nounlock+0x232/0x420 [ceph]
2012-06-13T13:58:21.450517-07:00 plana65 kernel: [ 3482.770704]    [<ffffffffa049ab2e>] ceph_writepage+0x3e/0x70 [ceph]
2012-06-13T13:58:21.450522-07:00 plana65 kernel: [ 3482.805237]    [<ffffffff81139d7f>] shrink_page_list+0x59f/0x900
2012-06-13T13:58:21.450529-07:00 plana65 kernel: [ 3482.839584]    [<ffffffff8113a566>] shrink_inactive_list+0x176/0x440
2012-06-13T13:58:21.450534-07:00 plana65 kernel: [ 3482.873932]    [<ffffffff8113af45>] shrink_lruvec+0x3d5/0x490
2012-06-13T13:58:21.450539-07:00 plana65 kernel: [ 3482.907175]    [<ffffffff8113b066>] shrink_zone+0x66/0xc0
2012-06-13T13:58:21.450543-07:00 plana65 kernel: [ 3482.939840]    [<ffffffff8113c497>] balance_pgdat+0x6c7/0x8b0
2012-06-13T13:58:21.450548-07:00 plana65 kernel: [ 3482.972718]    [<ffffffff8113c86d>] kswapd+0x1ed/0x420
2012-06-13T13:58:21.450553-07:00 plana65 kernel: [ 3483.004721]    [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.450561-07:00 plana65 kernel: [ 3483.036233]    [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.450565-07:00 plana65 kernel: [ 3483.068782] 
2012-06-13T13:58:21.450569-07:00 plana65 kernel: [ 3483.095783] 
2012-06-13T13:58:21.450571-07:00 plana65 kernel: [ 3483.095783] stack backtrace:
2012-06-13T13:58:21.450575-07:00 plana65 kernel: [ 3483.149563] Pid: 41, comm: kswapd0 Not tainted 3.5.0-rc1-ceph-00031-gc9b7406 #1
2012-06-13T13:58:21.450579-07:00 plana65 kernel: [ 3483.208384] Call Trace:
2012-06-13T13:58:21.450584-07:00 plana65 kernel: [ 3483.236416]  [<ffffffff810ad473>] print_irq_inversion_bug+0x1d3/0x220
2012-06-13T13:58:21.450591-07:00 plana65 kernel: [ 3483.269540]  [<ffffffff810ad4c0>] ? print_irq_inversion_bug+0x220/0x220
2012-06-13T13:58:21.450596-07:00 plana65 kernel: [ 3483.302474]  [<ffffffff810ad550>] check_usage_forwards+0x90/0x100
2012-06-13T13:58:21.450603-07:00 plana65 kernel: [ 3483.334197]  [<ffffffff810ae3b4>] mark_lock+0x1b4/0x410
2012-06-13T13:58:21.450608-07:00 plana65 kernel: [ 3483.365056]  [<ffffffff810af1a1>] __lock_acquire+0x431/0x15d0
2012-06-13T13:58:21.450612-07:00 plana65 kernel: [ 3483.396131]  [<ffffffffa027b47a>] ? msgpool_alloc+0x2a/0xa0 [libceph]
2012-06-13T13:58:21.450617-07:00 plana65 kernel: [ 3483.428147]  [<ffffffff81328986>] ? vsnprintf+0x416/0x5a0
2012-06-13T13:58:21.450622-07:00 plana65 kernel: [ 3483.459048]  [<ffffffffa028068b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450627-07:00 plana65 kernel: [ 3483.517918]  [<ffffffff810b03e2>] lock_acquire+0xa2/0x120
2012-06-13T13:58:21.450637-07:00 plana65 kernel: [ 3483.550068]  [<ffffffffa028068b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450642-07:00 plana65 kernel: [ 3483.610380]  [<ffffffff8162acdb>] __mutex_lock_common+0x4b/0x3d0
2012-06-13T13:58:21.450647-07:00 plana65 kernel: [ 3483.643156]  [<ffffffffa028068b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450652-07:00 plana65 kernel: [ 3483.703757]  [<ffffffffa028068b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450657-07:00 plana65 kernel: [ 3483.767007]  [<ffffffff8162b187>] mutex_lock_nested+0x37/0x50
2012-06-13T13:58:21.450663-07:00 plana65 kernel: [ 3483.802021]  [<ffffffffa028068b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-13T13:58:21.450668-07:00 plana65 kernel: [ 3483.868658]  [<ffffffffa0282844>] ceph_osdc_writepages+0xe4/0x190 [libceph]
2012-06-13T13:58:21.450673-07:00 plana65 kernel: [ 3483.907250]  [<ffffffffa049a902>] writepage_nounlock+0x232/0x420 [ceph]
2012-06-13T13:58:21.450680-07:00 plana65 kernel: [ 3483.945893]  [<ffffffff810aea8d>] ? trace_hardirqs_on+0xd/0x10
2012-06-13T13:58:21.450686-07:00 plana65 kernel: [ 3483.983797]  [<ffffffff81133048>] ? clear_page_dirty_for_io+0xc8/0x120
2012-06-13T13:58:21.450693-07:00 plana65 kernel: [ 3484.022320]  [<ffffffffa049ab2e>] ceph_writepage+0x3e/0x70 [ceph]
2012-06-13T13:58:21.450698-07:00 plana65 kernel: [ 3484.061212]  [<ffffffff81139d7f>] shrink_page_list+0x59f/0x900
2012-06-13T13:58:21.450703-07:00 plana65 kernel: [ 3484.100326]  [<ffffffff8113a566>] shrink_inactive_list+0x176/0x440
2012-06-13T13:58:21.450708-07:00 plana65 kernel: [ 3484.139249]  [<ffffffff8113ac78>] ? shrink_lruvec+0x108/0x490
2012-06-13T13:58:21.450713-07:00 plana65 kernel: [ 3484.177027]  [<ffffffff8113af45>] shrink_lruvec+0x3d5/0x490
2012-06-13T13:58:21.450721-07:00 plana65 kernel: [ 3484.214808]  [<ffffffff810c5677>] ? css_get_next+0x87/0x120
2012-06-13T13:58:21.450726-07:00 plana65 kernel: [ 3484.252029]  [<ffffffff8113b066>] shrink_zone+0x66/0xc0
2012-06-13T13:58:21.450731-07:00 plana65 kernel: [ 3484.288746]  [<ffffffff8113c497>] balance_pgdat+0x6c7/0x8b0
2012-06-13T13:58:21.450735-07:00 plana65 kernel: [ 3484.325640]  [<ffffffff810aea8d>] ? trace_hardirqs_on+0xd/0x10
2012-06-13T13:58:21.450740-07:00 plana65 kernel: [ 3484.362716]  [<ffffffff8113c86d>] kswapd+0x1ed/0x420
2012-06-13T13:58:21.450745-07:00 plana65 kernel: [ 3484.398774]  [<ffffffff81077010>] ? wake_up_bit+0x40/0x40
2012-06-13T13:58:21.450750-07:00 plana65 kernel: [ 3484.435071]  [<ffffffff8113c680>] ? balance_pgdat+0x8b0/0x8b0
2012-06-13T13:58:21.450754-07:00 plana65 kernel: [ 3484.471581]  [<ffffffff81076a9e>] kthread+0xbe/0xd0
2012-06-13T13:58:21.450761-07:00 plana65 kernel: [ 3484.506724]  [<ffffffff81637534>] kernel_thread_helper+0x4/0x10
2012-06-13T13:58:21.450769-07:00 plana65 kernel: [ 3484.542445]  [<ffffffff8162dfb0>] ? retint_restore_args+0x13/0x13
2012-06-13T13:58:21.450774-07:00 plana65 kernel: [ 3484.577975]  [<ffffffff810769e0>] ? __init_kthread_worker+0x70/0x70
2012-06-13T13:58:21.450779-07:00 plana65 kernel: [ 3484.613409]  [<ffffffff81637530>] ? gs_change+0x13/0x13

- chef: null
- ceph: null
- kclient: null
- workunit:
    clients:
      all:
      - suites/ffsb.sh

has happened several times, pretty reproducible.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Tamilarasi muthamizhan almost 12 years ago

Recent log location: /a/teuthology-2012-06-27_00:00:07-regression-next-testing-basic/3076

2012-06-27T01:25:05.114511-07:00 plana55 kernel: [17845.956769] =========================================================
2012-06-27T01:25:05.114517-07:00 plana55 kernel: [17845.974258] [ INFO: possible irq lock inversion dependency detected ]
2012-06-27T01:25:05.114519-07:00 plana55 kernel: [17845.992325] 3.4.3-ceph-00244-gfb3f3d7 #1 Not tainted
2012-06-27T01:25:05.114521-07:00 plana55 kernel: [17846.009097] ---------------------------------------------------------
2012-06-27T01:25:05.114523-07:00 plana55 kernel: [17846.027638] kswapd0/41 just changed the state of lock:
2012-06-27T01:25:05.114528-07:00 plana55 kernel: [17846.045606] (&osdc->request_mutex){+.+.-.}, at: [<ffffffffa028d34b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.114530-07:00 plana55 kernel: [17846.085324] but this lock took another, RECLAIM_FS-unsafe lock in the past:
2012-06-27T01:25:05.114532-07:00 plana55 kernel: [17846.085327] (&con->mutex){+.+.+.}
2012-06-27T01:25:05.114539-07:00 plana55 kernel: [17846.085330]
2012-06-27T01:25:05.114542-07:00 plana55 kernel: [17846.085331] and interrupts could create inverse lock ordering between them.
2012-06-27T01:25:05.114543-07:00 plana55 kernel: [17846.085333]
2012-06-27T01:25:05.114544-07:00 plana55 kernel: [17846.085334]
2012-06-27T01:25:05.114546-07:00 plana55 kernel: [17846.085334] other info that might help us debug this:
2012-06-27T01:25:05.114548-07:00 plana55 kernel: [17846.085336] Possible interrupt unsafe locking scenario:
2012-06-27T01:25:05.114549-07:00 plana55 kernel: [17846.085338]
2012-06-27T01:25:05.114550-07:00 plana55 kernel: [17846.085339] CPU0 CPU1
2012-06-27T01:25:05.114554-07:00 plana55 kernel: [17846.085340] ---- ----
2012-06-27T01:25:05.114556-07:00 plana55 kernel: [17846.085342] lock(&con->mutex);
2012-06-27T01:25:05.114558-07:00 plana55 kernel: [17846.085344] local_irq_disable();
2012-06-27T01:25:05.114559-07:00 plana55 kernel: [17846.085346] lock(&osdc->request_mutex);
2012-06-27T01:25:05.114561-07:00 plana55 kernel: [17846.085349] lock(&con->mutex);
2012-06-27T01:25:05.114562-07:00 plana55 kernel: [17846.085352] <Interrupt>
2012-06-27T01:25:05.114564-07:00 plana55 kernel: [17846.085353] lock(&osdc->request_mutex);
2012-06-27T01:25:05.114567-07:00 plana55 kernel: [17846.085356]
2012-06-27T01:25:05.114568-07:00 plana55 kernel: [17846.085356] * DEADLOCK *
2012-06-27T01:25:05.114570-07:00 plana55 kernel: [17846.085357]
2012-06-27T01:25:05.114571-07:00 plana55 kernel: [17846.085359] no locks held by kswapd0/41.
2012-06-27T01:25:05.114572-07:00 plana55 kernel: [17846.085360]
2012-06-27T01:25:05.114574-07:00 plana55 kernel: [17846.085361] the shortest dependencies between 2nd lock and 1st lock:
2012-06-27T01:25:05.114576-07:00 plana55 kernel: [17846.085376] ~~> (&con~~>mutex){+.+.+.} ops: 1498502 {
2012-06-27T01:25:05.114577-07:00 plana55 kernel: [17846.085380] HARDIRQ-ON-W at:
2012-06-27T01:25:05.114611-07:00 plana55 kernel: [17846.085383] [<ffffffff810abb2b>] __lock_acquire+0x5ab/0x15d0
2012-06-27T01:25:05.114616-07:00 plana55 kernel: [17846.085391] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.114618-07:00 plana55 kernel: [17846.085395] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.114621-07:00 plana55 kernel: [17846.085402] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.114623-07:00 plana55 kernel: [17846.085406] [<ffffffffa02862db>] ceph_con_revoke+0x3b/0x160 [libceph]
2012-06-27T01:25:05.114626-07:00 plana55 kernel: [17846.085417] [<ffffffffa0288ed4>] __send_prepared_auth_request+0x44/0x90 [libceph]
2012-06-27T01:25:05.114631-07:00 plana55 kernel: [17846.085426] [<ffffffffa02890e4>] __open_session+0x114/0x180 [libceph]
2012-06-27T01:25:05.114634-07:00 plana55 kernel: [17846.085438] [<ffffffffa028925f>] ceph_monc_open_session+0x2f/0x50 [libceph]
2012-06-27T01:25:05.114637-07:00 plana55 kernel: [17846.085452] [<ffffffffa0282259>] __ceph_open_session+0x39/0x270 [libceph]
2012-06-27T01:25:05.114639-07:00 plana55 kernel: [17846.085465] [<ffffffffa04ae2e3>] ceph_mount+0x423/0x800 [ceph]
2012-06-27T01:25:05.114641-07:00 plana55 kernel: [17846.085480] [<ffffffff811806e3>] mount_fs+0x43/0x1b0
2012-06-27T01:25:05.114644-07:00 plana55 kernel: [17846.085487] [<ffffffff8119d182>] vfs_kern_mount+0x72/0x110
2012-06-27T01:25:05.114646-07:00 plana55 kernel: [17846.085495] [<ffffffff8119d2a2>] do_kern_mount+0x52/0x110
2012-06-27T01:25:05.114650-07:00 plana55 kernel: [17846.085501] [<ffffffff8119e9aa>] do_mount+0x25a/0x7d0
2012-06-27T01:25:05.114653-07:00 plana55 kernel: [17846.085507] [<ffffffff8119efb0>] sys_mount+0x90/0xe0
2012-06-27T01:25:05.114655-07:00 plana55 kernel: [17846.085513] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:25:05.114656-07:00 plana55 kernel: [17846.085521] SOFTIRQ-ON-W at:
2012-06-27T01:25:05.114659-07:00 plana55 kernel: [17846.085524] [<ffffffff810abb5d>] __lock_acquire+0x5dd/0x15d0
2012-06-27T01:25:05.114661-07:00 plana55 kernel: [17846.085531] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.114665-07:00 plana55 kernel: [17846.085536] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.114668-07:00 plana55 kernel: [17846.085543] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.114670-07:00 plana55 kernel: [17846.085549] [<ffffffffa02862db>] ceph_con_revoke+0x3b/0x160 [libceph]
2012-06-27T01:25:05.115287-07:00 plana55 kernel: [17846.085563] [<ffffffffa0288ed4>] __send_prepared_auth_request+0x44/0x90 [libceph]
2012-06-27T01:25:05.115289-07:00 plana55 kernel: [17846.085578] [<ffffffffa02890e4>] __open_session+0x114/0x180 [libceph]
2012-06-27T01:25:05.115292-07:00 plana55 kernel: [17846.085591] [<ffffffffa028925f>] ceph_monc_open_session+0x2f/0x50 [libceph]
2012-06-27T01:25:05.115297-07:00 plana55 kernel: [17846.085605] [<ffffffffa0282259>] __ceph_open_session+0x39/0x270 [libceph]
2012-06-27T01:25:05.115299-07:00 plana55 kernel: [17846.085619] [<ffffffffa04ae2e3>] ceph_mount+0x423/0x800 [ceph]
2012-06-27T01:25:05.115301-07:00 plana55 kernel: [17846.085632] [<ffffffff811806e3>] mount_fs+0x43/0x1b0
2012-06-27T01:25:05.115304-07:00 plana55 kernel: [17846.085639] [<ffffffff8119d182>] vfs_kern_mount+0x72/0x110
2012-06-27T01:25:05.115306-07:00 plana55 kernel: [17846.085645] [<ffffffff8119d2a2>] do_kern_mount+0x52/0x110
2012-06-27T01:25:05.115308-07:00 plana55 kernel: [17846.085651] [<ffffffff8119e9aa>] do_mount+0x25a/0x7d0
2012-06-27T01:25:05.115310-07:00 plana55 kernel: [17846.085656] [<ffffffff8119efb0>] sys_mount+0x90/0xe0
2012-06-27T01:25:05.115315-07:00 plana55 kernel: [17846.085660] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:25:05.115316-07:00 plana55 kernel: [17846.085666] RECLAIM_FS-ON-W at:
2012-06-27T01:25:05.115318-07:00 plana55 kernel: [17846.085668] [<ffffffff810aae9d>] mark_held_locks+0x7d/0x120
2012-06-27T01:25:05.115321-07:00 plana55 kernel: [17846.085673] [<ffffffff810aafdf>] lockdep_trace_alloc+0x9f/0xd0
2012-06-27T01:25:05.115323-07:00 plana55 kernel: [17846.085677] [<ffffffff8116deba>] kmem_cache_alloc+0x3a/0x140
2012-06-27T01:25:05.115325-07:00 plana55 kernel: [17846.085682] [<ffffffff815049cd>] sock_alloc_inode+0x1d/0xd0
2012-06-27T01:25:05.115330-07:00 plana55 kernel: [17846.085688] [<ffffffff81198107>] alloc_inode+0x27/0xa0
2012-06-27T01:25:05.115332-07:00 plana55 kernel: [17846.085692] [<ffffffff8119819b>] new_inode_pseudo+0x1b/0x70
2012-06-27T01:25:05.115335-07:00 plana55 kernel: [17846.085696] [<ffffffff8150551e>] sock_alloc+0x1e/0x70
2012-06-27T01:25:05.115337-07:00 plana55 kernel: [17846.085701] [<ffffffff81505976>] __sock_create+0x96/0x280
2012-06-27T01:25:05.115339-07:00 plana55 kernel: [17846.085705] [<ffffffff81505b84>] sock_create_kern+0x24/0x30
2012-06-27T01:25:05.115342-07:00 plana55 kernel: [17846.085710] [<ffffffffa0285cc1>] try_write+0xe41/0xfd0 [libceph]
2012-06-27T01:25:05.115344-07:00 plana55 kernel: [17846.085719] [<ffffffffa0287110>] con_work+0xbe0/0x1b40 [libceph]
2012-06-27T01:25:05.115349-07:00 plana55 kernel: [17846.085728] [<ffffffff8106bfe6>] process_one_work+0x1a6/0x520
2012-06-27T01:25:05.115351-07:00 plana55 kernel: [17846.085733] [<ffffffff8106e323>] worker_thread+0x173/0x400
2012-06-27T01:25:05.115353-07:00 plana55 kernel: [17846.085737] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.115355-07:00 plana55 kernel: [17846.085742] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.115357-07:00 plana55 kernel: [17846.085746] INITIAL USE at:
2012-06-27T01:25:05.115359-07:00 plana55 kernel: [17846.085748] [<ffffffff810ab790>] __lock_acquire+0x210/0x15d0
2012-06-27T01:25:05.115363-07:00 plana55 kernel: [17846.085753] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.115366-07:00 plana55 kernel: [17846.085757] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.115368-07:00 plana55 kernel: [17846.085762] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.115370-07:00 plana55 kernel: [17846.085768] [<ffffffffa02862db>] ceph_con_revoke+0x3b/0x160 [libceph]
2012-06-27T01:25:05.115373-07:00 plana55 kernel: [17846.085781] [<ffffffffa0288ed4>] __send_prepared_auth_request+0x44/0x90 [libceph]
2012-06-27T01:25:05.115376-07:00 plana55 kernel: [17846.085795] [<ffffffffa02890e4>] __open_session+0x114/0x180 [libceph]
2012-06-27T01:25:05.115378-07:00 plana55 kernel: [17846.085809] [<ffffffffa028925f>] ceph_monc_open_session+0x2f/0x50 [libceph]
2012-06-27T01:25:05.115383-07:00 plana55 kernel: [17846.085824] [<ffffffffa0282259>] __ceph_open_session+0x39/0x270 [libceph]
2012-06-27T01:25:05.115385-07:00 plana55 kernel: [17846.085837] [<ffffffffa04ae2e3>] ceph_mount+0x423/0x800 [ceph]
2012-06-27T01:25:05.115997-07:00 plana55 kernel: [17846.085853] [<ffffffff811806e3>] mount_fs+0x43/0x1b0
2012-06-27T01:25:05.116000-07:00 plana55 kernel: [17846.085859] [<ffffffff8119d182>] vfs_kern_mount+0x72/0x110
2012-06-27T01:25:05.116002-07:00 plana55 kernel: [17846.085866] [<ffffffff8119d2a2>] do_kern_mount+0x52/0x110
2012-06-27T01:25:05.116004-07:00 plana55 kernel: [17846.085872] [<ffffffff8119e9aa>] do_mount+0x25a/0x7d0
2012-06-27T01:25:05.116011-07:00 plana55 kernel: [17846.085879] [<ffffffff8119efb0>] sys_mount+0x90/0xe0
2012-06-27T01:25:05.116014-07:00 plana55 kernel: [17846.085885] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:25:05.116015-07:00 plana55 kernel: [17846.085891] }
2012-06-27T01:25:05.116017-07:00 plana55 kernel: [17846.085893] ... key at: [<ffffffffa02a6cd8>] __key.63399+0x0/0xffffffffffff1f5f [libceph]
2012-06-27T01:25:05.116019-07:00 plana55 kernel: [17846.085908] ... acquired at:
2012-06-27T01:25:05.116020-07:00 plana55 kernel: [17846.085910] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.116022-07:00 plana55 kernel: [17846.085916] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.116027-07:00 plana55 kernel: [17846.085921] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.116029-07:00 plana55 kernel: [17846.085927] [<ffffffffa0286177>] ceph_con_send+0xa7/0x1d0 [libceph]
2012-06-27T01:25:05.116031-07:00 plana55 kernel: [17846.085940] [<ffffffffa028c565>] __send_request+0xc5/0x120 [libceph]
2012-06-27T01:25:05.116033-07:00 plana55 kernel: [17846.085954] [<ffffffffa028d3d3>] ceph_osdc_start_request+0x113/0x190 [libceph]
2012-06-27T01:25:05.116035-07:00 plana55 kernel: [17846.085969] [<ffffffffa04b5ed9>] ceph_aio_write+0x629/0xab0 [ceph]
2012-06-27T01:25:05.116037-07:00 plana55 kernel: [17846.085983] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:25:05.116039-07:00 plana55 kernel: [17846.085989] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:25:05.116040-07:00 plana55 kernel: [17846.085994] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:25:05.116045-07:00 plana55 kernel: [17846.085998] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:25:05.116046-07:00 plana55 kernel: [17846.086005]
2012-06-27T01:25:05.116047-07:00 plana55 kernel: [17846.086007] ~~> (&osdc~~>request_mutex){+.+.-.} ops: 1266856 {
2012-06-27T01:25:05.116049-07:00 plana55 kernel: [17846.086013] HARDIRQ-ON-W at:
2012-06-27T01:25:05.116051-07:00 plana55 kernel: [17846.086016] [<ffffffff810abb2b>] __lock_acquire+0x5ab/0x15d0
2012-06-27T01:25:05.116053-07:00 plana55 kernel: [17846.086023] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.116056-07:00 plana55 kernel: [17846.086029] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.116060-07:00 plana55 kernel: [17846.086036] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.116062-07:00 plana55 kernel: [17846.086041] [<ffffffffa028d48e>] kick_requests+0x3e/0x2b0 [libceph]
2012-06-27T01:25:05.116065-07:00 plana55 kernel: [17846.086058] [<ffffffffa028db44>] ceph_osdc_handle_map+0x444/0x560 [libceph]
2012-06-27T01:25:05.116067-07:00 plana55 kernel: [17846.086073] [<ffffffffa0289ca0>] dispatch+0x120/0x730 [libceph]
2012-06-27T01:25:05.116069-07:00 plana55 kernel: [17846.086087] [<ffffffffa0287f39>] con_work+0x1a09/0x1b40 [libceph]
2012-06-27T01:25:05.116072-07:00 plana55 kernel: [17846.086101] [<ffffffff8106bfe6>] process_one_work+0x1a6/0x520
2012-06-27T01:25:05.116076-07:00 plana55 kernel: [17846.086107] [<ffffffff8106e323>] worker_thread+0x173/0x400
2012-06-27T01:25:05.116078-07:00 plana55 kernel: [17846.086113] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.116081-07:00 plana55 kernel: [17846.086119] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.116082-07:00 plana55 kernel: [17846.086125] SOFTIRQ-ON-W at:
2012-06-27T01:25:05.116084-07:00 plana55 kernel: [17846.086128] [<ffffffff810abb5d>] __lock_acquire+0x5dd/0x15d0
2012-06-27T01:25:05.116086-07:00 plana55 kernel: [17846.086134] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.116089-07:00 plana55 kernel: [17846.086141] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.116093-07:00 plana55 kernel: [17846.086147] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.116096-07:00 plana55 kernel: [17846.086153] [<ffffffffa028d48e>] kick_requests+0x3e/0x2b0 [libceph]
2012-06-27T01:25:05.116098-07:00 plana55 kernel: [17846.086168] [<ffffffffa028db44>] ceph_osdc_handle_map+0x444/0x560 [libceph]
2012-06-27T01:25:05.116100-07:00 plana55 kernel: [17846.086182] [<ffffffffa0289ca0>] dispatch+0x120/0x730 [libceph]
2012-06-27T01:25:05.116103-07:00 plana55 kernel: [17846.086191] [<ffffffffa0287f39>] con_work+0x1a09/0x1b40 [libceph]
2012-06-27T01:25:05.116105-07:00 plana55 kernel: [17846.086200] [<ffffffff8106bfe6>] process_one_work+0x1a6/0x520
2012-06-27T01:25:05.116109-07:00 plana55 kernel: [17846.086204] [<ffffffff8106e323>] worker_thread+0x173/0x400
2012-06-27T01:25:05.116721-07:00 plana55 kernel: [17846.086208] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.116724-07:00 plana55 kernel: [17846.086212] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.116725-07:00 plana55 kernel: [17846.086217] IN-RECLAIM_FS-W at:
2012-06-27T01:25:05.116727-07:00 plana55 kernel: [17846.086219] [<ffffffff810ab9b1>] __lock_acquire+0x431/0x15d0
2012-06-27T01:25:05.116730-07:00 plana55 kernel: [17846.086223] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.116732-07:00 plana55 kernel: [17846.086228] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.116737-07:00 plana55 kernel: [17846.086232] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.116739-07:00 plana55 kernel: [17846.086238] [<ffffffffa028d34b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.116742-07:00 plana55 kernel: [17846.086253] [<ffffffffa028f444>] ceph_osdc_writepages+0xe4/0x190 [libceph]
2012-06-27T01:25:05.116744-07:00 plana55 kernel: [17846.086264] [<ffffffffa04b8912>] writepage_nounlock+0x232/0x420 [ceph]
2012-06-27T01:25:05.116747-07:00 plana55 kernel: [17846.086275] [<ffffffffa04b8b3e>] ceph_writepage+0x3e/0x70 [ceph]
2012-06-27T01:25:05.116749-07:00 plana55 kernel: [17846.086286] [<ffffffff81134d0a>] shrink_page_list+0x59a/0x980
2012-06-27T01:25:05.116754-07:00 plana55 kernel: [17846.086291] [<ffffffff81135b3b>] shrink_inactive_list+0x1cb/0x580
2012-06-27T01:25:05.116756-07:00 plana55 kernel: [17846.086295] [<ffffffff811362f9>] shrink_mem_cgroup_zone+0x409/0x5b0
2012-06-27T01:25:05.116758-07:00 plana55 kernel: [17846.086299] [<ffffffff81136503>] shrink_zone+0x63/0xb0
2012-06-27T01:25:05.116761-07:00 plana55 kernel: [17846.086303] [<ffffffff811377b5>] balance_pgdat+0x685/0x900
2012-06-27T01:25:05.116763-07:00 plana55 kernel: [17846.086307] [<ffffffff81137d77>] kswapd+0x217/0x410
2012-06-27T01:25:05.116765-07:00 plana55 kernel: [17846.086310] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.116767-07:00 plana55 kernel: [17846.086315] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.116771-07:00 plana55 kernel: [17846.086319] INITIAL USE at:
2012-06-27T01:25:05.116773-07:00 plana55 kernel: [17846.086321] [<ffffffff810ab790>] __lock_acquire+0x210/0x15d0
2012-06-27T01:25:05.116775-07:00 plana55 kernel: [17846.086325] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.116778-07:00 plana55 kernel: [17846.086330] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.116780-07:00 plana55 kernel: [17846.086334] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.116782-07:00 plana55 kernel: [17846.086338] [<ffffffffa028d48e>] kick_requests+0x3e/0x2b0 [libceph]
2012-06-27T01:25:05.116787-07:00 plana55 kernel: [17846.086349] [<ffffffffa028db44>] ceph_osdc_handle_map+0x444/0x560 [libceph]
2012-06-27T01:25:05.116789-07:00 plana55 kernel: [17846.086363] [<ffffffffa0289ca0>] dispatch+0x120/0x730 [libceph]
2012-06-27T01:25:05.116792-07:00 plana55 kernel: [17846.086377] [<ffffffffa0287f39>] con_work+0x1a09/0x1b40 [libceph]
2012-06-27T01:25:05.116794-07:00 plana55 kernel: [17846.086390] [<ffffffff8106bfe6>] process_one_work+0x1a6/0x520
2012-06-27T01:25:05.116796-07:00 plana55 kernel: [17846.086396] [<ffffffff8106e323>] worker_thread+0x173/0x400
2012-06-27T01:25:05.116798-07:00 plana55 kernel: [17846.086402] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.116800-07:00 plana55 kernel: [17846.086407] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.116804-07:00 plana55 kernel: [17846.086413] }
2012-06-27T01:25:05.116806-07:00 plana55 kernel: [17846.086415] ... key at: [<ffffffffa02a7960>] __key.38051+0x0/0xffffffffffff12d7 [libceph]
2012-06-27T01:25:05.116807-07:00 plana55 kernel: [17846.086429] ... acquired at:
2012-06-27T01:25:05.116809-07:00 plana55 kernel: [17846.086431] [<ffffffff810a9d60>] check_usage_forwards+0x90/0x100
2012-06-27T01:25:05.116811-07:00 plana55 kernel: [17846.086437] [<ffffffff810aabc4>] mark_lock+0x1b4/0x410
2012-06-27T01:25:05.116813-07:00 plana55 kernel: [17846.086442] [<ffffffff810ab9b1>] __lock_acquire+0x431/0x15d0
2012-06-27T01:25:05.116815-07:00 plana55 kernel: [17846.086447] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.116819-07:00 plana55 kernel: [17846.086452] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.116821-07:00 plana55 kernel: [17846.086458] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.116823-07:00 plana55 kernel: [17846.086463] [<ffffffffa028d34b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.116825-07:00 plana55 kernel: [17846.086477] [<ffffffffa028f444>] ceph_osdc_writepages+0xe4/0x190 [libceph]
2012-06-27T01:25:05.116827-07:00 plana55 kernel: [17846.086492] [<ffffffffa04b8912>] writepage_nounlock+0x232/0x420 [ceph]
2012-06-27T01:25:05.117317-07:00 plana55 kernel: [17846.086508] [<ffffffffa04b8b3e>] ceph_writepage+0x3e/0x70 [ceph]
2012-06-27T01:25:05.117319-07:00 plana55 kernel: [17846.086524] [<ffffffff81134d0a>] shrink_page_list+0x59a/0x980
2012-06-27T01:25:05.117321-07:00 plana55 kernel: [17846.086529] [<ffffffff81135b3b>] shrink_inactive_list+0x1cb/0x580
2012-06-27T01:25:05.117325-07:00 plana55 kernel: [17846.086533] [<ffffffff811362f9>] shrink_mem_cgroup_zone+0x409/0x5b0
2012-06-27T01:25:05.117327-07:00 plana55 kernel: [17846.086537] [<ffffffff81136503>] shrink_zone+0x63/0xb0
2012-06-27T01:25:05.117329-07:00 plana55 kernel: [17846.086542] [<ffffffff811377b5>] balance_pgdat+0x685/0x900
2012-06-27T01:25:05.117331-07:00 plana55 kernel: [17846.086547] [<ffffffff81137d77>] kswapd+0x217/0x410
2012-06-27T01:25:05.117332-07:00 plana55 kernel: [17846.086551] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.117334-07:00 plana55 kernel: [17846.086557] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.117335-07:00 plana55 kernel: [17846.086561]
2012-06-27T01:25:05.117340-07:00 plana55 kernel: [17846.086562]
2012-06-27T01:25:05.117342-07:00 plana55 kernel: [17846.086562] stack backtrace:
2012-06-27T01:25:05.117344-07:00 plana55 kernel: [17846.086566] Pid: 41, comm: kswapd0 Not tainted 3.4.3-ceph-00244-gfb3f3d7 #1
2012-06-27T01:25:05.117345-07:00 plana55 kernel: [17846.086568] Call Trace:
2012-06-27T01:25:05.117347-07:00 plana55 kernel: [17846.086573] [<ffffffff810a9c83>] print_irq_inversion_bug+0x1d3/0x220
2012-06-27T01:25:05.117349-07:00 plana55 kernel: [17846.086578] [<ffffffff810a9cd0>] ? print_irq_inversion_bug+0x220/0x220
2012-06-27T01:25:05.117351-07:00 plana55 kernel: [17846.086582] [<ffffffff810a9d60>] check_usage_forwards+0x90/0x100
2012-06-27T01:25:05.117353-07:00 plana55 kernel: [17846.086586] [<ffffffff810aabc4>] mark_lock+0x1b4/0x410
2012-06-27T01:25:05.117357-07:00 plana55 kernel: [17846.086591] [<ffffffff810ab9b1>] __lock_acquire+0x431/0x15d0
2012-06-27T01:25:05.117359-07:00 plana55 kernel: [17846.086599] [<ffffffffa028811a>] ? msgpool_alloc+0x2a/0xa0 [libceph]
2012-06-27T01:25:05.117361-07:00 plana55 kernel: [17846.086605] [<ffffffff8131fab6>] ? vsnprintf+0x416/0x5a0
2012-06-27T01:25:05.117363-07:00 plana55 kernel: [17846.086619] [<ffffffffa028d34b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.117365-07:00 plana55 kernel: [17846.086626] [<ffffffff810acbf2>] lock_acquire+0xa2/0x120
2012-06-27T01:25:05.117367-07:00 plana55 kernel: [17846.086635] [<ffffffffa028d34b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.117369-07:00 plana55 kernel: [17846.086640] [<ffffffff8161e64b>] __mutex_lock_common+0x4b/0x3d0
2012-06-27T01:25:05.117373-07:00 plana55 kernel: [17846.086650] [<ffffffffa028d34b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.117376-07:00 plana55 kernel: [17846.086660] [<ffffffffa028d34b>] ? ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.117378-07:00 plana55 kernel: [17846.086665] [<ffffffff812f75c6>] ? drive_stat_acct+0x56/0x1e0
2012-06-27T01:25:05.117379-07:00 plana55 kernel: [17846.086669] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:25:05.117382-07:00 plana55 kernel: [17846.086679] [<ffffffffa028d34b>] ceph_osdc_start_request+0x8b/0x190 [libceph]
2012-06-27T01:25:05.117384-07:00 plana55 kernel: [17846.086689] [<ffffffffa028f444>] ceph_osdc_writepages+0xe4/0x190 [libceph]
2012-06-27T01:25:05.117386-07:00 plana55 kernel: [17846.086700] [<ffffffffa04b8912>] writepage_nounlock+0x232/0x420 [ceph]
2012-06-27T01:25:05.117388-07:00 plana55 kernel: [17846.086706] [<ffffffff8112d4a0>] ? clear_page_dirty_for_io+0xd0/0x130
2012-06-27T01:25:05.117392-07:00 plana55 kernel: [17846.086716] [<ffffffffa04b8b3e>] ceph_writepage+0x3e/0x70 [ceph]
2012-06-27T01:25:05.117394-07:00 plana55 kernel: [17846.086720] [<ffffffff81134d0a>] shrink_page_list+0x59a/0x980
2012-06-27T01:25:05.117396-07:00 plana55 kernel: [17846.086724] [<ffffffff81135b3b>] shrink_inactive_list+0x1cb/0x580
2012-06-27T01:25:05.117398-07:00 plana55 kernel: [17846.086728] [<ffffffff81136046>] ? shrink_mem_cgroup_zone+0x156/0x5b0
2012-06-27T01:25:05.117400-07:00 plana55 kernel: [17846.086732] [<ffffffff811362f9>] shrink_mem_cgroup_zone+0x409/0x5b0
2012-06-27T01:25:05.117402-07:00 plana55 kernel: [17846.086736] [<ffffffff81136503>] shrink_zone+0x63/0xb0
2012-06-27T01:25:05.117403-07:00 plana55 kernel: [17846.086740] [<ffffffff811377b5>] balance_pgdat+0x685/0x900
2012-06-27T01:25:05.117805-07:00 plana55 kernel: [17846.086744] [<ffffffff81137d77>] kswapd+0x217/0x410
2012-06-27T01:25:05.117813-07:00 plana55 kernel: [17846.086748] [<ffffffff81073b10>] ? wake_up_bit+0x40/0x40
2012-06-27T01:25:05.117815-07:00 plana55 kernel: [17846.086751] [<ffffffff81137b60>] ? mem_cgroup_shrink_node_zone+0x130/0x130
2012-06-27T01:25:05.117817-07:00 plana55 kernel: [17846.086755] [<ffffffff8107359e>] kthread+0xbe/0xd0
2012-06-27T01:25:05.117819-07:00 plana55 kernel: [17846.086759] [<ffffffff8162adf4>] kernel_thread_helper+0x4/0x10
2012-06-27T01:25:05.117821-07:00 plana55 kernel: [17846.086764] [<ffffffff81621930>] ? retint_restore_args+0x13/0x13
2012-06-27T01:25:05.117823-07:00 plana55 kernel: [17846.086768] [<ffffffff810734e0>] ? __init_kthread_worker+0x70/0x70
2012-06-27T01:25:05.117825-07:00 plana55 kernel: [17846.086771] [<ffffffff8162adf0>] ? gs_change+0x13/0x13
2012-06-27T01:25:18.662661-07:00 plana55 kernel: [17854.191481] ------------[ cut here ]------------
2012-06-27T01:25:18.662680-07:00 plana55 kernel: [17854.191495] WARNING: at /srv/autobuild-ceph/gitbuilder.git/build/net/sched/sch_generic.c:256 dev_watchdog+0x26a/0x280()
2012-06-27T01:25:18.662688-07:00 plana55 kernel: [17854.191497] Hardware name: PowerEdge R410
2012-06-27T01:25:18.662690-07:00 plana55 kernel: [17854.191499] NETDEV WATCHDOG: eth0 (bnx2): transmit queue 2 timed out
2012-06-27T01:25:18.662697-07:00 plana55 kernel: [17854.191500] Modules linked in: ceph libceph ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs kvm_intel kvm coretemp crc32c_intel ghash_clmulni_intel aesni_intel cryptd lp aes_x86_64 aes_generic psmouse microcode i7core_edac parport serio_raw edac_core hed dcdbas joydev usbhid hid mptsas ixgbe mptscsih mptbase scsi_transport_sas dca mdio bnx2 btrfs zlib_deflate crc32c libcrc32c [last unloaded: rbd]
2012-06-27T01:25:18.662700-07:00 plana55 kernel: [17854.191528] Pid: 0, comm: swapper/0 Not tainted 3.4.3-ceph-00244-gfb3f3d7 #1
2012-06-27T01:25:18.662701-07:00 plana55 kernel: [17854.191530] Call Trace:
2012-06-27T01:25:18.662703-07:00 plana55 kernel: [17854.191531] <IRQ> [<ffffffff8104f5ef>] warn_slowpath_common+0x7f/0xc0
2012-06-27T01:25:18.662706-07:00 plana55 kernel: [17854.191537] [<ffffffff8104f6e6>] warn_slowpath_fmt+0x46/0x50
2012-06-27T01:25:18.662739-07:00 plana55 kernel: [17854.191541] [<ffffffff8153b09a>] dev_watchdog+0x26a/0x280
2012-06-27T01:25:18.662744-07:00 plana55 kernel: [17854.191544] [<ffffffff8105f219>] run_timer_softirq+0x1b9/0x3f0
2012-06-27T01:25:18.662745-07:00 plana55 kernel: [17854.191546] [<ffffffff8105f18a>] ? run_timer_softirq+0x12a/0x3f0
2012-06-27T01:25:18.662747-07:00 plana55 kernel: [17854.191549] [<ffffffff8153ae30>] ? __netdev_watchdog_up+0x80/0x80
2012-06-27T01:25:18.662748-07:00 plana55 kernel: [17854.191552] [<ffffffff81056d6f>] __do_softirq+0xcf/0x220
2012-06-27T01:25:18.662750-07:00 plana55 kernel: [17854.191556] [<ffffffff8162aeec>] call_softirq+0x1c/0x30
2012-06-27T01:25:18.662751-07:00 plana55 kernel: [17854.191559] [<ffffffff8101530d>] do_softirq+0x9d/0xd0
2012-06-27T01:25:18.662752-07:00 plana55 kernel: [17854.191561] [<ffffffff81056ae5>] irq_exit+0xd5/0xf0
2012-06-27T01:25:18.662753-07:00 plana55 kernel: [17854.191565] [<ffffffff81080d32>] scheduler_ipi+0x62/0x120
2012-06-27T01:25:18.662755-07:00 plana55 kernel: [17854.191568] [<ffffffff8103260a>] smp_reschedule_interrupt+0x2a/0x30
2012-06-27T01:25:18.662756-07:00 plana55 kernel: [17854.191570] [<ffffffff8162aa5f>] reschedule_interrupt+0x6f/0x80
2012-06-27T01:25:18.662758-07:00 plana55 kernel: [17854.191571] <EOI> [<ffffffff813710c2>] ? intel_idle+0xe2/0x150
2012-06-27T01:25:18.662759-07:00 plana55 kernel: [17854.191577] [<ffffffff813710be>] ? intel_idle+0xde/0x150
2012-06-27T01:25:18.662760-07:00 plana55 kernel: [17854.191581] [<ffffffff814e51b9>] cpuidle_enter+0x19/0x20
2012-06-27T01:25:18.662761-07:00 plana55 kernel: [17854.191583] [<ffffffff814e5a59>] cpuidle_idle_call+0xd9/0x1e0
2012-06-27T01:25:18.662763-07:00 plana55 kernel: [17854.191586] [<ffffffff8101bc35>] cpu_idle+0xd5/0x130
2012-06-27T01:25:18.662764-07:00 plana55 kernel: [17854.191589] [<ffffffff81605517>] rest_init+0xb7/0xc0
2012-06-27T01:25:18.662765-07:00 plana55 kernel: [17854.191591] [<ffffffff81605460>] ? csum_partial_copy_generic+0x170/0x170
2012-06-27T01:25:18.662767-07:00 plana55 kernel: [17854.191594] [<ffffffff81cf1da5>] start_kernel+0x3db/0x3e8
2012-06-27T01:25:18.662768-07:00 plana55 kernel: [17854.191596] [<ffffffff81cf17e8>] ? kernel_init+0x1d3/0x1d3
2012-06-27T01:25:18.662769-07:00 plana55 kernel: [17854.191599] [<ffffffff81cf1346>] x86_64_start_reservations+0x131/0x136
2012-06-27T01:25:18.662771-07:00 plana55 kernel: [17854.191601] [<ffffffff81cf144e>] x86_64_start_kernel+0x103/0x112
2012-06-27T01:25:18.662772-07:00 plana55 kernel: [17854.191603] ---[ end trace c6287fa2456a9316 ]---
2012-06-27T01:25:18.662773-07:00 plana55 kernel: [17854.191607] bnx2 0000:01:00.0: eth0: DEBUG: intr_sem⁰ PCI_CMD^00100406
2012-06-27T01:25:18.662774-07:00 plana55 kernel: [17854.191612] bnx2 0000:01:00.0: eth0: DEBUG: PCI_PM^19002008 PCI_MISC_CFG^92000088
2012-06-27T01:25:18.662775-07:00 plana55 kernel: [17854.191618] bnx2 0000:01:00.0: eth0: DEBUG: EMAC_TX_STATUS^00000008 EMAC_RX_STATUS^00000000
2012-06-27T01:25:18.662777-07:00 plana55 kernel: [17854.191621] bnx2 0000:01:00.0: eth0: DEBUG: RPM_MGMT_PKT_CTRL^40000088
2012-06-27T01:25:18.662778-07:00 plana55 kernel: [17854.191623] bnx2 0000:01:00.0: eth0: DEBUG: HC_STATS_INTERRUPT_STATUS[01fb0004]
2012-06-27T01:25:18.662785-07:00 plana55 kernel: [17854.191626] bnx2 0000:01:00.0: eth0: DEBUG: PBA^00000000
2012-06-27T01:25:18.662787-07:00 plana55 kernel: [17854.191627] bnx2 0000:01:00.0: eth0: <--- start MCP states dump --->
2012-06-27T01:25:18.662788-07:00 plana55 kernel: [17854.191631] bnx2 0000:01:00.0: eth0: DEBUG: MCP_STATE_P0[0003610e] MCP_STATE_P1[0003600e]
2012-06-27T01:25:18.662790-07:00 plana55 kernel: [17854.191637] bnx2 0000:01:00.0: eth0: DEBUG: MCP mode[0000b880] state^80000000 evt_mask^00000500
2012-06-27T01:25:18.662791-07:00 plana55 kernel: [17854.191642] bnx2 0000:01:00.0: eth0: DEBUG: pc[0800d7cc] pc[08000c3c] instr[3c050004]
2012-06-27T01:25:18.662792-07:00 plana55 kernel: [17854.191643] bnx2 0000:01:00.0: eth0: DEBUG: shmem states:
2012-06-27T01:25:18.662794-07:00 plana55 kernel: [17854.191648] bnx2 0000:01:00.0: eth0: DEBUG: drv_mb^01030003 fw_mb^00000003 link_status[0000006f] drv_pulse_mb[000045c2]
2012-06-27T01:25:18.662796-07:00 plana55 kernel: [17854.191654] bnx2 0000:01:00.0: eth0: DEBUG: dev_info_signature^44564907 reset_type^01005254 condition[0003610e]
2012-06-27T01:25:18.662797-07:00 plana55 kernel: [17854.191662] bnx2 0000:01:00.0: eth0: DEBUG: 000003cc: 00000000 00000000 00000000 00000000
2012-06-27T01:25:18.662799-07:00 plana55 kernel: [17854.191668] bnx2 0000:01:00.0: eth0: DEBUG: 000003dc: 00000000 00000000 00000000 00000000
2012-06-27T01:25:18.662800-07:00 plana55 kernel: [17854.191675] bnx2 0000:01:00.0: eth0: DEBUG: 000003ec: 00000000 00000000 00000000 00000000
2012-06-27T01:25:18.662801-07:00 plana55 kernel: [17854.191678] bnx2 0000:01:00.0: eth0: DEBUG: 0x3fc^00000000
2012-06-27T01:25:18.662824-07:00 plana55 kernel: [17854.191679] bnx2 0000:01:00.0: eth0: <--- end MCP states dump --->
2012-06-27T01:25:18.662826-07:00 plana55 kernel: [17854.282359] bnx2 0000:01:00.0: eth0: NIC Copper Link is Down
2012-06-27T01:25:18.662827-07:00 plana55 kernel: [17857.550450] bnx2 0000:01:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex
2012-06-27T01:25:59.209129-07:00 plana55 kernel: [17900.101853] libceph: tid 4775 timed out on osd4, will reset osd
2012-06-27T01:25:59.212207-07:00 plana55 kernel: [17900.150469] libceph: tid 5384 timed out on osd1, will reset osd
2012-06-27T01:25:59.297059-07:00 plana55 kernel: [17900.196561] libceph: tid 5892 timed out on osd2, will reset osd
2012-06-27T01:25:59.340900-07:00 plana55 kernel: [17900.240768] libceph: tid 7052 timed out on osd0, will reset osd
2012-06-27T01:25:59.385171-07:00 plana55 kernel: [17900.285888] libceph: tid 7077 timed out on osd5, will reset osd
2012-06-27T01:27:09.681975-07:00 plana55 kernel: [17970.459778] libceph: tid 198911 timed out on osd5, will reset osd
2012-06-27T01:27:09.682915-07:00 plana55 kernel: [17970.499554] libceph: tid 94865 timed out on osd0, will reset osd
2012-06-27T01:27:09.763299-07:00 plana55 kernel: [17970.542452] libceph: tid 5892 timed out on osd2, will reset osd
2012-06-27T01:27:09.802693-07:00 plana55 kernel: [17970.583014] libceph: tid 5384 timed out on osd1, will reset osd
2012-06-27T01:27:09.842908-07:00 plana55 kernel: [17970.624315] libceph: tid 4775 timed out on osd4, will reset osd
2012-06-27T01:27:10.588419-07:00 plana55 kernel: [17971.368167] INFO: task ffsb:10938 blocked for more than 120 seconds.
2012-06-27T01:27:10.588454-07:00 plana55 kernel: [17971.404185] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.392613-07:00 plana55 kernel: [17971.469712] ffsb D 00000000ffffffff 0 10938 9250 0x00000000
2012-06-27T01:27:22.392630-07:00 plana55 kernel: [17971.469718] ffff8802168d5ac8 0000000000000046 ffff88020d8b9f70 0000000000013d80
2012-06-27T01:27:22.392631-07:00 plana55 kernel: [17971.469723] ffff8802168d5fd8 ffff8802168d4010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.392647-07:00 plana55 kernel: [17971.469727] ffff8802168d5fd8 0000000000013d80 ffff880223dd3ee0 ffff88020d8b9f70
2012-06-27T01:27:22.392649-07:00 plana55 kernel: [17971.469732] Call Trace:
2012-06-27T01:27:22.392656-07:00 plana55 kernel: [17971.469740] [<ffffffff8161e81d>] ? __mutex_lock_common+0x21d/0x3d0
2012-06-27T01:27:22.392658-07:00 plana55 kernel: [17971.469745] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.392659-07:00 plana55 kernel: [17971.469749] [<ffffffff8162030e>] schedule_preempt_disabled+0xe/0x10
2012-06-27T01:27:22.392664-07:00 plana55 kernel: [17971.469753] [<ffffffff8161e787>] __mutex_lock_common+0x187/0x3d0
2012-06-27T01:27:22.392668-07:00 plana55 kernel: [17971.469759] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.392669-07:00 plana55 kernel: [17971.469773] [<ffffffffa04bda2f>] ? __cap_delay_requeue+0x5f/0x120 [ceph]
2012-06-27T01:27:22.392671-07:00 plana55 kernel: [17971.469777] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.393071-07:00 plana55 kernel: [17971.469781] [<ffffffff8161eaf7>] mutex_lock_nested+0x37/0x50
2012-06-27T01:27:22.393075-07:00 plana55 kernel: [17971.469785] [<ffffffff81124b7c>] generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.393080-07:00 plana55 kernel: [17971.469802] [<ffffffffa04c0a8b>] ? ceph_put_cap_refs+0x13b/0x2b0 [ceph]
2012-06-27T01:27:22.393083-07:00 plana55 kernel: [17971.469815] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]
2012-06-27T01:27:22.393085-07:00 plana55 kernel: [17971.469822] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.393092-07:00 plana55 kernel: [17971.469827] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.393093-07:00 plana55 kernel: [17971.469832] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:27:22.393094-07:00 plana55 kernel: [17971.469836] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.393309-07:00 plana55 kernel: [17971.469840] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.393316-07:00 plana55 kernel: [17971.469845] [<ffffffff812e5c88>] ? apparmor_file_permission+0x18/0x20
2012-06-27T01:27:22.393318-07:00 plana55 kernel: [17971.469850] [<ffffffff812ab3c3>] ? security_file_permission+0x23/0x90
2012-06-27T01:27:22.393319-07:00 plana55 kernel: [17971.469854] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:27:22.393320-07:00 plana55 kernel: [17971.469857] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:27:22.393326-07:00 plana55 kernel: [17971.469862] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:27:22.393328-07:00 plana55 kernel: [17971.469864] INFO: lockdep is turned off.
2012-06-27T01:27:22.393333-07:00 plana55 kernel: [17971.469867] INFO: task ffsb:10939 blocked for more than 120 seconds.
2012-06-27T01:27:22.393335-07:00 plana55 kernel: [17971.469869] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.393520-07:00 plana55 kernel: [17971.469871] ffsb D ffffffff81122320 0 10939 9250 0x00000000
2012-06-27T01:27:22.393526-07:00 plana55 kernel: [17971.469876] ffff8802168d7858 0000000000000046 ffff880223d48000 0000000000013d80
2012-06-27T01:27:22.393527-07:00 plana55 kernel: [17971.469880] ffff8802168d7fd8 ffff8802168d6010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.393528-07:00 plana55 kernel: [17971.469884] ffff8802168d7fd8 0000000000013d80 ffffffff81c13020 ffff880223d48000
2012-06-27T01:27:22.393529-07:00 plana55 kernel: [17971.469888] Call Trace:
2012-06-27T01:27:22.393534-07:00 plana55 kernel: [17971.469891] [<ffffffff81122320>] ? __lock_page+0x70/0x70
2012-06-27T01:27:22.393540-07:00 plana55 kernel: [17971.469895] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.393542-07:00 plana55 kernel: [17971.469899] [<ffffffff8162007c>] io_schedule+0x8c/0xd0
2012-06-27T01:27:22.393543-07:00 plana55 kernel: [17971.469902] [<ffffffff8112232e>] sleep_on_page+0xe/0x20
2012-06-27T01:27:22.393548-07:00 plana55 kernel: [17971.469906] [<ffffffff8161e0ba>] __wait_on_bit_lock+0x5a/0xc0
2012-06-27T01:27:22.393550-07:00 plana55 kernel: [17971.469910] [<ffffffff81122317>] __lock_page+0x67/0x70
2012-06-27T01:27:22.393551-07:00 plana55 kernel: [17971.469916] [<ffffffff81073b50>] ? autoremove_wake_function+0x40/0x40
2012-06-27T01:27:22.393553-07:00 plana55 kernel: [17971.469919] [<ffffffff811235a0>] find_lock_page+0x50/0x80
2012-06-27T01:27:22.393557-07:00 plana55 kernel: [17971.469923] [<ffffffff8112363f>] grab_cache_page_write_begin+0x6f/0xf0
2012-06-27T01:27:22.393560-07:00 plana55 kernel: [17971.469934] [<ffffffffa04bad3e>] ceph_write_begin+0x4e/0xd0 [ceph]
2012-06-27T01:27:22.393561-07:00 plana55 kernel: [17971.469938] [<ffffffff811218cb>] generic_file_buffered_write+0x11b/0x2a0
2012-06-27T01:27:22.393562-07:00 plana55 kernel: [17971.469943] [<ffffffff81055c07>] ? current_fs_time+0x27/0x30
2012-06-27T01:27:22.393564-07:00 plana55 kernel: [17971.469947] [<ffffffff811248e8>] __generic_file_aio_write+0x238/0x470
2012-06-27T01:27:22.393565-07:00 plana55 kernel: [17971.469959] [<ffffffffa04c098e>] ? ceph_put_cap_refs+0x3e/0x2b0 [ceph]
2012-06-27T01:27:22.393566-07:00 plana55 kernel: [17971.469962] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.393568-07:00 plana55 kernel: [17971.469966] [<ffffffff81124b98>] generic_file_aio_write+0x78/0xf0
2012-06-27T01:27:22.393569-07:00 plana55 kernel: [17971.469976] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]
2012-06-27T01:27:22.393570-07:00 plana55 kernel: [17971.469981] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.393577-07:00 plana55 kernel: [17971.469985] [<ffffffff8161e882>] ? __mutex_lock_common+0x282/0x3d0
2012-06-27T01:27:22.393578-07:00 plana55 kernel: [17971.469989] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:27:22.393580-07:00 plana55 kernel: [17971.469993] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.393581-07:00 plana55 kernel: [17971.469996] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.393582-07:00 plana55 kernel: [17971.470000] [<ffffffff812e5c88>] ? apparmor_file_permission+0x18/0x20
2012-06-27T01:27:22.393973-07:00 plana55 kernel: [17971.470004] [<ffffffff812ab3c3>] ? security_file_permission+0x23/0x90
2012-06-27T01:27:22.393975-07:00 plana55 kernel: [17971.470008] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:27:22.393976-07:00 plana55 kernel: [17971.470011] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:27:22.393978-07:00 plana55 kernel: [17971.470015] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:27:22.393979-07:00 plana55 kernel: [17971.470017] INFO: lockdep is turned off.
2012-06-27T01:27:22.393980-07:00 plana55 kernel: [17971.470020] INFO: task ffsb:10940 blocked for more than 120 seconds.
2012-06-27T01:27:22.393981-07:00 plana55 kernel: [17971.470021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.393982-07:00 plana55 kernel: [17971.470023] ffsb D ffffffff81122320 0 10940 9250 0x00000000
2012-06-27T01:27:22.393983-07:00 plana55 kernel: [17971.470028] ffff8801c1f09858 0000000000000046 ffff88020d678000 0000000000013d80
2012-06-27T01:27:22.393985-07:00 plana55 kernel: [17971.470032] ffff8801c1f09fd8 ffff8801c1f08010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.393986-07:00 plana55 kernel: [17971.470036] ffff8801c1f09fd8 0000000000013d80 ffff880223d1bee0 ffff88020d678000
2012-06-27T01:27:22.393987-07:00 plana55 kernel: [17971.470040] Call Trace:
2012-06-27T01:27:22.393988-07:00 plana55 kernel: [17971.470045] [<ffffffff81122320>] ? __lock_page+0x70/0x70
2012-06-27T01:27:22.393989-07:00 plana55 kernel: [17971.470051] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.393990-07:00 plana55 kernel: [17971.470057] [<ffffffff8162007c>] io_schedule+0x8c/0xd0
2012-06-27T01:27:22.393992-07:00 plana55 kernel: [17971.470061] [<ffffffff8112232e>] sleep_on_page+0xe/0x20
2012-06-27T01:27:22.393993-07:00 plana55 kernel: [17971.470064] [<ffffffff8161e0ba>] __wait_on_bit_lock+0x5a/0xc0
2012-06-27T01:27:22.393994-07:00 plana55 kernel: [17971.470068] [<ffffffff81122317>] __lock_page+0x67/0x70
2012-06-27T01:27:22.393996-07:00 plana55 kernel: [17971.470074] [<ffffffff81073b50>] ? autoremove_wake_function+0x40/0x40
2012-06-27T01:27:22.393997-07:00 plana55 kernel: [17971.470080] [<ffffffff811235a0>] find_lock_page+0x50/0x80
2012-06-27T01:27:22.393998-07:00 plana55 kernel: [17971.470085] [<ffffffff8112363f>] grab_cache_page_write_begin+0x6f/0xf0
2012-06-27T01:27:22.393999-07:00 plana55 kernel: [17971.470096] [<ffffffffa04bad3e>] ceph_write_begin+0x4e/0xd0 [ceph]
2012-06-27T01:27:22.394001-07:00 plana55 kernel: [17971.470100] [<ffffffff811218cb>] generic_file_buffered_write+0x11b/0x2a0
2012-06-27T01:27:22.394002-07:00 plana55 kernel: [17971.470104] [<ffffffff81055c07>] ? current_fs_time+0x27/0x30
2012-06-27T01:27:22.394004-07:00 plana55 kernel: [17971.470108] [<ffffffff811248e8>] __generic_file_aio_write+0x238/0x470
2012-06-27T01:27:22.394005-07:00 plana55 kernel: [17971.470120] [<ffffffffa04c098e>] ? ceph_put_cap_refs+0x3e/0x2b0 [ceph]
2012-06-27T01:27:22.394006-07:00 plana55 kernel: [17971.470123] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.394008-07:00 plana55 kernel: [17971.470127] [<ffffffff81124b98>] generic_file_aio_write+0x78/0xf0
2012-06-27T01:27:22.394009-07:00 plana55 kernel: [17971.470139] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]
2012-06-27T01:27:22.394010-07:00 plana55 kernel: [17971.470147] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.394012-07:00 plana55 kernel: [17971.470152] [<ffffffff8161e882>] ? __mutex_lock_common+0x282/0x3d0
2012-06-27T01:27:22.394013-07:00 plana55 kernel: [17971.470156] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:27:22.394014-07:00 plana55 kernel: [17971.470160] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394016-07:00 plana55 kernel: [17971.470163] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394017-07:00 plana55 kernel: [17971.470167] [<ffffffff812e5c88>] ? apparmor_file_permission+0x18/0x20
2012-06-27T01:27:22.394018-07:00 plana55 kernel: [17971.470171] [<ffffffff812ab3c3>] ? security_file_permission+0x23/0x90
2012-06-27T01:27:22.394020-07:00 plana55 kernel: [17971.470175] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:27:22.394021-07:00 plana55 kernel: [17971.470180] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:27:22.394022-07:00 plana55 kernel: [17971.470186] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:27:22.394023-07:00 plana55 kernel: [17971.470190] INFO: lockdep is turned off.
2012-06-27T01:27:22.394025-07:00 plana55 kernel: [17971.470193] INFO: task ffsb:10941 blocked for more than 120 seconds.
2012-06-27T01:27:22.394026-07:00 plana55 kernel: [17971.470194] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.394027-07:00 plana55 kernel: [17971.470197] ffsb D ffffffff81122320 0 10941 9250 0x00000000
2012-06-27T01:27:22.394028-07:00 plana55 kernel: [17971.470201] ffff8801c1f0b858 0000000000000046 ffff88020d67bee0 0000000000013d80
2012-06-27T01:27:22.394029-07:00 plana55 kernel: [17971.470205] ffff8801c1f0bfd8 ffff8801c1f0a010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.394031-07:00 plana55 kernel: [17971.470209] ffff8801c1f0bfd8 0000000000013d80 ffffffff81c13020 ffff88020d67bee0
2012-06-27T01:27:22.394054-07:00 plana55 kernel: [17971.470213] Call Trace:
2012-06-27T01:27:22.394058-07:00 plana55 kernel: [17971.470216] [<ffffffff81122320>] ? __lock_page+0x70/0x70
2012-06-27T01:27:22.394059-07:00 plana55 kernel: [17971.470220] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.394060-07:00 plana55 kernel: [17971.470224] [<ffffffff8162007c>] io_schedule+0x8c/0xd0
2012-06-27T01:27:22.394062-07:00 plana55 kernel: [17971.470227] [<ffffffff8112232e>] sleep_on_page+0xe/0x20
2012-06-27T01:27:22.394063-07:00 plana55 kernel: [17971.470231] [<ffffffff8161e0ba>] __wait_on_bit_lock+0x5a/0xc0
2012-06-27T01:27:22.394064-07:00 plana55 kernel: [17971.470234] [<ffffffff81122317>] __lock_page+0x67/0x70
2012-06-27T01:27:22.394065-07:00 plana55 kernel: [17971.470238] [<ffffffff81073b50>] ? autoremove_wake_function+0x40/0x40
2012-06-27T01:27:22.394067-07:00 plana55 kernel: [17971.470242] [<ffffffff811235a0>] find_lock_page+0x50/0x80
2012-06-27T01:27:22.394068-07:00 plana55 kernel: [17971.470245] [<ffffffff8112363f>] grab_cache_page_write_begin+0x6f/0xf0
2012-06-27T01:27:22.394069-07:00 plana55 kernel: [17971.470257] [<ffffffffa04bad3e>] ceph_write_begin+0x4e/0xd0 [ceph]
2012-06-27T01:27:22.394071-07:00 plana55 kernel: [17971.470260] [<ffffffff811218cb>] generic_file_buffered_write+0x11b/0x2a0
2012-06-27T01:27:22.394459-07:00 plana55 kernel: [17971.470264] [<ffffffff81055c07>] ? current_fs_time+0x27/0x30
2012-06-27T01:27:22.394461-07:00 plana55 kernel: [17971.470268] [<ffffffff811248e8>] __generic_file_aio_write+0x238/0x470
2012-06-27T01:27:22.394462-07:00 plana55 kernel: [17971.470280] [<ffffffffa04c098e>] ? ceph_put_cap_refs+0x3e/0x2b0 [ceph]
2012-06-27T01:27:22.394464-07:00 plana55 kernel: [17971.470283] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.394465-07:00 plana55 kernel: [17971.470287] [<ffffffff81124b98>] generic_file_aio_write+0x78/0xf0
2012-06-27T01:27:22.394466-07:00 plana55 kernel: [17971.470297] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]
2012-06-27T01:27:22.394468-07:00 plana55 kernel: [17971.470303] [<ffffffff8132209e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
2012-06-27T01:27:22.394469-07:00 plana55 kernel: [17971.470308] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.394470-07:00 plana55 kernel: [17971.470312] [<ffffffff8161e882>] ? __mutex_lock_common+0x282/0x3d0
2012-06-27T01:27:22.394471-07:00 plana55 kernel: [17971.470315] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:27:22.394473-07:00 plana55 kernel: [17971.470319] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394474-07:00 plana55 kernel: [17971.470323] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394475-07:00 plana55 kernel: [17971.470329] [<ffffffff812e5c88>] ? apparmor_file_permission+0x18/0x20
2012-06-27T01:27:22.394477-07:00 plana55 kernel: [17971.470335] [<ffffffff812ab3c3>] ? security_file_permission+0x23/0x90
2012-06-27T01:27:22.394478-07:00 plana55 kernel: [17971.470339] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:27:22.394479-07:00 plana55 kernel: [17971.470344] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:27:22.394481-07:00 plana55 kernel: [17971.470350] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:27:22.394482-07:00 plana55 kernel: [17971.470354] INFO: lockdep is turned off.
2012-06-27T01:27:22.394483-07:00 plana55 kernel: [17971.470358] INFO: task ffsb:10943 blocked for more than 120 seconds.
2012-06-27T01:27:22.394484-07:00 plana55 kernel: [17971.470361] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.394485-07:00 plana55 kernel: [17971.470363] ffsb D ffffffff81122320 0 10943 9250 0x00000000
2012-06-27T01:27:22.394487-07:00 plana55 kernel: [17971.470368] ffff88018f493858 0000000000000046 ffff88021b469f70 0000000000013d80
2012-06-27T01:27:22.394488-07:00 plana55 kernel: [17971.470372] ffff88018f493fd8 ffff88018f492010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.394489-07:00 plana55 kernel: [17971.470376] ffff88018f493fd8 0000000000013d80 ffff880223d2bee0 ffff88021b469f70
2012-06-27T01:27:22.394490-07:00 plana55 kernel: [17971.470380] Call Trace:
2012-06-27T01:27:22.394491-07:00 plana55 kernel: [17971.470383] [<ffffffff81122320>] ? __lock_page+0x70/0x70
2012-06-27T01:27:22.394493-07:00 plana55 kernel: [17971.470387] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.394494-07:00 plana55 kernel: [17971.470391] [<ffffffff8162007c>] io_schedule+0x8c/0xd0
2012-06-27T01:27:22.394495-07:00 plana55 kernel: [17971.470394] [<ffffffff8112232e>] sleep_on_page+0xe/0x20
2012-06-27T01:27:22.394497-07:00 plana55 kernel: [17971.470397] [<ffffffff8161e0ba>] __wait_on_bit_lock+0x5a/0xc0
2012-06-27T01:27:22.394498-07:00 plana55 kernel: [17971.470401] [<ffffffff81122317>] __lock_page+0x67/0x70
2012-06-27T01:27:22.394499-07:00 plana55 kernel: [17971.470405] [<ffffffff81073b50>] ? autoremove_wake_function+0x40/0x40
2012-06-27T01:27:22.394500-07:00 plana55 kernel: [17971.470408] [<ffffffff811235a0>] find_lock_page+0x50/0x80
2012-06-27T01:27:22.394502-07:00 plana55 kernel: [17971.470412] [<ffffffff8112363f>] grab_cache_page_write_begin+0x6f/0xf0
2012-06-27T01:27:22.394503-07:00 plana55 kernel: [17971.470423] [<ffffffffa04bad3e>] ceph_write_begin+0x4e/0xd0 [ceph]
2012-06-27T01:27:22.394504-07:00 plana55 kernel: [17971.470427] [<ffffffff811218cb>] generic_file_buffered_write+0x11b/0x2a0
2012-06-27T01:27:22.394506-07:00 plana55 kernel: [17971.470431] [<ffffffff81055c07>] ? current_fs_time+0x27/0x30
2012-06-27T01:27:22.394507-07:00 plana55 kernel: [17971.470435] [<ffffffff811248e8>] __generic_file_aio_write+0x238/0x470
2012-06-27T01:27:22.394508-07:00 plana55 kernel: [17971.470446] [<ffffffffa04c098e>] ? ceph_put_cap_refs+0x3e/0x2b0 [ceph]
2012-06-27T01:27:22.394510-07:00 plana55 kernel: [17971.470450] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.394511-07:00 plana55 kernel: [17971.470453] [<ffffffff81124b98>] generic_file_aio_write+0x78/0xf0
2012-06-27T01:27:22.394512-07:00 plana55 kernel: [17971.470463] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]
2012-06-27T01:27:22.394513-07:00 plana55 kernel: [17971.470468] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.394515-07:00 plana55 kernel: [17971.470472] [<ffffffff8161e882>] ? __mutex_lock_common+0x282/0x3d0
2012-06-27T01:27:22.394516-07:00 plana55 kernel: [17971.470476] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:27:22.394517-07:00 plana55 kernel: [17971.470480] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394518-07:00 plana55 kernel: [17971.470483] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394520-07:00 plana55 kernel: [17971.470487] [<ffffffff812e5c88>] ? apparmor_file_permission+0x18/0x20
2012-06-27T01:27:22.394521-07:00 plana55 kernel: [17971.470491] [<ffffffff812ab3c3>] ? security_file_permission+0x23/0x90
2012-06-27T01:27:22.394523-07:00 plana55 kernel: [17971.470495] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:27:22.394524-07:00 plana55 kernel: [17971.470498] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:27:22.394525-07:00 plana55 kernel: [17971.470502] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:27:22.394526-07:00 plana55 kernel: [17971.470504] INFO: lockdep is turned off.
2012-06-27T01:27:22.394527-07:00 plana55 kernel: [17971.470507] INFO: task ffsb:10949 blocked for more than 120 seconds.
2012-06-27T01:27:22.394529-07:00 plana55 kernel: [17971.470509] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.394530-07:00 plana55 kernel: [17971.470511] ffsb D ffffffff81122320 0 10949 9250 0x00000000
2012-06-27T01:27:22.394531-07:00 plana55 kernel: [17971.470515] ffff88018f48f858 0000000000000046 ffff8802227fbee0 0000000000013d80
2012-06-27T01:27:22.394532-07:00 plana55 kernel: [17971.470519] ffff88018f48ffd8 ffff88018f48e010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.394921-07:00 plana55 kernel: [17971.470523] ffff88018f48ffd8 0000000000013d80 ffff88021b46de50 ffff8802227fbee0
2012-06-27T01:27:22.394922-07:00 plana55 kernel: [17971.470527] Call Trace:
2012-06-27T01:27:22.394924-07:00 plana55 kernel: [17971.470530] [<ffffffff81122320>] ? __lock_page+0x70/0x70
2012-06-27T01:27:22.394925-07:00 plana55 kernel: [17971.470533] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.394926-07:00 plana55 kernel: [17971.470537] [<ffffffff8162007c>] io_schedule+0x8c/0xd0
2012-06-27T01:27:22.394927-07:00 plana55 kernel: [17971.470540] [<ffffffff8112232e>] sleep_on_page+0xe/0x20
2012-06-27T01:27:22.394929-07:00 plana55 kernel: [17971.470544] [<ffffffff8161e0ba>] __wait_on_bit_lock+0x5a/0xc0
2012-06-27T01:27:22.394930-07:00 plana55 kernel: [17971.470547] [<ffffffff81122317>] __lock_page+0x67/0x70
2012-06-27T01:27:22.394931-07:00 plana55 kernel: [17971.470551] [<ffffffff81073b50>] ? autoremove_wake_function+0x40/0x40
2012-06-27T01:27:22.394932-07:00 plana55 kernel: [17971.470555] [<ffffffff811235a0>] find_lock_page+0x50/0x80
2012-06-27T01:27:22.394934-07:00 plana55 kernel: [17971.470558] [<ffffffff8112363f>] grab_cache_page_write_begin+0x6f/0xf0
2012-06-27T01:27:22.394935-07:00 plana55 kernel: [17971.470569] [<ffffffffa04bad3e>] ceph_write_begin+0x4e/0xd0 [ceph]
2012-06-27T01:27:22.394937-07:00 plana55 kernel: [17971.470573] [<ffffffff811218cb>] generic_file_buffered_write+0x11b/0x2a0
2012-06-27T01:27:22.394938-07:00 plana55 kernel: [17971.470578] [<ffffffff811a603c>] ? __mark_inode_dirty+0x19c/0x230
2012-06-27T01:27:22.394939-07:00 plana55 kernel: [17971.470582] [<ffffffff811248e8>] __generic_file_aio_write+0x238/0x470
2012-06-27T01:27:22.394941-07:00 plana55 kernel: [17971.470593] [<ffffffffa04c098e>] ? ceph_put_cap_refs+0x3e/0x2b0 [ceph]
2012-06-27T01:27:22.394942-07:00 plana55 kernel: [17971.470597] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.394943-07:00 plana55 kernel: [17971.470601] [<ffffffff81124b98>] generic_file_aio_write+0x78/0xf0
2012-06-27T01:27:22.394945-07:00 plana55 kernel: [17971.470611] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]
2012-06-27T01:27:22.394946-07:00 plana55 kernel: [17971.470615] [<ffffffff81621930>] ? retint_restore_args+0x13/0x13
2012-06-27T01:27:22.394947-07:00 plana55 kernel: [17971.470620] [<ffffffff810ab790>] ? __lock_acquire+0x210/0x15d0
2012-06-27T01:27:22.394949-07:00 plana55 kernel: [17971.470624] [<ffffffff8161e882>] ? __mutex_lock_common+0x282/0x3d0
2012-06-27T01:27:22.394950-07:00 plana55 kernel: [17971.470627] [<ffffffff8117d472>] do_sync_write+0xe2/0x120
2012-06-27T01:27:22.394951-07:00 plana55 kernel: [17971.470631] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394953-07:00 plana55 kernel: [17971.470634] [<ffffffff8117f560>] ? fget_light+0x40/0x130
2012-06-27T01:27:22.394954-07:00 plana55 kernel: [17971.470638] [<ffffffff812e5c88>] ? apparmor_file_permission+0x18/0x20
2012-06-27T01:27:22.394955-07:00 plana55 kernel: [17971.470642] [<ffffffff812ab3c3>] ? security_file_permission+0x23/0x90
2012-06-27T01:27:22.394957-07:00 plana55 kernel: [17971.470646] [<ffffffff8117da08>] vfs_write+0xc8/0x190
2012-06-27T01:27:22.394958-07:00 plana55 kernel: [17971.470649] [<ffffffff8117dbc1>] sys_write+0x51/0x90
2012-06-27T01:27:22.394959-07:00 plana55 kernel: [17971.470653] [<ffffffff816299a9>] system_call_fastpath+0x16/0x1b
2012-06-27T01:27:22.394960-07:00 plana55 kernel: [17971.470655] INFO: lockdep is turned off.
2012-06-27T01:27:22.394961-07:00 plana55 kernel: [17971.470657] INFO: task ffsb:10950 blocked for more than 120 seconds.
2012-06-27T01:27:22.394963-07:00 plana55 kernel: [17971.470659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-06-27T01:27:22.394964-07:00 plana55 kernel: [17971.470661] ffsb D ffffffff81122320 0 10950 9250 0x00000000
2012-06-27T01:27:22.394965-07:00 plana55 kernel: [17971.470665] ffff88018f451858 0000000000000046 ffff880223db8000 0000000000013d80
2012-06-27T01:27:22.394967-07:00 plana55 kernel: [17971.470669] ffff88018f451fd8 ffff88018f450010 0000000000013d80 0000000000013d80
2012-06-27T01:27:22.394968-07:00 plana55 kernel: [17971.470673] ffff88018f451fd8 0000000000013d80 ffff88021c7d0000 ffff880223db8000
2012-06-27T01:27:22.394969-07:00 plana55 kernel: [17971.470677] Call Trace:
2012-06-27T01:27:22.394970-07:00 plana55 kernel: [17971.470680] [<ffffffff81122320>] ? __lock_page+0x70/0x70
2012-06-27T01:27:22.394971-07:00 plana55 kernel: [17971.470684] [<ffffffff8161ffa9>] schedule+0x29/0x70
2012-06-27T01:27:22.394973-07:00 plana55 kernel: [17971.470688] [<ffffffff8162007c>] io_schedule+0x8c/0xd0
2012-06-27T01:27:22.394974-07:00 plana55 kernel: [17971.470691] [<ffffffff8112232e>] sleep_on_page+0xe/0x20
2012-06-27T01:27:22.394975-07:00 plana55 kernel: [17971.470694] [<ffffffff8161e0ba>] __wait_on_bit_lock+0x5a/0xc0
2012-06-27T01:27:22.394976-07:00 plana55 kernel: [17971.470698] [<ffffffff81122317>] __lock_page+0x67/0x70
2012-06-27T01:27:22.394978-07:00 plana55 kernel: [17971.470702] [<ffffffff81073b50>] ? autoremove_wake_function+0x40/0x40
2012-06-27T01:27:22.394979-07:00 plana55 kernel: [17971.470705] [<ffffffff811235a0>] find_lock_page+0x50/0x80
2012-06-27T01:27:22.394980-07:00 plana55 kernel: [17971.470709] [<ffffffff8112363f>] grab_cache_page_write_begin+0x6f/0xf0
2012-06-27T01:27:22.394982-07:00 plana55 kernel: [17971.470720] [<ffffffffa04bad3e>] ceph_write_begin+0x4e/0xd0 [ceph]
2012-06-27T01:27:22.394983-07:00 plana55 kernel: [17971.470724] [<ffffffff811218cb>] generic_file_buffered_write+0x11b/0x2a0
2012-06-27T01:27:22.394984-07:00 plana55 kernel: [17971.470727] [<ffffffff811a603c>] ? __mark_inode_dirty+0x19c/0x230
2012-06-27T01:27:22.395471-07:00 plana55 kernel: [17971.470731] [<ffffffff811248e8>] __generic_file_aio_write+0x238/0x470
2012-06-27T01:27:22.395587-07:00 plana55 kernel: [17971.470743] [<ffffffffa04c098e>] ? ceph_put_cap_refs+0x3e/0x2b0 [ceph]
2012-06-27T01:27:22.395590-07:00 plana55 kernel: [17971.470746] [<ffffffff81124b7c>] ? generic_file_aio_write+0x5c/0xf0
2012-06-27T01:27:22.395592-07:00 plana55 kernel: [17971.470750] [<ffffffff81124b98>] generic_file_aio_write+0x78/0xf0
2012-06-27T01:27:22.395593-07:00 plana55 kernel: [17971.470760] [<ffffffffa04b5c1f>] ceph_aio_write+0x36f/0xab0 [ceph]