Actions
Bug #2764
closedxfstest hang; osd socket closed messages
Status:
Closed
Priority:
Normal
Assignee:
-
Category:
libceph
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
[223564.772925] libceph: osd1 10.214.132.12:6803 socket closed [223565.551727] libceph: osd4 10.214.132.13:6803 socket closed [224463.542820] libceph: osd1 10.214.132.12:6803 socket closed [224464.256620] libceph: osd4 10.214.132.13:6803 socket closed [225362.312671] libceph: osd1 10.214.132.12:6803 socket closed [225363.040916] libceph: osd4 10.214.132.13:6803 socket closed [226261.082746] libceph: osd1 10.214.132.12:6803 socket closed [226261.818194] libceph: osd4 10.214.132.13:6803 socket closed [227159.849469] libceph: osd1 10.214.132.12:6803 socket closed [227160.591953] libceph: osd4 10.214.132.13:6803 socket closed [228058.616188] libceph: osd1 10.214.132.12:6803 socket closed [228059.366723] libceph: osd4 10.214.132.13:6803 socket closed [228957.385373] libceph: osd1 10.214.132.12:6803 socket closed [228958.145662] libceph: osd4 10.214.132.13:6803 socket closed [229856.156362] libceph: osd1 10.214.132.12:6803 socket closed [229856.925149] libceph: osd4 10.214.132.13:6803 socket closed [230754.927726] libceph: osd1 10.214.132.12:6803 socket closed [230755.705742] libceph: osd4 10.214.132.13:6803 socket closed [231653.698026] libceph: osd1 10.214.132.12:6803 socket closed [231654.486141] libceph: osd4 10.214.132.13:6803 socket closed [232552.458092] libceph: osd1 10.214.132.12:6803 socket closed [232552.475194] libceph: osd1 10.214.132.12:6803 connect authorization failure [232553.247748] libceph: osd4 10.214.132.13:6803 socket closed [232553.265260] libceph: osd4 10.214.132.13:6803 connect authorization failure [233452.098267] libceph: osd1 10.214.132.12:6803 socket closed [233453.105762] libceph: osd4 10.214.132.13:6803 socket closed [234350.858738] libceph: osd1 10.214.132.12:6803 socket closed [234350.875846] libceph: osd1 10.214.132.12:6803 connect authorization failure [234351.867907] libceph: osd4 10.214.132.13:6803 socket closed [234351.885196] libceph: osd4 10.214.132.13:6803 connect authorization failure [235250.857388] libceph: osd1 10.214.132.12:6803 socket closed [235250.987315] libceph: osd4 10.214.132.13:6803 socket closed [235251.004955] libceph: osd4 10.214.132.13:6803 connect authorization failure [236149.618319] libceph: osd1 10.214.132.12:6803 socket closed [236149.636781] libceph: osd1 10.214.132.12:6803 connect authorization failure [236150.276918] libceph: osd4 10.214.132.13:6803 socket closed [237048.639043] libceph: osd1 10.214.132.12:6803 socket closed [237049.047246] libceph: osd4 10.214.132.13:6803 socket closed [237049.066697] libceph: osd4 10.214.132.13:6803 connect authorization failure [237947.408077] libceph: osd1 10.214.132.12:6803 socket closed [237947.428231] libceph: osd1 10.214.132.12:6803 connect authorization failure [237948.126775] libceph: osd4 10.214.132.13:6803 socket closed [238846.736883] libceph: osd1 10.214.132.12:6803 socket closed [238846.758182] libceph: osd1 10.214.132.12:6803 connect authorization failure [238846.897194] libceph: osd4 10.214.132.13:6803 socket closed [238846.919355] libceph: osd4 10.214.132.13:6803 connect authorization failure [239745.976770] libceph: osd4 10.214.132.13:6803 socket closed [239746.175294] libceph: osd1 10.214.132.12:6803 socket closed [240644.745103] libceph: osd4 10.214.132.13:6803 socket closed [240644.768313] libceph: osd4 10.214.132.13:6803 connect authorization failure [240644.944420] libceph: osd1 10.214.132.12:6803 socket closed [240644.968030] libceph: osd1 10.214.132.12:6803 connect authorization failure [241544.061204] libceph: osd4 10.214.132.13:6803 socket closed [241544.901360] libceph: osd1 10.214.132.12:6803 socket closed [242442.729929] libceph: osd4 10.214.132.13:6803 socket closed [242442.752860] libceph: osd4 10.214.132.13:6803 connect authorization failure [242443.670571] libceph: osd1 10.214.132.12:6803 socket closed [242443.694228] libceph: osd1 10.214.132.12:6803 connect authorization failure [243341.938215] libceph: osd4 10.214.132.13:6803 socket closed [243342.878488] libceph: osd1 10.214.132.12:6803 socket closed [243342.901684] libceph: osd1 10.214.132.12:6803 connect authorization failure [244240.700553] libceph: osd4 10.214.132.13:6803 socket closed [244240.724281] libceph: osd4 10.214.132.13:6803 connect authorization failure [244242.211031] libceph: osd1 10.214.132.12:6803 socket closed [245140.688321] libceph: osd4 10.214.132.13:6803 socket closed [245140.977989] libceph: osd1 10.214.132.12:6803 socket closed [245141.001021] libceph: osd1 10.214.132.12:6803 connect authorization failure [246039.444867] libceph: osd4 10.214.132.13:6803 socket closed [246039.468376] libceph: osd4 10.214.132.13:6803 connect authorization failure [246040.083501] libceph: osd1 10.214.132.12:6803 socket closed [246040.107216] libceph: osd1 10.214.132.12:6803 connect authorization failure [246938.680424] libceph: osd4 10.214.132.13:6803 socket closed [246939.668001] libceph: osd1 10.214.132.12:6803 socket closed [247837.374400] libceph: osd4 10.214.132.13:6803 socket closed [247837.397713] libceph: osd4 10.214.132.13:6803 connect authorization failure [247838.436308] libceph: osd1 10.214.132.12:6803 socket closed [247838.459458] libceph: osd1 10.214.132.12:6803 connect authorization failure [248736.555714] libceph: osd4 10.214.132.13:6803 socket closed [248738.422406] libceph: osd1 10.214.132.12:6803 socket closed [248738.446330] libceph: osd1 10.214.132.12:6803 connect authorization failure [249635.323764] libceph: osd4 10.214.132.13:6803 socket closed [249635.347633] libceph: osd4 10.214.132.13:6803 connect authorization failure [249637.724051] libceph: osd1 10.214.132.12:6803 socket closed [250534.432012] libceph: osd4 10.214.132.13:6803 socket closed [250536.404949] libceph: osd1 10.214.132.12:6803 socket closed [250536.428861] libceph: osd1 10.214.132.12:6803 connect authorization failure [251433.201815] libceph: osd4 10.214.132.13:6803 socket closed [251433.225705] libceph: osd4 10.214.132.13:6803 connect authorization failure [251435.737624] libceph: osd1 10.214.132.12:6803 socket closed [252333.099897] libceph: osd4 10.214.132.13:6803 socket closed [252334.472220] libceph: osd1 10.214.132.12:6803 socket closed [252334.495534] libceph: osd1 10.214.132.12:6803 connect authorization failure [253231.869504] libceph: osd4 10.214.132.13:6803 socket closed [253231.892891] libceph: osd4 10.214.132.13:6803 connect authorization failure [253233.735924] libceph: osd1 10.214.132.12:6803 socket closed [254131.068997] libceph: osd4 10.214.132.13:6803 socket closed [254132.507521] libceph: osd1 10.214.132.12:6803 socket closed [254132.530810] libceph: osd1 10.214.132.12:6803 connect authorization failure [255029.816652] libceph: osd4 10.214.132.13:6803 socket closed [255029.839967] libceph: osd4 10.214.132.13:6803 connect authorization failure [255031.587655] libceph: osd1 10.214.132.12:6803 socket closed [255031.611062] libceph: osd1 10.214.132.12:6803 connect authorization failure [255928.922133] libceph: osd4 10.214.132.13:6803 socket closed [255930.938476] libceph: osd1 10.214.132.12:6803 socket closed [256827.694937] libceph: osd4 10.214.132.13:6803 socket closed [256827.718337] libceph: osd4 10.214.132.13:6803 connect authorization failure [256829.710442] libceph: osd1 10.214.132.12:6803 socket closed [256829.733738] libceph: osd1 10.214.132.12:6803 connect authorization failure [257726.868955] libceph: osd4 10.214.132.13:6803 socket closed [257728.761629] libceph: osd1 10.214.132.12:6803 socket closed [258625.638183] libceph: osd4 10.214.132.13:6803 socket closed [258625.661165] libceph: osd4 10.214.132.13:6803 connect authorization failure [258627.533556] libceph: osd1 10.214.132.12:6803 socket closed [258627.556778] libceph: osd1 10.214.132.12:6803 connect authorization failure [259524.716029] libceph: osd4 10.214.132.13:6803 socket closed [259526.704617] libceph: osd1 10.214.132.12:6803 socket closed [259526.727868] libceph: osd1 10.214.132.12:6803 connect authorization failure [260423.490645] libceph: osd4 10.214.132.13:6803 socket closed [260423.514092] libceph: osd4 10.214.132.13:6803 connect authorization failure [260426.265991] libceph: osd1 10.214.132.12:6803 socket closed [261323.448994] libceph: osd4 10.214.132.13:6803 socket closed [261325.035092] libceph: osd1 10.214.132.12:6803 socket closed [261325.058539] libceph: osd1 10.214.132.12:6803 connect authorization failure [262222.145504] libceph: osd4 10.214.132.13:6803 socket closed [262222.168860] libceph: osd4 10.214.132.13:6803 connect authorization failure [262224.134678] libceph: osd1 10.214.132.12:6803 socket closed [262224.158992] libceph: osd1 10.214.132.12:6803 connect authorization failure [263121.470566] libceph: osd4 10.214.132.13:6803 socket closed [263123.572666] libceph: osd1 10.214.132.12:6803 socket closed [264020.241985] libceph: osd4 10.214.132.13:6803 socket closed [264020.265299] libceph: osd4 10.214.132.13:6803 connect authorization failure [264022.271957] libceph: osd1 10.214.132.12:6803 socket closed [264022.295175] libceph: osd1 10.214.132.12:6803 connect authorization failure [264919.316845] libceph: osd4 10.214.132.13:6803 socket closed [264922.243590] libceph: osd1 10.214.132.12:6803 socket closed [265818.091136] libceph: osd4 10.214.132.13:6803 socket closed [265818.114437] libceph: osd4 10.214.132.13:6803 connect authorization failure [265820.955322] libceph: osd1 10.214.132.12:6803 socket closed [265820.978595] libceph: osd1 10.214.132.12:6803 connect authorization failure [266717.171049] libceph: osd4 10.214.132.13:6803 socket closed [266720.125106] libceph: osd1 10.214.132.12:6803 socket closed [266720.148977] libceph: osd1 10.214.132.12:6803 connect authorization failure [267615.901722] libceph: osd4 10.214.132.13:6803 socket closed [267615.925581] libceph: osd4 10.214.132.13:6803 connect authorization failure [267619.683623] libceph: osd1 10.214.132.12:6803 socket closed [267619.708236] libceph: osd1 10.214.132.12:6803 connect authorization failure [268515.231851] libceph: osd4 10.214.132.13:6803 socket closed [268519.215217] libceph: osd1 10.214.132.12:6803 socket closed ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2012-07-09_00:00:07-regression-next-testing-basic/7880$ cat config.yaml kernel: &id001 kdb: true sha1: f26d9cb6552416dad978ca1f84bdcf3a0166b75e nuke-on-error: true overrides: ceph: fs: btrfs log-whitelist: - slow request sha1: 97c1562dda2b0f3370bb186e7acbcd9c273210bc roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - - client.0 targets: ubuntu@plana65.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTwfkF9asvpySXF/DOk10UkRDNtRwgGgLww/I/3E2r+JpsfYtW62TA1HMXjtB1g7SrcIolqCiiMd+5MIURIND94n76JiZ2o4DplLKIqUB6ys46gro7mwoeFnZNOuwdAA5bO4dfgeQ3yPtfIqpWTejkCB7ai/kG04C4ekz6EgplwtqWIfvXnij4fNaqvm3s/IxGhnO40DOGNwsAldEJo2fuJN8KHnYzsU/Dx5kJ85jQl2eQJI74VpMoh2Ge7+n9Q8rJhegfcHYPLJsX/Uyrf7Rtk1RfeTyZbIOSJIQDbQNepu278kvc9IEnFg3WfvWespfrUExVgnXq53xd1RIFcx6L ubuntu@plana66.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9QpKzRhLR+H4oCXhZtMxFDAC7E9cmtqcEPmQiW1ycxHNpk+Vy/uNNtlSh8Ljz7/R595beZe4JKgAkxZCCDmmWIEcm01bJgxVq0cbFLz+9dvyiirmr+RxbWqayu2VDC6uuiVAuxz5RdAw6+5Y/4gCrSdQKfQ8dUJZb4e/4Kz/TLr/+R5z+WqCGeutwb9QvK2anvhPJy+wa/JWHwtpTBjZVa5RFOkz9lfNDYayw3j1rylKk0d39J4VB/ch/qIBqUfxD7Hc4exu9sbG9bt5VKdomqdQvTjQXRw083+Nlj6RqJxYZfdJTWG/gYV3MXC4pwqP/ovcjoqZ9cvrsicdSezKp ubuntu@plana67.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqiZsiT7h3fNR9yzwK2WToaotO4olIxmVdh+aSf3ILwEpHjFYbWXymL0C77hn0MdGRbaOzWOSMMng3MAHKy9xR3/CGNXXqO7iEK1fJOvSfmypkvJDyrMY/RuSvdifcXJyREvFsSK6cdmRpO235ODhfui4FC5BLmgv/VvasH/1Ur4ALfe7UE9L+cU4VeoJdl082oYeo1nn1beERgaypX67MXepG2NKbEY77jG5FXbGVpKWmsgIEWiiX8p6+afTOP+8cGsM3vsAG7nTJeFVKkEHc7A8cPkT4l/iXKjSiwWAtU5NV0QmRC/1ad78+xTOWNzJaTrIxoKuuGpB+DjdvrJgN tasks: - internal.lock_machines: 3 - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock: null - ceph: null - rbd.xfstests: null ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2012-07-09_00:00:07-regression-next-testing-basic/7880$ less teuthology.log ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2012-07-09_00:00:07-regression-next-testing-basic/7880$ grep role teuthology.log 2012-07-09T00:19:40.934 INFO:teuthology.task.internal:roles: ubuntu@plana66.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2012-07-09T00:19:40.934 INFO:teuthology.task.internal:roles: ubuntu@plana65.front.sepia.ceph.com - ['mon.b', 'mds.a', 'osd.3', 'osd.4', 'osd.5'] 2012-07-09T00:19:40.935 INFO:teuthology.task.internal:roles: ubuntu@plana67.front.sepia.ceph.com - ['client.0'] ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2012-07-09_00:00:07-regression-next-testing-basic/7880$ tail teuthology.log 2012-07-09T01:00:51.060 INFO:teuthology.orchestra.run.out:137 17s 2012-07-09T01:01:48.502 INFO:teuthology.orchestra.run.out:138 56s 2012-07-09T01:02:50.892 INFO:teuthology.orchestra.run.out:139 60s 2012-07-09T01:03:40.139 INFO:teuthology.orchestra.run.out:140 48s 2012-07-09T01:03:45.886 INFO:teuthology.orchestra.run.out:141 4s 2012-07-09T01:03:51.603 INFO:teuthology.orchestra.run.out:166 3s 2012-07-09T01:03:59.320 INFO:teuthology.orchestra.run.out:169 6s 2012-07-09T01:04:58.436 INFO:teuthology.orchestra.run.out:179 58s 2012-07-09T01:05:25.113 INFO:teuthology.orchestra.run.out:182 24s 2012-07-09T01:05:33.988 INFO:teuthology.orchestra.run.out:183 6s
Actions