Project

General

Profile

Backport #19704

Updated by Nathan Cutler over 2 years ago

https://github.com/ceph/ceph/pull/14960 We're encountering this crash when nessus scans our gateways. I'll attempt to track down the exact request which is causing it, if that's useful.

<pre>
-23> 2017-04-19 18:50:47.985157 7f441c6fc700 1 ====== starting new request req=0x7f441c6f6340 =====
-22> 2017-04-19 18:50:47.985186 7f441c6fc700 2 req 12676657:0.000029::GET /::initializing for trans_id = tx000000000000000c16e31-0058f813f7-41ca13a-default
-21> 2017-04-19 18:50:47.985535 7f441c6fc700 2 req 12676657:0.000378:s3:GET /::getting op 0
-20> 2017-04-19 18:50:47.985546 7f441c6fc700 2 req 12676657:0.000389:s3:GET /:list_buckets:authorizing
-19> 2017-04-19 18:50:47.985552 7f441c6fc700 2 req 12676657:0.000395:s3:GET /:list_buckets:normalizing buckets and tenants
-18> 2017-04-19 18:50:47.985556 7f441c6fc700 2 req 12676657:0.000399:s3:GET /:list_buckets:init permissions
-17> 2017-04-19 18:50:47.985571 7f441c6fc700 2 req 12676657:0.000414:s3:GET /:list_buckets:recalculating target
-16> 2017-04-19 18:50:47.985577 7f441c6fc700 2 req 12676657:0.000419:s3:GET /:list_buckets:reading permissions
-15> 2017-04-19 18:50:47.985585 7f441c6fc700 2 req 12676657:0.000428:s3:GET /:list_buckets:init op
-14> 2017-04-19 18:50:47.985588 7f441c6fc700 2 req 12676657:0.000431:s3:GET /:list_buckets:verifying op mask
-13> 2017-04-19 18:50:47.985603 7f441c6fc700 2 req 12676657:0.000440:s3:GET /:list_buckets:verifying op permissions
-12> 2017-04-19 18:50:47.985613 7f441c6fc700 2 req 12676657:0.000456:s3:GET /:list_buckets:verifying op params
-11> 2017-04-19 18:50:47.985615 7f441c6fc700 2 req 12676657:0.000458:s3:GET /:list_buckets:pre-executing
-10> 2017-04-19 18:50:47.985617 7f441c6fc700 2 req 12676657:0.000460:s3:GET /:list_buckets:executing
-9> 2017-04-19 18:50:47.985692 7f441c6fc700 1 -- 10.30.1.13:0/1280116751 --> 10.30.1.124:6814/26162 -- osd_op(unknown.0.0:619994 7.d72ef9c4 anonymous.buckets [call user.list_buckets] snapc 0=[] ack+read+known_if_redirected e136543) v7 -- 0x7f45ac354000 con 0
-8> 2017-04-19 18:50:47.985719 7f441c6fc700 2 Event(0x7f459819f480 nevent=5000 time_id=304012).wakeup
-7> 2017-04-19 18:50:47.986499 7f457b9ba700 5 -- 10.30.1.13:0/1280116751 >> 10.30.1.124:6814/26162 conn(0x7f45ad4e7800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=197075 cs=1 l=1). rx osd.108 seq 76 0x7f45ac354000 osd_op_reply(619994 anonymous.buckets [call] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7
-6> 2017-04-19 18:50:47.986532 7f457b9ba700 1 -- 10.30.1.13:0/1280116751 <== osd.108 10.30.1.124:6814/26162 76 ==== osd_op_reply(619994 anonymous.buckets [call] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 137+0+0 (542233904 0 0) 0x7f45ac354000 con 0x7f45ad4e7800
-5> 2017-04-19 18:50:47.986706 7f441c6fc700 2 req 12676657:0.001548:s3:GET /:list_buckets:completing
-4> 2017-04-19 18:50:47.986745 7f441c6fc700 2 req 12676657:0.001588:s3:GET /:list_buckets:op status=0
-3> 2017-04-19 18:50:47.986751 7f441c6fc700 2 req 12676657:0.001594:s3:GET /:list_buckets:http status=200
-2> 2017-04-19 18:50:47.986757 7f441c6fc700 1 ====== req done req=0x7f441c6f6340 op status=0 http_status=200 ======
-1> 2017-04-19 18:50:47.986809 7f441c6fc700 1 civetweb: 0x7f45994d1000: 10.30.1.69 - - [19/Apr/2017:18:50:47 -0700] "GET / HTTP/1.1" 1 0 - Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
0> 2017-04-19 18:50:47.986759 7f4466790700 -1 *** Caught signal (Segmentation fault) **
in thread 7f4466790700 thread_name:civetweb-worker

ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (()+0x50bdca) [0x7f458e8e3dca]
2: (()+0xf100) [0x7f4583b41100]
3: (()+0x162961) [0x7f4582405961]
4: (()+0x2520f2) [0x7f458e62a0f2]
5: (()+0x253eb9) [0x7f458e62beb9]
6: (()+0x7dc5) [0x7f4583b39dc5]
7: (clone()+0x6d) [0x7f458239928d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/client.<>.log
--- end dump of recent events ---

</pre>

Back