2015-07-01 10:24:54.311787 7ff95edb57c0 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process radosgw, pid 26980 2015-07-01 10:24:54.793573 7ff95edb57c0 0 framework: fastcgi 2015-07-01 10:24:54.793693 7ff95edb57c0 0 starting handler: fastcgi 2015-07-01 10:24:54.815589 7ff92effd700 0 ERROR: can't read user header: ret=-2 2015-07-01 10:24:54.815604 7ff92effd700 0 ERROR: sync_user() failed, user=tw-west ret=-2 2015-07-01 10:25:22.237515 7ff927fef700 1 ====== starting new request req=0x7ff94000bfd0 ===== 2015-07-01 10:25:22.243321 7ff927fef700 1 ====== req done req=0x7ff94000bfd0 http_status=200 ====== 2015-07-01 10:34:38.656534 7ff9147c8700 1 ====== starting new request req=0x7ff94006a620 ===== 2015-07-01 10:34:38.660966 7ff9147c8700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:39.629302 7ff9147c8700 0 > HTTP_DATE -> Wed Jul 1 02:34:39 2015 2015-07-01 10:34:39.629304 7ff9147c8700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:39.733257 7ff9147c8700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:39.733353 7ff9147c8700 1 ====== req done req=0x7ff94006a620 http_status=500 ====== 2015-07-01 10:34:40.291928 7ff8fff9f700 1 ====== starting new request req=0x7ff94005b9e0 ===== 2015-07-01 10:34:40.292565 7ff8fff9f700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:40.304926 7ff8fff9f700 0 > HTTP_DATE -> Wed Jul 1 02:34:40 2015 2015-07-01 10:34:40.304928 7ff8fff9f700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:40.317349 7ff8fff9f700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:40.317401 7ff8fff9f700 1 ====== req done req=0x7ff94005b9e0 http_status=500 ====== 2015-07-01 10:34:41.903723 7ff9087b0700 1 ====== starting new request req=0x7ff940034db0 ===== 2015-07-01 10:34:41.904485 7ff9087b0700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:41.922114 7ff9087b0700 0 > HTTP_DATE -> Wed Jul 1 02:34:41 2015 2015-07-01 10:34:41.922118 7ff9087b0700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:41.931607 7ff9087b0700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:41.931678 7ff9087b0700 1 ====== req done req=0x7ff940034db0 http_status=500 ====== 2015-07-01 10:34:44.485520 7ff902fa5700 1 ====== starting new request req=0x7ff94000dde0 ===== 2015-07-01 10:34:44.486038 7ff902fa5700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:44.508509 7ff902fa5700 0 > HTTP_DATE -> Wed Jul 1 02:34:44 2015 2015-07-01 10:34:44.508511 7ff902fa5700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:44.549616 7ff902fa5700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:44.549777 7ff902fa5700 1 ====== req done req=0x7ff94000dde0 http_status=500 ====== 2015-07-01 10:34:46.394077 7ff918fd1700 1 ====== starting new request req=0x7ff9400397b0 ===== 2015-07-01 10:34:46.398290 7ff918fd1700 1 ====== req done req=0x7ff9400397b0 http_status=200 ====== 2015-07-01 10:34:46.404346 7ff9027a4700 1 ====== starting new request req=0x7ff940027840 ===== 2015-07-01 10:34:46.404853 7ff9027a4700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:46.482770 7ff9027a4700 0 > HTTP_DATE -> Wed Jul 1 02:34:46 2015 2015-07-01 10:34:46.482771 7ff9027a4700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:46.552883 7ff9027a4700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:46.552971 7ff9027a4700 1 ====== req done req=0x7ff940027840 http_status=500 ====== 2015-07-01 10:34:47.550229 7ff90d7ba700 1 ====== starting new request req=0x7ff940015950 ===== 2015-07-01 10:34:47.550688 7ff90d7ba700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:47.558518 7ff90d7ba700 0 > HTTP_DATE -> Wed Jul 1 02:34:47 2015 2015-07-01 10:34:47.558520 7ff90d7ba700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:47.566081 7ff90d7ba700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:47.566126 7ff90d7ba700 1 ====== req done req=0x7ff940015950 http_status=500 ====== 2015-07-01 10:34:48.932688 7ff913fc7700 1 ====== starting new request req=0x7ff94001c370 ===== 2015-07-01 10:34:48.933439 7ff913fc7700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:48.946175 7ff913fc7700 0 > HTTP_DATE -> Wed Jul 1 02:34:48 2015 2015-07-01 10:34:48.946179 7ff913fc7700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:48.956077 7ff913fc7700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:48.956261 7ff913fc7700 1 ====== req done req=0x7ff94001c370 http_status=500 ====== 2015-07-01 10:34:51.904977 7ff929ff3700 1 ====== starting new request req=0x7ff94005b9e0 ===== 2015-07-01 10:34:51.905556 7ff929ff3700 0 WARNING: couldn't find acl header for bucket, generating default 2015-07-01 10:34:51.922573 7ff929ff3700 0 > HTTP_DATE -> Wed Jul 1 02:34:51 2015 2015-07-01 10:34:51.922575 7ff929ff3700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json 2015-07-01 10:34:51.932717 7ff929ff3700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2015-07-01 10:34:51.932838 7ff929ff3700 1 ====== req done req=0x7ff94005b9e0 http_status=500 ====== 2015-07-01 10:34:54.096336 7ff91c7d8700 1 ====== starting new request req=0x7ff94000dde0 ===== 2015-07-01 10:34:54.099972 7ff91c7d8700 1 ====== req done req=0x7ff94000dde0 http_status=200 ====== 2015-07-01 10:34:54.112960 7ff91b7d6700 1 ====== starting new request req=0x7ff940034db0 ===== 2015-07-01 10:34:54.116891 7ff91b7d6700 -1 *** Caught signal (Aborted) ** in thread 7ff91b7d6700 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047) 1: (()+0x1f18af) [0x7ff95efcb8af] 2: (()+0x10340) [0x7ff95cfcd340] 3: (gsignal()+0x39) [0x7ff95c0d4cc9] 4: (abort()+0x148) [0x7ff95c0d80d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff95c9df535] 6: (()+0x5e6d6) [0x7ff95c9dd6d6] 7: (()+0x5e703) [0x7ff95c9dd703] 8: (()+0x5e922) [0x7ff95c9dd922] 9: (bool JSONDecoder::decode_json(char const*, utime_t&, JSONObj*, bool)+0x204) [0x7ff95eed35b4] 10: (void decode_json_obj(std::list >&, JSONObj*)+0x73) [0x7ff95eea9e13] 11: (int rgw_rest_get_json_input > >(CephContext*, req_state*, std::list >&, int, bool*)+0xa5) [0x7ff95eea9f75] 12: (RGWOp_BILog_SetBounds::execute()+0x390) [0x7ff95eea8e20] 13: (()+0xde63d) [0x7ff95eeb863d] 14: (RGWFCGXProcess::handle_request(RGWRequest*)+0x80) [0x7ff95eeb8d30] 15: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x30) [0x7ff95eeba340] 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0xaf1) [0x7ff95deca531] 17: (ThreadPool::WorkThread::entry()+0x10) [0x7ff95decb420] 18: (()+0x8182) [0x7ff95cfc5182] 19: (clone()+0x6d) [0x7ff95c19847d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -9999> 2015-07-01 10:33:49.893762 7ff9067ac700 2 req 6306:0.000109::POST /admin/log:unlock_datalog_object:authorizing -9998> 2015-07-01 10:33:49.893948 7ff9067ac700 2 req 6306:0.000294::POST /admin/log:unlock_datalog_object:reading permissions -9997> 2015-07-01 10:33:49.893954 7ff9067ac700 2 req 6306:0.000301::POST /admin/log:unlock_datalog_object:init op -9996> 2015-07-01 10:33:49.893957 7ff9067ac700 2 req 6306:0.000303::POST /admin/log:unlock_datalog_object:verifying op mask -9995> 2015-07-01 10:33:49.893959 7ff9067ac700 2 req 6306:0.000306::POST /admin/log:unlock_datalog_object:verifying op permissions -9994> 2015-07-01 10:33:49.893962 7ff9067ac700 2 overriding permissions due to system operation -9993> 2015-07-01 10:33:49.893963 7ff9067ac700 2 req 6306:0.000310::POST /admin/log:unlock_datalog_object:verifying op params -9992> 2015-07-01 10:33:49.893965 7ff9067ac700 2 req 6306:0.000312::POST /admin/log:unlock_datalog_object:executing -9991> 2015-07-01 10:33:49.894001 7ff9067ac700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:6478 data_log.106 [call lock.unlock] 59.d2dd3d41 ondisk+write e425) v4 -- ?+0 0x7ff950021c70 con 0x7ff95f92e520 -9990> 2015-07-01 10:33:49.902968 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2522 ==== osd_op_reply(6478 data_log.106 [call] v425'3938 uv3938 ondisk = 0) v6 ==== 179+0+0 (2424285253 0 0) 0x7ff9380101b0 con 0x7ff95f92e520 -9989> 2015-07-01 10:33:49.903068 7ff9067ac700 2 req 6306:0.009414::POST /admin/log:unlock_datalog_object:http status=200 -9988> 2015-07-01 10:33:49.903076 7ff9067ac700 1 ====== req done req=0x7ff940028200 http_status=200 ====== -307> 2015-07-01 10:34:38.588833 7ff91efdd700 1 ====== starting new request req=0x7ff94000dde0 ===== -306> 2015-07-01 10:34:38.588850 7ff91efdd700 2 req 6949:0.000017::GET /admin/replica_log::initializing -305> 2015-07-01 10:34:38.588877 7ff91efdd700 2 req 6949:0.000045::GET /admin/replica_log::getting op -304> 2015-07-01 10:34:38.588880 7ff91efdd700 2 req 6949:0.000048::GET /admin/replica_log:replicabilog_getbounds:authorizing -303> 2015-07-01 10:34:38.588998 7ff91efdd700 2 req 6949:0.000166::GET /admin/replica_log:replicabilog_getbounds:reading permissions -302> 2015-07-01 10:34:38.589003 7ff91efdd700 2 req 6949:0.000171::GET /admin/replica_log:replicabilog_getbounds:init op -301> 2015-07-01 10:34:38.589005 7ff91efdd700 2 req 6949:0.000172::GET /admin/replica_log:replicabilog_getbounds:verifying op mask -300> 2015-07-01 10:34:38.589006 7ff91efdd700 2 req 6949:0.000174::GET /admin/replica_log:replicabilog_getbounds:verifying op permissions -299> 2015-07-01 10:34:38.589008 7ff91efdd700 2 overriding permissions due to system operation -298> 2015-07-01 10:34:38.589009 7ff91efdd700 2 req 6949:0.000177::GET /admin/replica_log:replicabilog_getbounds:verifying op params -297> 2015-07-01 10:34:38.589011 7ff91efdd700 2 req 6949:0.000179::GET /admin/replica_log:replicabilog_getbounds:executing -296> 2015-07-01 10:34:38.589071 7ff91efdd700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7121 replica_log.BASKET1 [call replica_log.get] 59.933a1b8d ack+read e425) v4 -- ?+0 0x7ff93c042930 con 0x7ff95f9302e0 -295> 2015-07-01 10:34:38.590105 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1797 ==== osd_op_reply(7121 replica_log.BASKET1 [call] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 186+0+0 (2444263206 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -294> 2015-07-01 10:34:38.590208 7ff91efdd700 2 req 6949:0.001376::GET /admin/replica_log:replicabilog_getbounds:http status=404 -293> 2015-07-01 10:34:38.590214 7ff91efdd700 1 ====== req done req=0x7ff94000dde0 http_status=404 ====== -292> 2015-07-01 10:34:38.612810 7ff91e7dc700 1 ====== starting new request req=0x7ff9400261b0 ===== -291> 2015-07-01 10:34:38.612847 7ff91e7dc700 2 req 6950:0.000036::GET /admin/replica_log::initializing -290> 2015-07-01 10:34:38.612899 7ff91e7dc700 2 req 6950:0.000089::GET /admin/replica_log::getting op -289> 2015-07-01 10:34:38.612905 7ff91e7dc700 2 req 6950:0.000094::GET /admin/replica_log:replicabilog_getbounds:authorizing -288> 2015-07-01 10:34:38.613124 7ff91e7dc700 2 req 6950:0.000313::GET /admin/replica_log:replicabilog_getbounds:reading permissions -287> 2015-07-01 10:34:38.613134 7ff91e7dc700 2 req 6950:0.000324::GET /admin/replica_log:replicabilog_getbounds:init op -286> 2015-07-01 10:34:38.613137 7ff91e7dc700 2 req 6950:0.000327::GET /admin/replica_log:replicabilog_getbounds:verifying op mask -285> 2015-07-01 10:34:38.613140 7ff91e7dc700 2 req 6950:0.000330::GET /admin/replica_log:replicabilog_getbounds:verifying op permissions -284> 2015-07-01 10:34:38.613145 7ff91e7dc700 2 overriding permissions due to system operation -283> 2015-07-01 10:34:38.613148 7ff91e7dc700 2 req 6950:0.000338::GET /admin/replica_log:replicabilog_getbounds:verifying op params -282> 2015-07-01 10:34:38.613151 7ff91e7dc700 2 req 6950:0.000341::GET /admin/replica_log:replicabilog_getbounds:executing -281> 2015-07-01 10:34:38.613255 7ff91e7dc700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7122 replica_log.BASKET1 [call replica_log.get] 59.933a1b8d ack+read e425) v4 -- ?+0 0x7ff948045710 con 0x7ff95f9302e0 -280> 2015-07-01 10:34:38.614654 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1798 ==== osd_op_reply(7122 replica_log.BASKET1 [call] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 186+0+0 (2444263206 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -279> 2015-07-01 10:34:38.614788 7ff91e7dc700 2 req 6950:0.001978::GET /admin/replica_log:replicabilog_getbounds:http status=404 -278> 2015-07-01 10:34:38.614796 7ff91e7dc700 1 ====== req done req=0x7ff9400261b0 http_status=404 ====== -277> 2015-07-01 10:34:38.635376 7ff9057aa700 1 ====== starting new request req=0x7ff9400397b0 ===== -276> 2015-07-01 10:34:38.635411 7ff9057aa700 2 req 6951:0.000035::GET /admin/replica_log::initializing -275> 2015-07-01 10:34:38.635460 7ff9057aa700 2 req 6951:0.000084::GET /admin/replica_log::getting op -274> 2015-07-01 10:34:38.635466 7ff9057aa700 2 req 6951:0.000091::GET /admin/replica_log:replicabilog_getbounds:authorizing -273> 2015-07-01 10:34:38.635680 7ff9057aa700 2 req 6951:0.000304::GET /admin/replica_log:replicabilog_getbounds:reading permissions -272> 2015-07-01 10:34:38.635690 7ff9057aa700 2 req 6951:0.000315::GET /admin/replica_log:replicabilog_getbounds:init op -271> 2015-07-01 10:34:38.635693 7ff9057aa700 2 req 6951:0.000318::GET /admin/replica_log:replicabilog_getbounds:verifying op mask -270> 2015-07-01 10:34:38.635697 7ff9057aa700 2 req 6951:0.000322::GET /admin/replica_log:replicabilog_getbounds:verifying op permissions -269> 2015-07-01 10:34:38.635700 7ff9057aa700 2 overriding permissions due to system operation -268> 2015-07-01 10:34:38.635702 7ff9057aa700 2 req 6951:0.000327::GET /admin/replica_log:replicabilog_getbounds:verifying op params -267> 2015-07-01 10:34:38.635705 7ff9057aa700 2 req 6951:0.000330::GET /admin/replica_log:replicabilog_getbounds:executing -266> 2015-07-01 10:34:38.635800 7ff9057aa700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.75:6800/21154 -- osd_op(client.7211.0:7123 .bucket.meta.BASKET1:tw-east.6930.1 [call version.read,getxattrs,stat] 55.b017dbf8 ack+read e425) v4 -- ?+0 0x7ff93403d9b0 con 0x7ff94405f860 -265> 2015-07-01 10:34:38.638411 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.4 10.35.64.75:6800/21154 12 ==== osd_op_reply(7123 .bucket.meta.BASKET1:tw-east.6930.1 [call,getxattrs,stat] v0'0 uv3 ondisk = 0) v6 ==== 286+0+322 (3643147009 0 1252381838) 0x7ff94006a1e0 con 0x7ff94405f860 -264> 2015-07-01 10:34:38.638614 7ff9057aa700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7124 replica_log.BASKET1 [call replica_log.get] 59.933a1b8d ack+read e425) v4 -- ?+0 0x7ff934018ca0 con 0x7ff95f9302e0 -263> 2015-07-01 10:34:38.640517 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1799 ==== osd_op_reply(7124 replica_log.BASKET1 [call] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 186+0+0 (2444263206 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -262> 2015-07-01 10:34:38.640644 7ff9057aa700 2 req 6951:0.005269::GET /admin/replica_log:replicabilog_getbounds:http status=404 -261> 2015-07-01 10:34:38.640652 7ff9057aa700 1 ====== req done req=0x7ff9400397b0 http_status=404 ====== -260> 2015-07-01 10:34:38.656534 7ff9147c8700 1 ====== starting new request req=0x7ff94006a620 ===== -259> 2015-07-01 10:34:38.656585 7ff9147c8700 2 req 6952:0.000052::PUT /BASKET1/tw.json::initializing -258> 2015-07-01 10:34:38.656669 7ff9147c8700 2 req 6952:0.000136:s3:PUT /BASKET1/tw.json::getting op -257> 2015-07-01 10:34:38.656705 7ff9147c8700 2 req 6952:0.000173:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -256> 2015-07-01 10:34:38.656993 7ff9147c8700 2 req 6952:0.000460:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -255> 2015-07-01 10:34:38.657131 7ff9147c8700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7125 BASKET1 [call version.read,getxattrs,stat] 55.385b1c1c ack+read e425) v4 -- ?+0 0x7ff93c042930 con 0x7ff95f92e520 -254> 2015-07-01 10:34:38.660558 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2779 ==== osd_op_reply(7125 BASKET1 [call,getxattrs,stat] v0'0 uv6 ondisk = 0) v6 ==== 258+0+139 (3447838219 0 2364625330) 0x7ff938036ec0 con 0x7ff95f92e520 -253> 2015-07-01 10:34:38.660871 7ff9147c8700 2 req 6952:0.004338:s3:PUT /BASKET1/tw.json:copy_obj:init op -252> 2015-07-01 10:34:38.660883 7ff9147c8700 2 req 6952:0.004351:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -251> 2015-07-01 10:34:38.660886 7ff9147c8700 2 req 6952:0.004354:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -250> 2015-07-01 10:34:38.660966 7ff9147c8700 0 WARNING: couldn't find acl header for bucket, generating default -249> 2015-07-01 10:34:38.661032 7ff9147c8700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7126 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e425) v4 -- ?+0 0x7ff93c042930 con 0x7ff95f9302e0 -248> 2015-07-01 10:34:38.662749 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1800 ==== osd_op_reply(7126 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (535621043 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -247> 2015-07-01 10:34:38.662928 7ff9147c8700 2 overriding permissions due to system operation -246> 2015-07-01 10:34:38.662936 7ff9147c8700 2 req 6952:0.006404:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -245> 2015-07-01 10:34:38.662943 7ff9147c8700 2 req 6952:0.006411:s3:PUT /BASKET1/tw.json:copy_obj:executing -244> 2015-07-01 10:34:38.663271 7ff9147c8700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -243> 2015-07-01 10:34:38.663347 7ff9147c8700 10 monclient: _send_mon_message to mon.node3 at 10.35.64.73:6789/0 -242> 2015-07-01 10:34:38.663358 7ff9147c8700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6789/0 -- pool_op(create pool 0 auid 0 tid 7127 name .rgw.buckets v424) v4 -- ?+0 0x7ff93c012540 con 0x7ff95f926bf0 -241> 2015-07-01 10:34:39.380079 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.72:6800/1502 -- ping v1 -- ?+0 0x7ff93000d760 con 0x7ff95f9240b0 -240> 2015-07-01 10:34:39.380131 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- ping v1 -- ?+0 0x7ff93001cb20 con 0x7ff95f9302e0 -239> 2015-07-01 10:34:39.380151 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.74:6800/27167 -- ping v1 -- ?+0 0x7ff93001ccc0 con 0x7ff95f934520 -238> 2015-07-01 10:34:39.380170 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- ping v1 -- ?+0 0x7ff930045eb0 con 0x7ff95f92e520 -237> 2015-07-01 10:34:39.397337 7ff954816700 1 -- 10.35.64.71:0/1026982 <== mon.2 10.35.64.73:6789/0 18 ==== osd_map(426..426 src has 1..426) v3 ==== 417+0+0 (3467348124 0 0) 0x7ff940034240 con 0x7ff95f926bf0 -236> 2015-07-01 10:34:39.397490 7ff954816700 1 -- 10.35.64.71:0/1026982 <== mon.2 10.35.64.73:6789/0 19 ==== pool_op_reply(tid 7127 (0) Success v426) v1 ==== 43+0+0 (3279357725 0 0) 0x7ff94001fde0 con 0x7ff95f926bf0 -235> 2015-07-01 10:34:39.397829 7ff9147c8700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7128 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e426) v4 -- ?+0 0x7ff93c042930 con 0x7ff95f92e520 -234> 2015-07-01 10:34:39.629124 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2780 ==== osd_op_reply(7128 statelog.obj_opstate.68 [call] v426'4701 uv4701 ondisk = 0) v6 ==== 190+0+0 (1892154604 0 0) 0x7ff938052680 con 0x7ff95f92e520 -233> 2015-07-01 10:34:39.629302 7ff9147c8700 0 > HTTP_DATE -> Wed Jul 1 02:34:39 2015 -232> 2015-07-01 10:34:39.629304 7ff9147c8700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -231> 2015-07-01 10:34:39.631885 7ff9147c8700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7129 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e426) v4 -- ?+0 0x7ff93c042930 con 0x7ff95f92e520 -230> 2015-07-01 10:34:39.733125 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2781 ==== osd_op_reply(7129 statelog.obj_opstate.68 [call] v426'4702 uv4702 ondisk = 0) v6 ==== 190+0+0 (3841990531 0 0) 0x7ff938052680 con 0x7ff95f92e520 -229> 2015-07-01 10:34:39.733257 7ff9147c8700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -228> 2015-07-01 10:34:39.733334 7ff9147c8700 2 req 6952:1.076802:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -227> 2015-07-01 10:34:39.733353 7ff9147c8700 1 ====== req done req=0x7ff94006a620 http_status=500 ====== -226> 2015-07-01 10:34:40.291928 7ff8fff9f700 1 ====== starting new request req=0x7ff94005b9e0 ===== -225> 2015-07-01 10:34:40.291975 7ff8fff9f700 2 req 6953:0.000048::PUT /BASKET1/tw.json::initializing -224> 2015-07-01 10:34:40.292056 7ff8fff9f700 2 req 6953:0.000129:s3:PUT /BASKET1/tw.json::getting op -223> 2015-07-01 10:34:40.292091 7ff8fff9f700 2 req 6953:0.000164:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -222> 2015-07-01 10:34:40.292376 7ff8fff9f700 2 req 6953:0.000449:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -221> 2015-07-01 10:34:40.292507 7ff8fff9f700 2 req 6953:0.000580:s3:PUT /BASKET1/tw.json:copy_obj:init op -220> 2015-07-01 10:34:40.292513 7ff8fff9f700 2 req 6953:0.000586:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -219> 2015-07-01 10:34:40.292516 7ff8fff9f700 2 req 6953:0.000588:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -218> 2015-07-01 10:34:40.292565 7ff8fff9f700 0 WARNING: couldn't find acl header for bucket, generating default -217> 2015-07-01 10:34:40.292654 7ff8fff9f700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7130 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e426) v4 -- ?+0 0x7ff93c012540 con 0x7ff95f9302e0 -216> 2015-07-01 10:34:40.294338 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1801 ==== osd_op_reply(7130 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (1239691441 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -215> 2015-07-01 10:34:40.294503 7ff8fff9f700 2 overriding permissions due to system operation -214> 2015-07-01 10:34:40.294509 7ff8fff9f700 2 req 6953:0.002582:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -213> 2015-07-01 10:34:40.294514 7ff8fff9f700 2 req 6953:0.002587:s3:PUT /BASKET1/tw.json:copy_obj:executing -212> 2015-07-01 10:34:40.294579 7ff8fff9f700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -211> 2015-07-01 10:34:40.294686 7ff8fff9f700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7131 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e426) v4 -- ?+0 0x7ff93c012540 con 0x7ff95f92e520 -210> 2015-07-01 10:34:40.304814 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2782 ==== osd_op_reply(7131 statelog.obj_opstate.68 [call] v426'4703 uv4703 ondisk = 0) v6 ==== 190+0+0 (2520956582 0 0) 0x7ff938052680 con 0x7ff95f92e520 -209> 2015-07-01 10:34:40.304926 7ff8fff9f700 0 > HTTP_DATE -> Wed Jul 1 02:34:40 2015 -208> 2015-07-01 10:34:40.304928 7ff8fff9f700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -207> 2015-07-01 10:34:40.307663 7ff8fff9f700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7132 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e426) v4 -- ?+0 0x7ff93c012540 con 0x7ff95f92e520 -206> 2015-07-01 10:34:40.317280 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2783 ==== osd_op_reply(7132 statelog.obj_opstate.68 [call] v426'4704 uv4704 ondisk = 0) v6 ==== 190+0+0 (699648542 0 0) 0x7ff938052680 con 0x7ff95f92e520 -205> 2015-07-01 10:34:40.317349 7ff8fff9f700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -204> 2015-07-01 10:34:40.317385 7ff8fff9f700 2 req 6953:0.025458:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -203> 2015-07-01 10:34:40.317401 7ff8fff9f700 1 ====== req done req=0x7ff94005b9e0 http_status=500 ====== -202> 2015-07-01 10:34:41.903723 7ff9087b0700 1 ====== starting new request req=0x7ff940034db0 ===== -201> 2015-07-01 10:34:41.903778 7ff9087b0700 2 req 6954:0.000057::PUT /BASKET1/tw.json::initializing -200> 2015-07-01 10:34:41.903880 7ff9087b0700 2 req 6954:0.000159:s3:PUT /BASKET1/tw.json::getting op -199> 2015-07-01 10:34:41.903892 7ff9087b0700 2 req 6954:0.000171:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -198> 2015-07-01 10:34:41.904225 7ff9087b0700 2 req 6954:0.000504:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -197> 2015-07-01 10:34:41.904406 7ff9087b0700 2 req 6954:0.000685:s3:PUT /BASKET1/tw.json:copy_obj:init op -196> 2015-07-01 10:34:41.904414 7ff9087b0700 2 req 6954:0.000693:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -195> 2015-07-01 10:34:41.904418 7ff9087b0700 2 req 6954:0.000697:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -194> 2015-07-01 10:34:41.904485 7ff9087b0700 0 WARNING: couldn't find acl header for bucket, generating default -193> 2015-07-01 10:34:41.904597 7ff9087b0700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7133 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e426) v4 -- ?+0 0x7ff9380569a0 con 0x7ff95f9302e0 -192> 2015-07-01 10:34:41.906680 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1802 ==== osd_map(427..427 src has 1..427) v3 ==== 238+0+0 (1866214008 0 0) 0x7ff95f97d180 con 0x7ff95f9302e0 -191> 2015-07-01 10:34:41.906838 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1803 ==== osd_op_reply(7133 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (2078508111 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -190> 2015-07-01 10:34:41.907032 7ff9087b0700 2 overriding permissions due to system operation -189> 2015-07-01 10:34:41.907041 7ff9087b0700 2 req 6954:0.003319:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -188> 2015-07-01 10:34:41.907047 7ff9087b0700 2 req 6954:0.003326:s3:PUT /BASKET1/tw.json:copy_obj:executing -187> 2015-07-01 10:34:41.907338 7ff9087b0700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -186> 2015-07-01 10:34:41.907508 7ff9087b0700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7134 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff93805c050 con 0x7ff95f92e520 -185> 2015-07-01 10:34:41.921940 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2784 ==== osd_op_reply(7134 statelog.obj_opstate.68 [call] v427'4705 uv4705 ondisk = 0) v6 ==== 190+0+0 (794285493 0 0) 0x7ff93805c050 con 0x7ff95f92e520 -184> 2015-07-01 10:34:41.922114 7ff9087b0700 0 > HTTP_DATE -> Wed Jul 1 02:34:41 2015 -183> 2015-07-01 10:34:41.922118 7ff9087b0700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -182> 2015-07-01 10:34:41.925432 7ff9087b0700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7135 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff938048240 con 0x7ff95f92e520 -181> 2015-07-01 10:34:41.931486 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2785 ==== osd_op_reply(7135 statelog.obj_opstate.68 [call] v427'4706 uv4706 ondisk = 0) v6 ==== 190+0+0 (3130013402 0 0) 0x7ff938056780 con 0x7ff95f92e520 -180> 2015-07-01 10:34:41.931607 7ff9087b0700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -179> 2015-07-01 10:34:41.931666 7ff9087b0700 2 req 6954:0.027945:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -178> 2015-07-01 10:34:41.931678 7ff9087b0700 1 ====== req done req=0x7ff940034db0 http_status=500 ====== -177> 2015-07-01 10:34:44.380483 7ff94e6fb700 10 monclient: renew_subs -176> 2015-07-01 10:34:44.380531 7ff94e6fb700 10 monclient: _send_mon_message to mon.node3 at 10.35.64.73:6789/0 -175> 2015-07-01 10:34:44.380558 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6789/0 -- mon_subscribe({monmap=2+,osdmap=428}) v2 -- ?+0 0x7ff93000d760 con 0x7ff95f926bf0 -174> 2015-07-01 10:34:44.380597 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.72:6800/1502 -- ping v1 -- ?+0 0x7ff93001cb20 con 0x7ff95f9240b0 -173> 2015-07-01 10:34:44.380617 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- ping v1 -- ?+0 0x7ff93001ccc0 con 0x7ff95f9302e0 -172> 2015-07-01 10:34:44.380634 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.74:6800/27167 -- ping v1 -- ?+0 0x7ff930045eb0 con 0x7ff95f934520 -171> 2015-07-01 10:34:44.380652 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- ping v1 -- ?+0 0x7ff9300460d0 con 0x7ff95f92e520 -170> 2015-07-01 10:34:44.383326 7ff954816700 1 -- 10.35.64.71:0/1026982 <== mon.2 10.35.64.73:6789/0 20 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2045553213 0 0) 0x7ff94001c4a0 con 0x7ff95f926bf0 -169> 2015-07-01 10:34:44.383361 7ff954816700 10 monclient: handle_subscribe_ack sent 2015-07-01 10:34:44.380522 renew after 2015-07-01 10:37:14.380522 -168> 2015-07-01 10:34:44.485520 7ff902fa5700 1 ====== starting new request req=0x7ff94000dde0 ===== -167> 2015-07-01 10:34:44.485567 7ff902fa5700 2 req 6955:0.000048::PUT /BASKET1/tw.json::initializing -166> 2015-07-01 10:34:44.485636 7ff902fa5700 2 req 6955:0.000117:s3:PUT /BASKET1/tw.json::getting op -165> 2015-07-01 10:34:44.485643 7ff902fa5700 2 req 6955:0.000123:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -164> 2015-07-01 10:34:44.485871 7ff902fa5700 2 req 6955:0.000351:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -163> 2015-07-01 10:34:44.485987 7ff902fa5700 2 req 6955:0.000468:s3:PUT /BASKET1/tw.json:copy_obj:init op -162> 2015-07-01 10:34:44.485992 7ff902fa5700 2 req 6955:0.000473:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -161> 2015-07-01 10:34:44.485994 7ff902fa5700 2 req 6955:0.000475:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -160> 2015-07-01 10:34:44.486038 7ff902fa5700 0 WARNING: couldn't find acl header for bucket, generating default -159> 2015-07-01 10:34:44.486109 7ff902fa5700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7136 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e427) v4 -- ?+0 0x7ff93c012540 con 0x7ff95f9302e0 -158> 2015-07-01 10:34:44.487426 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1804 ==== osd_op_reply(7136 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (2078508111 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -157> 2015-07-01 10:34:44.487563 7ff902fa5700 2 overriding permissions due to system operation -156> 2015-07-01 10:34:44.487569 7ff902fa5700 2 req 6955:0.002050:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -155> 2015-07-01 10:34:44.487573 7ff902fa5700 2 req 6955:0.002054:s3:PUT /BASKET1/tw.json:copy_obj:executing -154> 2015-07-01 10:34:44.487639 7ff902fa5700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -153> 2015-07-01 10:34:44.487732 7ff902fa5700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7137 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff93c014ff0 con 0x7ff95f92e520 -152> 2015-07-01 10:34:44.508404 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2786 ==== osd_op_reply(7137 statelog.obj_opstate.68 [call] v427'4707 uv4707 ondisk = 0) v6 ==== 190+0+0 (3386021887 0 0) 0x7ff938056b10 con 0x7ff95f92e520 -151> 2015-07-01 10:34:44.508509 7ff902fa5700 0 > HTTP_DATE -> Wed Jul 1 02:34:44 2015 -150> 2015-07-01 10:34:44.508511 7ff902fa5700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -149> 2015-07-01 10:34:44.512194 7ff902fa5700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7138 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff93c012540 con 0x7ff95f92e520 -148> 2015-07-01 10:34:44.549538 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2787 ==== osd_op_reply(7138 statelog.obj_opstate.68 [call] v427'4708 uv4708 ondisk = 0) v6 ==== 190+0+0 (2498987765 0 0) 0x7ff938052680 con 0x7ff95f92e520 -147> 2015-07-01 10:34:44.549616 7ff902fa5700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -146> 2015-07-01 10:34:44.549737 7ff902fa5700 2 req 6955:0.064218:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -145> 2015-07-01 10:34:44.549777 7ff902fa5700 1 ====== req done req=0x7ff94000dde0 http_status=500 ====== -144> 2015-07-01 10:34:46.394077 7ff918fd1700 1 ====== starting new request req=0x7ff9400397b0 ===== -143> 2015-07-01 10:34:46.394144 7ff918fd1700 2 req 6956:0.000069::GET /admin/opstate::initializing -142> 2015-07-01 10:34:46.394220 7ff918fd1700 2 req 6956:0.000145::GET /admin/opstate::getting op -141> 2015-07-01 10:34:46.394226 7ff918fd1700 2 req 6956:0.000151::GET /admin/opstate:opstate_list:authorizing -140> 2015-07-01 10:34:46.394567 7ff918fd1700 2 req 6956:0.000491::GET /admin/opstate:opstate_list:reading permissions -139> 2015-07-01 10:34:46.394576 7ff918fd1700 2 req 6956:0.000501::GET /admin/opstate:opstate_list:init op -138> 2015-07-01 10:34:46.394580 7ff918fd1700 2 req 6956:0.000505::GET /admin/opstate:opstate_list:verifying op mask -137> 2015-07-01 10:34:46.394583 7ff918fd1700 2 req 6956:0.000508::GET /admin/opstate:opstate_list:verifying op permissions -136> 2015-07-01 10:34:46.394588 7ff918fd1700 2 overriding permissions due to system operation -135> 2015-07-01 10:34:46.394590 7ff918fd1700 2 req 6956:0.000515::GET /admin/opstate:opstate_list:verifying op params -134> 2015-07-01 10:34:46.394593 7ff918fd1700 2 req 6956:0.000518::GET /admin/opstate:opstate_list:executing -133> 2015-07-01 10:34:46.394725 7ff918fd1700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7139 statelog.obj_opstate.68 [call statelog.list] 59.868ada28 ack+read e427) v4 -- ?+0 0x7ff93403d540 con 0x7ff95f92e520 -132> 2015-07-01 10:34:46.398127 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2788 ==== osd_op_reply(7139 statelog.obj_opstate.68 [call] v0'0 uv4708 ondisk = 0) v6 ==== 190+0+90 (366987647 0 1510955586) 0x7ff938052680 con 0x7ff95f92e520 -131> 2015-07-01 10:34:46.398283 7ff918fd1700 2 req 6956:0.004208::GET /admin/opstate:opstate_list:http status=200 -130> 2015-07-01 10:34:46.398290 7ff918fd1700 1 ====== req done req=0x7ff9400397b0 http_status=200 ====== -129> 2015-07-01 10:34:46.404346 7ff9027a4700 1 ====== starting new request req=0x7ff940027840 ===== -128> 2015-07-01 10:34:46.404375 7ff9027a4700 2 req 6957:0.000030::PUT /BASKET1/tw.json::initializing -127> 2015-07-01 10:34:46.404435 7ff9027a4700 2 req 6957:0.000090:s3:PUT /BASKET1/tw.json::getting op -126> 2015-07-01 10:34:46.404451 7ff9027a4700 2 req 6957:0.000106:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -125> 2015-07-01 10:34:46.404640 7ff9027a4700 2 req 6957:0.000295:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -124> 2015-07-01 10:34:46.404789 7ff9027a4700 2 req 6957:0.000443:s3:PUT /BASKET1/tw.json:copy_obj:init op -123> 2015-07-01 10:34:46.404795 7ff9027a4700 2 req 6957:0.000449:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -122> 2015-07-01 10:34:46.404798 7ff9027a4700 2 req 6957:0.000453:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -121> 2015-07-01 10:34:46.404853 7ff9027a4700 0 WARNING: couldn't find acl header for bucket, generating default -120> 2015-07-01 10:34:46.404914 7ff9027a4700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7140 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e427) v4 -- ?+0 0x7ff950019e10 con 0x7ff95f9302e0 -119> 2015-07-01 10:34:46.406435 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1805 ==== osd_op_reply(7140 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (2078508111 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -118> 2015-07-01 10:34:46.406592 7ff9027a4700 2 overriding permissions due to system operation -117> 2015-07-01 10:34:46.406598 7ff9027a4700 2 req 6957:0.002252:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -116> 2015-07-01 10:34:46.406603 7ff9027a4700 2 req 6957:0.002258:s3:PUT /BASKET1/tw.json:copy_obj:executing -115> 2015-07-01 10:34:46.406682 7ff9027a4700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -114> 2015-07-01 10:34:46.406799 7ff9027a4700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7141 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff950019e10 con 0x7ff95f92e520 -113> 2015-07-01 10:34:46.482671 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2789 ==== osd_op_reply(7141 statelog.obj_opstate.68 [call] v427'4709 uv4709 ondisk = 0) v6 ==== 190+0+0 (3887146960 0 0) 0x7ff938052680 con 0x7ff95f92e520 -112> 2015-07-01 10:34:46.482770 7ff9027a4700 0 > HTTP_DATE -> Wed Jul 1 02:34:46 2015 -111> 2015-07-01 10:34:46.482771 7ff9027a4700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -110> 2015-07-01 10:34:46.484974 7ff9027a4700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7142 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff950019e10 con 0x7ff95f92e520 -109> 2015-07-01 10:34:46.552722 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2790 ==== osd_op_reply(7142 statelog.obj_opstate.68 [call] v427'4710 uv4710 ondisk = 0) v6 ==== 190+0+0 (1920386239 0 0) 0x7ff938052680 con 0x7ff95f92e520 -108> 2015-07-01 10:34:46.552883 7ff9027a4700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -107> 2015-07-01 10:34:46.552950 7ff9027a4700 2 req 6957:0.148604:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -106> 2015-07-01 10:34:46.552971 7ff9027a4700 1 ====== req done req=0x7ff940027840 http_status=500 ====== -105> 2015-07-01 10:34:47.347840 7ff94f7fe700 10 monclient: tick -104> 2015-07-01 10:34:47.347907 7ff94f7fe700 10 monclient: renew subs? (now: 2015-07-01 10:34:47.347907; renew after: 2015-07-01 10:37:14.380522) -- no -103> 2015-07-01 10:34:47.550229 7ff90d7ba700 1 ====== starting new request req=0x7ff940015950 ===== -102> 2015-07-01 10:34:47.550275 7ff90d7ba700 2 req 6958:0.000046::PUT /BASKET1/tw.json::initializing -101> 2015-07-01 10:34:47.550333 7ff90d7ba700 2 req 6958:0.000104:s3:PUT /BASKET1/tw.json::getting op -100> 2015-07-01 10:34:47.550340 7ff90d7ba700 2 req 6958:0.000111:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -99> 2015-07-01 10:34:47.550544 7ff90d7ba700 2 req 6958:0.000315:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -98> 2015-07-01 10:34:47.550644 7ff90d7ba700 2 req 6958:0.000415:s3:PUT /BASKET1/tw.json:copy_obj:init op -97> 2015-07-01 10:34:47.550648 7ff90d7ba700 2 req 6958:0.000419:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -96> 2015-07-01 10:34:47.550651 7ff90d7ba700 2 req 6958:0.000421:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -95> 2015-07-01 10:34:47.550688 7ff90d7ba700 0 WARNING: couldn't find acl header for bucket, generating default -94> 2015-07-01 10:34:47.550752 7ff90d7ba700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7143 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e427) v4 -- ?+0 0x7ff940074d00 con 0x7ff95f9302e0 -93> 2015-07-01 10:34:47.552349 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1806 ==== osd_op_reply(7143 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (2078508111 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -92> 2015-07-01 10:34:47.552454 7ff90d7ba700 2 overriding permissions due to system operation -91> 2015-07-01 10:34:47.552458 7ff90d7ba700 2 req 6958:0.002228:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -90> 2015-07-01 10:34:47.552461 7ff90d7ba700 2 req 6958:0.002232:s3:PUT /BASKET1/tw.json:copy_obj:executing -89> 2015-07-01 10:34:47.552520 7ff90d7ba700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -88> 2015-07-01 10:34:47.552594 7ff90d7ba700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7144 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff940074d00 con 0x7ff95f92e520 -87> 2015-07-01 10:34:47.558448 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2791 ==== osd_op_reply(7144 statelog.obj_opstate.68 [call] v427'4711 uv4711 ondisk = 0) v6 ==== 190+0+0 (20211098 0 0) 0x7ff938052680 con 0x7ff95f92e520 -86> 2015-07-01 10:34:47.558518 7ff90d7ba700 0 > HTTP_DATE -> Wed Jul 1 02:34:47 2015 -85> 2015-07-01 10:34:47.558520 7ff90d7ba700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -84> 2015-07-01 10:34:47.560962 7ff90d7ba700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7145 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff940074d00 con 0x7ff95f92e520 -83> 2015-07-01 10:34:47.565839 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2792 ==== osd_op_reply(7145 statelog.obj_opstate.68 [call] v427'4712 uv4712 ondisk = 0) v6 ==== 190+0+0 (3358910123 0 0) 0x7ff938052680 con 0x7ff95f92e520 -82> 2015-07-01 10:34:47.566081 7ff90d7ba700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -81> 2015-07-01 10:34:47.566119 7ff90d7ba700 2 req 6958:0.015890:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -80> 2015-07-01 10:34:47.566126 7ff90d7ba700 1 ====== req done req=0x7ff940015950 http_status=500 ====== -79> 2015-07-01 10:34:48.336661 7ff94d6f9700 2 RGWDataChangesLog::ChangesRenewThread: start -78> 2015-07-01 10:34:48.932688 7ff913fc7700 1 ====== starting new request req=0x7ff94001c370 ===== -77> 2015-07-01 10:34:48.932755 7ff913fc7700 2 req 6959:0.000069::PUT /BASKET1/tw.json::initializing -76> 2015-07-01 10:34:48.932849 7ff913fc7700 2 req 6959:0.000162:s3:PUT /BASKET1/tw.json::getting op -75> 2015-07-01 10:34:48.932860 7ff913fc7700 2 req 6959:0.000174:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -74> 2015-07-01 10:34:48.933188 7ff913fc7700 2 req 6959:0.000501:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -73> 2015-07-01 10:34:48.933362 7ff913fc7700 2 req 6959:0.000676:s3:PUT /BASKET1/tw.json:copy_obj:init op -72> 2015-07-01 10:34:48.933370 7ff913fc7700 2 req 6959:0.000683:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -71> 2015-07-01 10:34:48.933373 7ff913fc7700 2 req 6959:0.000687:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -70> 2015-07-01 10:34:48.933439 7ff913fc7700 0 WARNING: couldn't find acl header for bucket, generating default -69> 2015-07-01 10:34:48.933536 7ff913fc7700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7146 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e427) v4 -- ?+0 0x7ff938049650 con 0x7ff95f9302e0 -68> 2015-07-01 10:34:48.935930 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1807 ==== osd_op_reply(7146 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (2078508111 0 0) 0x7ff95f944240 con 0x7ff95f9302e0 -67> 2015-07-01 10:34:48.936103 7ff913fc7700 2 overriding permissions due to system operation -66> 2015-07-01 10:34:48.936110 7ff913fc7700 2 req 6959:0.003424:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -65> 2015-07-01 10:34:48.936116 7ff913fc7700 2 req 6959:0.003430:s3:PUT /BASKET1/tw.json:copy_obj:executing -64> 2015-07-01 10:34:48.936209 7ff913fc7700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -63> 2015-07-01 10:34:48.936346 7ff913fc7700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7147 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff938048c60 con 0x7ff95f92e520 -62> 2015-07-01 10:34:48.945999 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2793 ==== osd_op_reply(7147 statelog.obj_opstate.68 [call] v427'4713 uv4713 ondisk = 0) v6 ==== 190+0+0 (3145090958 0 0) 0x7ff9380491d0 con 0x7ff95f92e520 -61> 2015-07-01 10:34:48.946175 7ff913fc7700 0 > HTTP_DATE -> Wed Jul 1 02:34:48 2015 -60> 2015-07-01 10:34:48.946179 7ff913fc7700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -59> 2015-07-01 10:34:48.949680 7ff913fc7700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7148 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff938054010 con 0x7ff95f92e520 -58> 2015-07-01 10:34:48.955950 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2794 ==== osd_op_reply(7148 statelog.obj_opstate.68 [call] v427'4714 uv4714 ondisk = 0) v6 ==== 190+0+0 (783394017 0 0) 0x7ff938054010 con 0x7ff95f92e520 -57> 2015-07-01 10:34:48.956077 7ff913fc7700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -56> 2015-07-01 10:34:48.956219 7ff913fc7700 2 req 6959:0.023532:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -55> 2015-07-01 10:34:48.956261 7ff913fc7700 1 ====== req done req=0x7ff94001c370 http_status=500 ====== -54> 2015-07-01 10:34:49.380928 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.72:6800/1502 -- ping v1 -- ?+0 0x7ff930045eb0 con 0x7ff95f9240b0 -53> 2015-07-01 10:34:49.380977 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- ping v1 -- ?+0 0x7ff9300460d0 con 0x7ff95f9302e0 -52> 2015-07-01 10:34:49.381002 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.74:6800/27167 -- ping v1 -- ?+0 0x7ff9300462f0 con 0x7ff95f934520 -51> 2015-07-01 10:34:49.381019 7ff94e6fb700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- ping v1 -- ?+0 0x7ff93000d760 con 0x7ff95f92e520 -50> 2015-07-01 10:34:51.904977 7ff929ff3700 1 ====== starting new request req=0x7ff94005b9e0 ===== -49> 2015-07-01 10:34:51.905033 7ff929ff3700 2 req 6960:0.000057::PUT /BASKET1/tw.json::initializing -48> 2015-07-01 10:34:51.905106 7ff929ff3700 2 req 6960:0.000130:s3:PUT /BASKET1/tw.json::getting op -47> 2015-07-01 10:34:51.905114 7ff929ff3700 2 req 6960:0.000139:s3:PUT /BASKET1/tw.json:copy_obj:authorizing -46> 2015-07-01 10:34:51.905370 7ff929ff3700 2 req 6960:0.000394:s3:PUT /BASKET1/tw.json:copy_obj:reading permissions -45> 2015-07-01 10:34:51.905500 7ff929ff3700 2 req 6960:0.000524:s3:PUT /BASKET1/tw.json:copy_obj:init op -44> 2015-07-01 10:34:51.905505 7ff929ff3700 2 req 6960:0.000530:s3:PUT /BASKET1/tw.json:copy_obj:verifying op mask -43> 2015-07-01 10:34:51.905508 7ff929ff3700 2 req 6960:0.000533:s3:PUT /BASKET1/tw.json:copy_obj:verifying op permissions -42> 2015-07-01 10:34:51.905556 7ff929ff3700 0 WARNING: couldn't find acl header for bucket, generating default -41> 2015-07-01 10:34:51.905684 7ff929ff3700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.71:6800/25970 -- osd_op(client.7211.0:7149 tw-east [getxattrs,stat] 57.24dd5f83 ack+read e427) v4 -- ?+0 0x7ff934014bd0 con 0x7ff95f9302e0 -40> 2015-07-01 10:34:51.907012 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.0 10.35.64.71:6800/25970 1808 ==== osd_op_reply(7149 tw-east [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 216+0+0 (2078508111 0 0) 0x7ff95f97c820 con 0x7ff95f9302e0 -39> 2015-07-01 10:34:51.907303 7ff929ff3700 2 overriding permissions due to system operation -38> 2015-07-01 10:34:51.907310 7ff929ff3700 2 req 6960:0.002335:s3:PUT /BASKET1/tw.json:copy_obj:verifying op params -37> 2015-07-01 10:34:51.907315 7ff929ff3700 2 req 6960:0.002340:s3:PUT /BASKET1/tw.json:copy_obj:executing -36> 2015-07-01 10:34:51.907387 7ff929ff3700 5 Copy object BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json => BASKET1(@{i=.rgw.buckets.index}.rgw.buckets[tw-east.6930.1]):tw.json -35> 2015-07-01 10:34:51.907490 7ff929ff3700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7150 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff934014bd0 con 0x7ff95f92e520 -34> 2015-07-01 10:34:51.922481 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2795 ==== osd_op_reply(7150 statelog.obj_opstate.68 [call] v427'4715 uv4715 ondisk = 0) v6 ==== 190+0+0 (1576207812 0 0) 0x7ff9380492a0 con 0x7ff95f92e520 -33> 2015-07-01 10:34:51.922573 7ff929ff3700 0 > HTTP_DATE -> Wed Jul 1 02:34:51 2015 -32> 2015-07-01 10:34:51.922575 7ff929ff3700 0 > HTTP_X_AMZ_COPY_SOURCE -> BASKET1/tw.json -31> 2015-07-01 10:34:51.925367 7ff929ff3700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7151 statelog.obj_opstate.68 [call statelog.add] 59.868ada28 ondisk+write e427) v4 -- ?+0 0x7ff934014bd0 con 0x7ff95f92e520 -30> 2015-07-01 10:34:51.932631 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2796 ==== osd_op_reply(7151 statelog.obj_opstate.68 [call] v427'4716 uv4716 ondisk = 0) v6 ==== 190+0+0 (13767886 0 0) 0x7ff938052680 con 0x7ff95f92e520 -29> 2015-07-01 10:34:51.932717 7ff929ff3700 0 WARNING: set_req_state_err err_no=5 resorting to 500 -28> 2015-07-01 10:34:51.932812 7ff929ff3700 2 req 6960:0.027837:s3:PUT /BASKET1/tw.json:copy_obj:http status=500 -27> 2015-07-01 10:34:51.932838 7ff929ff3700 1 ====== req done req=0x7ff94005b9e0 http_status=500 ====== -26> 2015-07-01 10:34:54.096336 7ff91c7d8700 1 ====== starting new request req=0x7ff94000dde0 ===== -25> 2015-07-01 10:34:54.096395 7ff91c7d8700 2 req 6961:0.000060::GET /admin/opstate::initializing -24> 2015-07-01 10:34:54.096482 7ff91c7d8700 2 req 6961:0.000147::GET /admin/opstate::getting op -23> 2015-07-01 10:34:54.096488 7ff91c7d8700 2 req 6961:0.000154::GET /admin/opstate:opstate_list:authorizing -22> 2015-07-01 10:34:54.096809 7ff91c7d8700 2 req 6961:0.000474::GET /admin/opstate:opstate_list:reading permissions -21> 2015-07-01 10:34:54.096818 7ff91c7d8700 2 req 6961:0.000484::GET /admin/opstate:opstate_list:init op -20> 2015-07-01 10:34:54.096823 7ff91c7d8700 2 req 6961:0.000489::GET /admin/opstate:opstate_list:verifying op mask -19> 2015-07-01 10:34:54.096827 7ff91c7d8700 2 req 6961:0.000493::GET /admin/opstate:opstate_list:verifying op permissions -18> 2015-07-01 10:34:54.096832 7ff91c7d8700 2 overriding permissions due to system operation -17> 2015-07-01 10:34:54.096835 7ff91c7d8700 2 req 6961:0.000501::GET /admin/opstate:opstate_list:verifying op params -16> 2015-07-01 10:34:54.096838 7ff91c7d8700 2 req 6961:0.000504::GET /admin/opstate:opstate_list:executing -15> 2015-07-01 10:34:54.096964 7ff91c7d8700 1 -- 10.35.64.71:0/1026982 --> 10.35.64.73:6800/29951 -- osd_op(client.7211.0:7152 statelog.obj_opstate.68 [call statelog.list] 59.868ada28 ack+read e427) v4 -- ?+0 0x7ff930049f60 con 0x7ff95f92e520 -14> 2015-07-01 10:34:54.099798 7ff954816700 1 -- 10.35.64.71:0/1026982 <== osd.2 10.35.64.73:6800/29951 2797 ==== osd_op_reply(7152 statelog.obj_opstate.68 [call] v0'0 uv4716 ondisk = 0) v6 ==== 190+0+90 (1631025820 0 1691987791) 0x7ff938052680 con 0x7ff95f92e520 -13> 2015-07-01 10:34:54.099959 7ff91c7d8700 2 req 6961:0.003625::GET /admin/opstate:opstate_list:http status=200 -12> 2015-07-01 10:34:54.099972 7ff91c7d8700 1 ====== req done req=0x7ff94000dde0 http_status=200 ====== -11> 2015-07-01 10:34:54.112960 7ff91b7d6700 1 ====== starting new request req=0x7ff940034db0 ===== -10> 2015-07-01 10:34:54.112996 7ff91b7d6700 2 req 6962:0.000036::POST /admin/replica_log::initializing -9> 2015-07-01 10:34:54.113064 7ff91b7d6700 2 req 6962:0.000105::POST /admin/replica_log::getting op -8> 2015-07-01 10:34:54.113071 7ff91b7d6700 2 req 6962:0.000112::POST /admin/replica_log:replicabilog_updatebounds:authorizing -7> 2015-07-01 10:34:54.113293 7ff91b7d6700 2 req 6962:0.000334::POST /admin/replica_log:replicabilog_updatebounds:reading permissions -6> 2015-07-01 10:34:54.113303 7ff91b7d6700 2 req 6962:0.000344::POST /admin/replica_log:replicabilog_updatebounds:init op -5> 2015-07-01 10:34:54.113307 7ff91b7d6700 2 req 6962:0.000348::POST /admin/replica_log:replicabilog_updatebounds:verifying op mask -4> 2015-07-01 10:34:54.113311 7ff91b7d6700 2 req 6962:0.000352::POST /admin/replica_log:replicabilog_updatebounds:verifying op permissions -3> 2015-07-01 10:34:54.113316 7ff91b7d6700 2 overriding permissions due to system operation -2> 2015-07-01 10:34:54.113318 7ff91b7d6700 2 req 6962:0.000359::POST /admin/replica_log:replicabilog_updatebounds:verifying op params -1> 2015-07-01 10:34:54.113321 7ff91b7d6700 2 req 6962:0.000362::POST /admin/replica_log:replicabilog_updatebounds:executing 0> 2015-07-01 10:34:54.116891 7ff91b7d6700 -1 *** Caught signal (Aborted) ** in thread 7ff91b7d6700 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047) 1: (()+0x1f18af) [0x7ff95efcb8af] 2: (()+0x10340) [0x7ff95cfcd340] 3: (gsignal()+0x39) [0x7ff95c0d4cc9] 4: (abort()+0x148) [0x7ff95c0d80d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff95c9df535] 6: (()+0x5e6d6) [0x7ff95c9dd6d6] 7: (()+0x5e703) [0x7ff95c9dd703] 8: (()+0x5e922) [0x7ff95c9dd922] 9: (bool JSONDecoder::decode_json(char const*, utime_t&, JSONObj*, bool)+0x204) [0x7ff95eed35b4] 10: (void decode_json_obj(std::list >&, JSONObj*)+0x73) [0x7ff95eea9e13] 11: (int rgw_rest_get_json_input > >(CephContext*, req_state*, std::list >&, int, bool*)+0xa5) [0x7ff95eea9f75] 12: (RGWOp_BILog_SetBounds::execute()+0x390) [0x7ff95eea8e20] 13: (()+0xde63d) [0x7ff95eeb863d] 14: (RGWFCGXProcess::handle_request(RGWRequest*)+0x80) [0x7ff95eeb8d30] 15: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x30) [0x7ff95eeba340] 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0xaf1) [0x7ff95deca531] 17: (ThreadPool::WorkThread::entry()+0x10) [0x7ff95decb420] 18: (()+0x8182) [0x7ff95cfc5182] 19: (clone()+0x6d) [0x7ff95c19847d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/radosgw.node1.log --- end dump of recent events ---