2015-07-16 18:14:32,923 2611 [radosgw_agent][INFO ] __ __ __ ___ ___ 2015-07-16 18:14:32,924 2611 [radosgw_agent][INFO ] /__` \ / |\ | / ` /\ / _` |__ |\ | | 2015-07-16 18:14:32,924 2611 [radosgw_agent][INFO ] .__/ | | \| \__, /~~\ \__> |___ | \| | 2015-07-16 18:14:32,925 2611 [radosgw_agent][INFO ] v1.2.3 2015-07-16 18:14:32,925 2611 [radosgw_agent][INFO ] agent options: 2015-07-16 18:14:32,926 2611 [radosgw_agent][INFO ] args: 2015-07-16 18:14:32,927 2611 [radosgw_agent][INFO ] conf : None 2015-07-16 18:14:32,927 2611 [radosgw_agent][INFO ] dest_access_key : **************** 2015-07-16 18:14:32,928 2611 [radosgw_agent][INFO ] dest_secret_key : **************** 2015-07-16 18:14:32,928 2611 [radosgw_agent][INFO ] destination : http://10.35.64.71:80 2015-07-16 18:14:32,929 2611 [radosgw_agent][INFO ] incremental_sync_delay : 30 2015-07-16 18:14:32,929 2611 [radosgw_agent][INFO ] lock_timeout : 60 2015-07-16 18:14:32,930 2611 [radosgw_agent][INFO ] log_file : /var/log/radosgw/radosgw-sync-tw-east-west.log 2015-07-16 18:14:32,930 2611 [radosgw_agent][INFO ] log_lock_time : 20 2015-07-16 18:14:32,931 2611 [radosgw_agent][INFO ] max_entries : 1000 2015-07-16 18:14:32,931 2611 [radosgw_agent][INFO ] metadata_only : False 2015-07-16 18:14:32,932 2611 [radosgw_agent][INFO ] num_workers : 1 2015-07-16 18:14:32,932 2611 [radosgw_agent][INFO ] object_sync_timeout : 216000 2015-07-16 18:14:32,933 2611 [radosgw_agent][INFO ] prepare_error_delay : 10 2015-07-16 18:14:32,933 2611 [radosgw_agent][INFO ] quiet : False 2015-07-16 18:14:32,933 2611 [radosgw_agent][INFO ] rgw_data_log_window : 30 2015-07-16 18:14:32,934 2611 [radosgw_agent][INFO ] source : http://10.35.64.81:80 2015-07-16 18:14:32,934 2611 [radosgw_agent][INFO ] src_access_key : **************** 2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] src_secret_key : **************** 2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] src_zone : None 2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] sync_scope : incremental 2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] test_server_host : None 2015-07-16 18:14:32,936 2611 [radosgw_agent][INFO ] test_server_port : 8080 2015-07-16 18:14:32,936 2611 [radosgw_agent][INFO ] verbose : False 2015-07-16 18:14:32,936 2611 [radosgw_agent][INFO ] versioned : False 2015-07-16 18:14:32,936 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.71:80 2015-07-16 18:14:32,937 2611 [boto][DEBUG ] Using access key provided by client. 2015-07-16 18:14:32,937 2611 [boto][DEBUG ] Using secret key provided by client. 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:14:32 GMT /admin/config 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Signature: AWS tw_access_key:hS2fQmwAJX2UEOAjV151DSvab1g= 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] url = 'http://10.35.64.71/admin/config' params={} headers={'Date': 'Thu, 16 Jul 2015 10:14:32 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:hS2fQmwAJX2UEOAjV151DSvab1g=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Method: GET 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Path: /admin/config 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Data: 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Headers: {} 2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Port: 80 2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Params: {} 2015-07-16 18:14:32,939 2611 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70} 2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Token: None 2015-07-16 18:14:32,939 2611 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:14:32 GMT /admin/config 2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Signature: AWS tw_access_key:hS2fQmwAJX2UEOAjV151DSvab1g= 2015-07-16 18:14:33,073 2611 [radosgw_agent][INFO ] http://10.35.64.81:80 endpoint does not support versioning 2015-07-16 18:14:33,075 2611 [radosgw_agent][WARNING] encountered issues reaching to endpoint http://10.35.64.81:80 2015-07-16 18:14:33,076 2611 [radosgw_agent][WARNING] HTTP Error 403: Forbidden 2015-07-16 18:14:33,078 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.81:80 2015-07-16 18:14:33,079 2611 [boto][DEBUG ] Using access key provided by client. 2015-07-16 18:14:33,079 2611 [boto][DEBUG ] Using secret key provided by client. 2015-07-16 18:14:33,080 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.71:80 2015-07-16 18:14:33,081 2611 [boto][DEBUG ] Using access key provided by client. 2015-07-16 18:14:33,081 2611 [boto][DEBUG ] Using secret key provided by client. 2015-07-16 18:14:33,082 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.81:80 2015-07-16 18:14:33,083 2611 [boto][DEBUG ] Using access key provided by client. 2015-07-16 18:14:33,084 2611 [boto][DEBUG ] Using secret key provided by client. 2015-07-16 18:14:33,084 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.71:80 2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Using access key provided by client. 2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Using secret key provided by client. 2015-07-16 18:14:33,086 2611 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:14:33 GMT /admin/log 2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Signature: AWS tw_access_key:2fWu2+WDCBFlNBPYRkd2BIC9gsM= 2015-07-16 18:14:33,086 2611 [boto][DEBUG ] url = 'http://10.35.64.81/admin/log' params={'type': 'metadata'} headers={'Date': 'Thu, 16 Jul 2015 10:14:33 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:2fWu2+WDCBFlNBPYRkd2BIC9gsM=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Method: GET 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Path: /admin/log?type=metadata 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Data: 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Headers: {} 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Host: 10.35.64.81 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Port: 80 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Params: {'type': 'metadata'} 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70} 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Token: None 2015-07-16 18:14:33,087 2611 [boto][DEBUG ] StringToSign: GET | | Thu, 16 Jul 2015 10:15:11 GMT /admin/log 2015-07-16 18:15:11,492 2621 [boto][DEBUG ] Signature: AWS tw_access_key:/jJFDjzjiQkvBPgexaGsG07bVgc= 2015-07-16 18:15:11,502 2621 [radosgw_agent.worker][DEBUG ] bucket instance "BUCKET1:tw-east.24117.1" has 2 entries after " " 2015-07-16 18:15:11,503 2621 [radosgw_agent.worker][INFO ] ******************************************************************************** 2015-07-16 18:15:11,504 2621 [radosgw_agent.worker][INFO ] syncing bucket "BUCKET1" 2015-07-16 18:15:11,505 2621 [radosgw_agent.worker][DEBUG ] syncing object BUCKET1/tw.json 2015-07-16 18:15:11,506 2621 [boto][DEBUG ] StringToSign: PUT application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:11 GMT x-amz-copy-source:BUCKET1/tw.json /BUCKET1/tw.json 2015-07-16 18:15:11,506 2621 [boto][DEBUG ] Signature: AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ= 2015-07-16 18:15:11,507 2621 [boto][DEBUG ] url = 'http://10.35.64.71/BUCKET1/tw.json' params={'rgwx-op-id': 'node1:2611:1', 'rgwx-source-zone': u'tw-east', 'rgwx-client-id': 'radosgw-agent'} headers={'Content-Length': '0', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic', 'x-amz-copy-source': 'BUCKET1/tw.json', 'Date': 'Thu, 16 Jul 2015 10:15:11 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Authorization': 'AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ='} data=None 2015-07-16 18:15:11,507 2621 [boto][DEBUG ] Method: PUT 2015-07-16 18:15:11,507 2621 [boto][DEBUG ] Path: /BUCKET1/tw.json?rgwx-op-id=node1%3A2611%3A1&rgwx-source-zone=tw-east&rgwx-client-id=radosgw-agent 2015-07-16 18:15:11,508 2621 [boto][DEBUG ] Data: 2015-07-16 18:15:11,508 2621 [boto][DEBUG ] Headers: {'Content-Type': 'application/json; charset=UTF-8', 'x-amz-copy-source': 'BUCKET1/tw.json'} 2015-07-16 18:15:11,509 2621 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:15:11,509 2621 [boto][DEBUG ] Port: 80 2015-07-16 18:15:11,510 2621 [boto][DEBUG ] Params: {'rgwx-op-id': 'node1%3A2611%3A1', 'rgwx-source-zone': 'tw-east', 'rgwx-client-id': 'radosgw-agent'} 2015-07-16 18:15:11,510 2621 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70} 2015-07-16 18:15:11,510 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:11,511 2621 [boto][DEBUG ] StringToSign: PUT application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:11 GMT x-amz-copy-source:BUCKET1/tw.json /BUCKET1/tw.json 2015-07-16 18:15:11,511 2621 [boto][DEBUG ] Signature: AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ= 2015-07-16 18:15:12,420 2621 [boto][DEBUG ] path=/admin/log?lock 2015-07-16 18:15:12,421 2621 [boto][DEBUG ] auth_path=/admin/log?lock 2015-07-16 18:15:12,421 2621 [boto][DEBUG ] StringToSign: POST application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:12 GMT /admin/log 2015-07-16 18:15:12,421 2621 [boto][DEBUG ] Signature: AWS tw_access_key:ZnHewFD7nyOTaPhpAHCh+CQrApc= 2015-07-16 18:15:12,421 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/log?lock' params={'locker-id': 'node1:2611', 'length': 60, 'zone-id': u'tw-west', 'type': 'data', 'id': 35} headers={'Date': 'Thu, 16 Jul 2015 10:15:12 GMT', 'Content-Length': '0', 'Content-Type': 'application/json; charset=UTF-8', 'Authorization': 'AWS tw_access_key:ZnHewFD7nyOTaPhpAHCh+CQrApc=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:15:12,422 2621 [boto][DEBUG ] path=/admin/log?lock 2015-07-16 18:15:12,422 2621 [boto][DEBUG ] auth_path=/admin/log?lock 2015-07-16 18:15:12,422 2621 [boto][DEBUG ] Method: POST 2015-07-16 18:15:12,422 2621 [boto][DEBUG ] Path: /admin/log?lock&length=60&locker-id=node1%3A2611&type=data&zone-id=tw-west&id=35 2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Data: 2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Headers: {'Content-Type': 'application/json; charset=UTF-8'} 2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Port: 80 2015-07-16 18:15:12,424 2621 [boto][DEBUG ] Params: {'length': '60', 'locker-id': 'node1%3A2611', 'type': 'data', 'zone-id': 'tw-west', 'id': '35'} 2015-07-16 18:15:12,425 2621 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70} 2015-07-16 18:15:12,426 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:12,426 2621 [boto][DEBUG ] StringToSign: POST application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:12 GMT /admin/log 2015-07-16 18:15:12,427 2621 [boto][DEBUG ] Signature: AWS tw_access_key:ZnHewFD7nyOTaPhpAHCh+CQrApc= 2015-07-16 18:15:12,783 2621 [boto][DEBUG ] Received 500 response. Retrying in 0.1 seconds 2015-07-16 18:15:12,907 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:12,907 2621 [boto][DEBUG ] StringToSign: PUT application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:11 GMT x-amz-copy-source:BUCKET1/tw.json /BUCKET1/tw.json 2015-07-16 18:15:12,908 2621 [boto][DEBUG ] Signature: AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ= 2015-07-16 18:15:14,372 2621 [boto][DEBUG ] Received 500 response. Retrying in 0.8 seconds 2015-07-16 18:15:15,212 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:15,212 2621 [boto][DEBUG ] StringToSign: PUT application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:11 GMT x-amz-copy-source:BUCKET1/tw.json /BUCKET1/tw.json 2015-07-16 18:15:15,212 2621 [boto][DEBUG ] Signature: AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ= 2015-07-16 18:15:16,997 2621 [boto][DEBUG ] Received 500 response. Retrying in 3.4 seconds 2015-07-16 18:15:20,392 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:20,393 2621 [boto][DEBUG ] StringToSign: PUT application/json; charset=UTF-8 Thu, 16 Jul 2015 10:15:11 GMT x-amz-copy-source:BUCKET1/tw.json /BUCKET1/tw.json 2015-07-16 18:15:20,393 2621 [boto][DEBUG ] Signature: AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ= 2015-07-16 18:15:20,420 2621 [boto][DEBUG ] Received 500 response. Retrying in 2.6 seconds 2015-07-16 18:15:22,993 2621 [dev.radosgw_agent.worker][WARNING] encountered an error during sync Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 232, in sync_object local_op_id) File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 411, in sync_object_intra_region expect_json=False) File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 210, in request check_result_status(error) File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 154, in check_result_status exc.HttpError)(result.status, result.reason) HttpError: Http error code 500 content Internal Server Error 2015-07-16 18:15:22,995 2621 [radosgw_agent.worker][WARNING] encountered an error during sync: Http error code 500 content Internal Server Error 2015-07-16 18:15:22,996 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:22 GMT /admin/opstate 2015-07-16 18:15:22,996 2621 [boto][DEBUG ] Signature: AWS tw_access_key:jhck8Nv6pv0dFvk0SjipsGGEtec= 2015-07-16 18:15:22,996 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate' params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'} headers={'Date': 'Thu, 16 Jul 2015 10:15:22 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:jhck8Nv6pv0dFvk0SjipsGGEtec=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:15:22,997 2621 [boto][DEBUG ] Method: GET 2015-07-16 18:15:22,997 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1 2015-07-16 18:15:22,997 2621 [boto][DEBUG ] Data: 2015-07-16 18:15:22,998 2621 [boto][DEBUG ] Headers: {} 2015-07-16 18:15:22,998 2621 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:15:22,998 2621 [boto][DEBUG ] Port: 80 2015-07-16 18:15:22,999 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'} 2015-07-16 18:15:22,999 2621 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70} 2015-07-16 18:15:22,999 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:23,000 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:23 GMT /admin/opstate 2015-07-16 18:15:23,000 2621 [boto][DEBUG ] Signature: AWS tw_access_key:EvJFrss7PEvy/4xPBVgWv+DwMLg= 2015-07-16 18:15:23,006 2621 [radosgw_agent.worker][DEBUG ] op state is [] 2015-07-16 18:15:23,007 2621 [radosgw_agent.worker][DEBUG ] error geting op state: list index out of range Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 275, in wait_for_object state = state[0]['state'] IndexError: list index out of range 2015-07-16 18:15:24,009 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:24 GMT /admin/opstate 2015-07-16 18:15:24,009 2621 [boto][DEBUG ] Signature: AWS tw_access_key:Att2BZKNC6pdw+YxpzAbxydyuis= 2015-07-16 18:15:24,010 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate' params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'} headers={'Date': 'Thu, 16 Jul 2015 10:15:24 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:Att2BZKNC6pdw+YxpzAbxydyuis=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:15:24,010 2621 [boto][DEBUG ] Method: GET 2015-07-16 18:15:24,010 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1 2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Data: 2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Headers: {} 2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Port: 80 2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'} 2015-07-16 18:15:24,012 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:24,012 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:24 GMT /admin/opstate 2015-07-16 18:15:24,012 2621 [boto][DEBUG ] Signature: AWS tw_access_key:Att2BZKNC6pdw+YxpzAbxydyuis= 2015-07-16 18:15:24,020 2621 [radosgw_agent.worker][DEBUG ] op state is [] 2015-07-16 18:15:24,020 2621 [radosgw_agent.worker][DEBUG ] error geting op state: list index out of range Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 275, in wait_for_object state = state[0]['state'] IndexError: list index out of range 2015-07-16 18:15:25,022 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:25 GMT /admin/opstate 2015-07-16 18:15:25,022 2621 [boto][DEBUG ] Signature: AWS tw_access_key:VGMI8nTYVHPvPuSjAQDPKgz9rKE= 2015-07-16 18:15:25,023 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate' params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'} headers={'Date': 'Thu, 16 Jul 2015 10:15:25 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:VGMI8nTYVHPvPuSjAQDPKgz9rKE=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:15:25,023 2621 [boto][DEBUG ] Method: GET 2015-07-16 18:15:25,023 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1 2015-07-16 18:15:25,023 2621 [boto][DEBUG ] Data: 2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Headers: {} 2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Port: 80 2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'} 2015-07-16 18:15:25,025 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:25,025 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:25 GMT /admin/opstate 2015-07-16 18:15:25,025 2621 [boto][DEBUG ] Signature: AWS tw_access_key:VGMI8nTYVHPvPuSjAQDPKgz9rKE= 2015-07-16 18:15:25,032 2621 [radosgw_agent.worker][DEBUG ] op state is [] 2015-07-16 18:15:25,032 2621 [radosgw_agent.worker][DEBUG ] error geting op state: list index out of range Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 275, in wait_for_object state = state[0]['state'] IndexError: list index out of range 2015-07-16 18:15:26,035 2621 [boto][DEBUG ] StringToSign: GET Thu, 16 Jul 2015 10:15:26 GMT /admin/opstate 2015-07-16 18:15:26,036 2621 [boto][DEBUG ] Signature: AWS tw_access_key:Nrt3TAfspX8c+cTM98jovGz2KT0= 2015-07-16 18:15:26,036 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate' params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'} headers={'Date': 'Thu, 16 Jul 2015 10:15:26 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:Nrt3TAfspX8c+cTM98jovGz2KT0=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'} data=None 2015-07-16 18:15:26,036 2621 [boto][DEBUG ] Method: GET 2015-07-16 18:15:26,036 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1 2015-07-16 18:15:26,037 2621 [boto][DEBUG ] Data: 2015-07-16 18:15:26,037 2621 [boto][DEBUG ] Headers: {} 2015-07-16 18:15:26,037 2621 [boto][DEBUG ] Host: 10.35.64.71 2015-07-16 18:15:26,039 2621 [boto][DEBUG ] Port: 80 2015-07-16 18:15:26,040 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'} 2015-07-16 18:15:26,040 2621 [boto][DEBUG ] Token: None 2015-07-16 18:15:26,041 2621 [boto][DEBUG ] StringToSign: GET