Project

General

Profile

Bug #12182 » radosgw-sync.log

Wen-Dwo Hwang Hwang, 07/17/2015 02:04 AM

 
2015-07-16 18:14:32,923 2611 [radosgw_agent][INFO ] __ __ __ ___ ___
2015-07-16 18:14:32,924 2611 [radosgw_agent][INFO ] /__` \ / |\ | / ` /\ / _` |__ |\ | |
2015-07-16 18:14:32,924 2611 [radosgw_agent][INFO ] .__/ | | \| \__, /~~\ \__> |___ | \| |
2015-07-16 18:14:32,925 2611 [radosgw_agent][INFO ] v1.2.3
2015-07-16 18:14:32,925 2611 [radosgw_agent][INFO ] agent options:
2015-07-16 18:14:32,926 2611 [radosgw_agent][INFO ] args:
2015-07-16 18:14:32,927 2611 [radosgw_agent][INFO ] conf : None
2015-07-16 18:14:32,927 2611 [radosgw_agent][INFO ] dest_access_key : ****************
2015-07-16 18:14:32,928 2611 [radosgw_agent][INFO ] dest_secret_key : ****************
2015-07-16 18:14:32,928 2611 [radosgw_agent][INFO ] destination : http://10.35.64.71:80
2015-07-16 18:14:32,929 2611 [radosgw_agent][INFO ] incremental_sync_delay : 30
2015-07-16 18:14:32,929 2611 [radosgw_agent][INFO ] lock_timeout : 60
2015-07-16 18:14:32,930 2611 [radosgw_agent][INFO ] log_file : /var/log/radosgw/radosgw-sync-tw-east-west.log
2015-07-16 18:14:32,930 2611 [radosgw_agent][INFO ] log_lock_time : 20
2015-07-16 18:14:32,931 2611 [radosgw_agent][INFO ] max_entries : 1000
2015-07-16 18:14:32,931 2611 [radosgw_agent][INFO ] metadata_only : False
2015-07-16 18:14:32,932 2611 [radosgw_agent][INFO ] num_workers : 1
2015-07-16 18:14:32,932 2611 [radosgw_agent][INFO ] object_sync_timeout : 216000
2015-07-16 18:14:32,933 2611 [radosgw_agent][INFO ] prepare_error_delay : 10
2015-07-16 18:14:32,933 2611 [radosgw_agent][INFO ] quiet : False
2015-07-16 18:14:32,933 2611 [radosgw_agent][INFO ] rgw_data_log_window : 30
2015-07-16 18:14:32,934 2611 [radosgw_agent][INFO ] source : http://10.35.64.81:80
2015-07-16 18:14:32,934 2611 [radosgw_agent][INFO ] src_access_key : ****************
2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] src_secret_key : ****************
2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] src_zone : None
2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] sync_scope : incremental
2015-07-16 18:14:32,935 2611 [radosgw_agent][INFO ] test_server_host : None
2015-07-16 18:14:32,936 2611 [radosgw_agent][INFO ] test_server_port : 8080
2015-07-16 18:14:32,936 2611 [radosgw_agent][INFO ] verbose : False
2015-07-16 18:14:32,936 2611 [radosgw_agent][INFO ] versioned : False
2015-07-16 18:14:32,936 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.71:80
2015-07-16 18:14:32,937 2611 [boto][DEBUG ] Using access key provided by client.
2015-07-16 18:14:32,937 2611 [boto][DEBUG ] Using secret key provided by client.
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:14:32 GMT
/admin/config
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Signature:
AWS tw_access_key:hS2fQmwAJX2UEOAjV151DSvab1g=
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] url = 'http://10.35.64.71/admin/config'
params={}
headers={'Date': 'Thu, 16 Jul 2015 10:14:32 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:hS2fQmwAJX2UEOAjV151DSvab1g=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Method: GET
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Path: /admin/config
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Data:
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Headers: {}
2015-07-16 18:14:32,938 2611 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Port: 80
2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Params: {}
2015-07-16 18:14:32,939 2611 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Token: None
2015-07-16 18:14:32,939 2611 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:14:32 GMT
/admin/config
2015-07-16 18:14:32,939 2611 [boto][DEBUG ] Signature:
AWS tw_access_key:hS2fQmwAJX2UEOAjV151DSvab1g=
2015-07-16 18:14:33,073 2611 [radosgw_agent][INFO ] http://10.35.64.81:80 endpoint does not support versioning
2015-07-16 18:14:33,075 2611 [radosgw_agent][WARNING] encountered issues reaching to endpoint http://10.35.64.81:80
2015-07-16 18:14:33,076 2611 [radosgw_agent][WARNING] HTTP Error 403: Forbidden
2015-07-16 18:14:33,078 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.81:80
2015-07-16 18:14:33,079 2611 [boto][DEBUG ] Using access key provided by client.
2015-07-16 18:14:33,079 2611 [boto][DEBUG ] Using secret key provided by client.
2015-07-16 18:14:33,080 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.71:80
2015-07-16 18:14:33,081 2611 [boto][DEBUG ] Using access key provided by client.
2015-07-16 18:14:33,081 2611 [boto][DEBUG ] Using secret key provided by client.
2015-07-16 18:14:33,082 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.81:80
2015-07-16 18:14:33,083 2611 [boto][DEBUG ] Using access key provided by client.
2015-07-16 18:14:33,084 2611 [boto][DEBUG ] Using secret key provided by client.
2015-07-16 18:14:33,084 2611 [radosgw_agent.client][INFO ] creating connection to endpoint: http://10.35.64.71:80
2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Using access key provided by client.
2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Using secret key provided by client.
2015-07-16 18:14:33,086 2611 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:14:33 GMT
/admin/log
2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Signature:
AWS tw_access_key:2fWu2+WDCBFlNBPYRkd2BIC9gsM=
2015-07-16 18:14:33,086 2611 [boto][DEBUG ] url = 'http://10.35.64.81/admin/log'
params={'type': 'metadata'}
headers={'Date': 'Thu, 16 Jul 2015 10:14:33 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:2fWu2+WDCBFlNBPYRkd2BIC9gsM=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:14:33,086 2611 [boto][DEBUG ] Method: GET
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Path: /admin/log?type=metadata
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Data:
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Headers: {}
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Host: 10.35.64.81
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Port: 80
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Params: {'type': 'metadata'}
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] Token: None
2015-07-16 18:14:33,087 2611 [boto][DEBUG ] StringToSign:
GET

|
|

Thu, 16 Jul 2015 10:15:11 GMT
/admin/log
2015-07-16 18:15:11,492 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:/jJFDjzjiQkvBPgexaGsG07bVgc=
2015-07-16 18:15:11,502 2621 [radosgw_agent.worker][DEBUG ] bucket instance "BUCKET1:tw-east.24117.1" has 2 entries after " "
2015-07-16 18:15:11,503 2621 [radosgw_agent.worker][INFO ] ********************************************************************************
2015-07-16 18:15:11,504 2621 [radosgw_agent.worker][INFO ] syncing bucket "BUCKET1"
2015-07-16 18:15:11,505 2621 [radosgw_agent.worker][DEBUG ] syncing object BUCKET1/tw.json
2015-07-16 18:15:11,506 2621 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:11 GMT
x-amz-copy-source:BUCKET1/tw.json
/BUCKET1/tw.json
2015-07-16 18:15:11,506 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ=
2015-07-16 18:15:11,507 2621 [boto][DEBUG ] url = 'http://10.35.64.71/BUCKET1/tw.json'
params={'rgwx-op-id': 'node1:2611:1', 'rgwx-source-zone': u'tw-east', 'rgwx-client-id': 'radosgw-agent'}
headers={'Content-Length': '0', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic', 'x-amz-copy-source': 'BUCKET1/tw.json', 'Date': 'Thu, 16 Jul 2015 10:15:11 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Authorization': 'AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ='}
data=None
2015-07-16 18:15:11,507 2621 [boto][DEBUG ] Method: PUT
2015-07-16 18:15:11,507 2621 [boto][DEBUG ] Path: /BUCKET1/tw.json?rgwx-op-id=node1%3A2611%3A1&rgwx-source-zone=tw-east&rgwx-client-id=radosgw-agent
2015-07-16 18:15:11,508 2621 [boto][DEBUG ] Data:
2015-07-16 18:15:11,508 2621 [boto][DEBUG ] Headers: {'Content-Type': 'application/json; charset=UTF-8', 'x-amz-copy-source': 'BUCKET1/tw.json'}
2015-07-16 18:15:11,509 2621 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:15:11,509 2621 [boto][DEBUG ] Port: 80
2015-07-16 18:15:11,510 2621 [boto][DEBUG ] Params: {'rgwx-op-id': 'node1%3A2611%3A1', 'rgwx-source-zone': 'tw-east', 'rgwx-client-id': 'radosgw-agent'}
2015-07-16 18:15:11,510 2621 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2015-07-16 18:15:11,510 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:11,511 2621 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:11 GMT
x-amz-copy-source:BUCKET1/tw.json
/BUCKET1/tw.json
2015-07-16 18:15:11,511 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ=
2015-07-16 18:15:12,420 2621 [boto][DEBUG ] path=/admin/log?lock
2015-07-16 18:15:12,421 2621 [boto][DEBUG ] auth_path=/admin/log?lock
2015-07-16 18:15:12,421 2621 [boto][DEBUG ] StringToSign:
POST

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:12 GMT
/admin/log
2015-07-16 18:15:12,421 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:ZnHewFD7nyOTaPhpAHCh+CQrApc=
2015-07-16 18:15:12,421 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/log?lock'
params={'locker-id': 'node1:2611', 'length': 60, 'zone-id': u'tw-west', 'type': 'data', 'id': 35}
headers={'Date': 'Thu, 16 Jul 2015 10:15:12 GMT', 'Content-Length': '0', 'Content-Type': 'application/json; charset=UTF-8', 'Authorization': 'AWS tw_access_key:ZnHewFD7nyOTaPhpAHCh+CQrApc=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:15:12,422 2621 [boto][DEBUG ] path=/admin/log?lock
2015-07-16 18:15:12,422 2621 [boto][DEBUG ] auth_path=/admin/log?lock
2015-07-16 18:15:12,422 2621 [boto][DEBUG ] Method: POST
2015-07-16 18:15:12,422 2621 [boto][DEBUG ] Path: /admin/log?lock&length=60&locker-id=node1%3A2611&type=data&zone-id=tw-west&id=35
2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Data:
2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Headers: {'Content-Type': 'application/json; charset=UTF-8'}
2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:15:12,423 2621 [boto][DEBUG ] Port: 80
2015-07-16 18:15:12,424 2621 [boto][DEBUG ] Params: {'length': '60', 'locker-id': 'node1%3A2611', 'type': 'data', 'zone-id': 'tw-west', 'id': '35'}
2015-07-16 18:15:12,425 2621 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2015-07-16 18:15:12,426 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:12,426 2621 [boto][DEBUG ] StringToSign:
POST

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:12 GMT
/admin/log
2015-07-16 18:15:12,427 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:ZnHewFD7nyOTaPhpAHCh+CQrApc=
2015-07-16 18:15:12,783 2621 [boto][DEBUG ] Received 500 response. Retrying in 0.1 seconds
2015-07-16 18:15:12,907 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:12,907 2621 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:11 GMT
x-amz-copy-source:BUCKET1/tw.json
/BUCKET1/tw.json
2015-07-16 18:15:12,908 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ=
2015-07-16 18:15:14,372 2621 [boto][DEBUG ] Received 500 response. Retrying in 0.8 seconds
2015-07-16 18:15:15,212 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:15,212 2621 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:11 GMT
x-amz-copy-source:BUCKET1/tw.json
/BUCKET1/tw.json
2015-07-16 18:15:15,212 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ=
2015-07-16 18:15:16,997 2621 [boto][DEBUG ] Received 500 response. Retrying in 3.4 seconds
2015-07-16 18:15:20,392 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:20,393 2621 [boto][DEBUG ] StringToSign:
PUT

application/json; charset=UTF-8
Thu, 16 Jul 2015 10:15:11 GMT
x-amz-copy-source:BUCKET1/tw.json
/BUCKET1/tw.json
2015-07-16 18:15:20,393 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:PM51wrFPVqvlzF1wZZDi3VCPZfQ=
2015-07-16 18:15:20,420 2621 [boto][DEBUG ] Received 500 response. Retrying in 2.6 seconds
2015-07-16 18:15:22,993 2621 [dev.radosgw_agent.worker][WARNING] encountered an error during sync
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 232, in sync_object
local_op_id)
File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 411, in sync_object_intra_region
expect_json=False)
File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 210, in request
check_result_status(error)
File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 154, in check_result_status
exc.HttpError)(result.status, result.reason)
HttpError: Http error code 500 content Internal Server Error
2015-07-16 18:15:22,995 2621 [radosgw_agent.worker][WARNING] encountered an error during sync: Http error code 500 content Internal Server Error
2015-07-16 18:15:22,996 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:22 GMT
/admin/opstate
2015-07-16 18:15:22,996 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:jhck8Nv6pv0dFvk0SjipsGGEtec=
2015-07-16 18:15:22,996 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate'
params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'}
headers={'Date': 'Thu, 16 Jul 2015 10:15:22 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:jhck8Nv6pv0dFvk0SjipsGGEtec=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:15:22,997 2621 [boto][DEBUG ] Method: GET
2015-07-16 18:15:22,997 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1
2015-07-16 18:15:22,997 2621 [boto][DEBUG ] Data:
2015-07-16 18:15:22,998 2621 [boto][DEBUG ] Headers: {}
2015-07-16 18:15:22,998 2621 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:15:22,998 2621 [boto][DEBUG ] Port: 80
2015-07-16 18:15:22,999 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'}
2015-07-16 18:15:22,999 2621 [boto][DEBUG ] establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2015-07-16 18:15:22,999 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:23,000 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:23 GMT
/admin/opstate
2015-07-16 18:15:23,000 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:EvJFrss7PEvy/4xPBVgWv+DwMLg=
2015-07-16 18:15:23,006 2621 [radosgw_agent.worker][DEBUG ] op state is []
2015-07-16 18:15:23,007 2621 [radosgw_agent.worker][DEBUG ] error geting op state: list index out of range
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 275, in wait_for_object
state = state[0]['state']
IndexError: list index out of range
2015-07-16 18:15:24,009 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:24 GMT
/admin/opstate
2015-07-16 18:15:24,009 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:Att2BZKNC6pdw+YxpzAbxydyuis=
2015-07-16 18:15:24,010 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate'
params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'}
headers={'Date': 'Thu, 16 Jul 2015 10:15:24 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:Att2BZKNC6pdw+YxpzAbxydyuis=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:15:24,010 2621 [boto][DEBUG ] Method: GET
2015-07-16 18:15:24,010 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1
2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Data:
2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Headers: {}
2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Port: 80
2015-07-16 18:15:24,011 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'}
2015-07-16 18:15:24,012 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:24,012 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:24 GMT
/admin/opstate
2015-07-16 18:15:24,012 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:Att2BZKNC6pdw+YxpzAbxydyuis=
2015-07-16 18:15:24,020 2621 [radosgw_agent.worker][DEBUG ] op state is []
2015-07-16 18:15:24,020 2621 [radosgw_agent.worker][DEBUG ] error geting op state: list index out of range
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 275, in wait_for_object
state = state[0]['state']
IndexError: list index out of range
2015-07-16 18:15:25,022 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:25 GMT
/admin/opstate
2015-07-16 18:15:25,022 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:VGMI8nTYVHPvPuSjAQDPKgz9rKE=
2015-07-16 18:15:25,023 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate'
params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'}
headers={'Date': 'Thu, 16 Jul 2015 10:15:25 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:VGMI8nTYVHPvPuSjAQDPKgz9rKE=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:15:25,023 2621 [boto][DEBUG ] Method: GET
2015-07-16 18:15:25,023 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1
2015-07-16 18:15:25,023 2621 [boto][DEBUG ] Data:
2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Headers: {}
2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Port: 80
2015-07-16 18:15:25,024 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'}
2015-07-16 18:15:25,025 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:25,025 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:25 GMT
/admin/opstate
2015-07-16 18:15:25,025 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:VGMI8nTYVHPvPuSjAQDPKgz9rKE=
2015-07-16 18:15:25,032 2621 [radosgw_agent.worker][DEBUG ] op state is []
2015-07-16 18:15:25,032 2621 [radosgw_agent.worker][DEBUG ] error geting op state: list index out of range
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/radosgw_agent/worker.py", line 275, in wait_for_object
state = state[0]['state']
IndexError: list index out of range
2015-07-16 18:15:26,035 2621 [boto][DEBUG ] StringToSign:
GET


Thu, 16 Jul 2015 10:15:26 GMT
/admin/opstate
2015-07-16 18:15:26,036 2621 [boto][DEBUG ] Signature:
AWS tw_access_key:Nrt3TAfspX8c+cTM98jovGz2KT0=
2015-07-16 18:15:26,036 2621 [boto][DEBUG ] url = 'http://10.35.64.71/admin/opstate'
params={'client-id': 'radosgw-agent', 'object': u'BUCKET1/tw.json', 'op-id': 'node1:2611:1'}
headers={'Date': 'Thu, 16 Jul 2015 10:15:26 GMT', 'Content-Length': '0', 'Authorization': 'AWS tw_access_key:Nrt3TAfspX8c+cTM98jovGz2KT0=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.13.0-57-generic'}
data=None
2015-07-16 18:15:26,036 2621 [boto][DEBUG ] Method: GET
2015-07-16 18:15:26,036 2621 [boto][DEBUG ] Path: /admin/opstate?client-id=radosgw-agent&object=BUCKET1/tw.json&op-id=node1%3A2611%3A1
2015-07-16 18:15:26,037 2621 [boto][DEBUG ] Data:
2015-07-16 18:15:26,037 2621 [boto][DEBUG ] Headers: {}
2015-07-16 18:15:26,037 2621 [boto][DEBUG ] Host: 10.35.64.71
2015-07-16 18:15:26,039 2621 [boto][DEBUG ] Port: 80
2015-07-16 18:15:26,040 2621 [boto][DEBUG ] Params: {'client-id': 'radosgw-agent', 'object': 'BUCKET1/tw.json', 'op-id': 'node1%3A2611%3A1'}
2015-07-16 18:15:26,040 2621 [boto][DEBUG ] Token: None
2015-07-16 18:15:26,041 2621 [boto][DEBUG ] StringToSign:
GET


(3-3/5)