Bug #15087: rgw zone set failure --log-to-stderr --format json -n client.1 -n client.1 zone set --rgw-zone r0z1' - rgw - Ceph

Actions

Copy link

Bug #15087

closed

rgw zone set failure --log-to-stderr --format json -n client.1 -n client.1 zone set --rgw-zone r0z1'

Added by Vasu Kulkarni about 8 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Orit Wasserman

Target version:

% Done:

Source:

other

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

cli fails during zone set

http://magna002.ceph.redhat.com/vasu-2016-03-11_00:54:46-rgw-jewel---basic-magna/214784/teuthology.log


2016-03-11T00:58:11.685 INFO:teuthology.orchestra.run.magna052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p .region0.r0z1.domain_root rm zone_info.default'
2016-03-11T00:58:11.749 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.749412 7fb91bb19a40  1 -- :/0 messenger.start
2016-03-11T00:58:11.755 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.749944 7fb91bb19a40  1 -- :/2061041454 --> 10.8.128.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x5606bde12d00 con 0x5606bde10900
2016-03-11T00:58:11.756 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.750124 7fb91bb0f700  1 -- 10.8.128.52:0/2061041454 learned my addr 10.8.128.52:0/2061041454
2016-03-11T00:58:11.757 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.750620 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7fb8ec000a60 con 0x5606bde10900
2016-03-11T00:58:11.758 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.750670 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (94782053 0 0) 0x7fb8ec000ee0 con 0x5606bde10900
2016-03-11T00:58:11.759 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.750761 7fb902d24700  1 -- 10.8.128.52:0/2061041454 --> 10.8.128.52:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7fb8f00013c0 con 0x5606bde10900
2016-03-11T00:58:11.760 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.751172 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3952013453 0 0) 0x7fb8ec000ee0 con 0x5606bde10900
2016-03-11T00:58:11.761 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.751256 7fb902d24700  1 -- 10.8.128.52:0/2061041454 --> 10.8.128.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7fb8f0001840 con 0x5606bde10900
2016-03-11T00:58:11.762 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.751670 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 393+0+0 (15478327 0 0) 0x7fb8ec001180 con 0x5606bde10900
2016-03-11T00:58:11.763 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.751744 7fb902d24700  1 -- 10.8.128.52:0/2061041454 --> 10.8.128.52:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x5606bde11cf0 con 0x5606bde10900
2016-03-11T00:58:11.764 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.751793 7fb91bb19a40  1 -- 10.8.128.52:0/2061041454 --> 10.8.128.52:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x5606bde11cf0 con 0x5606bde10900
2016-03-11T00:58:11.805 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.751966 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7fb8ec0010e0 con 0x5606bde10900
2016-03-11T00:58:11.805 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.752054 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 6 ==== osd_map(4..4 src has 1..4) v3 ==== 3560+0+0 (1109694628 0 0) 0x7fb8ec001f30 con 0x5606bde10900
2016-03-11T00:58:11.806 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.752166 7fb91bb19a40  1 -- 10.8.128.52:0/2061041454 --> 10.8.128.52:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x5606bde10bb0 con 0x5606bde10900
2016-03-11T00:58:11.807 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.752346 7fb902d24700  1 -- 10.8.128.52:0/2061041454 <== mon.0 10.8.128.52:6789/0 7 ==== mon_check_map_ack(handle=1 version=4) v2 ==== 24+0+0 (896996162 0 0) 0x7fb8ec0010e0 con 0x5606bde10900
2016-03-11T00:58:11.848 INFO:teuthology.orchestra.run.magna052.stderr:error opening pool .region0.r0z1.domain_root: (2) No such file or directory
2016-03-11T00:58:11.849 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.752671 7fb91bb19a40  1 -- 10.8.128.52:0/2061041454 mark_down 0x5606bde10900 -- 0x5606bde33a80
2016-03-11T00:58:11.889 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.752701 7fb91bb19a40  1 -- 10.8.128.52:0/2061041454 mark_down_all
2016-03-11T00:58:11.890 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:11.752838 7fb91bb19a40  1 -- 10.8.128.52:0/2061041454 shutdown complete.
2016-03-11T00:58:11.930 INFO:teuthology.orchestra.run.magna052:Running: 'sudo ceph osd pool create .region0.r0z1.index_pool 64 64'
2016-03-11T00:58:12.045 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.045045 7f92c96be700  1 -- :/0 messenger.start
2016-03-11T00:58:12.049 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.045992 7f92c96be700  1 -- :/3183417432 --> 10.8.128.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x7f92c405fdd0 con 0x7f92c4061100
2016-03-11T00:58:12.050 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.046237 7f92c81c1700  1 -- 10.8.128.52:0/3183417432 learned my addr 10.8.128.52:0/3183417432
2016-03-11T00:58:12.051 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.046737 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7f92a8000a60 con 0x7f92c4061100
2016-03-11T00:58:12.052 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.046790 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1943194608 0 0) 0x7f92a8000f20 con 0x7f92c4061100
2016-03-11T00:58:12.053 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.046884 7f92c257d700  1 -- 10.8.128.52:0/3183417432 --> 10.8.128.52:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f92ac0013c0 con 0x7f92c4061100
2016-03-11T00:58:12.054 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.047275 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1292356599 0 0) 0x7f92a8000f20 con 0x7f92c4061100
2016-03-11T00:58:12.055 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.047376 7f92c257d700  1 -- 10.8.128.52:0/3183417432 --> 10.8.128.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7f92ac001840 con 0x7f92c4061100
2016-03-11T00:58:12.056 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.047783 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 393+0+0 (3364664021 0 0) 0x7f92a8001240 con 0x7f92c4061100
2016-03-11T00:58:12.057 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.047880 7f92c257d700  1 -- 10.8.128.52:0/3183417432 --> 10.8.128.52:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f92c4060010 con 0x7f92c4061100
2016-03-11T00:58:12.058 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.047947 7f92c96be700  1 -- 10.8.128.52:0/3183417432 --> 10.8.128.52:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7f92c405fec0 con 0x7f92c4061100
2016-03-11T00:58:12.059 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.048092 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7f92a8000ba0 con 0x7f92c4061100
2016-03-11T00:58:12.060 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.048182 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 6 ==== osd_map(4..4 src has 1..4) v3 ==== 3560+0+0 (1109694628 0 0) 0x7f92a8000ba0 con 0x7f92c4061100
2016-03-11T00:58:12.062 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.050798 7f92c96be700  1 -- 10.8.128.52:0/3183417432 --> 10.8.128.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7f92c4050510 con 0x7f92c4061100
2016-03-11T00:58:12.063 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.053076 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+38616 (1092875540 0 4143127583) 0x7f92a8000c90 con 0x7f92c4061100
2016-03-11T00:58:12.143 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.143557 7f92c96be700  1 -- 10.8.128.52:0/3183417432 --> 10.8.128.52:6789/0 -- mon_command({"prefix": "osd pool create", "pg_num": 64, "pool": ".region0.r0z1.index_pool", "pgp_num": 64} v 0) v1 -- ?+0 0x7f92c4050510 con 0x7f92c4061100
2016-03-11T00:58:12.566 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.566226 7f92c257d700  1 -- 10.8.128.52:0/3183417432 <== mon.0 10.8.128.52:6789/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 64, "pool": ".region0.r0z1.index_pool", "pgp_num": 64}]=0 pool '.region0.r0z1.index_pool' created v5) v1 ==== 167+0+0 (2577262720 0 0) 0x7f92a8000ba0 con 0x7f92c4061100
2016-03-11T00:58:12.607 INFO:teuthology.orchestra.run.magna052.stderr:pool '.region0.r0z1.index_pool' created
2016-03-11T00:58:12.609 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.609279 7f92c96be700  1 -- 10.8.128.52:0/3183417432 mark_down 0x7f92c4061100 -- 0x7f92c4081c40
2016-03-11T00:58:12.611 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.609317 7f92c96be700  1 -- 10.8.128.52:0/3183417432 mark_down_all
2016-03-11T00:58:12.612 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.609439 7f92c96be700  1 -- 10.8.128.52:0/3183417432 shutdown complete.
2016-03-11T00:58:12.619 INFO:teuthology.orchestra.run.magna052:Running: 'sudo ceph osd erasure-code-profile set r0z1 k=2 m=1 ruleset-failure-domain=osd'
2016-03-11T00:58:12.720 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.720126 7fe274414700  1 -- :/0 messenger.start
2016-03-11T00:58:12.795 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.721520 7fe274414700  1 -- :/1544297514 --> 10.8.128.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x7fe26c05fdd0 con 0x7fe26c061100
2016-03-11T00:58:12.796 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.721677 7fe270493700  1 -- 10.8.128.52:0/1544297514 learned my addr 10.8.128.52:0/1544297514
2016-03-11T00:58:12.797 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.722136 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7fe25c000a60 con 0x7fe26c061100
2016-03-11T00:58:12.798 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.722187 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1007278725 0 0) 0x7fe25c000ee0 con 0x7fe26c061100
2016-03-11T00:58:12.798 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.722283 7fe271495700  1 -- 10.8.128.52:0/1544297514 --> 10.8.128.52:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7fe2500013c0 con 0x7fe26c061100
2016-03-11T00:58:12.799 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.722690 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1351250246 0 0) 0x7fe25c000ee0 con 0x7fe26c061100
2016-03-11T00:58:12.800 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.722775 7fe271495700  1 -- 10.8.128.52:0/1544297514 --> 10.8.128.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7fe250001840 con 0x7fe26c061100
2016-03-11T00:58:12.801 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.723158 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 393+0+0 (2443791311 0 0) 0x7fe25c001180 con 0x7fe26c061100
2016-03-11T00:58:12.802 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.723233 7fe271495700  1 -- 10.8.128.52:0/1544297514 --> 10.8.128.52:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7fe26c060010 con 0x7fe26c061100
2016-03-11T00:58:12.803 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.723283 7fe274414700  1 -- 10.8.128.52:0/1544297514 --> 10.8.128.52:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7fe26c05fec0 con 0x7fe26c061100
2016-03-11T00:58:12.803 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.723522 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7fe25c0010e0 con 0x7fe26c061100
2016-03-11T00:58:12.804 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.723557 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 6 ==== osd_map(5..5 src has 1..5) v3 ==== 3823+0+0 (1921257749 0 0) 0x7fe25c002250 con 0x7fe26c061100
2016-03-11T00:58:12.805 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.726795 7fe274414700  1 -- 10.8.128.52:0/1544297514 --> 10.8.128.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7fe26c050510 con 0x7fe26c061100
2016-03-11T00:58:12.806 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.729340 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+38616 (1092875540 0 4143127583) 0x7fe25c001820 con 0x7fe26c061100
2016-03-11T00:58:12.822 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:12.822993 7fe274414700  1 -- 10.8.128.52:0/1544297514 --> 10.8.128.52:6789/0 -- mon_command({"profile": ["k=2", "m=1", "ruleset-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "r0z1"} v 0) v1 -- ?+0 0x7fe26c050510 con 0x7fe26c061100
2016-03-11T00:58:14.238 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.238577 7fe271495700  1 -- 10.8.128.52:0/1544297514 <== mon.0 10.8.128.52:6789/0 8 ==== mon_command_ack([{"profile": ["k=2", "m=1", "ruleset-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "r0z1"}]=0  v6) v1 ==== 149+0+0 (4024971607 0 0) 0x7fe25c0010e0 con 0x7fe26c061100
2016-03-11T00:58:14.289 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.288729 7fe274414700  1 -- 10.8.128.52:0/1544297514 mark_down 0x7fe26c061100 -- 0x7fe26c081c40
2016-03-11T00:58:14.290 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.288781 7fe274414700  1 -- 10.8.128.52:0/1544297514 mark_down_all
2016-03-11T00:58:14.291 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.289048 7fe274414700  1 -- 10.8.128.52:0/1544297514 shutdown complete.
2016-03-11T00:58:14.299 INFO:teuthology.orchestra.run.magna052:Running: 'sudo ceph osd pool create .region0.r0z1.data_pool 64 64 erasure r0z1'
2016-03-11T00:58:14.407 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.407148 7fc700e7b700  1 -- :/0 messenger.start
2016-03-11T00:58:14.410 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.407963 7fc700e7b700  1 -- :/2539959485 --> 10.8.128.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x7fc6fc060c50 con 0x7fc6fc0610e0
2016-03-11T00:58:14.411 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.408135 7fc70017f700  1 -- 10.8.128.52:0/2539959485 learned my addr 10.8.128.52:0/2539959485
2016-03-11T00:58:14.412 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.408606 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7fc6e0000a60 con 0x7fc6fc0610e0
2016-03-11T00:58:14.413 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.408658 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2595920875 0 0) 0x7fc6e0000ee0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.414 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.408747 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 --> 10.8.128.52:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7fc6e40013c0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.415 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.409187 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (157786167 0 0) 0x7fc6e0000ee0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.416 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.409278 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 --> 10.8.128.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7fc6e4001840 con 0x7fc6fc0610e0
2016-03-11T00:58:14.417 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.409684 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 393+0+0 (1530148092 0 0) 0x7fc6e0001180 con 0x7fc6fc0610e0
2016-03-11T00:58:14.418 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.409761 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 --> 10.8.128.52:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7fc6fc05fab0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.419 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.409833 7fc700e7b700  1 -- 10.8.128.52:0/2539959485 --> 10.8.128.52:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7fc6fc0602d0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.420 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.409992 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 191+0+0 (1254111538 0 0) 0x7fc6e00010e0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.421 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.410098 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 6 ==== osd_map(6..6 src has 1..6) v3 ==== 4017+0+0 (4000935761 0 0) 0x7fc6e0002100 con 0x7fc6fc0610e0
2016-03-11T00:58:14.422 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.413753 7fc700e7b700  1 -- 10.8.128.52:0/2539959485 --> 10.8.128.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7fc6fc0504e0 con 0x7fc6fc0610e0
2016-03-11T00:58:14.462 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.416031 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+38616 (1092875540 0 4143127583) 0x7fc6e0001820 con 0x7fc6fc0610e0
2016-03-11T00:58:14.508 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:14.508421 7fc700e7b700  1 -- 10.8.128.52:0/2539959485 --> 10.8.128.52:6789/0 -- mon_command({"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 64, "erasure_code_profile": "r0z1", "pgp_num": 64, "pool": ".region0.r0z1.data_pool"} v 0) v1 -- ?+0 0x7fc6fc0504e0 con 0x7fc6fc0610e0
2016-03-11T00:58:17.361 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:17.361515 7fc6f9d7c700  1 -- 10.8.128.52:0/2539959485 <== mon.0 10.8.128.52:6789/0 8 ==== mon_command_ack([{"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 64, "erasure_code_profile": "r0z1", "pgp_num": 64, "pool": ".region0.r0z1.data_pool"}]=0 pool '.region0.r0z1.data_pool' created v8) v1 ==== 221+0+0 (674404171 0 0) 0x7fc6e00010e0 con 0x7fc6fc0610e0
2016-03-11T00:58:17.371 INFO:teuthology.orchestra.run.magna052.stderr:pool '.region0.r0z1.data_pool' created
2016-03-11T00:58:17.405 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:17.373176 7fc700e7b700  1 -- 10.8.128.52:0/2539959485 mark_down 0x7fc6fc0610e0 -- 0x7fc6fc081bf0
2016-03-11T00:58:17.406 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:17.373219 7fc700e7b700  1 -- 10.8.128.52:0/2539959485 mark_down_all
2016-03-11T00:58:17.446 INFO:teuthology.orchestra.run.magna052.stderr:2016-03-11 05:58:17.373360 7fc700e7b700  1 -- 10.8.128.52:0/2539959485 shutdown complete.
2016-03-11T00:58:17.447 INFO:tasks.util.rgw:rgwadmin: client.1 : ['-n', 'client.1', 'zone', 'set', '--rgw-zone', 'r0z1']
2016-03-11T00:58:17.487 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '-n', 'client.1', 'zone', 'set', '--rgw-zone', 'r0z1']
2016-03-11T00:58:17.488 INFO:teuthology.orchestra.run.magna052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 zone set --rgw-zone r0z1'
2016-03-11T00:58:19.236 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/ceph-qa-suite_jewel/tasks/rgw.py", line 678, in configure_regions_and_zones
    check_status=True)
  File "/home/teuthworker/src/ceph-qa-suite_jewel/tasks/util/rgw.py", line 38, in rgwadmin
    stdin=stdin,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthw