Bug #12021: hammer-x upgrade test.sh fails with ENOTEMPTY - Ceph - Ceph

Actions

Copy link

Bug #12021

closed

hammer-x upgrade test.sh fails with ENOTEMPTY

Added by Sage Weil almost 9 years ago. Updated over 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

upgrade/dumpling-firefly-x, upgrade/firefly-x, upgrade/giant-x, upgrade/hammer-x

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2015-06-13T21:36:05.996 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.PromoteSnapScrub
2015-06-13T21:36:05.996 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.PromoteSnapTrimRace
2015-06-13T21:36:05.996 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.Whiteout
2015-06-13T21:36:05.997 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate
2015-06-13T21:36:05.997 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.Evict
2015-06-13T21:36:05.997 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.EvictSnap
2015-06-13T21:36:05.997 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.EvictSnap2
2015-06-13T21:36:05.998 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.TryFlush
2015-06-13T21:36:05.998 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.Flush
2015-06-13T21:36:05.998 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.FlushSnap
2015-06-13T21:36:05.998 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.FlushTryFlushRaces
2015-06-13T21:36:05.998 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.TryFlushReadRace
2015-06-13T21:36:05.998 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.HitSetRead
2015-06-13T21:36:05.999 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.HitSetWrite
2015-06-13T21:36:05.999 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.HitSetTrim
2015-06-13T21:36:05.999 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.PromoteOn2ndRead
2015-06-13T21:36:05.999 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsPP.ProxyRead
2015-06-13T21:36:06.000 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace
2015-06-13T21:36:06.000 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.Whiteout
2015-06-13T21:36:06.000 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.Evict
2015-06-13T21:36:06.000 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.EvictSnap
2015-06-13T21:36:06.000 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.TryFlush
2015-06-13T21:36:06.000 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.Flush
2015-06-13T21:36:06.001 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.FlushSnap
2015-06-13T21:36:06.001 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.FlushTryFlushRaces
2015-06-13T21:36:06.001 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.TryFlushReadRace
2015-06-13T21:36:06.001 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.HitSetRead
2015-06-13T21:36:06.001 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.HitSetTrim
2015-06-13T21:36:06.002 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.PromoteOn2ndRead
2015-06-13T21:36:06.002 INFO:tasks.workunit.client.1.plana83.stdout:[ FAILED ] LibRadosTwoPoolsECPP.ProxyRead

for example,

2015-06-13T21:36:02.850 INFO:tasks.workunit.client.1.plana83.stdout:[ RUN      ] LibRadosTwoPoolsECPP.ProxyRead
2015-06-13T21:36:02.887 INFO:tasks.workunit.client.1.plana83.stdout:test/librados/tier.cc:4400: Failure
2015-06-13T21:36:02.887 INFO:tasks.workunit.client.1.plana83.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, 
__null)
2015-06-13T21:36:02.888 INFO:tasks.workunit.client.1.plana83.stdout:  Actual: -39
2015-06-13T21:36:02.888 INFO:tasks.workunit.client.1.plana83.stdout:Expected: 0
2015-06-13T21:36:02.964 INFO:tasks.workunit.client.1.plana83.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.ProxyRead (115 ms)

teuthology:sage-2015-06-13_20:20:33-upgrade:hammer-x-master---basic-multi

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

Release set to master
ceph-qa-suite upgrade/hammer-x added

Actions

Copy link

Updated by Sage Weil almost 9 years ago

Assignee set to Sage Weil

Actions

Copy link

Updated by Sage Weil almost 9 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

ceph-qa-suite upgrade/giant-x added

Run: http://pulpito.ceph.redhat.com/teuthology-2015-06-16_17:05:02-upgrade:giant-x-hammer-distro-basic-magna/

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

ceph-qa-suite upgrade/dumpling-firefly-x added

Run: http://pulpito.ceph.redhat.com/teuthology-2015-06-16_17:10:02-upgrade:dumpling-firefly-x-hammer-distro-basic-magna/
Jobs: 133048, 133050

Actions

Copy link

Updated by David Zafman almost 9 years ago

Either the --force-nonempty needs to override the check below (see below), or the test needs to be changed so it doesn't try to add a tier for a pool after snapshots have been created.

2015-06-19 18:16:01.235215 7f57344d1700 1 -- 10.214.133.23:6789/0 --> 10.214.132.24:0/3028365 -- mon_command_ack([{"prefix": "osd tier add", "pool": "test-rados-api-plana54-28365-6", "tierpool": "test-rados-api-plana54-28365-7", "force_nonempty": "--force-nonempty" }]=-39 tier pool 'test-rados-api-plana54-28365-7' has snapshot state; it cannot be added as a tier without breaking the pool v2048) v1 -- ?+0 0x41eec00 con 0x427b1e0

This might be caused by bbec53edf9e585af4e20bbc9ba9057d6fdfda342 in particular this part:

diff --git a/src/mon/OSDMonitor.cc b/src/mon/OSDMonitor.cc
index 10597d0..9d3f6e4 100644
--- a/src/mon/OSDMonitor.cc
+++ b/src/mon/OSDMonitor.cc
@@ -6463,6 +6463,11 @@ done:
       err = -ENOTEMPTY;
       goto reply;
     }
+    if (!tp->removed_snaps.empty() || !tp->snaps.empty()) {
+      ss << "tier pool '" << tierpoolstr << "' has snapshot state; it cannot be added as a tier without breaking the pool";
+      err = -ENOTEMPTY;
+      goto reply;
+    }
     // go
     pg_pool_t *np = pending_inc.get_new_pool(pool_id, p);
     pg_pool_t *ntp = pending_inc.get_new_pool(tierpool_id, tp);

Actions

Copy link

Updated by Samuel Just almost 9 years ago

We need to backport the changes to the test I made to avoid this.

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

Run: http://pulpito.ceph.redhat.com/teuthology-2015-06-20_17:10:02-upgrade:dumpling-firefly-x-hammer-distro-basic-magna/
Jobs: 135828, 135830
Logs: http://magna002.ceph.redhat.com/teuthology-2015-06-20_17:10:02-upgrade:dumpling-firefly-x-hammer-distro-basic-magna/135828/

2015-06-21T05:29:29.007 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:29.047 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:29.048 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:29.089 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.PromoteSnapScrub (46391 ms)
2015-06-21T05:29:29.090 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.PromoteSnapTrimRace
2015-06-21T05:29:29.827 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:556: Failure
2015-06-21T05:29:29.847 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:29.849 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:29.889 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:29.891 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (810 ms)
2015-06-21T05:29:29.895 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.Whiteout
2015-06-21T05:29:29.899 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:597: Failure
2015-06-21T05:29:29.901 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:29.903 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:29.908 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:29.936 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.Whiteout (67 ms)
2015-06-21T05:29:29.966 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.Evict
2015-06-21T05:29:29.968 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:660: Failure
2015-06-21T05:29:29.969 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:29.970 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:29.971 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:29.996 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.Evict (60 ms)
2015-06-21T05:29:30.034 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.EvictSnap
2015-06-21T05:29:31.136 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:803: Failure
2015-06-21T05:29:31.173 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:31.175 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:31.215 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:31.237 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.EvictSnap (1241 ms)
2015-06-21T05:29:31.257 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.EvictSnap2
2015-06-21T05:29:32.447 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:997: Failure
2015-06-21T05:29:32.612 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:32.613 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:32.614 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:32.617 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.EvictSnap2 (1256 ms)
2015-06-21T05:29:32.655 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.TryFlush
2015-06-21T05:29:32.656 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:1052: Failure
2015-06-21T05:29:32.696 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:32.697 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:32.737 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:32.738 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.TryFlush (50 ms)
2015-06-21T05:29:32.778 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.Flush
2015-06-21T05:29:32.779 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:1160: Failure
2015-06-21T05:29:32.819 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:32.820 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:32.860 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:32.861 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.Flush (50 ms)
2015-06-21T05:29:32.902 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.FlushSnap
2015-06-21T05:29:32.904 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:1320: Failure
2015-06-21T05:29:32.944 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:32.946 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:32.986 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:32.990 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.FlushSnap (50 ms)
2015-06-21T05:29:33.028 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.FlushTryFlushRaces
2015-06-21T05:29:33.029 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:1627: Failure
2015-06-21T05:29:33.069 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:33.070 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:33.111 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:33.112 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.FlushTryFlushRaces (50 ms)
2015-06-21T05:29:33.152 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.TryFlushReadRace
2015-06-21T05:29:33.153 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:1827: Failure
2015-06-21T05:29:33.193 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:33.194 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:33.234 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:33.235 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.TryFlushReadRace (51 ms)
2015-06-21T05:29:33.275 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.HitSetRead
2015-06-21T05:29:33.276 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:1923: Failure
2015-06-21T05:29:33.316 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:33.317 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:33.357 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:33.358 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.HitSetRead (50 ms)
2015-06-21T05:29:33.398 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.HitSetWrite
2015-06-21T05:29:33.399 INFO:tasks.workunit.client.0.magna056.stdout:pg_num = 8
2015-06-21T05:29:33.439 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2017: Failure
2015-06-21T05:29:33.440 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:33.480 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:33.481 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:33.521 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.HitSetWrite (52 ms)
2015-06-21T05:29:33.522 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsPP.HitSetTrim
2015-06-21T05:29:33.562 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2099: Failure
2015-06-21T05:29:33.563 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:29:33.603 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:29:33.604 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:29:33.645 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.HitSetTrim (50 ms)
2015-06-21T05:29:34.859 INFO:tasks.workunit.client.0.magna056.stdout:[----------] 17 tests from LibRadosTwoPoolsPP (69534 ms total)
2015-06-21T05:29:34.859 INFO:tasks.workunit.client.0.magna056.stdout:
2015-06-21T05:29:34.860 INFO:tasks.workunit.client.0.magna056.stdout:[----------] 3 tests from LibRadosTierECPP
2015-06-21T05:29:38.412 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTierECPP.Dirty
2015-06-21T05:29:40.433 INFO:tasks.workunit.client.0.magna056.stdout:[       OK ] LibRadosTierECPP.Dirty (2021 ms)
2015-06-21T05:29:40.440 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTierECPP.FlushWriteRaces
2015-06-21T05:29:50.763 INFO:tasks.workunit.client.0.magna056.stdout:[       OK ] LibRadosTierECPP.FlushWriteRaces (10330 ms)
2015-06-21T05:29:50.793 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTierECPP.HitSetNone
2015-06-21T05:29:50.795 INFO:tasks.workunit.client.0.magna056.stdout:[       OK ] LibRadosTierECPP.HitSetNone (12 ms)
2015-06-21T05:29:53.305 INFO:tasks.workunit.client.0.magna056.stdout:[----------] 3 tests from LibRadosTierECPP (12363 ms total)
2015-06-21T05:29:53.305 INFO:tasks.workunit.client.0.magna056.stdout:
2015-06-21T05:29:53.346 INFO:tasks.workunit.client.0.magna056.stdout:[----------] 14 tests from LibRadosTwoPoolsECPP
2015-06-21T05:29:57.890 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.Overlay
2015-06-21T05:30:04.404 INFO:tasks.workunit.client.0.magna056.stdout:[       OK ] LibRadosTwoPoolsECPP.Overlay (6514 ms)
2015-06-21T05:30:04.405 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.Promote
2015-06-21T05:30:06.347 INFO:tasks.ceph.mon.b.magna035.stderr:2015-06-21 05:30:06.348111 7f2ec11e6700 -1 mon.b@0(leader).mds e16 Missing health data for MDS 4434
2015-06-21T05:30:10.546 INFO:tasks.workunit.client.0.magna056.stdout:[       OK ] LibRadosTwoPoolsECPP.Promote (6142 ms)
2015-06-21T05:30:10.609 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.PromoteSnap
2015-06-21T05:30:15.611 INFO:tasks.workunit.client.0.magna056.stdout:waiting for scrub...
2015-06-21T05:30:26.586 DEBUG:teuthology.parallel:result is None
2015-06-21T05:30:30.611 INFO:tasks.workunit.client.0.magna056.stdout:done waiting
2015-06-21T05:30:32.228 INFO:tasks.workunit.client.0.magna056.stdout:[       OK ] LibRadosTwoPoolsECPP.PromoteSnap (21681 ms)
2015-06-21T05:30:32.646 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace
2015-06-21T05:30:33.623 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2572: Failure
2015-06-21T05:30:33.625 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:33.626 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:33.626 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:33.706 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (1478 ms)
2015-06-21T05:30:33.707 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.Whiteout
2015-06-21T05:30:33.729 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2613: Failure
2015-06-21T05:30:33.730 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:33.731 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:33.732 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:33.769 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.Whiteout (63 ms)
2015-06-21T05:30:33.771 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.Evict
2015-06-21T05:30:33.779 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2676: Failure
2015-06-21T05:30:33.780 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:33.781 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:33.782 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:33.824 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.Evict (55 ms)
2015-06-21T05:30:33.825 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.EvictSnap
2015-06-21T05:30:35.074 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2819: Failure
2015-06-21T05:30:35.076 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.077 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.077 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.468 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.EvictSnap (1644 ms)
2015-06-21T05:30:35.469 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.TryFlush
2015-06-21T05:30:35.471 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:2992: Failure
2015-06-21T05:30:35.472 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.473 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.474 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.511 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.TryFlush (43 ms)
2015-06-21T05:30:35.512 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.Flush
2015-06-21T05:30:35.514 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:3100: Failure
2015-06-21T05:30:35.515 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.516 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.516 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.554 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.Flush (43 ms)
2015-06-21T05:30:35.555 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.FlushSnap
2015-06-21T05:30:35.556 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:3260: Failure
2015-06-21T05:30:35.557 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.558 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.559 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.597 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.FlushSnap (43 ms)
2015-06-21T05:30:35.598 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.FlushTryFlushRaces
2015-06-21T05:30:35.599 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:3570: Failure
2015-06-21T05:30:35.600 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.601 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.602 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.640 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (43 ms)
2015-06-21T05:30:35.641 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.TryFlushReadRace
2015-06-21T05:30:35.643 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:3736: Failure
2015-06-21T05:30:35.644 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.644 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.645 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.683 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.TryFlushReadRace (43 ms)
2015-06-21T05:30:35.758 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.HitSetRead
2015-06-21T05:30:35.760 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:3818: Failure
2015-06-21T05:30:35.761 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.761 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.762 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.763 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.HitSetRead (42 ms)
2015-06-21T05:30:35.763 INFO:tasks.workunit.client.0.magna056.stdout:[ RUN      ] LibRadosTwoPoolsECPP.HitSetTrim
2015-06-21T05:30:35.764 INFO:tasks.workunit.client.0.magna056.stdout:test/librados/tier.cc:3959: Failure
2015-06-21T05:30:35.764 INFO:tasks.workunit.client.0.magna056.stdout:Value of: cluster.mon_command( "{\"prefix\": \"osd tier add\", \"pool\": \"" + pool_name + "\", \"tierpool\": \"" + cache_pool_name + "\", \"force_nonempty\": \"--force-nonempty\" }", inbl, __null, __null)
2015-06-21T05:30:35.765 INFO:tasks.workunit.client.0.magna056.stdout:  Actual: -39
2015-06-21T05:30:35.766 INFO:tasks.workunit.client.0.magna056.stdout:Expected: 0
2015-06-21T05:30:35.778 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.HitSetTrim (43 ms)
2015-06-21T05:30:38.375 INFO:tasks.workunit.client.0.magna056.stdout:[----------] 14 tests from LibRadosTwoPoolsECPP (37878 ms total)
2015-06-21T05:30:38.376 INFO:tasks.workunit.client.0.magna056.stdout:
2015-06-21T05:30:38.377 INFO:tasks.workunit.client.0.magna056.stdout:[----------] Global test environment tear-down
2015-06-21T05:30:38.378 INFO:tasks.workunit.client.0.magna056.stdout:[==========] 37 tests from 4 test cases ran. (152651 ms total)
2015-06-21T05:30:38.379 INFO:tasks.workunit.client.0.magna056.stdout:[  PASSED  ] 12 tests.
2015-06-21T05:30:38.380 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] 25 tests, listed below:
2015-06-21T05:30:38.380 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.PromoteSnapScrub
2015-06-21T05:30:38.381 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.PromoteSnapTrimRace
2015-06-21T05:30:38.382 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.Whiteout
2015-06-21T05:30:38.383 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.Evict
2015-06-21T05:30:38.384 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.EvictSnap
2015-06-21T05:30:38.384 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.EvictSnap2
2015-06-21T05:30:38.385 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.TryFlush
2015-06-21T05:30:38.386 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.Flush
2015-06-21T05:30:38.387 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.FlushSnap
2015-06-21T05:30:38.388 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.FlushTryFlushRaces
2015-06-21T05:30:38.388 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.TryFlushReadRace
2015-06-21T05:30:38.389 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.HitSetRead
2015-06-21T05:30:38.390 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.HitSetWrite
2015-06-21T05:30:38.391 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsPP.HitSetTrim
2015-06-21T05:30:38.391 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace
2015-06-21T05:30:38.392 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.Whiteout
2015-06-21T05:30:38.393 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.Evict
2015-06-21T05:30:38.394 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.EvictSnap
2015-06-21T05:30:38.394 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.TryFlush
2015-06-21T05:30:38.395 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.Flush
2015-06-21T05:30:38.396 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.FlushSnap
2015-06-21T05:30:38.397 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.FlushTryFlushRaces
2015-06-21T05:30:38.397 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.TryFlushReadRace
2015-06-21T05:30:38.398 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.HitSetRead
2015-06-21T05:30:38.399 INFO:tasks.workunit.client.0.magna056.stdout:[  FAILED  ] LibRadosTwoPoolsECPP.HitSetTrim
2015-06-21T05:30:38.400 INFO:tasks.workunit.client.0.magna056.stdout:
2015-06-21T05:30:38.400 INFO:tasks.workunit.client.0.magna056.stdout:25 FAILED TESTS

Actions

Copy link