Actions
Tasks #11644
closedfirefly v0.80.11
% Done:
0%
Tags:
Reviewed:
Affected Versions:
Pull request ID:
Description
Workflow¶
- Preparing the release IN PROGRESS
- Cutting the release
- Nathan asks Sage if a point release should be published OK
- Nathan gets approval from all leads
- Sage writes and commits the release notes IN PROGRESS
- Nathan informs Yuri that the branch is ready for testing DONE
- Yuri runs additional integration tests DONE
- If Yuri discovers new bugs with severity Critical, the release goes back to being prepared, it was not ready after all
- Yuri informs Alfredo that the branch is ready for release DONE
- Alfredo creates the packages and sets the release tag IN PROGERSS
Release information¶
- branch to build from: firefly, commit:c551622ca21fe044bc1083614c45d888a2a34aeb
- version: v0.80.11
- type of release: point release
- where to publish the release: http://ceph.com/debian-firefly and http://ceph.com/rpm-firefly
git --no-pager log --format='%H %s' --graph tags/v0.80.10..ceph/firefly | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
teuthology run commit:d135e397fa182cebde4d96f52c8eb3075e51b74c (firefly-backports august 2015)¶
git --no-pager log --format='%H %s' --graph ceph/firefly..ceph/firefly-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge (\d+)/) { s|\w+\s+Merge (\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
- + Pull request 4535
- |\
- | + rgw: use correct objv_tracker for bucket instance
- + Pull request 4765
- |\
- | + rgw: always check if token is expired
- + Pull request 4769
- |\
- | + mon: prevent bucket deletion when referenced by a rule
- + Pull request 4788
- |\
- | + mon/OSDMonitor: require mon_allow_pool_delete = true to remove pools
- + Pull request 4854
- |\
- | + tests: verify librbd blocking aio code path
- | + librbd: new rbd_non_blocking_aio config option
- | + PendingReleaseNotes: document changes to librbd's aio_read methods
- | + tests: update librbd AIO tests to remove result code
- | + librbd: AioRequest::send no longer returns a result
- | + librbd: internal AIO methods no longer return result
- | + Throttle: added pending_error method to SimpleThrottle
- | + librbd: add new fail method to AioCompletion
- | + librbd: avoid blocking AIO API methods
- | + librbd: add task pool / work queue for AIO requests
- | + WorkQueue: added virtual destructor
- | + WorkQueue: add new ContextWQ work queue
- | + common/ceph_context: don't import std namespace
- | + CephContext: Add AssociatedSingletonObject to allow CephContext's singleton
- + Pull request 4861
- |\
- | + rgw: Do not enclose the Bucket header in quotes
- + Pull request 5043
- |\
- | + For pgls OP, get/put budget on per list session basis, instead of per OP basis, which could lead to deadlock.
- + Pull request 5056
- |\
- | + rgw/logrotate.conf: Rename service name
- + Pull request 5199
- |\
- | + mon: always reply mdsbeacon
- | + mon/MDSMonitor: rename labels to a better name
- | + mon: send no_reply() to peon to drop ignored mdsbeacon
- | + mon: remove unnecessary error handling
- + Pull request 5200
- |\
- | + ReplicatedPG::maybe_handle_cache: do not skip promote for write_ordered
- | + osd: promotion on 2nd read for cache tiering
- | + ceph_test_rados_api_tier: test promote-on-second-read behavior
- | + osd/osd_types: be pedantic about encoding last_force_op_resend without feature bit
- + Pull request 5217
- |\
- | + ceph.spec.in: python-argparse only in Python 2.6
- + Pull request 5224
- |\
- | + ceph.spec.in: do not run fdupes, even on SLE/openSUSE
- + Pull request 5225
- |\
- | + ceph.spec.in: use _udevrulesdir to eliminate conditionals
- + Pull request 5232
- |\
- | + rgw: simplify content length handling
- | + rgw: make compatability deconfliction optional.
- | + rgw: improve content-length env var handling
- + Pull request 5233
- |\
- | + Unconditionally chown rados log file.
- + Pull request 5234
- |\
- | + rgw: error out if frontend did not send all data
- + Pull request 5235
- |\
- | + os/chain_xattr: handle read on chnk-aligned xattr
- + Pull request 5287
- |\
- | + TestPGLog: fix invalid proc_replica_log test case
- | + TestPGLog: fix noop log proc_replica_log test case
- | + TestPGLog: add test for 11358
- | + PGLog::proc_replica_log: handle split out overlapping entries
- + Pull request 5358
- |\
- | + mon: PaxosService: call post_refresh() instead of post_paxos_update()
- + Pull request 5360
- |\
- | + mon: MonitorDBStore: get_next_key() only if prefix matches
- + Pull request 5388
- |\
- | + UnittestBuffer: Add bufferlist zero test case
- | + buffer: Fix bufferlist::zero bug with special case
- + Pull request 5389
- |\
- | + OSDMonitor: disallow ec pools as tiers
- + Pull request 5390
- |\
- | + rgw/logrotate.conf: Rename service name
- + Pull request 5394
- |\
- | + ceph.spec.in: drop SUSE-specific %py_requires macro
- + Pull request 5403
- |\
- | + Mutex: fix leak of pthread_mutexattr
- + Pull request 5404
- |\
- | + mon/PGMonitor: bug fix pg monitor get crush rule
- + Pull request 5406
- |\
- | + Log::reopen_log_file: take m_flush_mutex
- + Pull request 5408
- |\
- | + mon: OSDMonitor: fix hex output on 'osd reweight'
- + Pull request 5409
- |\
- | + mon/PGMonitor: avoid uint64_t overflow when checking pool 'target/max' status. Fixes: #12401
- + Pull request 5410
- + Update OSDMonitor.cc
rados¶
./virtualenv/bin/teuthology-suite --priority 1000 --subset 5/18 --suite rados --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/smithfarm-2015-08-10_12:21:46-rados-firefly-backports---basic-multi/
- environmental noise (Connection failure: timed out) in two jobs: http://pulpito.ceph.com/smithfarm-2015-08-10_12:21:46-rados-firefly-backports---basic-multi/ and 6 more jobs dead
Re-running dead and failed jobs with:
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rados --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/smithfarm-2015-08-27_03:39:40-rados-firefly-backports---basic-multi/
- known bug http://tracker.ceph.com/issues/12870 'Failed to validate the SSL certificate for raw.githubusercontent.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed
- known bug http://tracker.ceph.com/issues/12870 Error getting key from: https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub
- plus 5 more dead -> according to jluis, these are caused by PR#5360.
Reset integration branch, omitting PR#5360. Re-running dead and failed jobs:
./virtualenv/bin/teuthology-suite --priority 101 --subset 5/18 --suite rados --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports --filter="$filter"
- red http://pulpito.ceph.com/smithfarm-2015-09-02_02:32:32-rados-firefly-backports---basic-multi/
- one job succeeded, killed the rest because I forgot to wait for gitbuilder
Re-run again (really omitting PR#5360 this time):
- red http://pulpito.ceph.com/smithfarm-2015-09-02_14:07:02-rados-firefly-backports---basic-multi/
- environmental problem The command was not found or was not executable: ansible-playbook. Reported the issue here: http://tracker.ceph.com/issues/12926
- just one test left; all others have passed
Re-run the last failed job on the assumption that missing ansible-playbook was "environmental noise":
rgw¶
./virtualenv/bin/teuthology-suite --priority 1000 --suite rgw --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports --subset 5/18
fs¶
./virtualenv/bin/teuthology-suite --priority 1000 --suite fs -k testing --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports --subset 5/18
- red http://pulpito.ceph.com/smithfarm-2015-08-10_12:31:36-fs-firefly-backports-testing-basic-multi/
- environmental noise ConnectionLostError: SSH connection to plana04 was lost: http://pulpito.ceph.com/smithfarm-2015-08-10_12:31:36-fs-firefly-backports-testing-basic-multi/1008824/ - plus 7 dead jobs
Re-running dead and failed jobs with:
./virtualenv/bin/teuthology-suite --priority 1000 --suite fs -k testing --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports --subset 5/18 --filter="$filter"
- green http://pulpito.ceph.com/smithfarm-2015-08-27_03:43:47-fs-firefly-backports-testing-basic-multi/
rbd¶
./virtualenv/bin/teuthology-suite --priority 1000 --suite rbd --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports --subset 5/18
- red http://pulpito.ceph.com/smithfarm-2015-08-10_12:32:34-rbd-firefly-backports---basic-multi/
- 1 failed and 1 dead job, otherwise all passed
Re-running 1 failed and 1 dead job:
./virtualenv/bin/teuthology-suite --priority 1000 --suite rbd --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports --subset 5/18 --filter="$filter"
powercycle¶
./virtualenv/bin/teuthology-suite -l2 -v -c firefly-backports -k testing -m plana,burnupi,mira -s powercycle -p 1000 --email ncutler@suse.cz
- red http://pulpito.ceph.com/smithfarm-2015-08-10_12:35:46-powercycle-firefly-backports-testing-basic-multi/
- environmental problem hopefully fixed by https://github.com/ceph/ceph-cm-ansible/pull/109
Re-run with the same command:
- red http://pulpito.ceph.com/smithfarm-2015-08-31_04:51:54-powercycle-firefly-backports-testing-basic-multi/
- both dead
upgrade¶
./virtualenv/bin/teuthology-suite --machine-type plana,burnupi,mira --suite upgrade/firefly --filter=ubuntu_14.04 --suite-branch firefly --ceph firefly-backports
- red http://pulpito.ceph.com/smithfarm-2015-08-10_12:44:04-upgrade:firefly-firefly-backports---basic-multi/
- 10 dead, 11 passed
Re-running the 10 dead jobs with:
./virtualenv/bin/teuthology-suite --machine-type plana,burnupi,mira --suite upgrade/firefly --suite-branch firefly --ceph firefly-backports --filter="$filter"
Running also on CentOS 6 to verify the sanity of the rpm packages
./virtualenv/bin/teuthology-suite --machine-type plana,burnupi,mira --suite upgrade/firefly --filter=centos_6 --suite-branch firefly --ceph firefly-backports --email ncutler@suse.cz
- red http://pulpito.ceph.com/loic-2015-09-02_07:36:47-upgrade:firefly-firefly-backports---basic-multi/
- all jobs dead
Re-running on CentOS with -m vps
:
- red http://pulpito.ceph.com/smithfarm-2015-09-04_05:32:56-upgrade:firefly-firefly-backports---basic-vps/
- environmental noise "yum -y install ..." failures
Re-running the four failed jobs on CentOS with -m vps
:
- green http://pulpito.ceph.com/smithfarm-2015-09-06_01:24:25-upgrade:firefly-firefly-backports---basic-vps/
ceph-deploy¶
./virtualenv/bin/teuthology-suite --machine-type vps --suite ceph-deploy --suite-branch firefly --ceph firefly-backports --email ncutler@suse.cz
teuthology run commit:b2aaddd3a06ac13c46df659e1f2b3119f5675802 (firefly-backports july 2015)¶
git --no-pager log --format='%H %s' --graph ceph/firefly..ceph/firefly-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge (\d+)/) { s|\w+\s+Merge (\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
- + Pull request 4535
- |\
- | + rgw: use correct objv_tracker for bucket instance
- + Pull request 4582
- |\
- | + ceph_argparse_flag has no regular 3rd parameter.
- + Pull request 4583
- |\
- | + Variable length array of std::strings (not legal in C++) changed to std::vector<std::string>
- + Pull request 4584
- |\
- | + rgw: swift GET / HEAD object returns X-Timestamp field
- + Pull request 4597
- |\
- | + osdc: add epoch_t last_force_resend in Op/LingerOp.
- + Pull request 4630
- |\
- | + ceph-disk: more robust parted output parser
- + Pull request 4631
- + Pull request 4632
- |\
- | + osd: refuse to write a new erasure coded object with an offset > 0
- + Pull request 4633
- + Pull request 4635
- |\
- | + json_sprit: fix the FTBFS on old gcc
- | + json_spirit: use utf8 intenally when parsing \uHHHH
- + Pull request 4636
- |\
- | + Fix disk zap sgdisk invocation
- + Pull request 4639
- |\
- | + librbd: updated cache max objects calculation
- + Pull request 4641
- |\
- | + rgw: remove meta file after deleting bucket The meta file is deleted only if the bucket meta data is not synced
- + Pull request 4642
- |\
- | + rgw: quota not respected in POST object
- + Pull request 4762
- |\
- | + rgw: Use attrs from source bucket on copy
- + Pull request 4765
- |\
- | + rgw: always check if token is expired
- + Pull request 4769
- |\
- | + mon: prevent bucket deletion when referenced by a rule
- + Pull request 4771
- |\
- | + ceph-disk: support NVMe device partitions
- + Pull request 4867
- |\
- | + Always provide summary for non-healthy cluster.
- + Pull request 5037
- |\
- | + Makefile: install ceph-post-file keys with mode 600
- | + ceph-post-file: improve check for a source install
- | + ceph-post-file: behave when sftp doesn't take -i
- + Pull request 5039
- |\
- | + osd: Cleanup boost optionals
- + Pull request 5044
- |\
- | + The fix for issue 9614 was not completed, as a result, for those erasure coded PGs with one OSD down, the state was wrongly marked as active+clean+degraded. This patch makes sure the clean flag is not set for such PG. Signed-off-by: Guang Yang <yguang@yahoo-inc.com>
- + Pull request 5051
- |\
- | + osd: cache pool: flush object ignoring cache min flush age when cache pool is full Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
- | + osd: add local_mtime to struct object_info_t
- + Pull request 5056
- |\
- | + rgw/logrotate.conf: Rename service name
- + Pull request 5062
- |\
- | + Objecter: resend linger ops on any interval change
- | + osd_types: factor out is_new_interval from check_new_interval
- + Pull request 5129
- |\
- | + mon: remove unused variable
- + Pull request 5170
- |\
- | + rgw: send Content-Length in response for HEAD on Swift account.
- | + rgw: send Content-Length in response for DELETE on Swift container.
- | + rgw: send Content-Length in response for PUT on Swift container.
- | + rgw: send Content-Length in response for GET on Swift container.
- | + rgw: enable end_header() to handle proposal of Content-Length.
- + Pull request 5171
- + librbd: assertion failure race condition if watch disconnected
rados¶
./virtualenv/bin/teuthology-suite --priority 1000 --subset 1/18 --suite rados --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
rgw¶
run=loic-2015-07-15_14:21:51-rgw-firefly-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --priority 1000 --filter="$filter" --suite rgw --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
run=loic-2015-07-09_21:11:18-rgw-firefly-backports---basic-multi eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') ./virtualenv/bin/teuthology-suite --priority 1000 --filter="$filter" --suite rgw --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/loic-2015-07-15_14:21:51-rgw-firefly-backports---basic-multi/
- environmental noise skipping: no hosts matched
- environmental noise FATAL: all hosts have already failed -- aborting
./virtualenv/bin/teuthology-suite --priority 1000 --suite rgw --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/loic-2015-07-09_21:11:18-rgw-firefly-backports---basic-multi
- known bug Couldn't find index page for 'boto'
fs¶
run="loic-2015-07-16_13:13:38-fs-firefly-backports-testing-basic-multi"
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/ | jq '.jobs[] | select(.status == "dead") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite fs -k testing --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- green http://pulpito.ceph.com/smithfarm-2015-07-21_02:16:37-fs-firefly-backports-testing-basic-multi/
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/ | jq '.jobs[] | select(.status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite fs -k testing --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- one dead, otherwise green http://pulpito.ceph.com/loic-2015-07-16_13:13:38-fs-firefly-backports-testing-basic-multi/
./virtualenv/bin/teuthology-suite --priority 1000 --suite fs -k testing --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/loic-2015-07-09_21:07:09-fs-firefly-backports-testing-basic-multi
- environmental problem http://pulpito.ceph.com/loic-2015-07-09_21:07:09-fs-firefly-backports-testing-basic-multi/967283/
2015-07-14T02:20:26.544 INFO:teuthology.misc:trying to connect to ubuntu@plana79.front.sepia.ceph.com 2015-07-14T02:20:26.544 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'plana79.front.sepia.ceph.com', 'timeout': 60} 2015-07-14T02:20:28.542 DEBUG:teuthology.orchestra.remote:[Errno 113] No route to host
- environmental problem http://pulpito.ceph.com/loic-2015-07-09_21:07:09-fs-firefly-backports-testing-basic-multi/967319/
2015-07-14T04:31:29.497 DEBUG:teuthology.misc:waited 63.8534479141 2015-07-14T04:31:30.498 INFO:teuthology.misc:trying to connect to ubuntu@plana69.front.sepia.ceph.com 2015-07-14T04:31:30.498 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'plana69.front.sepia.ceph.com', 'timeout': 60} 2015-07-14T04:32:30.498 DEBUG:teuthology.orchestra.remote:timed out
- environmental problem http://pulpito.ceph.com/loic-2015-07-09_21:07:09-fs-firefly-backports-testing-basic-multi/967283/
rbd¶
run=loic-2015-07-09_16:57:58-rbd-firefly-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --priority 1000 --filter="$filter" --suite rbd --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
./virtualenv/bin/teuthology-suite --priority 1000 --suite rbd --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/loic-2015-07-09_16:57:58-rbd-firefly-backports---basic-multi
- environmental noise adjust-ulimits ceph-coverage command fails in two different ways
2015-07-14T02:59:32.995 INFO:tasks.qemu:starting qemu... 2015-07-14T02:59:32.995 INFO:teuthology.orchestra.run.mira077:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term qemu-system-x86_64 -enable-kvm -nographic -m 4096 -drive file=/home/ubuntu/cephtest/qemu/base.client.0.qcow2,format=qcow2,if=virtio -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0:id=0,format=raw,if=virtio,cache=none' 2015-07-14T02:59:32.997 DEBUG:teuthology.run_tasks:Unwinding manager qemu 2015-07-14T02:59:32.998 INFO:tasks.qemu:waiting for qemu tests to finish... 2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:Traceback (most recent call last): 2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr: File "/usr/bin/daemon-helper", line 66, in <module> 2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr: preexec_fn=os.setsid,
- environmental noise adjust-ulimits ceph-coverage command fails in two different ways
powercyle¶
../virtualenv/bin/teuthology-suite -l2 -v -c firefly-backports -k testing -m plana,burnupi,mira -s powercycle -p 1000 --email ncutler@suse.cz
- green http://pulpito.ceph.com/loic-2015-07-09_21:15:54-powercycle-firefly-backports-testing-basic-multi
upgrade¶
teuthology-openstack --verbose --key-name loic --suite upgrade/firefly --filter=ubuntu_14.04 --suite-branch firefly --ceph firefly-backports
teuthology run commit:73cad63e89ebeb9de47c7f2a2187deff8c39d590 (firefly-backports may 2015)¶
rados¶
./virtualenv/bin/teuthology-suite --priority 1000 --subset 1/18 --suite rados --suite-branch firefly --machine-type plana,burnupi,mira --distro ubuntu --email ncutler@suse.cz --ceph firefly-backports
- red http://pulpito.ceph.com/loic-2015-05-29_16:20:09-rados-firefly-backports---basic-multi/
- known bug ceph-objectstore-tool: import failure with status 139
- environmental problem sudo rm
f -/etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart - known bug [ERR] 3.0s0 scrub stat mismatch, got 2005/2009 objects, 0/0 clones, 2005/2009 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 3981889330/3988381157 bytes. in cluster log
Actions