Ceph

03/20/2024

06:44 AM Ceph Bug #58965: radosgw: Stale entries in bucket indexes following successful DELETE request

I've detected that this issue appears when rgw/cluster gets high count requests, but not every time and I cannot debu... Jonas Nemeikšis

09/26/2023

06:39 AM Ceph Bug #58965: radosgw: Stale entries in bucket indexes following successful DELETE request

This scenario identical affected us too.
Ceph version: 16.2.13
One more fact we also use Restic and stale entri... Jonas Nemeikšis

09/25/2023

01:35 PM rgw Bug #53457: Stale bucket index entry remains after object deletion

Zhong, maybe found more reasons? What's version? Jonas Nemeikšis

08/11/2023

02:29 PM rgw Bug #61716: Production random data not accessible. Object not found on healthy cluster.

I've found that 100% does not depend on the client. I also can reproduce with the aws-cli client..
Conditions and... Jonas Nemeikšis

07:17 AM rgw Bug #61716: Production random data not accessible. Object not found on healthy cluster.

Sorry, I've wrong pasted here cmd. I use rados tool to inspect the missing rados objects. Jonas Nemeikšis

08/02/2023

05:41 AM RADOS Bug #61824: mgr/prometheus: Prometheus metrics type counter decreasing

Yes, I can confirm that metrics went wrong only in this scenario:
1. Added new osds nodes to pool
2. Increased pg... Jonas Nemeikšis

07/31/2023

07:08 PM RADOS Bug #61824: mgr/prometheus: Prometheus metrics type counter decreasing

We've turned off the autoscaler in all clusters. Yes, freshly added OSDs map into the affected pool. Maybe it is rela... Jonas Nemeikšis

07/26/2023

04:48 PM rgw Bug #61716: Production random data not accessible. Object not found on healthy cluster.

Updates, seems like now I can reproduce this issue. Our rgws nodes are accessible through haproxy nodes, we've `timeo... Jonas Nemeikšis

07/25/2023

03:33 PM rgw Bug #47866: Object not found on healthy cluster

Updates, seems like I can semi reproduce it. Repeating upload .tar file 8GB and sometimes broken uploads.
One of ... Jonas Nemeikšis

07/20/2023

02:53 PM rgw Bug #61716: Production random data not accessible. Object not found on healthy cluster.

nope, I don't have any logs and I cannot reproduce it, but I am still trying to do that. One more fact that I found i... Jonas Nemeikšis