Activity
From 11/06/2022 to 12/05/2022
12/05/2022
- 11:55 PM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- /a/lflores-2022-12-02_20:40:02-rados-wip-yuri6-testing-2022-11-23-1348-distro-default-smithi/7101847
- 11:53 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- imirc tw wrote:
> I think this is container related. I suspect the container has a default timeout of 10 seconds for... - 03:56 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- I think this is container related. I suspect the container has a default timeout of 10 seconds for commands, after th...
- 02:38 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- for the record, the shutdown log before the gap:...
- 02:25 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- again, a gap:
2022-12-05T14:23:33.437+0000 7efcbdb19700 10 bluestore.OnodeSpace(0x562debe3f9a0 in 0x562deb647000) ... - 02:21 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- I noticed the same, not sure why though. Will create a new log with debug to 10.
- 01:57 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- I can see ~17 seconds missing in the log:
2022-12-05T09:15:31.918+0000 7f820508e700 20 bluestore.BufferSpace(0x555cf... - 01:12 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- Needed to split it in several parts due to upload limit, took -100k + 100k to be sure.
- 12:42 PM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- well, let it be -50000 and +50000 lines from OSD restart point, below is a sample log line which indicates such a poi...
- 09:04 AM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- Igor Fedotov wrote:
> Could you please set fast osd shutdown to false and debug-bluestore to 20. Then restart OSD an... - 07:55 AM Orchestrator Feature #58158: OSD container's shutdown timeout should be raised (and configurable)
- Could you please set fast osd shutdown to false and debug-bluestore to 20. Then restart OSD and share shutdown/startu...
- 11:51 PM bluestore Backport #58181 (Resolved): pacific: KernelDevice::read doesn't translate error codes correctly
- 11:51 PM bluestore Backport #58180 (Resolved): quincy: KernelDevice::read doesn't translate error codes correctly
- 11:48 PM rgw Backport #58119: pacific: check-generated.sh failures for rgw_log_entry
- Casey Bodley wrote:
> https://github.com/ceph/ceph/pull/49142
merged - 11:46 PM bluestore Bug #57857 (Pending Backport): KernelDevice::read doesn't translate error codes correctly
- 11:39 PM rbd Bug #57726 (In Progress): [rbd_support] set_localized_module_option(..., None) is spamming the au...
- 11:37 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- https://shaman.ceph.com/builds/ceph/wip-revert-pr-48713/2b583578473c82604cfdab2faef9f161dc2fb0b9/
- 11:20 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- The bug reproduced on Yuri's test branch. The difference between the test branch and the main SHA is that the test br...
- 07:23 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- Laura Flores wrote:
> Scheduled 50x tests to run here: http://pulpito.front.sepia.ceph.com/lflores-2022-12-05_17:05:... - 07:22 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- I have a feeling that the tests I scheduled earlier on the main branch all passed since the SHA it picked up is older...
- 07:14 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- Wondering if there could have been a regression caused by https://github.com/ceph/ceph/pull/48713.
- 06:38 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- /a/yuriw-2022-11-28_21:26:12-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7095988
/a/lflores-2022-1... - 04:17 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- Scheduled 50x tests to run here: http://pulpito.front.sepia.ceph.com/lflores-2022-12-05_17:05:59-rados-wip-yuri10-tes...
- 04:10 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- Three recent instances of this bug in the main branch point to a regression. My next steps here will be to schedule m...
- 10:46 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- That is every log file from every node. There are no ceph-mgr* logs. :/
Even from inside the docker on the adm n... - 06:33 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- Hello. Thanks for response and the files....
- 09:11 PM RADOS Bug #58173: api_aio_pp: failure on LibRadosAio.SimplePoolEIOFlag and LibRadosAio.PoolEIOFlag
- Building a branch here with https://github.com/ceph/ceph/pull/49029 reverted, which can be used to verify whether it ...
- 09:03 PM RADOS Bug #58173: api_aio_pp: failure on LibRadosAio.SimplePoolEIOFlag and LibRadosAio.PoolEIOFlag
- Excuse my update Sam, I see you already added it as a duplicate.
- 08:55 PM RADOS Bug #58173: api_aio_pp: failure on LibRadosAio.SimplePoolEIOFlag and LibRadosAio.PoolEIOFlag
- Matan added that test within the last two weeks: https://github.com/ceph/ceph/pull/49029
- 07:10 PM RADOS Bug #58173 (Resolved): api_aio_pp: failure on LibRadosAio.SimplePoolEIOFlag and LibRadosAio.PoolE...
- The workunits/rados/test.sh script is run in the orch suite on some tests. In a few of them, these two tests were fai...
- 09:02 PM crimson Bug #58179: crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag
- Excuse my update Sam, I see you already added it as a duplicate.
- 08:49 PM crimson Bug #58179 (Duplicate): crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag
- 08:43 PM crimson Bug #58179 (Duplicate): crimson: api_aio_pp: [ FAILED ] LibRadosAio.SimplePoolEIOFlag
- Note, this is with https://github.com/ceph/ceph/pull/49029 merged.
https://pulpito.ceph.com/sjust-2022-12-01_22:28... - 08:52 PM rgw Bug #57562: multisite replication issue on Quincy
- Hey Adam,
Quick update on the testing that was done on both the latest PR commits (multimap and vector one), we did ... - 08:06 PM RADOS Bug #58178: FAILED ceph_assert(last_e.version.version < e.version.version)
- Noticed an osd, doing this, on a cluster over the weekend. Its been crashing consistently since.
- 08:05 PM RADOS Bug #58178 (Need More Info): FAILED ceph_assert(last_e.version.version < e.version.version)
- debug -4> 2022-12-05T19:14:03.556+0000 7fe51028a200 5 osd.57 pg_epoch: 261349 pg[1.573( v 261349'617978754 (2613...
- 07:41 PM Orchestrator Backport #58177 (Resolved): pacific: cephadm/ingress: support TLS RGW backend
- https://github.com/ceph/ceph/pull/49917
- 07:41 PM Orchestrator Backport #58176 (Resolved): quincy: cephadm/ingress: support TLS RGW backend
- https://github.com/ceph/ceph/pull/49865
- 07:40 PM Orchestrator Backport #58175 (Resolved): pacific: cephadm: cephadm always reports new or changed devices even ...
- https://github.com/ceph/ceph/pull/49916
- 07:40 PM Orchestrator Backport #58174 (Resolved): quincy: cephadm: cephadm always reports new or changed devices even i...
- https://github.com/ceph/ceph/pull/49864
- 07:39 PM Orchestrator Bug #57999 (Pending Backport): cephadm: cephadm always reports new or changed devices even if dev...
- 07:35 PM Orchestrator Feature #51972 (Pending Backport): cephadm/ingress: support TLS RGW backend
- 07:07 PM RADOS Bug #56733: Since Pacific upgrade, sporadic latencies plateau on random OSD/disks
- I've just let Mark and Ronen know about this issue.
- 07:06 PM rbd Bug #58172 (Duplicate): get_rollback_snap_id throws bad variant access
- The function get_rollback_snap_id generates core dump when force promoting image...
- 07:05 PM RADOS Bug #58156: Monitors do not permit OSD to join after upgrading to Quincy
- Radoslaw Zarzynski wrote:
> Hi Igor! What was the intermediary version during the upgrade? We merged https://github.... - 06:40 PM RADOS Bug #58156: Monitors do not permit OSD to join after upgrading to Quincy
- Hi Igor! What was the intermediary version during the upgrade? We merged https://github.com/ceph/ceph/pull/44090 but ...
- 07:00 PM RADOS Bug #58142 (In Progress): rbd-python snaps-many-objects: deep-scrub : stat mismatch
- Moving to @In progress@ basing the core standup 1 Dec.
- 06:58 PM rgw Backport #58171 (Resolved): quincy: RGW misplaces index entries after dynamically resharding bucket
- https://github.com/ceph/ceph/pull/49795
- 06:58 PM rgw Backport #58170 (Duplicate): pacific: RGW misplaces index entries after dynamically resharding bu...
- 06:56 PM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- Hello!
what is on disk is actually serialized from the the in-memory representation. We don't see huge numbers of ... - 06:51 PM rgw Bug #58034 (Pending Backport): RGW misplaces index entries after dynamically resharding bucket
- 06:24 PM RADOS Bug #58166 (Need More Info): mon:DAEMON_OLD_VERSION newer versions is considered older than earlier
- If your cluster is in the same state, can you please share mon logs with debug_mon=20? The following code snippet in ...
- 02:53 PM RADOS Bug #58166: mon:DAEMON_OLD_VERSION newer versions is considered older than earlier
- This was probably introduced in https://github.com/ceph/ceph/pull/36759
- 02:52 PM RADOS Bug #58166 (Need More Info): mon:DAEMON_OLD_VERSION newer versions is considered older than earlier
- We have a cluster with most mon/mgr/osd are running 16.2.10 and some OSDs are running 16.2.9
The healthcheck does ... - 06:24 PM RADOS Backport #58169 (Resolved): quincy: extra debugs for: [mon] high cpu usage by fn_monstore thread
- https://github.com/ceph/ceph/pull/50406
- 06:16 PM RADOS Feature #58168 (Pending Backport): extra debugs for: [mon] high cpu usage by fn_monstore thread
- 06:16 PM RADOS Feature #58168 (Pending Backport): extra debugs for: [mon] high cpu usage by fn_monstore thread
- 06:10 PM RADOS Bug #53806: unessesarily long laggy PG state
- > I think as long as `acting` does not have duplicate entries, the logic is exactly the same as before.
Yeah. I'm ... - 05:51 PM RADOS Backport #55768: pacific: rados_api_tests: LibRadosWatchNotify.AioWatchNotify2 fails
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/46499
merged - 05:34 PM RADOS Backport #56648: quincy: [Progress] Do not show NEW PG_NUM value for pool if autoscaler is set to...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/47925
merged - 05:15 PM RADOS Fix #57963: osd: Misleading information displayed for the running configuration of osd_mclock_max...
- https://github.com/ceph/ceph/pull/48708 merged
- 05:12 PM RADOS Bug #57782: [mon] high cpu usage by fn_monstore thread
- Radoslaw Zarzynski wrote:
> NOT A FIX (extra debugs): https://github.com/ceph/ceph/pull/48513
merged - 05:04 PM rgw Bug #58167: No Authentication/Authorization for creating topics on RGW
- In my example in the original comment the curl was run on a node inside the Ceph test cluster (of Apple M1 Max VMs).
... - 04:34 PM rgw Bug #58167: No Authentication/Authorization for creating topics on RGW
- * creating a topic by using curl without any user credential is a critical securuty issue.
* since topics are global... - 04:09 PM rgw Bug #58167 (Pending Backport): No Authentication/Authorization for creating topics on RGW
- I'm on a containerized Ceph 17.2.5 serving only RGW/S3 clients.
I'm experimenting with notifications for S3 bucket... - 04:56 PM CephFS Backport #57875: pacific: Permissions of the .snap directory do not inherit ACLs
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48553
merged - 04:54 PM CephFS Backport #57880: pacific: NFS client unable to see newly created files when listing directory con...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48521
merged - 04:53 PM bluestore Backport #57687 (Resolved): pacific: unable to read osd superblock on AArch64 with page size 64K
- 04:52 PM bluestore Backport #57687: pacific: unable to read osd superblock on AArch64 with page size 64K
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/48278
merged - 04:53 PM CephFS Backport #57848: pacific: mgr/volumes: addition of human-readable flag to volume info command
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48468
merged - 04:53 PM Backport #57593 (Resolved): pacific: STORE==USED in ceph df
- 04:51 PM Backport #57593: pacific: STORE==USED in ceph df
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/48250
merged - 04:50 PM bluestore Backport #57027 (Resolved): pacific: Bluefs might put an orpan op_update record in the log
- 04:45 PM bluestore Backport #57027: pacific: Bluefs might put an orpan op_update record in the log
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/48168
merged - 04:49 PM bluestore Backport #55300 (Resolved): pacific: Hybrid allocator might return duplicate extents when perform...
- 04:44 PM bluestore Backport #55300: pacific: Hybrid allocator might return duplicate extents when performing online ...
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/48167
merged - 04:48 PM bluestore Bug #54554 (Resolved): Bluestore volume selector improperly tracks bluefs log size
- 04:42 PM bluestore Bug #54554: Bluestore volume selector improperly tracks bluefs log size
- https://github.com/ceph/ceph/pull/45408 merged
- 04:02 PM RADOS Bug #58165 (Fix Under Review): rados: fix extra tabs on warning for pool copy
- 12:57 PM RADOS Bug #58165 (Resolved): rados: fix extra tabs on warning for pool copy
- BZ link: https://bugzilla.redhat.com/show_bug.cgi?id=2148242
- 04:00 PM rbd Backport #58076: pacific: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = ...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/49054
merged - 03:52 PM RADOS Bug #57632 (Fix Under Review): test_envlibrados_for_rocksdb: free(): invalid pointer
- 09:42 AM rbd Bug #56724 (New): [rbd_support] recover from RADOS instance blocklisting
- 07:52 AM mgr Feature #58164 (In Progress): mgr/prometheus: introduce fetch_perf_counters_metrics module option
- 07:43 AM mgr Feature #58164 (Pending Backport): mgr/prometheus: introduce fetch_perf_counters_metrics module o...
- By default, starting from ceph version `17.2.6`, prometheus module no more export perf counters as prometheus metrics...
- 07:37 AM RADOS Bug #57940: ceph osd crashes with FAILED ceph_assert(clone_overlap.count(clone)) when nobackfill ...
- Thomas Le Gentil wrote:
> I could avoid this crash by removing all pg for which ceph could not get the clone_bytes, ... - 07:34 AM Tasks #58163 (In Progress): doc/monitoring: add documentation for ceph-exporter daemon
- 03:16 AM crimson Bug #58162 (New): seastar sharded object destructor assert when crimosd handle error
- The backtrace :...
- 02:31 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Now it's an occasional problem, and I'm watching to see if it happens again. I'm not sure what the trigger condition ...
12/04/2022
- 04:57 PM rbd Tasks #58161 (New): [luks] exercise layered encryption with ceph_test_librbd_fsx
- fsx has proven very useful in weeding out clone and flatten-related bugs in layering code, both in librbd and krbd.
- 04:40 PM rbd Bug #58160 (New): [luks] encryption-aware flatten/resize can be proxied to a peer that doesn't ha...
- Currently, if this happens, all encryption awareness is stripped and the operation is carried out as if the user didn...
- 02:41 PM Dashboard Feature #58159: mgr/dashboard: add option to set frame ancestors in CSP header mgr/dashboard: sh...
- Avan Thakkar wrote:
> h3. Description
Introduce a dashboard module option to allow to set frame ancestors host sour... - 02:40 PM Dashboard Feature #58159 (Fix Under Review): mgr/dashboard: add option to set frame ancestors in CSP heade...
- h3. Description
Introduce a dashboard module option to allow to set frame ancestors host sources so the page can be ... - 12:11 PM Orchestrator Bug #57771: orch/cephadm suite: 'TESTDIR=/home/ubuntu/cephtest bash -s' fails
- /a/yuriw-2022-11-28_21:13:47-rados-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7095058
/a/yuriw-2022-12... - 12:06 PM Orchestrator Bug #52321: qa/tasks/rook times out: 'check osd count' reached maximum tries (90) after waiting f...
- /a/yuriw-2022-11-28_21:13:47-rados-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7095046/
- 12:00 PM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- /a/yuriw-2022-11-28_21:13:47-rados-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7095033/
- 11:56 AM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- /a/yuriw-2022-11-28_21:13:47-rados-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7095031/
- 11:46 AM RADOS Bug #53789: CommandFailedError (rados/test_python.sh): "RADOS object not found" causes test_rados...
- /a/yuriw-2022-11-23_21:36:17-rados-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7089814/
- 09:41 AM RADOS Backport #58144 (In Progress): pacific: mon/MonCommands: Support dump_historic_slow_ops
- 09:37 AM Orchestrator Feature #58158 (Resolved): OSD container's shutdown timeout should be raised (and configurable)
- Currently OSD starts are very slow in quincy v.17.2.5. Disabling osd fast shutdown has no impact.
2022-12-04T09:22... - 09:37 AM RADOS Backport #58143 (In Progress): quincy: mon/MonCommands: Support dump_historic_slow_ops
12/03/2022
- 06:18 AM rbd Bug #57765 (Resolved): rbd CLI inconsistencies affecting "--namespace" arg
- 06:17 AM rbd Backport #57844 (Resolved): pacific: rbd CLI inconsistencies affecting "--namespace" arg
12/02/2022
- 09:49 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- In a passed job, the crashes are posted:...
- 09:33 PM RADOS Bug #58098 (In Progress): qa/workunits/rados/test_crash.sh: crashes are never posted
- In the job that passed, the mgr.server reports a recent crash:
/a/lflores-2022-11-30_22:53:49-rados-main-distro-de... - 09:06 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- In one of the jobs that passed, the OSDs were also failed for 31 seconds, but this time, the crashes were detected. S...
- 09:02 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- Didn't reproduce in the 20x run above, but it did reproduce a second time here:
/a/yuriw-2022-11-28_21:09:37-rados... - 09:26 PM Bug #58157 (New): Ceph Orchestrator fails to recognize partition.
- I have a system that has an unmounted XFS volume (it was failing and was unmounted) that was created using the root d...
- 08:57 PM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- /a/yuriw-2022-11-28_21:09:37-rados-wip-yuri4-testing-2022-11-10-1051-distro-default-smithi/7094882
- 08:56 PM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- @John might be. I'll mark it as related.
- 02:57 PM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- Duplicate of https://tracker.ceph.com/issues/58145 perhaps?
- 08:53 PM Orchestrator Bug #56000: task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`
- /a/yuriw-2022-11-28_21:09:37-rados-wip-yuri4-testing-2022-11-10-1051-distro-default-smithi/7094866
- 08:29 PM rbd Backport #57844: pacific: rbd CLI inconsistencies affecting "--namespace" arg
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48459
merged - 06:09 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- Attaching server2 to this message.
- 06:09 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- I am realizing those logs are from a single host (server4).
server3 got removed today.
Attaching server1 to this me... - 05:42 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- Radoslaw Zarzynski wrote:
> Well, I think the command you mentioned did effect for RGW, not MGR. I'm providing the c... - 04:22 PM rgw Bug #57562: multisite replication issue on Quincy
- Thanks Adam, we looked at the new PR and see that you are using a vector instead of multi-map. And then do a find on ...
- 03:28 PM RADOS Bug #58156 (In Progress): Monitors do not permit OSD to join after upgrading to Quincy
- 03:28 PM RADOS Bug #58156 (Resolved): Monitors do not permit OSD to join after upgrading to Quincy
- The Nautilus cluster has been eventually upgraded to Quincy and at the end OSDs stopped joining the cluster.
The i... - 03:24 PM RADOS Bug #58155 (Resolved): mon:ceph_assert(m < ranks.size()) `different code path than tracker 50089`
- Same problem with https://tracker.ceph.com/issues/50089, but it is a different code path.
We opened a new tracker ... - 02:51 PM rbd Backport #58084 (Resolved): pacific: [rbd_support] pool removal can wedge the task queue
- 02:49 PM rbd Backport #58084: pacific: [rbd_support] pool removal can wedge the task queue
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/49056
merged - 02:39 PM rbd Backport #58151 (In Progress): quincy: [rbd-nbd] add --snap-id option to "rbd device map" to allo...
- 12:24 PM rbd Backport #58151 (Resolved): quincy: [rbd-nbd] add --snap-id option to "rbd device map" to allow m...
- https://github.com/ceph/ceph/pull/49197
- 02:30 PM rbd Backport #58152 (In Progress): quincy: rbd snap list not change the last read when more than 64 g...
- 12:24 PM rbd Backport #58152 (Resolved): quincy: rbd snap list not change the last read when more than 64 grou...
- https://github.com/ceph/ceph/pull/49196
- 02:28 PM rbd Backport #58153 (In Progress): pacific: rbd snap list not change the last read when more than 64 ...
- 12:24 PM rbd Backport #58153 (Resolved): pacific: rbd snap list not change the last read when more than 64 gro...
- https://github.com/ceph/ceph/pull/49195
- 02:12 PM CephFS Feature #58154 (Resolved): mds: add minor segment boundaries
- See PR/commits.
- 12:18 PM rbd Bug #57066 (Pending Backport): rbd snap list not change the last read when more than 64 group snaps
- 12:12 PM rbd Bug #57902 (Pending Backport): [rbd-nbd] add --snap-id option to "rbd device map" to allow mappin...
- 10:58 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- xianpao chen wrote:
> Is there a good way to monitor the read/write speed of the fuse and kernel client?
Is this ... - 09:48 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Is there a good way to monitor the read/write speed of the fuse and kernel client?
- 03:52 AM Orchestrator Feature #58150 (Resolved): Addhigh level host related information to the orch host ls command
- This feature provides a --detail flag on the 'orch host ls' command to show high level information that cephadm colle...
- 02:59 AM Bug #58120: watcher remains after "rados watch" is interrupted
- Ilya Dryomov wrote:
> Hi Wang,
>
> When the client is taken down gracefully, it closes its watch(es). When the c... - 01:31 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- Nitzan Mordechai wrote:
> 王子敬 wang wrote:
> > Nitzan Mordechai wrote:
> > > Since you attached part of the pglog, ... - 01:30 AM bluestore Feature #57785: fragmentation score in metrics
- I didn't know it was a problem until I tripped across it. The warning I think does more help then harm. Having a docu...
- 01:06 AM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- Linked a possible solution for skipping ubuntu with this test. I scheduled a teuthology test for it, which I will use...
- 12:43 AM bluestore Bug #58022: Fragmentation score rising by seemingly stuck thread
- These run away osds are not on the heavy delete workload cluster. Its a relatively lightly loaded cluster. though I c...
12/01/2022
- 09:44 PM RADOS Bug #58130: LibRadosAio.SimpleWrite hang and pkill
- Thanks for your observations, Brad! I'm going to dedicate this Tracker to `LibRadosAio.SimpleWrite` and mark it as re...
- 09:20 PM RADOS Bug #58130: LibRadosAio.SimpleWrite hang and pkill
- The issue appears to be in the api_aio test as it gets started but doesn't complete....
- 08:04 PM RADOS Bug #58130: LibRadosAio.SimpleWrite hang and pkill
- Ran into another instance of this here:
/a/yuriw-2022-11-30_23:13:27-rados-wip-yuri2-testing-2022-11-30-0724-pacif... - 09:43 PM RADOS Bug #57618: rados/test.sh hang and pkilled (LibRadosWatchNotifyEC.WatchNotify)
- /a/yuriw-2022-11-29_22:29:58-rados-wip-yuri10-testing-2022-11-29-1005-pacific-distro-default-smithi/7097464/
- 09:23 PM RADOS Bug #57751: LibRadosAio.SimpleWritePP hang and pkill
- possibly 58130 is related
- 08:44 PM bluestore Bug #58022: Fragmentation score rising by seemingly stuck thread
- Igor/Adam - "But the behavior stops immediately on restart. So feels like some thread in the osd is doing something u...
- 07:30 PM RADOS Cleanup #58149 (Resolved): Clarify pool creation failure message due to exceeding max_pgs_per_osd
- This was inspired by the Re: [ceph-users] proxmox hyperconverged pg calculations in ceph pacific, pve 7.2 thread.
- 07:30 PM RADOS Bug #50089 (Resolved): mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when reducing number of...
- 06:59 PM RADOS Bug #50089 (New): mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when reducing number of moni...
- 07:05 PM Orchestrator Bug #57303: rados/cephadm: Failed to fetch package version from https://shaman.ceph.com/api/searc...
- @Adam in the rados suite, this test seems to have additional distros: https://github.com/ceph/ceph/tree/pacific/qa/su...
- 05:54 PM rgw Bug #57562: multisite replication issue on Quincy
- I have a more thoroughly cleaned up and refactored fix at.
Apart from other changes, it collapses identical journa... - 05:30 PM Orchestrator Backport #58148 (Resolved): pacific: test_cephadm.sh: Error: Error initializing source docker://q...
- https://github.com/ceph/ceph/pull/49285
- 05:30 PM Orchestrator Bug #58145: orch/cephadm: nfs tests failing to mount exports (mount -t nfs 10.0.31.120:/fake /mnt...
- I attempted to debug this situation locally on a 3-node VM cluster. I am able to reproduce the case where mount.nfs f...
- 04:33 PM Orchestrator Bug #58145 (Pending Backport): orch/cephadm: nfs tests failing to mount exports (mount -t nfs 10....
- Currently, since the sepia lab has recovered, all tests that attempt to mount NFS exports are no longer passing. All ...
- 05:30 PM Orchestrator Backport #58147 (Resolved): quincy: test_cephadm.sh: Error: Error initializing source docker://qu...
- https://github.com/ceph/ceph/pull/49284
- 05:27 PM Fix #58135 (Fix Under Review): radosgw:Fix Statement repetition
- 03:18 AM Fix #58135 (Fix Under Review): radosgw:Fix Statement repetition
- There are two statements "using rgw::IAM::s3ListBucket;" in “ceph/src/test/rgw/test_rgw_iam_policy.cc”, it's repeated.
- 05:21 PM Orchestrator Bug #58146 (Pending Backport): test_cephadm.sh: Error: Error initializing source docker://quay.ce...
- 05:02 PM Orchestrator Bug #58146 (Resolved): test_cephadm.sh: Error: Error initializing source docker://quay.ceph.io/ce...
- /a/yuriw-2022-11-28_21:21:22-rados-wip-yuri5-testing-2022-11-18-1554-quincy-distro-default-smithi/7095613...
- 05:09 PM Orchestrator Bug #56000: task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`
- /a/yuriw-2022-11-28_21:21:22-rados-wip-yuri5-testing-2022-11-18-1554-quincy-distro-default-smithi/7095434
- 04:20 PM Orchestrator Bug #57287 (Closed): orch/cephadm: 'mount -t nfs 10.0.31.109:/foouser /mnt/foo' failing in nfs-rg...
- closing as this stopped happening a long time ago. There does happen to be another similar looking issue right now (y...
- 04:12 PM RADOS Backport #58144 (Resolved): pacific: mon/MonCommands: Support dump_historic_slow_ops
- https://github.com/ceph/ceph/pull/49233
- 04:12 PM RADOS Backport #58143 (Resolved): quincy: mon/MonCommands: Support dump_historic_slow_ops
- https://github.com/ceph/ceph/pull/49232
- 04:02 PM RADOS Bug #58141 (Pending Backport): mon/MonCommands: Support dump_historic_slow_ops
- 12:42 PM RADOS Bug #58141 (Resolved): mon/MonCommands: Support dump_historic_slow_ops
- Slow ops are being tracked in the mon while `dump_historic_slow_ops` command is not registered:
```
$ ceph daemon .... - 03:56 PM RADOS Bug #58142 (In Progress): rbd-python snaps-many-objects: deep-scrub : stat mismatch
- ...
- 03:47 PM rgw Bug #58127 (Fix Under Review): multisite: test_zg_master_zone_delete fails
- 03:45 PM RADOS Bug #56733: Since Pacific upgrade, sporadic latencies plateau on random OSD/disks
- It seems more like generic RADOS issue.
- 03:14 PM bluestore Bug #58099 (Need More Info): ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixPreferDeferred/2 ...
- 03:12 PM rgw Bug #58104 (Won't Fix - EOL): `putlc` failed in slave zonegroup
- 03:12 PM rgw Bug #58105 (Won't Fix - EOL): `DeleteBucketPolicy` can not delete policy in slave zonegroup
- the nautilus release is no longer supported. this was fixed in pacific
- 03:08 PM rgw Bug #58125 (Won't Fix - EOL): In the nautilus version ceph, the notification message "awsRegion" ...
- the nautilus release is no longer supported so won't receive any more backports
- 06:14 AM rgw Bug #58125: In the nautilus version ceph, the notification message "awsRegion" parameter is null
- https://tracker.ceph.com/issues/53186, can be backport to N.
- 03:07 PM rgw Bug #58136 (Fix Under Review): usage trim has infinite loop problem
- 09:04 AM rgw Bug #58136: usage trim has infinite loop problem
https://github.com/ceph/ceph/pull/49168- 06:35 AM rgw Bug #58136 (Fix Under Review): usage trim has infinite loop problem
- try usage trim only specifying "--bucket", when first MAX_USAGE_TRIM_ENTRIES entries in cls method RGW_USER_USAGE_LOG...
- 03:06 PM Orchestrator Bug #58140 (Need More Info): quay.ceph.io/ceph-ci/ceph: manifest unknown
- hi Aishwarya, this isn't an rgw bug - could you please move it to the appropriate project?
- 12:01 PM Orchestrator Bug #58140: quay.ceph.io/ceph-ci/ceph: manifest unknown
- /a/yuriw-2022-11-28_16:28:53-rados-wip-yuri-testing-2022-11-18-1500-pacific-distro-default-smithi/7094314/
/a/yuriw-... - 11:20 AM Orchestrator Bug #58140 (Need More Info): quay.ceph.io/ceph-ci/ceph: manifest unknown
- Seeing the following error in cephadm/test_cephadm.sh failure:
2022-11-30T02:06:35.681 INFO:tasks.workunit.client... - 01:37 PM Bug #58120 (Need More Info): watcher remains after "rados watch" is interrupted
- Hi Wang,
When the client is taken down gracefully, it closes its watch(es). When the client is shut down ungracef... - 01:33 PM CephFS Feature #58129: mon/FSCommands: support swapping file systems by name
- Venky Shankar wrote:
> The operation also needs to swap the fsid and no clients should we interfering when the swap ... - 04:10 AM CephFS Feature #58129: mon/FSCommands: support swapping file systems by name
- The operation also needs to swap the fsid and no clients should we interfering when the swap is under execution.
- 01:23 PM Linux kernel client Bug #58126 (Fix Under Review): kclient: WARNING: fs/crypto/keyring.c:242 fscrypt_destroy_keyring+...
- The patchwork link: https://patchwork.kernel.org/project/ceph-devel/patch/20221201065800.18149-1-xiubli@redhat.com/
- 12:27 PM RADOS Bug #57757 (Fix Under Review): ECUtil: terminate called after throwing an instance of 'ceph::buff...
- 11:04 AM CephFS Bug #58138 (In Progress): "ceph nfs cluster info" shows junk data for non-existent cluster
- 09:18 AM CephFS Bug #58138 (Resolved): "ceph nfs cluster info" shows junk data for non-existent cluster
- BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2149415
Steps to Reproduce(we will use a non-existent cluster name... - 09:56 AM Bug #58139 (New): MON spams with Refuse to add /dev/sdq due to limit policy
- Hi,
Ceph version 17.2.5, freshly installed with cephadm
All OSDs are fully installed and functional but the MON... - 09:39 AM Orchestrator Bug #57311: rook: ensure CRDs are installed first
- /a/yuriw-2022-11-28_16:28:53-rados-wip-yuri-testing-2022-11-18-1500-pacific-distro-default-smithi/7094422/
/a/yuriw-... - 09:32 AM Dashboard Backport #57829: pacific: cephadm/test_dashboard_e2e.sh: Expected to find content: '/^foo$/' with...
- /a/yuriw-2022-11-30_23:13:27-rados-wip-yuri2-testing-2022-11-30-0724-pacific-distro-default-smithi/7099298
- 09:25 AM Orchestrator Bug #53501: Exception when running 'rook' task.
- /a/yuriw-2022-11-28_16:28:53-rados-wip-yuri-testing-2022-11-18-1500-pacific-distro-default-smithi/7094123/
- 09:23 AM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- /a/yuriw-2022-11-28_21:26:12-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7096011
- 09:08 AM ceph-volume Bug #58137 (Fix Under Review): OSD prepare job fails with KeyError: 'KNAME'
- 08:46 AM ceph-volume Bug #58137 (Resolved): OSD prepare job fails with KeyError: 'KNAME'
- ...
- 08:18 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- 王子敬 wang wrote:
> Nitzan Mordechai wrote:
> > Since you attached part of the pglog, i can't see how many entries yo... - 01:50 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- Nitzan Mordechai wrote:
> Since you attached part of the pglog, i can't see how many entries you have for log and ho... - 07:55 AM bluestore Backport #58102 (In Progress): pacific: BlueStore doesn't defer small writes for pre-pacific hdd ...
- 03:41 AM RADOS Bug #53806: unessesarily long laggy PG state
- Radoslaw Zarzynski wrote:
> OK, Aishwarya has found in testing that the @break@-related commit (https://github.com/c... - 02:15 AM Documentation #58134 (New): doc/radosgw:index error
- There is a sentence in the note of chapter "Pool Placement and storage classes": "If you have not done any previous M...
- 01:28 AM CephFS Feature #58133 (Fix Under Review): qa: add test cases for fscrypt feature in kernel CephFS client
- 01:22 AM CephFS Feature #58133 (Resolved): qa: add test cases for fscrypt feature in kernel CephFS client
- As per the documentation fscrypt is a (kernel) "library which filesystems can hook into to support transparent encryp...
- 12:51 AM RADOS Backport #58040: quincy: osd: add created_at and ceph_version_when_created metadata
- please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/49159
ceph-backport.sh versi...
11/30/2022
- 11:15 PM RADOS Bug #58132 (In Progress): qa/standalone/mon: --mon-initial-members setting causes us to populate ...
- 11:08 PM RADOS Bug #58132 (Resolved): qa/standalone/mon: --mon-initial-members setting causes us to populate rem...
- Problem:
--mon-initial-members does nothing but cause monmap
to populate ``removed_ranks`` because the way we sta... - 10:57 PM RADOS Bug #58098: qa/workunits/rados/test_crash.sh: crashes are never posted
- Neha suggested we see how reproducible this is, so as not to mask any underlying problems by sleeping longer. I sched...
- 10:56 PM Orchestrator Cleanup #58131 (New): orchestrator: infrastructure for tracking dependencies
- We have a few dependencies that affect the CI testing. Occasionally problems will pop up where a new version of that ...
- 10:34 PM RADOS Bug #58130 (In Progress): LibRadosAio.SimpleWrite hang and pkill
- A rados api test experienced a failure after the last global tests had successfully run.
/a/yuriw-2022-11-29_22:29... - 09:48 PM CephFS Bug #54643 (Duplicate): crash: void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*): ass...
- 09:48 PM CephFS Bug #53179 (Duplicate): Crash when unlink in corrupted cephfs
- 09:47 PM CephFS Bug #38452 (Need More Info): mds: assert crash loop while unlinking file
- 07:31 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- Well, I think the command you mentioned did effect for RGW, not MGR. I'm providing the commands increasing log verbos...
- 07:25 PM RADOS Bug #57977: osd:tick checking mon for new map
- The issue during the upgrade looks awfully similar to a downstream Prashant has working on.
Prashant, would find som... - 07:14 PM Orchestrator Bug #57374: orch/cephadm: nfs-ingress-rgw tests failing with "Command failed on smithi158 with st...
- /a/yuriw-2022-11-29_22:29:58-rados-wip-yuri10-testing-2022-11-29-1005-pacific-distro-default-smithi/7097649
- 07:09 PM RADOS Bug #58106 (Need More Info): when a large number of error ops appear in the OSDs,pglog does not t...
- 10:43 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- Since you attached part of the pglog, i can't see how many entries you have for log and how many for dups
can you pl... - 08:38 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- 王子敬 wang wrote:
> Nitzan Mordechai wrote:
> > @王子敬 wang, can you please send us the output for one of the pgs from ... - 08:32 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- Nitzan Mordechai wrote:
> @王子敬 wang, can you please send us the output for one of the pgs from ceph-objectstore-tool... - 07:30 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- @王子敬 wang, can you please send us the output for one of the pgs from ceph-objectstore-tool?...
- 02:16 AM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- Nitzan Mordechai wrote:
> @王子敬 wang can you please provide the output of 'ceph pg dump' ?
ok, the output in the pg_... - 07:07 PM RADOS Bug #57546: rados/thrash-erasure-code: wait_for_recovery timeout due to "active+clean+remapped+la...
- I think the invariant here is that the @acting@ container should not have duplicates. If it is broken, we have a more...
- 01:55 PM RADOS Bug #57546: rados/thrash-erasure-code: wait_for_recovery timeout due to "active+clean+remapped+la...
- If there are indeed duplicated entries in the acting set, should there be a 'break' at all in this loop? It seems lik...
- 07:00 PM RADOS Bug #53806: unessesarily long laggy PG state
- OK, Aishwarya has found in testing that the @break@-related commit (https://github.com/ceph/ceph/pull/44499/commits/9...
- 02:02 PM RADOS Bug #53806: unessesarily long laggy PG state
- FWIW, we've seen this happen very frequently during Nautilus->{Octopus,Pacific} upgrades. I had just tracked down the...
- 05:05 PM CephFS Feature #58129 (Pending Backport): mon/FSCommands: support swapping file systems by name
- Storage operators like Rook constantly do "reconciliation" to ensure that the desired state of the system (e.g. file ...
- 04:44 PM Orchestrator Bug #57311: rook: ensure CRDs are installed first
- /a/yuriw-2022-11-29_22:29:58-rados-wip-yuri10-testing-2022-11-29-1005-pacific-distro-default-smithi/7097553
- 04:03 PM Bug #58128 (New): FTBFS with fmtlib 9.1.0
- two recent commits:
01f6bef6e5250bf568d9d5f6acf37de64236159a and
3f67d67ede11605b78f9e3abeacd9c46c37d3794
(con... - 04:02 PM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Krunal Chheda wrote:
> > Thanks once again for explanation, i was trying to figure out how t... - 02:48 AM rgw Bug #57562: multisite replication issue on Quincy
- Krunal Chheda wrote:
> Thanks once again for explanation, i was trying to figure out how the assert_exists() works ... - 02:15 AM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Krunal Chheda wrote:
> > Regarding the ENOENT, with assert_exists in place now, will the rea... - 01:38 AM rgw Bug #57562: multisite replication issue on Quincy
- Specifically in cls_fifo_legacy.cc/push_part()
- 01:37 AM rgw Bug #57562: multisite replication issue on Quincy
- Krunal Chheda wrote:
> Regarding the ENOENT, with assert_exists in place now, will the read_part_header on the trim... - 01:19 AM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> I've pushed a commit that uses assert_exists, then fetches metadata on -ENOENT.
Thanks for ... - 01:07 AM rgw Bug #57562: multisite replication issue on Quincy
- I've pushed a commit that uses assert_exists, then fetches metadata on -ENOENT.
- 03:47 PM rgw Bug #58127 (Resolved): multisite: test_zg_master_zone_delete fails
- ex. http://qa-proxy.ceph.com/teuthology/cbodley-2022-09-29_01:41:10-rgw-wip-rgw-sal-bootstrap-distro-default-smithi/7...
- 03:38 PM CephFS Bug #24403: mon failed to return metadata for mds
- Was discussion about this tracker with Patrick - there are separate paxos proposals for fsmap update and the metadata...
- 02:27 PM CephFS Bug #24403: mon failed to return metadata for mds
- The MDS is identified using a nonce as well as an IP in the map, right? After the containerized OSDs managed to clobb...
- 02:23 PM CephFS Bug #24403: mon failed to return metadata for mds
- Venky Shankar wrote:
> It seems the MDS can miss sending beacon in up:boot state. This state encodes the MDS metadat... - 12:52 PM CephFS Bug #24403: mon failed to return metadata for mds
- It seems the MDS can miss sending beacon in up:boot state. This state encodes the MDS metadata and includes that in t...
- 03:36 PM RADOS Bug #58114 (Closed): mon: FAILED ceph_assert(rank == new_rank)
- Close due to this issue is found pre-merge testing from PR: https://github.com/ceph/ceph/pull/48698/
- 01:57 PM rgw Backport #58119 (In Progress): pacific: check-generated.sh failures for rgw_log_entry
- https://github.com/ceph/ceph/pull/49142
- 01:22 PM Dashboard Bug #54992: pacific: rados/dashboard: tasks/dashboard: cannot stat '/etc/containers/registries.co...
- /a/yuriw-2022-11-29_22:29:58-rados-wip-yuri10-testing-2022-11-29-1005-pacific-distro-default-smithi/7097412
- 11:53 AM bluestore Bug #58113 (In Progress): BLK/Kernel: Improve protection against running one OSD twice
- 09:53 AM Linux kernel client Bug #58126: kclient: WARNING: fs/crypto/keyring.c:242 fscrypt_destroy_keyring+0x7e/0xd0
- Xiubo Li wrote:
> By reading the code it should be a bug in *fs/crypto/* code. When closing the file it will be dela... - 09:52 AM Linux kernel client Bug #58126: kclient: WARNING: fs/crypto/keyring.c:242 fscrypt_destroy_keyring+0x7e/0xd0
- By reading the code it should be a bug in *fs/crypto/* code. When closing the file it will be delayed in kernel space...
- 09:46 AM Linux kernel client Bug #58126 (Fix Under Review): kclient: WARNING: fs/crypto/keyring.c:242 fscrypt_destroy_keyring+...
- In https://pulpito.ceph.com/xiubli-2022-11-30_03:04:20-fs:fscrypt-wip-lxb-fscrypt-20221122-0933-distro-default-smithi...
- 09:25 AM CephFS Bug #57014 (Fix Under Review): cephfs-top: add an option to dump the computed values to stdout
- 09:09 AM rbd Bug #58110: Live migration from a QCOW2 source
- There are several issues:
1.
librbd does not detect that a qcow2 is actually zlib compressed. There's no eviden... - 08:26 AM rgw Bug #58125 (Won't Fix - EOL): In the nautilus version ceph, the notification message "awsRegion" ...
- The content of the message is as follows:
{
"Records": [{
"eventVersion": "2.2",
"eventSource... - 07:43 AM Documentation #58124 (Resolved): doc:Word repetition
- Word repetition in swift API
- 06:38 AM Documentation #58123 (Resolved): ceph:correct help message of "--admin-daemon ADMIN_SOCKET"
- The description ‘("help" for help’ after the command "--admin-daemon ADMIN_SOCKET" in the ceph help document is redun...
- 06:23 AM Dashboard Backport #58121 (In Progress): pacific: mgr/dashboard: relax the cors policies
- 06:18 AM Dashboard Backport #58121 (Resolved): pacific: mgr/dashboard: relax the cors policies
- https://github.com/ceph/ceph/pull/49151
- 06:22 AM Dashboard Backport #58122 (In Progress): quincy: mgr/dashboard: relax the cors policies
- 06:18 AM Dashboard Backport #58122 (Resolved): quincy: mgr/dashboard: relax the cors policies
- https://github.com/ceph/ceph/pull/49150
- 06:16 AM Dashboard Bug #58086 (Pending Backport): mgr/dashboard: relax the cors policies
- 06:16 AM Dashboard Bug #58086 (Fix Under Review): mgr/dashboard: relax the cors policies
- 06:02 AM Dashboard Bug #58086 (Pending Backport): mgr/dashboard: relax the cors policies
- 06:02 AM Dashboard Bug #58086 (Fix Under Review): mgr/dashboard: relax the cors policies
- 04:26 AM Dashboard Bug #58086 (Pending Backport): mgr/dashboard: relax the cors policies
- 04:14 AM RADOS Backport #58039: pacific: osd: add created_at and ceph_version_when_created metadata
- please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/49144
ceph-backport.sh versi... - 02:48 AM Bug #58120 (Fix Under Review): watcher remains after "rados watch" is interrupted
- when client is down, rbd image still has watchers. I take the down client into osd blacklist ,but still has watchers....
11/29/2022
- 11:40 PM rgw Bug #57562: multisite replication issue on Quincy
- This is very useful, thank you, and might explain why it's happening.
So from your logs, does it seem like the par... - 11:18 PM rgw Bug #57562: multisite replication issue on Quincy
- So coming back to EIO analysis, what we have found so far is this issue happens for more than 1 RGW instances running...
- 09:08 PM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Can you point me to the PR for retrying on EIO?
https://github.com/adamemerson/ceph/pull/4/... - 04:39 PM rgw Bug #57562: multisite replication issue on Quincy
- Can you point me to the PR for retrying on EIO?
- 11:18 PM RADOS Bug #54438: test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid].data, ...
- /a/yuriw-2022-11-28_16:28:53-rados-wip-yuri-testing-2022-11-18-1500-pacific-distro-default-smithi/7094026
- 09:35 PM rgw Backport #58119 (Resolved): pacific: check-generated.sh failures for rgw_log_entry
- 09:35 PM rgw Backport #58118 (In Progress): quincy: check-generated.sh failures for rgw_log_entry
- 09:28 PM rgw Bug #58115 (Pending Backport): check-generated.sh failures for rgw_log_entry
- 06:36 PM rgw Bug #58115 (Fix Under Review): check-generated.sh failures for rgw_log_entry
- 06:35 PM rgw Bug #58115 (Pending Backport): check-generated.sh failures for rgw_log_entry
- ...
- 07:44 PM rgw Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- J. Eric Ivancich wrote:
> The code on the PR seems to address the issue. My colleague Mark Kogan ran it through a te... - 07:21 PM rgw Bug #58111 (Fix Under Review): crash: verify_bucket_owner_or_policy
- 09:42 AM rgw Bug #58111: crash: verify_bucket_owner_or_policy
- ...
- 09:28 AM rgw Bug #58111 (Resolved): crash: verify_bucket_owner_or_policy
- When executing 's3cmd ls s3://a:', rgw was terminated. ...
- 07:14 PM RADOS Backport #58117 (In Progress): quincy: qa/workunits/rados/test_librados_build.sh: specify redirec...
- https://github.com/ceph/ceph/pull/49140
- 06:58 PM RADOS Backport #58117 (In Progress): quincy: qa/workunits/rados/test_librados_build.sh: specify redirec...
- 07:11 PM RADOS Backport #58116 (In Progress): pacific: qa/workunits/rados/test_librados_build.sh: specify redire...
- https://github.com/ceph/ceph/pull/49139
- 06:58 PM RADOS Backport #58116 (Resolved): pacific: qa/workunits/rados/test_librados_build.sh: specify redirect ...
- 06:52 PM RADOS Bug #58046 (Pending Backport): qa/workunits/rados/test_librados_build.sh: specify redirect in cur...
- 05:37 PM RADOS Bug #58046: qa/workunits/rados/test_librados_build.sh: specify redirect in curl command
- Seen in Pacific run: /a/yuriw-2022-11-28_21:10:48-rados-wip-yuri10-testing-2022-11-28-1042-pacific-distro-default-smi...
- 06:33 PM Orchestrator Bug #58066 (Fix Under Review): config key for cephadm hosts doesn't have values for all network i...
- 06:12 PM Feature #58112: Add OpenMPI and ROMIO modules to support "striping_factor" from MPI standard
- People at OpenMPI created skeleton for CephFS support:
https://github.com/open-mpi/ompi/pull/11122
If anyone wa... - 12:46 PM Feature #58112 (New): Add OpenMPI and ROMIO modules to support "striping_factor" from MPI standard
- Hi,
I am creating this feature request but maybe there is already an ongoing work about adding CephFS support for ... - 05:52 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- We discussed this tracker in the RADOS meeting. Sam pointed out that this set of tests doesn't have any actual users,...
- 05:52 PM Orchestrator Bug #54071: rados/cephadm/osds: Invalid command: missing required parameter hostname(<string>)
- /a/yuriw-2022-11-28_21:10:48-rados-wip-yuri10-testing-2022-11-28-1042-pacific-distro-default-smithi/7095174
- 05:47 PM cephsqlite Bug #51372: pacific: libcephsqlite: segmentation fault
- /a/yuriw-2022-11-28_21:10:48-rados-wip-yuri10-testing-2022-11-28-1042-pacific-distro-default-smithi/7095357
Coredump... - 05:24 PM RADOS Bug #58114 (Closed): mon: FAILED ceph_assert(rank == new_rank)
- /a/yuriw-2022-11-28_21:10:48-rados-wip-yuri10-testing-2022-11-28-1042-pacific-distro-default-smithi/7095280/remote/sm...
- 05:07 PM bluestore Bug #58113 (Resolved): BLK/Kernel: Improve protection against running one OSD twice
- 04:59 PM RADOS Bug #44595: cache tiering: Error: oid 48 copy_from 493 returned error code -2
- ...
- 03:05 PM RADOS Bug #58107: mon-stretch: old stretch_marked_down_mons leads to ceph unresponsive
- Therefore, there is nothing we can do but wait for the other site to come back up, so pgs can complete peering and th...
- 03:04 PM RADOS Bug #58107 (Closed): mon-stretch: old stretch_marked_down_mons leads to ceph unresponsive
- Closed due to this is not a corner case but quote from Greg Farnum:
``it’s that electing those two monitors means ... - 04:15 AM RADOS Bug #58107 (In Progress): mon-stretch: old stretch_marked_down_mons leads to ceph unresponsive
- 04:14 AM RADOS Bug #58107 (Closed): mon-stretch: old stretch_marked_down_mons leads to ceph unresponsive
- h1. How to reproduce the issue
h2. Set up:
mon.a (zone 1) rank=0
mon.b (zone 1) rank=1
mon.c (zone 2) rank=2
... - 01:07 PM RADOS Bug #58106: when a large number of error ops appear in the OSDs,pglog does not trim.
- @王子敬 wang can you please provide the output of 'ceph pg dump' ?
- 01:42 AM RADOS Bug #58106 (Need More Info): when a large number of error ops appear in the OSDs,pglog does not t...
- When We use the s3 interface append and copy of the object gateway, a large number of error ops appear in the OSDs wh...
- 11:44 AM Dashboard Bug #48686 (Resolved): mgr/dashboard: permission denied when creating a NFS export
- 11:44 AM Dashboard Backport #57692 (Resolved): quincy: mgr/dashboard: permission denied when creating a NFS export
- 11:23 AM crimson Bug #58089 (In Progress): get_acting_recovery_backfill().empty() assertion
- 11:12 AM RADOS Bug #57940: ceph osd crashes with FAILED ceph_assert(clone_overlap.count(clone)) when nobackfill ...
- I could avoid this crash by removing all pg for which ceph could not get the clone_bytes, except the one I was sure t...
- 10:40 AM rgw Bug #44660: Multipart re-uploads cause orphan data
- It is very big problem for us.
We have a lot of big buckets with orphaned parts which use hundreds TBs of space.
... - 10:39 AM rgw Bug #16767: RadosGW Multipart Cleanup Failure
- It is very big problem for us.
We have a lot of big buckets with orphaned parts which use hundreds TBs of space.
... - 09:02 AM RADOS Backport #57496 (Resolved): quincy: Invalid read of size 8 in handle_recovery_delete()
- 08:16 AM rbd Bug #58110 (New): Live migration from a QCOW2 source
- When using certain QCOW2 images as the source format for live migration, I encounter errors. With some QCOW2 images, ...
- 07:13 AM CephFS Feature #56489: qa: test mgr plugins with standby mgr failover
- New pull request with mgr thrasher.
- 07:05 AM RADOS Bug #50042 (Fix Under Review): rados/test.sh: api_watch_notify failures
- 05:10 AM CephFS Bug #58109 (Pending Backport): ceph-fuse: doesn't work properly when the version of libfuse is 3....
- I want to use ceph-fuse with libfuse which version is 3.6 or later, because it supports for fuse kernel feature `max_...
- 04:53 AM Dashboard Cleanup #58108 (Fix Under Review): mgr/dashboard: add an env variable to chose custom build folder
- 04:51 AM Dashboard Cleanup #58108 (Fix Under Review): mgr/dashboard: add an env variable to chose custom build folder
- as title says
- 04:50 AM CephFS Bug #58095: snap-schedule: handle non-existent path gracefully during snapshot creation
- Venky Shankar wrote:
> Milind Changire wrote:
> > The most common mistake that users tend to do is include the moun... - 02:13 AM rgw Bug #54908: crash: double const md_config_t::get_val<double>(ConfigValues const&, std::basic_stri...
- Similar problem.
(gdb) bt full ... - 02:04 AM CephFS Bug #58090: Non-existent pending clone shows up in snapshot info
- The `/volumes/_index/clone/` directory is empty, by the way. But that's after the snapshot was deleted successfully. ...
- 02:01 AM CephFS Bug #58090: Non-existent pending clone shows up in snapshot info
- Now the snapshot is deleted (finally). From the logs of our CSI provisioner, it seems that the snapshot was deleted s...
- 01:35 AM rgw Bug #58105 (Won't Fix - EOL): `DeleteBucketPolicy` can not delete policy in slave zonegroup
- 01:26 AM CephFS Feature #58070: qa: add test suite to test old kernels
- Patrick Donnelly wrote:
> This is certainly a good thing to add. Where do we want to put it? fs:workload?
I was ... - 01:16 AM CephFS Feature #58070: qa: add test suite to test old kernels
- This is certainly a good thing to add. Where do we want to put it? fs:workload? We need to be careful to avoid testin...
- 01:18 AM CephFS Feature #58072: enable 'ceph fs new' use 'ceph fs set' options
- I think at this point we should consider making it possible to set arbitrary settings on a fs during creation. i.e. a...
- 01:11 AM rgw Bug #58104 (Won't Fix - EOL): `putlc` failed in slave zonegroup
- radosgw version: nautilus(14.2.15)
11/28/2022
- 10:24 PM RADOS Bug #58098 (Fix Under Review): qa/workunits/rados/test_crash.sh: crashes are never posted
- 05:34 PM RADOS Bug #58098 (Resolved): qa/workunits/rados/test_crash.sh: crashes are never posted
- /a/yuriw-2022-11-23_15:09:06-rados-wip-yuri10-testing-2022-11-22-1711-distro-default-smithi/7087281...
- 09:57 PM bluestore Backport #58103 (Resolved): quincy: BlueStore doesn't defer small writes for pre-pacific hdd osds
- https://github.com/ceph/ceph/pull/49333
- 09:56 PM bluestore Backport #58102 (Resolved): pacific: BlueStore doesn't defer small writes for pre-pacific hdd osds
- https://github.com/ceph/ceph/pull/49170
- 09:56 PM bluestore Bug #56488: BlueStore doesn't defer small writes for pre-pacific hdd osds
- Konstantin Shalygin wrote:
> Igor, PR should be replaced to 48490?
yep! done. - 09:56 PM bluestore Bug #56488 (Pending Backport): BlueStore doesn't defer small writes for pre-pacific hdd osds
- 07:38 PM bluestore Bug #56488: BlueStore doesn't defer small writes for pre-pacific hdd osds
- Igor, PR should be replaced to 48490?
- 09:43 PM RADOS Bug #56733: Since Pacific upgrade, sporadic latencies plateau on random OSD/disks
- Just a follow-up.
Finally, what's helping us the best is increasing osd_scrub_sleep to 0.4. - 08:38 PM Orchestrator Feature #58101 (Resolved): cephadm: allow setting mon crush locations
- Useful for integrating with stretch mode, particularly for stretch mode, and even more specifically, is necessary for...
- 07:29 PM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Yes, I just pushed it. We're still testing for regression, but you're welcome to try it.
Than... - 05:31 PM rgw Bug #57562: multisite replication issue on Quincy
- Yes, I just pushed it. We're still testing for regression, but you're welcome to try it.
- 05:13 PM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Also the goal is to remove tags as part of the fix.
>
> We had some confusion over where a ... - 03:36 PM rgw Bug #57562: multisite replication issue on Quincy
- So currently without the tag changes we see a issue as mentioned here in "comment":https://tracker.ceph.com/issues/57...
- 03:17 PM rgw Bug #57562: multisite replication issue on Quincy
- Also the goal is to remove tags as part of the fix.
We had some confusion over where a regression is and it's conf... - 03:17 PM rgw Bug #57562: multisite replication issue on Quincy
- > Also based on our previous test analysis, we think removal of tags will still not completely solve the race conditi...
- 07:17 PM bluestore Bug #58022: Fragmentation score rising by seemingly stuck thread
- Newer picture, after I had just restarted the current batch of runaways.
- 05:24 PM bluestore Bug #58022: Fragmentation score rising by seemingly stuck thread
- 11 more osds started doing this over the holiday weekend.
- 07:02 PM Orchestrator Feature #58100 (Resolved): cephadm: redo service level configuration when service spec is changed
- The specific case in mind writing this is that if you have an rgw frontend ssl cert specified in your spec, and then ...
- 06:23 PM bluestore Bug #58099 (Need More Info): ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixPreferDeferred/2 ...
- /a/yuriw-2022-10-19_18:35:19-rados-wip-yuri10-testing-2022-10-19-0810-distro-default-smithi/7074995/...
- 06:13 PM Orchestrator Bug #57998 (Resolved): cephadm stuck trying to download "mon"
- Shawn Iverson wrote:
> I executed the following:
>
> [...]
>
> and now cephadmn is working!
was away the la... - 05:19 PM CephFS Bug #58041 (Duplicate): mds: src/mds/Server.cc: 3231: FAILED ceph_assert(straydn->get_name() == s...
- Milind Changire wrote:
> Due to unavailability of debug logs, there has been some speculation about the issue during... - 04:58 PM CephFS Bug #58095: snap-schedule: handle non-existent path gracefully during snapshot creation
- Milind Changire wrote:
> The most common mistake that users tend to do is include the mount point path along with th... - 03:39 PM CephFS Bug #58095 (Resolved): snap-schedule: handle non-existent path gracefully during snapshot creation
- The most common mistake that users tend to do is include the mount point path along with the file-system path when us...
- 04:29 PM Orchestrator Bug #58096: test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- Laura Flores wrote:
> [...]
/a/yuriw-2022-11-23_15:09:06-rados-wip-yuri10-testing-2022-11-22-1711-distro-default-... - 04:28 PM Orchestrator Bug #58096 (New): test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory
- ...
- 04:01 PM Bug #58048: «EPERM: error calling ceph_mount» when trying to use subvolume commands
- The project should be CephFS on this issue but I can't seem to be able to change it myself.
- 03:26 PM CephFS Bug #54017: Problem with ceph fs snapshot mirror and read-only folders
- Milind, this was discussed here - https://www.mail-archive.com/ceph-users@ceph.io/msg14364.html
Related bz - https... - 03:06 PM CephFS Bug #58090: Non-existent pending clone shows up in snapshot info
- Hi Sebastian,
There is a stray index causing this issue. Could you list the contents of `/volumes/_index/clone/` (... - 02:47 PM RADOS Bug #52657: MOSDPGLog::encode_payload(uint64_t): Assertion `HAVE_FEATURE(features, SERVER_NAUTILUS)'
- Aishwarya Mathuria wrote:
> We suspect that this assert failure is hit in cases when we try to encode a message befo... - 02:44 PM CephFS Bug #58058: CephFS Snapshot Mirroring slow due to repeating attribute sync
- Mathias Kuhring wrote:
> We might have found a major performance bug in the cephfs snapshot mirroring.
> We already... - 01:41 PM CephFS Bug #58058 (Triaged): CephFS Snapshot Mirroring slow due to repeating attribute sync
- 02:37 PM Dashboard Bug #58094 (Resolved): mgr/dashboard: expose slow ops per osd
- h3. Description of problem
Expose the number of slow operations per osd in prometheus and create a grafana panel.
... - 02:20 PM Support #58093: recover deleted bucket during replication
- Shafiq M wrote:
> recover deleted bucket during replication
- 12:01 PM Support #58093 (New): recover deleted bucket during replication
- recover deleted bucket during replication
- 11:42 AM rgw Documentation #58092 (New): rgw_enable_gc_threads / lc_threads not documented on web
- Options rgw_enable_gc_threads and rgw_enable_lc_threads are not rendered for docs.ceph.com.
I would expect those t... - 10:07 AM ceph-volume Bug #57767: ceph-volume should check if device is locked prior to zapping it
- A recent thread [1] addresses this issue again. A simple test with Nautilus shows that in previous versions ceph-volu...
- 09:11 AM rgw Backport #57238 (In Progress): pacific: crash: RGWCoroutinesStack::wakeup()
- 09:11 AM rgw Backport #57237 (In Progress): quincy: crash: RGWCoroutinesStack::wakeup()
- 09:08 AM rgw Backport #55228 (In Progress): pacific: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_s...
- 09:06 AM rgw Backport #55227 (In Progress): quincy: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_st...
- 09:05 AM rgw Backport #54497 (In Progress): pacific: bucket index completions may not retry after reshard
- 09:04 AM rgw Backport #54496 (In Progress): quincy: bucket index completions may not retry after reshard
- 09:03 AM rgw Backport #54155 (In Progress): pacific: rgw: "reshard cancel" errors with "invalid argument"
- 09:03 AM rgw Backport #54157 (In Progress): quincy: rgw: "reshard cancel" errors with "invalid argument"
- 09:00 AM rgw Backport #55505 (In Progress): pacific: radosgw rejects some requests without Content-MD5 Header
- 08:59 AM rgw Backport #55506 (In Progress): quincy: radosgw rejects some requests without Content-MD5 Header
- 08:44 AM Backport #57463 (In Progress): quincy: pacific: Could NOT find Python3
- 08:37 AM rgw Backport #57409 (In Progress): pacific: rgw: bucket list operation slow down in special scenario
- 08:32 AM rgw Backport #57410 (In Progress): quincy: rgw: bucket list operation slow down in special scenario
- 08:20 AM rgw Backport #57752 (In Progress): quincy: Log status of individual object deletions for multi-object...
- 07:55 AM rgw Backport #54493 (In Progress): quincy: segmentation fault in UserAsyncRefreshHandler::init_fetch
- 05:32 AM CephFS Bug #58082 (Fix Under Review): cephfs:filesystem became read only after Quincy upgrade
- 05:05 AM RADOS Support #58091 (New): osd: reduce default value of osd_heartbeat_grace
- Client io hang 20s when peer osd ping failure, 20s is too long. In case of network jitter, it generally does not exce...
11/27/2022
- 10:10 PM bluestore Feature #57785: fragmentation score in metrics
- I think having the metric available opens the door for monitoring escalation for prometheus and less frequently used ...
- 02:08 PM CephFS Bug #58090 (New): Non-existent pending clone shows up in snapshot info
- Ceph version: v17.2.5
My CephFS somehow got in a state where a snapshot has a pending clone, but the pending clone... - 11:16 AM crimson Bug #58089: get_acting_recovery_backfill().empty() assertion
- https://pulpito.ceph.com/matan-2022-11-22_15:04:25-crimson-rados-main-distro-crimson-smithi/7086464/
- 11:13 AM crimson Bug #58089 (Resolved): get_acting_recovery_backfill().empty() assertion
- ...
- 07:46 AM Documentation #58080 (Resolved): Update RBD documentation - Ceph iSCSI Gateway does not support r...
11/26/2022
- 05:23 PM CephFS Bug #58088 (Fix Under Review): qa/tasks/vstart_runner: TypeError: LocalFuseMount._run_mount_cmd()...
- 05:08 PM CephFS Bug #58088 (Resolved): qa/tasks/vstart_runner: TypeError: LocalFuseMount._run_mount_cmd() takes 3...
- Hit this error,...
11/25/2022
- 03:57 PM rgw Backport #58087 (In Progress): quincy: rgw/cloud-tranistion: Issues with MCG cloud endpoint
- 03:56 PM rgw Backport #58087: quincy: rgw/cloud-tranistion: Issues with MCG cloud endpoint
- https://github.com/ceph/ceph/pull/49061
- 03:23 PM rgw Backport #58087 (In Progress): quincy: rgw/cloud-tranistion: Issues with MCG cloud endpoint
- 03:17 PM rgw Bug #57979 (Pending Backport): rgw/cloud-tranistion: Issues with MCG cloud endpoint
- 03:07 PM Dashboard Bug #58086 (Fix Under Review): mgr/dashboard: relax the cors policies
- 02:55 PM Dashboard Bug #58086 (Resolved): mgr/dashboard: relax the cors policies
- cors policy is strictly disabled now in the api. Relax it when needed.
- 12:05 PM Bug #58069: flake8 fails since Nov 23 2022
- ... and quincy too:
https://jenkins.ceph.com/job/ceph-pull-requests/107099/consoleText
pacific backport PR: htt... - 11:44 AM Bug #58069: flake8 fails since Nov 23 2022
- Also seen on pacific:...
- 11:40 AM rbd Backport #58085 (In Progress): quincy: [rbd_support] pool removal can wedge the task queue
- 10:53 AM rbd Backport #58085 (Resolved): quincy: [rbd_support] pool removal can wedge the task queue
- https://github.com/ceph/ceph/pull/49057
- 11:37 AM rbd Backport #58084 (In Progress): pacific: [rbd_support] pool removal can wedge the task queue
- 10:53 AM rbd Backport #58084 (Resolved): pacific: [rbd_support] pool removal can wedge the task queue
- https://github.com/ceph/ceph/pull/49056
- 11:17 AM rbd Backport #58077 (In Progress): quincy: [pwl] inconsistent "rbd status" output (clean = true but d...
- 11:15 AM rbd Backport #58076 (In Progress): pacific: [pwl] inconsistent "rbd status" output (clean = true but ...
- 10:49 AM rbd Bug #52932 (Pending Backport): [rbd_support] pool removal can wedge the task queue
- 10:12 AM CephFS Bug #58041: mds: src/mds/Server.cc: 3231: FAILED ceph_assert(straydn->get_name() == straydname)
- Due to unavailability of debug logs, there has been some speculation about the issue during discussion with Venky.
T... - 06:07 AM Dashboard Feature #58083 (Resolved): mgr/dashboard: Add metric relative to osd blocklist
- It would be good to have a metric with number of clients "blocklisted" of command: "ceph osd blocklist ls". The reaso...
- 05:07 AM CephFS Bug #58082: cephfs:filesystem became read only after Quincy upgrade
- From the logs, the *dir(0x1)* will submit the *volumes* Dentry to metadata pool: ...
- 04:51 AM CephFS Bug #58082 (Resolved): cephfs:filesystem became read only after Quincy upgrade
- Copy the info from ceph-user mail list by Adrien:...
- 04:54 AM CephFS Bug #52260 (Duplicate): 1 MDSs are read only | pacific 16.2.5
- Will tracker and fix it in https://tracker.ceph.com/issues/58082.
11/24/2022
- 06:00 PM Documentation #58080 (Resolved): Update RBD documentation - Ceph iSCSI Gateway does not support r...
- From https://pad.ceph.com/p/Report_Documentation_Bugs:
"Applications that use SCSI 7 perstistent group reserva... - 05:27 PM CephFS Backport #58079 (Resolved): quincy: cephfs-top: Sorting doesn't work when the filesystems are rem...
- https://github.com/ceph/ceph/pull/50151
- 05:27 PM CephFS Backport #58078 (Resolved): pacific: cephfs-top: Sorting doesn't work when the filesystems are re...
- https://github.com/ceph/ceph/pull/49303
- 05:26 PM CephFS Bug #58028 (Pending Backport): cephfs-top: Sorting doesn't work when the filesystems are removed ...
- 03:28 PM Orchestrator Bug #58063 (In Progress): osd encryption is failing on ubuntu 22.04
- 03:19 PM Orchestrator Bug #58063: osd encryption is failing on ubuntu 22.04
- Ubuntu 22.04 minimal**
- 03:17 PM Orchestrator Bug #58063: osd encryption is failing on ubuntu 22.04
- This issue is only reproducible on ubuntu 22.04 mini.
- 03:15 PM rbd Backport #58077 (Resolved): quincy: [pwl] inconsistent "rbd status" output (clean = true but dirt...
- https://github.com/ceph/ceph/pull/49055
- 03:15 PM rbd Backport #58076 (Resolved): pacific: [pwl] inconsistent "rbd status" output (clean = true but dir...
- https://github.com/ceph/ceph/pull/49054
- 03:13 PM rbd Bug #57872 (Pending Backport): [pwl] inconsistent "rbd status" output (clean = true but dirty_byt...
- 02:44 PM Orchestrator Bug #58066: config key for cephadm hosts doesn't have values for all network interfaces present i...
- Raised following PR - https://github.com/ceph/ceph/pull/49043
- 02:21 PM Support #58075 (New): bucket not able to read from s3 says no bucket and replication not working ...
- bucket not able to read from s3 says no bucket and replication not working for such bucket
- 01:39 PM CephFS Backport #58074 (Resolved): quincy: cephfs-top: sorting/limit excepts when the filesystems are re...
- https://github.com/ceph/ceph/pull/50151
- 01:39 PM CephFS Backport #58073 (Resolved): pacific: cephfs-top: sorting/limit excepts when the filesystems are r...
- https://github.com/ceph/ceph/pull/49303
- 01:36 PM CephFS Bug #58031 (Pending Backport): cephfs-top: sorting/limit excepts when the filesystems are removed...
- 01:05 PM CephFS Feature #58072 (Fix Under Review): enable 'ceph fs new' use 'ceph fs set' options
- As discussed in PR [1], this flag would come handy in situations like 'ceph fs new --recover'. Need to push this enha...
- 12:51 PM Bug #58069 (Resolved): flake8 fails since Nov 23 2022
- 06:06 AM Bug #58069 (In Progress): flake8 fails since Nov 23 2022
- 05:42 AM Bug #58069 (Resolved): flake8 fails since Nov 23 2022
- ...
- 08:41 AM Bug #58071 (New): Bucket keys do not exist in omap
- In a cluster one bucket(not found more yet) cannot list the owner because key do not exist in omap. Here: rados -p de...
- 07:33 AM CephFS Feature #58070 (New): qa: add test suite to test old kernels
- Currently there is test case will test old ceph-fuse clients with new ceph, but we also need to test the old kclient ...
- 05:36 AM CephFS Feature #55940 (Fix Under Review): quota: accept values in human readable format as well
- 03:54 AM RADOS Bug #57977: osd:tick checking mon for new map
- The more I dig, the more I'm thinking that this might be some race to do with noup, and probably has nothing to do wi...
- 03:42 AM RADOS Bug #57977: osd:tick checking mon for new map
- Something that's probably worth mentioning - we had noup set in the cluster for each upgrade, and we wait until all O...
- 03:12 AM RADOS Bug #57977: osd:tick checking mon for new map
- We saw this happen to roughly a dozen OSDs (1-2 per host for some hosts) during a recent upgrade from Nautilus to Pac...
11/23/2022
- 06:08 PM rbd Bug #52932 (Fix Under Review): [rbd_support] pool removal can wedge the task queue
- 06:06 PM rgw Bug #58059 (Resolved): s3tests v2 SignatureDoesNotMatch failures on ubuntu
- 05:31 PM CephFS Bug #24403: mon failed to return metadata for mds
- FYI - restarting the MDS fixes the issue.
- 05:30 PM CephFS Bug #24403: mon failed to return metadata for mds
- This was seen in pacific installation. MDS entries in FSMap are fine - that serves `fs dump` and `fs status` commands...
- 04:32 PM Documentation #57001: RBD Rewrite Project
- https://github.com/ceph/ceph/pull/49035
This PR swaps the order of the first two paragraphs in the front matter.
... - 04:26 PM Documentation #57001: RBD Rewrite Project
- https://medium.com/system-design-blog/eventual-consistency-vs-strong-consistency-b4de1f92534d - A medium.com article ...
- 04:20 PM bluestore Bug #48216: Spanning blobs list might have zombie blobs that aren't of use any more
- Gilles Mocellin wrote:
> Hello,
>
> No news on that ?
> Does someone knows if the problem also happens on Quincy... - 09:46 AM bluestore Bug #48216: Spanning blobs list might have zombie blobs that aren't of use any more
- Hello,
No news on that ?
Does someone knows if the problem also happens on Quincy ? - 04:19 PM rgw Bug #57853 (Fix Under Review): multisite sync process block after long time running
- 04:12 PM bluestore Bug #54019: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
- Just made a topic for potential fix discussion at https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHO...
- 01:19 PM Orchestrator Bug #58067 (Fix Under Review): Service discovery are not reported consistently in ceph orch ps
- 09:09 AM Orchestrator Bug #58067 (Resolved): Service discovery are not reported consistently in ceph orch ps
- Ceph orch ps lists all the daemons and open ports for each one of them. Right now cephadm service discovery port is o...
- 01:18 PM rbd Bug #58062 (Duplicate): RBD tasks will stop if a pool is deleted, blocking further queue
- Hi Miodrag,
Thanks for the report! This is a known issue, I'll bump the priority on the older ticket and we will ... - 12:05 PM CephFS Bug #58031 (Fix Under Review): cephfs-top: sorting/limit excepts when the filesystems are removed...
- 10:24 AM Orchestrator Bug #58066: config key for cephadm hosts doesn't have values for all network interfaces present i...
- same problem is observed in v17.2.5 as well
- 08:17 AM Orchestrator Bug #58066: config key for cephadm hosts doesn't have values for all network interfaces present i...
- Affected version: v16.2.10
- 08:17 AM Orchestrator Bug #58066 (Fix Under Review): config key for cephadm hosts doesn't have values for all network i...
- with cephadm, it stores key-value for each host that are part of the cluster.
For ex:
ceph config-key ls | grep m... - 09:23 AM ceph-ansible Bug #53684: The ingress daemon uses "*" as frontend network even though a virtual_ip is specified...
- Did anyone think about the fact that HAProxy configuration is created by keepalived as per https://tracker.ceph.com/i...
- 09:20 AM Orchestrator Bug #57910: ingress: HAProxy fails to start because keepalived IP address not yet available on ne...
- The HAProxy service does not start until it is manually started or the server is restarted.
The problem root is th... - 08:05 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- xianpao chen wrote:
> I've heard that too large mds_cache_memory_limit may cause problems, so I use mds_cache_memory... - 07:40 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- I've heard that too large mds_cache_memory_limit may cause problems, so I use mds_cache_memory_limit = 16GB, no speci...
- 07:25 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Any reason you are using mds_cache_memory_limit = 16GB when you have memory to spare?
- 06:32 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- the "free -h" of the mds node(after restart the mds): ...
- 06:06 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- BTW, do you see any performance degradation on clients in general over the course and/or when the MDS is about to get...
- 05:27 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- I checked the session info, there is just one client which is holding ~1M caps. But that should not bother the MDS th...
- 05:57 AM Dashboard Tasks #58065 (Resolved): mgr/dashboard: inventory card
- 12:57 AM mgr Feature #58064 (New): mgr/restful: support for http requests
- Support ceph config set mgr mgr/restful/https_mode false/true to control HTTP/HTTPS access.
11/22/2022
- 11:10 PM rgw Bug #58053: bucket is list in s3cmd but can not be queried after deleting
- *ceph-qa-suite* should be RGW
- 06:20 AM rgw Bug #58053: bucket is list in s3cmd but can not be queried after deleting
- Prepare a policy.json file that contains the bucket policy for testing....
- 06:13 AM rgw Bug #58053: bucket is list in s3cmd but can not be queried after deleting
- The bug can be reproduced with the following script:
- 07:48 PM rgw Bug #58059 (Fix Under Review): s3tests v2 SignatureDoesNotMatch failures on ubuntu
- https://github.com/ceph/s3-tests/pull/476
- 05:03 PM rgw Bug #58059: s3tests v2 SignatureDoesNotMatch failures on ubuntu
- bisected botocore versions down to good=botocore-1.27.96 bad=botocore-1.28.0
botocore debug log output from good=b... - 03:11 PM rgw Bug #58059: s3tests v2 SignatureDoesNotMatch failures on ubuntu
- boto versions from a failing run on ubuntu:...
- 07:01 PM bluestore Feature #57785: fragmentation score in metrics
- ❤️
- 06:52 PM bluestore Feature #57785: fragmentation score in metrics
- After syncing with Adam Kupczyk today:
In the shorter term we will make the fragmentation score, both for bluefs ... - 06:28 PM Orchestrator Bug #58063 (In Progress): osd encryption is failing on ubuntu 22.04
- Hallo folks,
i am deploying a quincy ceph cluster 17.2.0 on openstack vm with ubuntu
22.04 minimal with cephadm.
... - 06:17 PM RADOS Bug #57977: osd:tick checking mon for new map
- I already restart osd daemon, but have no reproduct. If it happens again, I will collect more logs
- 03:54 PM RADOS Bug #58052: Empty Pool (zero objects) shows usage.
- Radoslaw Zarzynski wrote:
> Could you please provide a log from an active mgr with @debug_ms=1@ and @debug_mgr=20@?
... - 03:42 PM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- I happened to have a memory problem today, then I changed mds_session_cache_liveness_decay_rate to 150s, tried "ceph ...
- 11:19 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Venky Shankar wrote:
> Did you get to applying the suggested config?
Thanks for your suggestion, I will try it to... - 11:07 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Did you get to applying the suggested config?
- 03:26 PM rbd Bug #58062 (Duplicate): RBD tasks will stop if a pool is deleted, blocking further queue
- Hi.
I believe we found a bug with RBD tasks in Ceph MGR, dating somewhere between versions 16.2.0 and 17.2.1 (thes... - 03:00 PM Orchestrator Feature #58061 (New): Add support for HA for ceph mgr
- In the current design ceph can run multiple instances of mgr. The mgr is a core component of ceph that runs important...
- 10:19 AM CephFS Bug #57523: CephFS performance degredation in mountpoint
- Guys this can't be only a thing in our setup. Every time a connection puts more than a few GB into cephfs the perform...
- 04:42 AM rgw Bug #57562: multisite replication issue on Quincy
- Hi Adam, another question regarding your changes to remove the use of "tags". I'd like to understand your opinion/pla...
- 04:34 AM rgw Bug #57562: multisite replication issue on Quincy
- Did some investigation on the latest failure of the tests on `lastest - 1` "PR":https://github.com/ceph/ceph/pull/486...
- 03:32 AM rbd Bug #58060 (Resolved): avoid EUCLEAN error after "rbd rm" is interrupted
- If rbd rm process is interrupted (for example, rbd client killed, or ceph_assert fault injection in rbd code), rbd st...
- 01:19 AM CephFS Bug #58056: ceph-fuse - fuse failed to start on CentOS 7 host machine
- It seems passing invalidate flag ?...
11/21/2022
- 09:03 PM rgw Bug #58059 (Resolved): s3tests v2 SignatureDoesNotMatch failures on ubuntu
- from main branch results: https://pulpito.ceph.com/cbodley-2022-11-21_18:00:47-rgw-main-distro-default-smithi/
s3t... - 08:41 PM rgw Bug #57562: multisite replication issue on Quincy
- Hi Adam,
Wanted to provide you with an update about the testing that we did over the weekend,
We took 2 PR's, one w... - 08:17 PM CephFS Bug #58058 (Triaged): CephFS Snapshot Mirroring slow due to repeating attribute sync
- We might have found a major performance bug in the cephfs snapshot mirroring.
We already reported it to the mailing ... - 08:17 PM bluestore Bug #58022: Fragmentation score rising by seemingly stuck thread
- Saw this on 8 more osds over the weekend.
- 06:35 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- @Radek I have been trying to reproduce this locally with no luck. I'll try your suggestion and update if I'm successful.
- 06:34 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- Thanks for the link, Matan! I'm a bit worried the experiment there involved changing 2 parameters the same: compiler ...
- 06:29 PM RADOS Bug #58044 (Need More Info): ceph-osd: osd numa affinity setting doesn't work
- How do you check the affinity?
Have you rebooted the OSD after the injecting the setting?
Could you please provide ... - 06:22 PM RADOS Bug #58046 (Resolved): qa/workunits/rados/test_librados_build.sh: specify redirect in curl command
- 06:21 PM RADOS Bug #58052 (Need More Info): Empty Pool (zero objects) shows usage.
- Could you please provide a log from an active mgr with @debug_ms=1@ and @debug_mgr=20@? We would like to see which OS...
- 06:11 PM CephFS Support #38374: Crash when using cephfs as /var/lib/docker in devicemapper mode
- We're not using this kind of setup anymore and won't be troubleshooting further. We can close this for now, probably ...
- 03:56 PM rgw Bug #55310 (Duplicate): [pacific] RadosGW instance of Cloud Sync zone crashes when objects are up...
- 01:53 PM CephFS Feature #58057: cephfs-top: enhance fstop tests to cover testing displayed data
- The Dashboard folks could point us to tools for testing console UI based apps.
- 11:30 AM CephFS Feature #58057 (Resolved): cephfs-top: enhance fstop tests to cover testing displayed data
- Right now the tests are pretty rudimentary. cephfs-top is a UI tool and writing tests can be a bit hard. Due to this ...
- 10:42 AM Orchestrator Bug #57897: ceph mgr restart causes restart of all iscsi daemons in a loop
- Hi Mykola and Redouane
Thanks for looking into this - we initially thought the same regarding the commit in the [2... - 09:52 AM rbd Bug #57941 (Rejected): Severe performance drop after writing 100 GB of data to RBD volume, depend...
- Thanks for following up, Guillaume!
- 09:05 AM CephFS Bug #58056 (New): ceph-fuse - fuse failed to start on CentOS 7 host machine
- Hello,
We were previously using ceph v16.2.10 (docker container) with rook under kubernetes which was installed on... - 08:42 AM Linux kernel client Feature #58055 (New): feature to block kclient from establishing session
- We need to implement https://github.com/ceph/ceph/pull/48720 for kclient too.
- 07:18 AM RADOS Bug #58027: op slow from throttled to header_read
- Radoslaw Zarzynski wrote:
> Hello! The most important thing is Octopus is EOL. Second, I'm also not sure whether thi... - 06:31 AM CephFS Bug #57014 (In Progress): cephfs-top: add an option to dump the computed values to stdout
- 02:02 AM Linux kernel client Bug #58054 (Resolved): kclient: xfstests-dev generic/684 fails
- ...
- 01:47 AM Linux kernel client Bug #57986 (Fix Under Review): ceph: ceph_fl_release_lock cause "unable to handle kernel paging r...
- The patchwork links:
Jeff's VFS locks patch:
https://patchwork.kernel.org/project/ceph-devel/list/?series=695950
... - 01:43 AM Bug #57613 (Resolved): Kernel Oops, kernel NULL pointer dereference
- David Howells has fixed it in netfs layer:
https://git.kernel.org/torvalds/c/59d0d52c30d4991ac4b329f049cc37118e00f5b0 - 12:00 AM rgw Bug #58053 (Need More Info): bucket is list in s3cmd but can not be queried after deleting
- ceph version 16.2.10
There will be a race between s3.DeleteBucket and s3.DeleteBucketPolicy. When the race happens...
11/20/2022
- 10:59 PM rbd Bug #57941: Severe performance drop after writing 100 GB of data to RBD volume, dependent on RAM ...
- :facepalm: the issue has nothing to do with Ceph and everything to do with smaller EC2 instances having burstable net...
- 05:23 PM RADOS Bug #58052 (Need More Info): Empty Pool (zero objects) shows usage.
- I have a pool that was/is being used in a CephFS. I have migrated all of the files off of the pool and was preparing...
- 01:24 PM Orchestrator Bug #57897: ceph mgr restart causes restart of all iscsi daemons in a loop
- David mentioned [1] as a potential cause when the issue was introduced. But actually I think it is [2]. And apart tha...
- 01:10 PM Orchestrator Bug #58051 (Resolved): mgr/cephadm: try to avoid pull when getting container image info
- Currently, when getting a container image info the cephadm module always runs pull command.
There may be environme...
11/18/2022
- 08:49 PM mgr Bug #58050 (Fix Under Review): mgr/telemetry: perf histograms are not formatted in `all` commands
- 08:47 PM mgr Bug #58050 (Resolved): mgr/telemetry: perf histograms are not formatted in `all` commands
- I noticed that perf histograms weren't getting formatted in `preview-all` and `show-all`.
- 04:16 PM rbd Bug #57941: Severe performance drop after writing 100 GB of data to RBD volume, dependent on RAM ...
- In order to rule out if client side caching is a factor can you do a few tests?
Run tests without client side cach... - 03:29 PM RADOS Bug #58049 (Resolved): mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker...
- First encountered in the downstream: https://bugzilla.redhat.com/show_bug.cgi?id=2142674
When we failover monitors... - 02:27 PM Bug #58048: «EPERM: error calling ceph_mount» when trying to use subvolume commands
- Here is a log of the issue when running ceph-mds manually with the following command:
ceph-mds --setuser ceph --setg... - 02:21 PM Bug #58048 (New): «EPERM: error calling ceph_mount» when trying to use subvolume commands
- When trying to issue commands such as «ceph fs subvolume ls cephfsv2», we get a return of «Error EPERM: error calling...
- 11:38 AM CephFS Bug #58028 (Fix Under Review): cephfs-top: Sorting doesn't work when the filesystems are removed ...
- 03:23 AM Linux kernel client Bug #57686: general protection fault and CephFS kernel client hangs after MDS failover
- Greg Farnum wrote:
> Xiubo, I know the patch addresses the immediate fault, but do we have any idea how or why the s... - 02:54 AM Linux kernel client Bug #57686: general protection fault and CephFS kernel client hangs after MDS failover
- Xiubo, I know the patch addresses the immediate fault, but do we have any idea how or why the snaprealm was corrupt i...
- 03:02 AM Bug #58047 (New): ceph-osd: osd numa affinity setting doesn't work
- After setting osd_numa_node parameter, the osd numa is not as expected.
* cpu numa node... - 12:40 AM RADOS Bug #58046 (Fix Under Review): qa/workunits/rados/test_librados_build.sh: specify redirect in cur...
- 12:36 AM RADOS Bug #58046 (Pending Backport): qa/workunits/rados/test_librados_build.sh: specify redirect in cur...
- The workunit currently grabs files with:...
11/17/2022
- 11:46 PM Orchestrator Bug #57287: orch/cephadm: 'mount -t nfs 10.0.31.109:/foouser /mnt/foo' failing in nfs-rgw-ingress...
- Tracker #57163 was resolved by compiling with c++20. Perhaps that could also help in this case?
- 10:37 PM bluestore Feature #57785: fragmentation score in metrics
- We have a meeting scheduled for next week to discuss this topic.
- 06:30 PM bluestore Feature #57785: fragmentation score in metrics
- ❤️
- 06:28 PM bluestore Feature #57785: fragmentation score in metrics
- Thanks, Kevin. Let me talk this over with Adam and Paul, and we will decide a course of action.
- 06:15 PM bluestore Feature #57785: fragmentation score in metrics
- A ceph warning for it would also be quite useful I think.
https://access.redhat.com/documentation/fr-fr/red_hat_ceph... - 06:09 PM bluestore Feature #57785: fragmentation score in metrics
- Thanks for sharing this, Kevin. We discussed this Tracker more in the Telemetry huddle, and we are curious if you wou...
- 05:11 PM bluestore Feature #57785: fragmentation score in metrics
- We've had to hack a script together to monitor one of our clusters, and it has been useful to catch an issue:
https:... - 04:25 PM bluestore Feature #57785: fragmentation score in metrics
- @Kevin I have asked Paul Cuzner to take a look at this tracker and offer his opinion, as he has done a lot of work fo...
- 06:14 PM Orchestrator Bug #58045 (Resolved): orchestrator: incorrect size listed for devices in device ls
- currently when printing out the size in `ceph orch device ls` the orchestrator is calling a function that continually...
- 05:07 PM RADOS Bug #52657: MOSDPGLog::encode_payload(uint64_t): Assertion `HAVE_FEATURE(features, SERVER_NAUTILUS)'
- We suspect that this assert failure is hit in cases when we try to encode a message before the connection is in a sta...
- 03:30 PM RADOS Bug #56147: snapshots will not be deleted after upgrade from nautilus to pacific
- > For already-converted clusters: Separate PR will be issued to remove/update the malformed SnapMapper keys.
https... - 03:10 PM rgw Bug #58034 (In Progress): RGW misplaces index entries after dynamically resharding bucket
- 02:15 PM Dashboard Backport #58043 (In Progress): quincy: mgr/dashboard: missing data on hosts Grafana dashboard
- 01:50 PM Dashboard Backport #58043 (Resolved): quincy: mgr/dashboard: missing data on hosts Grafana dashboard
- https://github.com/ceph/ceph/pull/48934
- 02:12 PM Dashboard Backport #58042 (In Progress): pacific: mgr/dashboard: missing data on hosts Grafana dashboard
- 01:50 PM Dashboard Backport #58042 (Resolved): pacific: mgr/dashboard: missing data on hosts Grafana dashboard
- https://github.com/ceph/ceph/pull/48933
- 02:09 PM RADOS Bug #58044 (Need More Info): ceph-osd: osd numa affinity setting doesn't work
- After setting osd_numa_node parameter, the osd numa is not as expected.
- 01:42 PM Dashboard Bug #57987 (Pending Backport): mgr/dashboard: missing data on hosts Grafana dashboard
- 01:20 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- Radoslaw Zarzynski wrote:
> Do we know the reason why switching g++11 helps? Is it a known compiler's bug?
See Br... - 12:15 PM RADOS Bug #57940: ceph osd crashes with FAILED ceph_assert(clone_overlap.count(clone)) when nobackfill ...
- Thomas Le Gentil wrote:
> the osd process does not crash if it is marked 'out'
Sorry, this is false. The OSD cras... - 12:05 PM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Hey,
Thanks for the update. You should try adjusting `mds_session_cache_liveness_decay_rate` to a lower value (def... - 10:17 AM CephFS Bug #58041: mds: src/mds/Server.cc: 3231: FAILED ceph_assert(straydn->get_name() == straydname)
- and another side note, the crash was seen when a directory pin was removed from rank-0 mds. Pinning it back again cea...
- 10:16 AM CephFS Bug #58041: mds: src/mds/Server.cc: 3231: FAILED ceph_assert(straydn->get_name() == straydname)
- oh, and btw this was seen in ceph-16.2.8.
- 10:15 AM CephFS Bug #58041 (Duplicate): mds: src/mds/Server.cc: 3231: FAILED ceph_assert(straydn->get_name() == s...
- ...
- 09:42 AM RADOS Backport #58040 (Resolved): quincy: osd: add created_at and ceph_version_when_created metadata
- 09:42 AM RADOS Backport #58039 (Resolved): pacific: osd: add created_at and ceph_version_when_created metadata
- 09:34 AM RADOS Feature #58038 (Pending Backport): osd: add created_at and ceph_version_when_created metadata
- 07:24 AM RADOS Feature #58038: osd: add created_at and ceph_version_when_created metadata
- PR#48298 has already been merged. Could you change the status of this issue to "Pending Backport"?
I'll create backp... - 07:15 AM RADOS Feature #58038 (Resolved): osd: add created_at and ceph_version_when_created metadata
- Add the following two OSD metadata.
- created_at: the timestamp when OSD was created. It's useful when getting som... - 09:21 AM CephFS Feature #55215 (Fix Under Review): mds: fragment directory snapshots
- 08:30 AM rgw Bug #55498 (Duplicate): "AssertionError: rgw multisite test failures" in upgrade:octopus
- will be fixed as part of: https://tracker.ceph.com/issues/58036
- 08:25 AM rgw Backport #58036 (In Progress): pacific: pubsub test failures
- 06:18 AM mgr Bug #58037: mgr crash
- this crash happen after cmd "rbd perf image stats"...
- 05:10 AM mgr Bug #58037 (Need More Info): mgr crash
- ...
11/16/2022
- 07:30 PM rgw Backport #58036 (Resolved): pacific: pubsub test failures
- https://github.com/ceph/ceph/pull/48928
- 07:11 PM RADOS Bug #57977: osd:tick checking mon for new map
- Thanks for the update! Yeah, it might stuck there. To confirm we would logs with increased debugs (maybe @debug_mon =...
- 07:09 PM rgw Bug #56572: pubsub test failures
- this should also be backported to pacific, in order to fix the upgrade issues. see: https://tracker.ceph.com/issues/5...
- 07:06 PM RADOS Bug #51729: Upmap verification fails for multi-level crush rule
- Thanks for formulating the hypothesis!
Just updating to keep this ticket in the front of the tracker. - 07:02 PM RADOS Bug #57546: rados/thrash-erasure-code: wait_for_recovery timeout due to "active+clean+remapped+la...
- Yeah, worth looking the msgr encode issue has the priority.
- 07:00 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- Discussed during the RADOS Team Meeting on 15 Nov.
Linking the Nitzan's gist: https://gist.github.com/NitzanMordhai/... - 06:58 PM RADOS Bug #57989: test-erasure-eio.sh fails since pg is not in unfound
- Definitely a low priority.
- 06:52 PM RADOS Bug #58027 (Closed): op slow from throttled to header_read
- Hello! The most important thing is Octopus is EOL. Second, I'm also not sure whether this is really a bug. Seeing 0,5...
- 06:48 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- Do we know the reason why switching g++11 helps? Is it a known compiler's bug?
- 05:47 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- I was able to schedule a teuthology run: http://pulpito.front.sepia.ceph.com/lflores-2022-11-16_15:49:13-rados:single...
- 06:28 PM rgw Bug #57562: multisite replication issue on Quincy
- Hey Adam,
So after all of our current testing and debugging of issue, the current race condition is that the same pa... - 04:43 PM rgw Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Hold off for now, I've introduced one problem I need to debug.
Ack.
We are currently tes... - 11:02 AM rgw Bug #57562: multisite replication issue on Quincy
- Hold off for now, I've introduced one problem I need to debug.
- 10:47 AM rgw Bug #57562: multisite replication issue on Quincy
- I have pushed a new commit. It disables all use of the part tags. I believe this should eliminate any remaining diffi...
- 06:08 PM rgw Bug #58020 (Fix Under Review): notifications: zero timestamp in complete multipart upload event
- 06:06 PM rgw Backport #57561 (In Progress): quincy: pubsub test failures
- 03:26 PM rgw Bug #58035 (Fix Under Review): Copying an object to itself crashes de RGW if executed as admin user.
- 10:44 AM rgw Bug #58035 (Pending Backport): Copying an object to itself crashes de RGW if executed as admin user.
- This was observed after executing the test *s3tests_boto3.functional.test_s3:test_object_copy_to_itself* with a user ...
- 01:11 PM RADOS Bug #57940: ceph osd crashes with FAILED ceph_assert(clone_overlap.count(clone)) when nobackfill ...
- the osd process does not crash if it is marked 'out'
- 12:24 PM Orchestrator Bug #58021 (Fix Under Review): Promteheus haproxy targets are down
- 10:09 AM Dashboard Bug #56970 (Resolved): mgr/dashboard: dashboard connects via ssl to an ip address instead of a dn...
- 04:56 AM Documentation #57001: RBD Rewrite Project
- As of 16 Nov 2022 AEST, the front matter of the RBD book reads as follows:
> A block is a sequence of bytes (often... - 04:53 AM Documentation #57001: RBD Rewrite Project
- https://docs.ceph.com/en/quincy/rbd/
- 04:50 AM Documentation #57108: add ".. prompt :: bash $" to /doc/rados
- I slit the sheet, the sheet I slit, and on the slitted sheet I sit.
- 03:28 AM rgw Bug #57980 (Fix Under Review): rgw/cloud-transition: transition fails when using MCG Azure Namesp...
- >>> From the http packets -> MCG is returning 403 for HEAD request and 400 for PUT request (failed). Maybe the header...
11/15/2022
- 09:56 PM rgw Bug #58034 (Resolved): RGW misplaces index entries after dynamically resharding bucket
- When RGW reshards buckets with ~250k index entries*, I've noticed some s3:PutObject requests that return 200 end up w...
- 09:54 PM rgw Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- The code on the PR seems to address the issue. My colleague Mark Kogan ran it through a test at scale and it behaved ...
- 07:34 PM rgw Bug #58033 (New): multipart copy part: use refcount optimization when possible
- rgw only supports CopyObject[1] for object sizes up to rgw_max_put_size=5GB, and requires multipart with UploadPartCo...
- 07:07 PM rgw Bug #50076 (Fix Under Review): route librdkafka log messages to rgw log
- 05:45 PM rgw Bug #50076 (In Progress): route librdkafka log messages to rgw log
- 06:27 PM Bug #58032 (New): ceph-base: main(): ceph-crash:109:main:TypeError: handler() takes 1 positional ...
- https://bugzilla.redhat.com/show_bug.cgi?id=2141246
Description of problem:
crash happened in the background
V... - 01:49 PM CephFS Bug #58031 (Resolved): cephfs-top: sorting/limit excepts when the filesystems are removed and cre...
- This happens in the main branch. Please check.
1. cephfs-top is launched and the clients are sorted by 'mlatavg(ms... - 01:42 PM CephFS Bug #58000 (Fix Under Review): mds: switch submit_mutex to fair mutex for MDLog
- 01:41 PM CephFS Bug #58008 (Fix Under Review): mds/PurgeQueue: don't consider filer_max_purge_ops when _calculate...
- 01:41 PM CephFS Bug #58028 (Triaged): cephfs-top: Sorting doesn't work when the filesystems are removed and created
- 10:12 AM CephFS Bug #58028 (Resolved): cephfs-top: Sorting doesn't work when the filesystems are removed and created
- Sorting doesn't work in the following scenario
1. cephfs-top is launched and the clients are sorted by 'mlatavg(ms... - 11:08 AM CephFS Bug #58030 (Resolved): mds: avoid ~mdsdir's scrubbing and reporting damage health status
- We are supposed to handle the case of mdsdir, where we
are not having any backtrace actually.We should prevent the
... - 10:49 AM CephFS Bug #58029 (Fix Under Review): cephfs-data-scan: multiple data pools are not supported
- 10:46 AM CephFS Bug #58029 (Resolved): cephfs-data-scan: multiple data pools are not supported
- The tool cannot properly recover if a fs has extra data pools. We need access to all data pools on `scan_extents` ste...
- 09:56 AM bluestore Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- In the customer case running luminous when OSD process was run twice (let's skip how), the assert
'file->fnode.ino' ... - 08:44 AM RADOS Bug #56772: crash: uint64_t SnapSet::get_clone_bytes(snapid_t) const: assert(clone_overlap.count(...
- This bug is present in v17.2.5
- 07:32 AM RADOS Bug #58027 (Closed): op slow from throttled to header_read
- ceph version 15.2.7
Op spend 500ms from throttled to header_read... - 06:47 AM Dashboard Feature #58016 (Fix Under Review): mgr/dashboard: expose grafana port and initial_admin_password ...
- 04:40 AM Dashboard Backport #58026 (Resolved): quincy: mgr/dashboard: update to angular 13
- https://github.com/ceph/ceph/pull/50124
- 04:40 AM Dashboard Backport #58025 (Resolved): quincy: mgr/dashboard: migrate bootstrap 4 to 5
- https://github.com/ceph/ceph/pull/50124
- 04:40 AM Dashboard Backport #58024 (Resolved): quincy: mgr/dashboard: update jest to 28
- https://github.com/ceph/ceph/pull/50124
- 04:28 AM Dashboard Bug #48258 (Pending Backport): mgr/dashboard: Switch from tslint to eslint
- 04:27 AM Dashboard Cleanup #57972 (Pending Backport): mgr/dashboard: update jest to 28
- 04:27 AM Dashboard Cleanup #57867 (Pending Backport): mgr/dashboard: migrate bootstrap 4 to 5
- 04:26 AM Dashboard Cleanup #57866 (Pending Backport): mgr/dashboard: update to angular 13
- 12:24 AM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- There is also a coredump located at `/a/matan-2022-09-08_11:12:20-rados:singleton-main-distro-default-smithi/7020422/...
- 12:01 AM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- Some relevant frames:...
11/14/2022
- 11:39 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- I followed Brad's ubuntu 20.04 coredump tutorial: https://source.redhat.com/personal_blogs/debugging_a_ceph_osd_cored...
- 08:20 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- The original build is by now expired, so I'm rebuilding it here: https://shaman.ceph.com/builds/ceph/wip-kefu-testing...
- 08:14 PM RADOS Bug #57632: test_envlibrados_for_rocksdb: free(): invalid pointer
- Ran the test locally in an ubuntu 20.04 environment, and the test ran fine.
There is a coredump located under /a/k... - 10:51 PM Bug #58019: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
- Found this https://github.com/ceph/ceph/pull/47341
Would love to know when we'll be getting a 16.2.11 release with... - 03:50 PM Bug #58019 (New): Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
- The public access URL:
https://spvip-ch2-h01.ece.comcast.net/swift/v1/AUTH_{UUID_REDACTED}/{bucket-name}
Returns "N... - 09:32 PM CephFS Fix #58023 (Pending Backport): mds: do not evict clients if OSDs are laggy
- Monitoring perf dumps from the MDS can sometimes show that OSDs are laggy, "objecter.op_laggy" and "objecter.osd_lagg...
- 05:06 PM bluestore Bug #58022 (Pending Backport): Fragmentation score rising by seemingly stuck thread
- Due to issue https://tracker.ceph.com/issues/57672 we've been monitoring our clusters closely ensure it doesn't run i...
- 04:43 PM Orchestrator Bug #58021 (Resolved): Promteheus haproxy targets are down
- Using the latest code from main (v18.0) prometheus is not able to scrap the metrics of haproxy. This happens because ...
- 04:18 PM Orchestrator Bug #57998: cephadm stuck trying to download "mon"
- I executed the following:...
- 04:16 PM Orchestrator Bug #57998: cephadm stuck trying to download "mon"
- Here's the dump, there is definitely something fishy here, how do I remove it?...
- 11:04 AM Orchestrator Bug #57998 (Need More Info): cephadm stuck trying to download "mon"
- 03:57 PM rgw Bug #58020 (Pending Backport): notifications: zero timestamp in complete multipart upload event
- this is a regression due to: https://github.com/ceph/ceph/pull/42266
(original fix was: https://github.com/ceph/ceph... - 03:51 PM Orchestrator Bug #51642: cephadm/rgw : RGW server is not coming up: Initialization timeout, failed to initialize
- I just ran into this issue and would like to propose reopening this.
radosgw should clearly log something when sta... - 03:13 PM rgw Bug #58014 (Fix Under Review): notifications: metadata does not work for COPY events
- 01:42 PM rgw Bug #57562: multisite replication issue on Quincy
- Adam, I submitted a PR on top of yours. My 8h test has passed with this PR. The same test usually failed on earlier v...
- 01:37 PM rgw Bug #57562: multisite replication issue on Quincy
- Two more racing conditions found. These two are all on the journal entries.
h3. *Race condition 1:*... - 01:27 PM CephFS Bug #58018 (Fix Under Review): mount.ceph: will fail with old kernels
- 10:09 AM CephFS Bug #58018 (Pending Backport): mount.ceph: will fail with old kernels
- ...
- 12:10 PM bluestore Bug #53466 (Fix Under Review): OSD is unable to allocate free space for BlueFS
- 12:09 PM Orchestrator Bug #57910: ingress: HAProxy fails to start because keepalived IP address not yet available on ne...
- As of the current design of the ingress service, the keepalived starts before haproxy because the daemon depends on t...
- 11:37 AM RADOS Bug #55750: mon: slow request of very long time
- {
"description": "osd_failure(failed timeout osd.6 [v2:10.172.98.151:6800/39,v1:10.172.98.151:68... - 11:08 AM Orchestrator Bug #57897: ceph mgr restart causes restart of all iscsi daemons in a loop
- I tried to reproduce the issue with the same setup but wasn't successful so far. I started from a cluster running v17...
- 10:09 AM Dashboard Bug #58017 (Pending Backport): mgr/dashboard: Error "Evaluating rule failed" for CephPoolGrowthWa...
- Prometheus reports an error - many-to-many matching not allowed: matching labels must be unique on one side for CephP...
- 05:20 AM Linux kernel client Bug #57898 (In Progress): ceph client extremely slow kernel version between 5.15 and 6.0
- 05:20 AM Linux kernel client Bug #57898: ceph client extremely slow kernel version between 5.15 and 6.0
- I saw the same issue when testing the `6.1.0-rc1` upstream code:...
- 02:30 AM Dashboard Feature #58016 (Resolved): mgr/dashboard: expose grafana port and initial_admin_password in servi...
- as title says
11/13/2022
- 09:37 AM rgw Bug #58014 (Pending Backport): notifications: metadata does not work for COPY events
- this is a regression due to: https://github.com/ceph/ceph/pull/39192/commits/35a4eb4410394a0014648dda7df92642f3b536d3...
11/12/2022
- 04:18 PM Bug #58013 (New): Osdmap too big lead osd crash
- My cluster has failed, a large number of osds cannot be started,
and the troubleshooting found that the size of osd... - 03:40 PM Bug #58012: OpTracker event duration calculation errror
- with https://github.com/ceph/ceph/pull/48860 applied:
v2:... - 03:05 PM Bug #58012 (Duplicate): OpTracker event duration calculation errror
- h1. ceph version...
11/11/2022
- 08:31 PM RADOS Bug #56101: Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_timer
- Also to note: We set `ceph config set mgr mgr_stats_period 1` on the gibba cluster to reproduce this bug. (This occur...
- 07:00 PM devops Bug #56411 (Closed): Workaround for ceph-mgr breaks Cython builds
- 06:27 PM RADOS Bug #49689: osd/PeeringState.cc: ceph_abort_msg("past_interval start interval mismatch") start
- I think https://tracker.ceph.com/issues/49689#note-31 makes sense and the following logs also show what max_oldest_ma...
- 05:50 PM Feature #58010 (New): Add the OpenSSF Scorecard Action
- Hey, I'm Pedro and I'm working for Google and the "OpenSSF":https://openssf.org/ to improve the supply-chain security...
- 03:09 PM Dashboard Bug #57987 (In Progress): mgr/dashboard: missing data on hosts Grafana dashboard
- 02:11 PM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Venky Shankar wrote:
> xianpao chen wrote:
> > Venky Shankar wrote:
> > > Could you share the output of
> > >
>... - 01:02 PM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- xianpao chen wrote:
> Venky Shankar wrote:
> > Could you share the output of
> >
> > [...]
> >
> > Also, does... - 10:52 AM Dashboard Tasks #58009 (Resolved): mgr/dashboard: style cards on the page
- h3. Description
Currently on the dashboard revamp we are placing the cards within a Bootstrap grid of two rows w... - 10:08 AM RADOS Backport #58007: pacific: bail from handle_command() if _generate_command_map() fails
- please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/48846
ceph-backport.sh versi... - 09:07 AM RADOS Backport #58007 (Resolved): pacific: bail from handle_command() if _generate_command_map() fails
- https://github.com/ceph/ceph/pull/48846
- 10:03 AM RADOS Backport #58006: quincy: bail from handle_command() if _generate_command_map() fails
- please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/48845
ceph-backport.sh versi... - 09:07 AM RADOS Backport #58006 (Resolved): quincy: bail from handle_command() if _generate_command_map() fails
- https://github.com/ceph/ceph/pull/48845
- 09:14 AM CephFS Bug #58008: mds/PurgeQueue: don't consider filer_max_purge_ops when _calculate_ops
- When increasing filer_max_purge_ops on a pacific version mds, pq_executing_ops/pq_executing_ops_high_water of purge_q...
- 09:13 AM CephFS Bug #58008 (Resolved): mds/PurgeQueue: don't consider filer_max_purge_ops when _calculate_ops
- _calculate_ops relying on a config which can be modified on the fly will cause a bug. e.g.
# A file has 20 objects... - 09:01 AM RADOS Bug #57859 (Pending Backport): bail from handle_command() if _generate_command_map() fails
- PR https://github.com/ceph/ceph/pull/48044 has been merged in main.
- 06:46 AM Bug #57973: rook:rook module failed to connect k8s api server because of self-signed cert with se...
- It seems due to bad k8s cert trust chain. ceph is fine. This bug could be closed.
- 01:01 AM rgw Bug #57562: multisite replication issue on Quincy
- It should, thank you. I don't think it's the underlying cause, but it's a good catch.
11/10/2022
- 11:57 PM crimson Bug #58005 (Resolved): release-built osd failed to mkfs
- It seems that when "seastar::need_preempt" is true, "crimson::do_for_each" will turn into a long recursive function. ...
- 11:37 PM RADOS Bug #56101 (Fix Under Review): Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function s...
- 11:21 PM RADOS Bug #56101 (In Progress): Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_t...
- 04:52 AM RADOS Bug #56101: Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_timer
- Thanks for your work in capturing the core Laura.
I had a look at the coredump and it shows exactly what we had sp... - 09:06 PM rgw Bug #57562: multisite replication issue on Quincy
- A potential bug?
https://github.com/ceph/ceph/blob/main/src/cls/fifo/cls_fifo_types.h#L66
Should it be the follow... - 07:14 PM RADOS Bug #52657: MOSDPGLog::encode_payload(uint64_t): Assertion `HAVE_FEATURE(features, SERVER_NAUTILUS)'
- /a/yuriw-2022-10-17_17:31:25-rados-wip-yuri7-testing-2022-10-17-0814-distro-default-smithi/7071031
- 07:09 PM Orchestrator Bug #57311: rook: ensure CRDs are installed first
- /a/yuriw-2022-10-17_17:31:25-rados-wip-yuri7-testing-2022-10-17-0814-distro-default-smithi/7070926
- 05:11 PM Orchestrator Backport #58004 (In Progress): quincy: rook/k8s: nfs cluster creation ends up with no daemons dep...
- 05:00 PM Orchestrator Backport #58004 (Resolved): quincy: rook/k8s: nfs cluster creation ends up with no daemons deploy...
- https://github.com/ceph/ceph/pull/48830
- 04:54 PM Orchestrator Bug #57954 (Pending Backport): rook/k8s: nfs cluster creation ends up with no daemons deployment
- 03:44 PM Fix #58003 (Pending Backport): mon: add exception handling to ceph health mute
- Running ceph health mute with an invalid TTL causes the mon to crash, because the exception thrown by parse_timespan(...
- 03:25 PM rgw Bug #57706 (Need More Info): When creating a new user, if the 'uid' is not provided, error report...
- Hi Kevin Wang,
Could I get what version of Ceph this issue occurred on? The issue does seem to be resolved in the ... - 03:07 PM rgw Bug #57724 (Fix Under Review): Keys returned by Admin API during user creation on secondary zone ...
- 01:47 PM Orchestrator Bug #58001 (Fix Under Review): haproxy targets are not updated correctly in prometheus.yaml file
- 09:57 AM Orchestrator Bug #58001 (Resolved): haproxy targets are not updated correctly in prometheus.yaml file
- steps to reproduce the issue:
1) Bootstrap a new cluster (with monitoring enabled)
2) Wait until Prometheus is up... - 01:14 PM Bug #58002 (New): mon_max_pg_per_osd is not checked per OSD
- The warning for exceeding mon_max_pg_per_osd seems to be triggered only when the average PG count over all OSDs excee...
- 11:50 AM RADOS Bug #57989: test-erasure-eio.sh fails since pg is not in unfound
- For some reason, the pool already exist...
- 08:44 AM RADOS Bug #57757 (In Progress): ECUtil: terminate called after throwing an instance of 'ceph::buffer::v...
- 08:42 AM RADOS Bug #57618 (Fix Under Review): rados/test.sh hang and pkilled (LibRadosWatchNotifyEC.WatchNotify)
- 08:34 AM RADOS Bug #57618: rados/test.sh hang and pkilled (LibRadosWatchNotifyEC.WatchNotify)
- Some of the OSDs stopped due to valgrind errors. This is duplicate of other bug
- 08:39 AM RADOS Bug #57751 (Fix Under Review): LibRadosAio.SimpleWritePP hang and pkill
- 08:18 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Venky Shankar wrote:
> BTW, are you *not* seeing any "oversized cache" warning for the MDS?
there is no "oversize... - 04:06 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- BTW, are you *not* seeing any "oversized cache" warning for the MDS?
- 02:42 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Do you have lots of small files and frequently scan them?
- 01:12 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Venky Shankar wrote:
> Have you tried running `heap release`?
yes,but it didn't seem to work. - 07:38 AM RADOS Bug #57546: rados/thrash-erasure-code: wait_for_recovery timeout due to "active+clean+remapped+la...
- Thanks for taking a look Radek! That's a good point since we are seeing this issue with rados/thrash-erasure-code tes...
- 01:45 AM CephFS Bug #58000: mds: switch submit_mutex to fair mutex for MDLog
- From Patrick's comment in https://github.com/ceph/ceph/pull/44180#pullrequestreview-1174516711.
- 01:44 AM CephFS Bug #58000 (Resolved): mds: switch submit_mutex to fair mutex for MDLog
- The implementations of the Mutex (e.g. std::mutex in C++) do not
guarantee fairness, they do not guarantee that the ...
11/09/2022
- 10:56 PM RADOS Bug #56101: Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_timer
- Managed to reproduce this on the Gibba cluster and produce a coredump!
The core file is located on gibba001 under ... - 10:11 PM rgw Bug #57706: When creating a new user, if the 'uid' is not provided, error reported as 'Permission...
- On a branch close to the master branch from a vstart cluster when I try this same scenario I see:
[ali@acadia buil... - 09:51 PM rgw Bug #57562: multisite replication issue on Quincy
- We also found a place that might potentially cause issues.
Rgw locks the mutex and gets some data from "info" befo... - 09:22 PM rgw Bug #57562: multisite replication issue on Quincy
- Here is some more detailed explanation on how the -EINVAL(-22) error (hence datalog writing failure) happens based on...
- 08:18 PM RADOS Backport #57704 (Resolved): quincy: mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when reduc...
- https://github.com/ceph/ceph/pull/48321
- 08:17 PM RADOS Backport #57705 (Resolved): pacific: mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when redu...
- https://github.com/ceph/ceph/pull/48320
- 08:17 PM RADOS Bug #50089 (Resolved): mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when reducing number of...
- 07:08 PM CephFS Feature #57090 (Fix Under Review): MDSMonitor,mds: add MDSMap flag to prevent clients from connec...
- 06:23 PM Orchestrator Bug #57999 (Resolved): cephadm: cephadm always reports new or changed devices even if devices are...
- This appears to be an issue with the "created" field changing, which should not affect equality in this case...
- 06:19 PM Orchestrator Bug #57998: cephadm stuck trying to download "mon"
- hmm, can I see what "ceph config dump" spits out (feel free to remove anything sensitive if if necessary)? All the im...
- 10:51 AM Orchestrator Bug #57998 (Resolved): cephadm stuck trying to download "mon"
- Entire cluster cephadm management is stuck and repeatedly tries to download an unqualified "mon" instead of the ceph ...
- 04:34 PM RADOS Bug #51729: Upmap verification fails for multi-level crush rule
- Thanks again for looking at this.
I haven't looked further, but I suspect the issue will come down to the variable... - 01:22 PM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Have you tried running `heap release`?
- 09:35 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Venky Shankar wrote:
> Could you share the output of
>
> [...]
>
> Also, does running
>
> [...]
>
> redu... - 09:23 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Venky Shankar wrote:
> Could you share the output of
>
> [...]
>
> Also, does running
>
> [...]
>
> redu... - 08:56 AM CephFS Support #57952: Pacific: the buffer_anon_bytes of ceph-mds is too large
- Could you share the output of...
- 08:32 AM crimson Bug #57990: Crimson OSD crashes when trying to bring it up
- Crimson is not production ready yet, and there will be no backport to Quincy.
It is expected that there were bugs ... - 07:46 AM Dashboard Backport #57983 (In Progress): quincy: mgr/dashboard: error message displaying when editing journ...
- 07:43 AM Dashboard Backport #57982 (In Progress): pacific: mgr/dashboard: error message displaying when editing jour...
- 03:01 AM Backport #57997 (In Progress): quincy: ceph-crash service should run as unprivileged user, not ro...
- 02:18 AM Backport #57997 (Resolved): quincy: ceph-crash service should run as unprivileged user, not root ...
- https://github.com/ceph/ceph/pull/48805
- 02:58 AM Backport #57996 (In Progress): pacific: ceph-crash service should run as unprivileged user, not r...
- 02:18 AM Backport #57996 (Resolved): pacific: ceph-crash service should run as unprivileged user, not root...
- https://github.com/ceph/ceph/pull/48804
- 01:54 AM Bug #57967 (Pending Backport): ceph-crash service should run as unprivileged user, not root (CVE-...
11/08/2022
- 09:23 PM RADOS Bug #57017: mon-stretched_cluster: degraded stretched mode lead to Monitor crash
- pacific backport: https://github.com/ceph/ceph/pull/48803
- 08:59 PM RADOS Bug #57017: mon-stretched_cluster: degraded stretched mode lead to Monitor crash
- quincy backport: https://github.com/ceph/ceph/pull/48802
- 09:20 PM bluestore Feature #57785: fragmentation score in metrics
- @Vikhyat, no worries. Based on Kevin's comment, I think this metric might be better suited for Prometheus than Teleme...
- 06:37 PM bluestore Feature #57785: fragmentation score in metrics
- Laura - sorry I missed the update. Can you please ping Adam and Igor?
- 08:59 PM rbd Bug #57941: Severe performance drop after writing 100 GB of data to RBD volume, dependent on RAM ...
- Thanks for looking into this Christopher. You are right, this is a 100% sequential workload, just filling a volume wi...
- 08:23 PM rbd Bug #57941 (In Progress): Severe performance drop after writing 100 GB of data to RBD volume, dep...
- 08:23 PM rbd Bug #57941: Severe performance drop after writing 100 GB of data to RBD volume, dependent on RAM ...
- I'm not familiar with PVE and how it sets up Ceph. I took a look at your testcase and it appears to be a sequential w...
- 07:37 PM bluestore Fix #54299 (Need More Info): osd error restart
- 07:34 PM bluestore Bug #57672 (Duplicate): SSD OSD won't start after high framentation score!
- 07:27 PM bluestore Bug #53466 (In Progress): OSD is unable to allocate free space for BlueFS
- 07:23 PM RADOS Bug #51729: Upmap verification fails for multi-level crush rule
- I believe I've reproduced the issue using the osdmaps that Chris provided.
First, I used the osdmaptool to run the... - 02:49 PM rgw Bug #57911 (Fix Under Review): Segmentation fault when uploading file with bucket policy on Quincy
- 02:08 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- after rechecking the logs it looks like we are taking 2 different versions of smithi01231941-9:head
All chunks with ... - 05:44 AM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- @Laura, thanks for confirm that in the coredump, yes, shard0 also showing that when it get the chunk from bluestore:
... - 12:07 AM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- Brad and I did some more debugging today.
Here is the end of the log associated with the coredump:... - 12:03 PM Orchestrator Bug #57897: ceph mgr restart causes restart of all iscsi daemons in a loop
- Adam King wrote:
> this is a painful one. @David at least until we have a fix for this, I will mention that setting ... - 09:42 AM Bug #57956: Ceph monitors in crash loop
- liu jun wrote:
> Creating a pool causes mon to restart
>
> This is the detailed question:https://github.com/rook/... - 09:32 AM Dashboard Backport #57995 (New): quincy: mgr/dashboard: paginate services
- 09:31 AM Dashboard Backport #57994 (Rejected): pacific: mgr/dashboard: paginate services
- 09:21 AM Dashboard Feature #56512 (Pending Backport): mgr/dashboard: paginate services
- 08:53 AM Support #57992: Stuck in linking when I comiple the CEPH
- And when I ctrl+c to stop it and restart it with ninja, it will stuck in "dashboard nodeenv is being installed".
- 02:14 AM Support #57992 (New): Stuck in linking when I comiple the CEPH
- I follow the README to compile the CEPH on Github(https://github.com/ceph/ceph/tree/v17.2.5#readme). When I ninja the...
- 08:39 AM Orchestrator Feature #51971 (New): cephadm/ingress: update keepalived container image
- Reopning to check/address some of the concerns about the current keepalived image
- 05:00 AM Dashboard Backport #57993 (New): quincy: mgr/dashboard: Improve level AA color contrast accessibility for d...
- 04:51 AM Dashboard Bug #56023 (Pending Backport): mgr/dashboard: Improve level AA color contrast accessibility for d...
- 01:20 AM Orchestrator Documentation #57991 (New): Migration documentation about osd service
- The documentation doesn't mention how to make the osds in the cluster managed, nor how to add new osds. A new cephadm...
- 12:24 AM crimson Bug #57990 (New): Crimson OSD crashes when trying to bring it up
- Hello,
Using the `crimson-osd` Ubuntu package for Quincy, we're seeing somewhat recurrent crashes when trying to b...
11/07/2022
- 09:45 PM rgw Bug #57562: multisite replication issue on Quincy
- > I think if the create_part is made exclusive, one of them would fail at part creation and let the other complete pa...
- 09:27 PM RADOS Bug #57977: osd:tick checking mon for new map
- Radoslaw Zarzynski wrote:
> Octopus is EOL. Does it happen on a supported release?
>
> Regardless of that, could ... - 06:13 PM RADOS Bug #57977 (Need More Info): osd:tick checking mon for new map
- Octopus is EOL. Does it happen on a supported release?
Regardless of that, could you please provide logs from this... - 07:30 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- Also to note, we can see information about argument `to_read` here:...
- 07:27 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- @Nitzan, what do you think about this analysis? Or are there any other frames/locals you'd like me to check?
- 07:12 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- Looking at frame 12, I can see that the incorrect length (262144) for shard 0 is evident in the local variable "from"...
- 06:02 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- Got it to detect the right symbols with the new build!
I will attempt to analyze this coredump at a deeper level, ... - 03:16 PM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- According to Brad, the build needs to be as close to the test branch that originally experienced the crash as possibl...
- 07:18 PM RADOS Bug #51729: Upmap verification fails for multi-level crush rule
- Thanks Chris! @Radek I have been taking some time to analyze this scenario, and will post updates soon.
- 06:36 PM RADOS Bug #51729: Upmap verification fails for multi-level crush rule
- Thanks for the info! Laura, would you mind retaking a look?
- 06:36 PM RADOS Bug #51729 (New): Upmap verification fails for multi-level crush rule
- 06:43 PM RADOS Bug #50219 (Closed): qa/standalone/erasure-code/test-erasure-eio.sh fails since pg is not in reco...
- The original issue was caused by a commit in a wip branch being tested, so it's highly unprobable it's a reoccurence....
- 06:42 PM RADOS Bug #57989 (New): test-erasure-eio.sh fails since pg is not in unfound
- /a/lflores-2022-10-17_18:19:55-rados:standalone-main-distro-default-smithi/7071287...
- 06:35 PM RADOS Bug #57845: MOSDRepOp::encode_payload(uint64_t): Assertion `HAVE_FEATURE(features, SERVER_OCTOPUS...
- Likely it's even a duplicate of https://tracker.ceph.com/issues/52657.
- 06:28 PM RADOS Bug #52136 (Fix Under Review): Valgrind reports memory "Leak_DefinitelyLost" errors.
- 06:26 PM RADOS Bug #57940 (Duplicate): ceph osd crashes with FAILED ceph_assert(clone_overlap.count(clone)) when...
- Looks like a duplicate of 56772.
- 06:24 PM RADOS Bug #55141: thrashers/fastread: assertion failure: rollback_info_trimmed_to == head
- Nitzan Mordechai wrote:
> Radoslaw Zarzynski wrote:
> > Well, just found a new occurance.
> Where can i find it?
... - 06:12 PM RADOS Bug #56101: Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_timer
- Brad and I ran a reproducer on the gibba cluster (restarting OSDs with `for osd in $(systemctl -l |grep osd|gawk '{pr...
- 06:01 PM RADOS Bug #56101: Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_timer
- Is there any news on that?
- 05:59 PM RADOS Bug #49689: osd/PeeringState.cc: ceph_abort_msg("past_interval start interval mismatch") start
- Updated the PR link.
- 05:37 PM Dashboard Cleanup #57984 (In Progress): mgr/dashboard: Add tooltip
- 07:27 AM Dashboard Cleanup #57984 (Resolved): mgr/dashboard: Add tooltip
- h3. Description of problem
Add tooltip for '# Local' and '# Remote' columns in rbd mirroring page. - 03:40 PM rbd Feature #57988 (In Progress): [rbd-mirror] checksumming for snapshot-based mirroring
- This is similar to an older feature request for journal-based mirroring but for snapshot-based mirroring the hope is ...
- 01:48 PM CephFS Bug #57985 (Triaged): mds: warning `clients failing to advance oldest client/flush tid` seen with...
- 09:06 AM CephFS Bug #57985 (Pending Backport): mds: warning `clients failing to advance oldest client/flush tid` ...
- https://bugzilla.redhat.com/show_bug.cgi?id=2134709
Generally seen when the MDS is heavily loaded with I/Os. Inter... - 01:16 PM Dashboard Bug #57987 (Resolved): mgr/dashboard: missing data on hosts Grafana dashboard
- h3. Description of problem
A lot of data is missing on the hosts Grafana dashboard (host-detail) and an error mess... - 09:51 AM Linux kernel client Bug #57986: ceph: ceph_fl_release_lock cause "unable to handle kernel paging request at fffffffff...
- There should be a race in 'filp_close()`, for example in a single process a file is opened twice with two different f...
- 09:48 AM Linux kernel client Bug #57986 (Resolved): ceph: ceph_fl_release_lock cause "unable to handle kernel paging request a...
- ...
- 09:37 AM Linux kernel client Bug #57686 (Fix Under Review): general protection fault and CephFS kernel client hangs after MDS ...
- The patchwork: https://patchwork.kernel.org/project/ceph-devel/patch/20221107071759.32000-1-xiubli@redhat.com/
<pr... - 06:54 AM Linux kernel client Bug #57686 (In Progress): general protection fault and CephFS kernel client hangs after MDS failover
- 09:23 AM Bug #57976: ceph-volume lvm activate removes /var/lib/ceph/osd/ceph-XXX folder and then chokes on...
- Looks like the problem is gone after a full reboot. No idea what was going on, but it was reproducible on all nodes.
- 07:22 AM Linux kernel client Bug #57898: ceph client extremely slow kernel version between 5.15 and 6.0
- Minjong Kim wrote:
> https://gist.github.com/caffeinism/dbfd974374d620911a6c0c3dd1daadfb
>
> I am not good at wri... - 06:54 AM Linux kernel client Bug #57817 (Duplicate): general protection fault and CephFS kernel client hangs after MDS failover
- This is exactly the same issue with tracker#57686.
- 06:37 AM Dashboard Backport #57983 (Resolved): quincy: mgr/dashboard: error message displaying when editing journam ...
- https://github.com/ceph/ceph/pull/48807
- 06:37 AM Dashboard Backport #57982 (Resolved): pacific: mgr/dashboard: error message displaying when editing journam...
- https://github.com/ceph/ceph/pull/48806
- 06:25 AM Dashboard Bug #57922 (Pending Backport): mgr/dashboard: error message displaying when editing journam mirro...
- 06:23 AM Backport #57981 (New): quincy: ceph-mixin: Add Prometheus Alert for Degraded Bond
- 06:23 AM rgw Bug #57980: rgw/cloud-transition: transition fails when using MCG Azure Namespacestore with a pre...
- Few observations:
- 2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to check object on the ... - 06:21 AM rgw Bug #57980 (Pending Backport): rgw/cloud-transition: transition fails when using MCG Azure Namesp...
- Reported by - dparkes@redhat.com
>>>>
Found Errors during cloud transition when using MCG Azure Namespacestore wit... - 06:18 AM Feature #57962 (Pending Backport): ceph-mixin: Add Prometheus Alert for Degraded Bond
- 06:07 AM rgw Bug #57979 (Pending Backport): rgw/cloud-tranistion: Issues with MCG cloud endpoint
- Below issues were observed while testing cloud-transition feature using MCG (Noobaa) endpoint
1) Creation of targe... - 04:35 AM Bug #57966: Ceph cluster osds failed when ms_cluster_type=async+rdma is used
- the same problem on ceph 17.2.5:
root@ceph01:~# ceph crash info 2022-11-07T03:29:36.731174Z_bb6f8fea-ea87-4f83-a28a-... - 01:08 AM RADOS Bug #57937: pg autoscaler of rgw pools doesn't work after creating otp pool
- Is there any updates? Please let me know if I can do something.
11/06/2022
- 02:27 PM Dashboard Feature #57978 (Fix Under Review): mgr/dashboard: allow to get/update RBD image metadata via REST...
- h3. Description of problem
Currently we are missing an ability to get/update RBD image metadata via REST API. We c... - 05:47 AM RADOS Bug #57757: ECUtil: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::end_of...
- @brad, maybe it's a good candidate for another blog for upstream core dump analysis that you talked about (ubuntu 20.04)
Also available in: Atom