Project

General

Profile

Activity

From 01/15/2021 to 02/13/2021

02/13/2021

01:01 AM Bug #47921: Bad auth caps for orchestrated mds daemon
the problem is probably not related to the missing caps. Are you *really* sure this is the correct solution? Sebastian Wagner
12:54 AM Bug #49287 (New): podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.s...
... Sebastian Wagner

02/12/2021

05:24 PM Tasks #47369: Ceph scales to 100's of hosts, 1000's of OSDs....can orchestrator?
I was talking with Yaarit for getting real figures from Telemetry, and she mentioned the following ones:
* RBD ima...
Ernesto Puerta
04:15 PM Bug #49280: mds/orch: bare/short hostname as a number is not supported
... David Casier
04:13 PM Bug #49280 (Duplicate): mds/orch: bare/short hostname as a number is not supported
If the bare/short hostname is a simple id (example: 2.storage.domain):
----...
David Casier
03:46 PM Bug #49277: cephadm bootstrap --apply-spec <cluster.yaml> hangs
Reproduced just now and attaching logs and versions.
using latest cephadm from https://download.ceph.com/rpm-octo...
John Fulton
02:40 PM Bug #49277 (Duplicate): cephadm bootstrap --apply-spec <cluster.yaml> hangs
The feature introduced by https://tracker.ceph.com/issues/44873 seems to have the following flaw.
If I bootstrap...
John Fulton
01:48 PM Bug #49276 (Duplicate): Create multiple RGW instances in the same realm , same zone fails using c...
The scenario is to create 2 rgw instances on the same node (different ports) under the same realm and same zone.
Cu...
Juan Miguel Olmo Martínez
01:13 PM Bug #49273 (Resolved): cephadm fails deployment of node-exporter when ipv6 is disabled
Which is due to the check in `port_in_use` method checking for both, ipv4 and ipv6 support in the same try except block. Patrick Seidensal
12:45 PM Feature #49249 (Duplicate): cephadm: Automatically create OSDs after reinstalling base os
Sebastian Wagner
12:43 PM Feature #49159: "cephadm ceph-volume activate" does not support cephadm
Great! Please verify that the container image used is consistent across the cluster after running the adoption process. Sebastian Wagner
09:31 AM Feature #49159: "cephadm ceph-volume activate" does not support cephadm
Hi Sebastian,
Since I would need to do this for +500 OSDs, doing this manually is not really an option..
I guess ...
Kenneth Waegeman
12:30 PM Feature #49269: cephadm: upgrade stuck in repeating sleep when a host is offline
Did you verify that the upgrade continues, if the host is online again?
I'm a bit inclined to close this as works...
Sebastian Wagner
09:51 AM Feature #49269 (New): cephadm: upgrade stuck in repeating sleep when a host is offline
Even though the documentation clearly mentions that all hosts should be online before you initiate an upgrade I never... Gunther Heinrich
12:01 AM Feature #47885: Add networking checks
Currently preparing a PR which deals with requirements 1 and 4. Paul Cuzner

02/11/2021

07:34 PM Bug #49239: cephadm cannot deploy OSDs with selinux-policy-minimum
Follow-on fix for systems that do not have /usr/share/empty (eg. SUSE): https://github.com/ceph/ceph/pull/39424
An...
Ken Dreyer
01:34 AM Bug #49239 (Pending Backport): cephadm cannot deploy OSDs with selinux-policy-minimum
Ken Dreyer
07:12 PM Bug #48142 (Fix Under Review): rados:cephadm/upgrade/mon_election tests are failing: CapAdd and p...
Sage Weil
06:24 PM Feature #49159: "cephadm ceph-volume activate" does not support cephadm
does https://tracker.ceph.com/issues/46691#note-1 help? Sebastian Wagner
05:55 PM Feature #49159: "cephadm ceph-volume activate" does not support cephadm
Hi,
Thanks for looking into this!
Well, it's a chicken-egg problem:
This is a node that I just reinstalled to a...
Kenneth Waegeman
10:22 AM Feature #49159: "cephadm ceph-volume activate" does not support cephadm
there should not be such a big difference between running ceph-volume naively vs in a container.
Please have a lo...
Sebastian Wagner
04:49 PM Feature #49249 (Duplicate): cephadm: Automatically create OSDs after reinstalling base os
#46691 provides the manual process of deploying cephadm OSDs.
we should probably provide an automated way to do t...
Sebastian Wagner
03:51 PM Bug #48598: "ceph orch daemon redeploy" fails with [errno 13] RADOS permission denied
might be related to a wrong or non-upt-do-date ceph.conf??? Sebastian Wagner
12:25 PM Feature #43696 (Rejected): cephadm: check that units start
this would make the daemon deployment of cephadm super slow Sebastian Wagner
12:23 PM Documentation #45383 (Can't reproduce): Cephadm.py OSD deployment fails: full device path or just...
Sebastian Wagner
12:19 PM Support #49247 (Resolved): cephadm: Add support for single daemon redeployment
already done: https://docs.ceph.com/en/latest/api/mon_command_api/#orch-daemon-redeploy Sebastian Wagner
12:11 PM Support #49247 (Resolved): cephadm: Add support for single daemon redeployment
Currently, cephadm allows to redeploy services as a whole which sometimes might be a little over the top if only one ... Gunther Heinrich
12:17 PM Feature #45770 (Rejected): cephadm: allow count=0 to have services without daemons
why? Sebastian Wagner
12:15 PM Feature #43691 (Resolved): cephadm: upgrade major releases
done in the meantime Sebastian Wagner
12:14 PM Feature #44606 (Resolved): cephadm: RGW firewall + static port
resolved in the meantime Sebastian Wagner
12:14 PM Bug #46134 (Can't reproduce): ceph mgr should fail if it cannot add osd
Sebastian Wagner
12:13 PM Feature #45769 (Resolved): cephadm: Don't deploy on offline hosts
done in the meantime Sebastian Wagner
12:12 PM Feature #46265 (Duplicate): test cephadm MDS deployment
Sebastian Wagner
12:11 PM Documentation #46335: Document "Using cephadm to set up rgw-nfs"
thing is, I don't want users to use nfs-rgw. performance and usability is just poor. Sebastian Wagner
12:06 PM Bug #46568 (Can't reproduce): cephadm: Sometimes setting global container_image does not work
please reopen, if it is still reproducible Sebastian Wagner
12:05 PM Support #46547 (Resolved): cephadm: Exception adding host via FQDN if host was already added
Sebastian Wagner
12:01 PM Documentation #46691: Document manually deploment of OSDs
h3. How to manually (re-)deploy OSDs
In order to manually deploy cephadm OSDs, first run ceph-volume (skip this ...
Sebastian Wagner
11:56 AM Documentation #44354 (Duplicate): cephadm: Log messages are missing
fixed by using journald as log driver Sebastian Wagner
11:55 AM Bug #46990 (Can't reproduce): execnet: EOFError: couldnt load message header, expected 9 bytes, g...
Sebastian Wagner
11:54 AM Bug #44673 (Rejected): cephadm: `orch apply` and `orch daemon add` use completely different code ...
Sebastian Wagner
11:53 AM Bug #47401 (Can't reproduce): improve drive group validation
workaround: Use *ceph orch apply -i* instead of *ceph orch apply osd -i* Sebastian Wagner
11:52 AM Feature #47533 (Rejected): Scan for dangling ceph auth entries
we now clean up auth entities. Sebastian Wagner
11:50 AM Bug #47702 (Can't reproduce): upgrading via ceph orch upgrade start results in partial applicatio...
Sebastian Wagner
11:49 AM Bug #47694 (Won't Fix): downgrading via ceph orch upgrade start results in partial application an...
we have to support downgrades to some degree. closing as it worked eventually Sebastian Wagner
11:48 AM Bug #47726 (Resolved): disk selector should pass all devices to ceph-volume (available and unavai...
Sebastian Wagner
11:45 AM Bug #46665 (Resolved): cephadm plugin: Failure to start service stops service loop; no other inst...
Sebastian Wagner
11:45 AM Bug #47916 (Pending Backport): podman containers running in a detached state do not output logs t...
Sebastian Wagner
11:44 AM Bug #48105 (Can't reproduce): cephadm.py: failure on interactive on error for archive file handling
that code changed in the meantime Sebastian Wagner
11:42 AM Bug #48171 (Resolved): catatonit not available on CentOS
pacific now uses... Sebastian Wagner
11:41 AM Bug #45808 (Resolved): cephadm/test_adoption.sh: Error parsing image configuration: Invalid statu...
Sebastian Wagner
11:36 AM Bug #47500 (Won't Fix): Feature <encryption> is not supported" with having it set it to "False"
downstream issue not upstream Sebastian Wagner
11:36 AM Tasks #47369: Ceph scales to 100's of hosts, 1000's of OSDs....can orchestrator?
yes, we have users with > 1000 osds. that works already :-) Sebastian Wagner
11:27 AM Feature #45712 (Duplicate): Add 'state' attribute to ServiceSpec
Sebastian Wagner
11:25 AM Bug #46247 (Can't reproduce): cephadm mon failure: Error: no container with name or ID ... no suc...
This was fixed in the meantime Sebastian Wagner
11:12 AM Feature #47261 (New): cephadm integration for cephfs-mirror daemon
Sebastian Wagner
11:10 AM Feature #47261: cephadm integration for cephfs-mirror daemon
cephfs-mirror daemon
https://github.com/ceph/ceph/blob/72c3b5e6a3a88c40f6b8286cd4b2d6f1a335ed63/doc/man/8/cephfs-mir...
Sebastian Wagner
11:11 AM Feature #48560 (Need More Info): Spec files for each daemon in the monitoring stack
Sebastian Wagner
11:11 AM Feature #48560 (New): Spec files for each daemon in the monitoring stack
Sebastian Wagner
11:06 AM Feature #48560 (Need More Info): Spec files for each daemon in the monitoring stack
Sebastian Wagner
11:10 AM Bug #47107 (Resolved): device-health-metrics unavailable because image ceph/ceph:latest has smart...
resolved in ceph-container Sebastian Wagner
11:08 AM Feature #48822: Add proper port management to mgr/cephadm
... Sebastian Wagner
11:05 AM Feature #49246 (Duplicate): cephadm: Display error message when given service name is wrong
When executing a service command with a wrong service name like... Gunther Heinrich
11:05 AM Feature #47145: cephadm: Multiple daemons of the same service on single host
in order to co-locate daemons, we have to use different ports for those new daemons. Sebastian Wagner
11:02 AM Bug #48656: cephadm botched install of ceph-fuse (symbol lookup error)
tbh, this is somewhat out of scope for cephadm. cephadm mainly cares about containers, not so much about keeping pack... Sebastian Wagner
10:57 AM Bug #48442: cephadm: upgrade loops on mixed x86_64/arm64 cluster
right now, this is somewhat low on our priority list. But in pacific, this should be improved by using repo_digest fo... Sebastian Wagner
10:55 AM Bug #48261 (Won't Fix): cephadm ceph-volume inventory -- --format json-pretty: INFO:cephadm:/usr...
*workaround*: ... Sebastian Wagner
10:54 AM Bug #48799 (Can't reproduce): test_cephadm: stderr Job for container.alertmanager.a.service faile...
gone Sebastian Wagner
10:54 AM Subtask #45116 (Resolved): cephadm: RGW Load balancer using HAproxy
Sebastian Wagner
10:54 AM Documentation #48333 (Rejected): cephadm: document the image used by cephadm to call ceph-volume ...
Sebastian Wagner
10:53 AM Bug #48894: cephadm e2e: ceph device monitoring off: Error EINVAL
somehow this is gone now Sebastian Wagner
10:52 AM Bug #48894 (Can't reproduce): cephadm e2e: ceph device monitoring off: Error EINVAL
Sebastian Wagner
10:51 AM Bug #48694 (Resolved): ceph-volume: unrecognized arguments: --filter-for-batch
Sebastian Wagner
10:50 AM Bug #45973: Adopted MDS daemons are removed by the orchestrator because they're orphans
prio=low. probably easier to simply redeploy MDS for upstream and find a typical downstream solution for downstream. Sebastian Wagner
10:49 AM Bug #45465 (Resolved): cephadm: `ceph orch restart osd` has the potential to break your cluster
Sebastian Wagner
10:48 AM Support #48630 (Resolved): non-LVM OSD do not start after upgrade from 15.2.4 -> 15.2.7
Sebastian Wagner
10:47 AM Bug #45628 (Resolved): cephadm qa: smoke should verify daemons are actually running
Sebastian Wagner
10:47 AM Bug #48925 (Resolved): cephadm: iscsi missing mgr permissions
Sebastian Wagner
10:47 AM Bug #48947 (Resolved): cephadm: fix rgw osd cap tag
Sebastian Wagner
10:46 AM Bug #48594 (Resolved): cephadm: too many osd privileges for osd caps
Sebastian Wagner
10:45 AM Bug #48870 (Resolved): cephadm: Several services in error status after upgrade to 15.2.8: unrecog...
https://github.com/ceph/ceph/pull/39300 Sebastian Wagner
10:44 AM Bug #44559 (Can't reproduce): cephadm logs an invalid stat command
please reopen, if you see this again Sebastian Wagner
10:42 AM Bug #49016 (Resolved): find multiple coredumps of conmon
resolved upstream Sebastian Wagner
10:41 AM Bug #49056 (Resolved): faulty behaviour running ceph orch apply mds with missing fsname
Sebastian Wagner
10:41 AM Bug #49056: faulty behaviour running ceph orch apply mds with missing fsname
always use yaml files! ... Sebastian Wagner
10:39 AM Bug #48916 (Duplicate): "File system None does not exist in the map" in upgrade:octopus-x:paralle...
Sebastian Wagner
10:38 AM Bug #49014 (Resolved): OSD service specifications ignore "rotational: 0"
Sebastian Wagner
10:37 AM Feature #47139 (Resolved): Require a minimum version for podman/docker
Sebastian Wagner
10:37 AM Feature #47139: Require a minimum version for podman/docker
we can't backport podman >= 2.0 to octopus! Old octopus versions don#t support podman 2, thus we have to have a way f... Sebastian Wagner
10:26 AM Bug #48164 (Resolved): Orchestrator: failed deployments leave orphaned auth entries
Sebastian Wagner
10:25 AM Bug #45279: cephadm bootstrap: monmaptool --create: error writing to '/tmp/monmap': (21) Is a dir...
Could you please verify that /tmp/ceph-tmp6sp3jhv3 is in fact a file?
Then, could you please run the docker comman...
Sebastian Wagner
02:09 AM Bug #49228 (Resolved): qa/tasks/cephadm.py: file changed as we read it
pacific backport: https://github.com/ceph/ceph/pull/39403 Neha Ojha

02/10/2021

05:05 PM Bug #49143: rados/upgrade/pacific-x/parallel: monclient(hunting): authenticate timed out after 30...
The problem seems to occur when the first mon is restarted after upgrade.... Neha Ojha
03:23 PM Bug #49239 (Resolved): cephadm cannot deploy OSDs with selinux-policy-minimum
When the following conditions are true:
# A host has @selinux-policy-targeted@,
# We mount the host's @/sys@ int...
Ken Dreyer
12:47 PM Bug #48142 (New): rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are...
Sebastian Wagner
12:35 PM Feature #49235 (Resolved): cephadm: Log number of already upgraded daemons during upgrade process
During the upgrade process it would be helpful if cephadm displays the number of daemons which have aleady been upgra... Gunther Heinrich
11:26 AM Bug #49233 (New): cephadm shell: TLS handshake timeout
https://pulpito.ceph.com/swagner-2021-02-09_10:28:14-rados:cephadm-wip-swagner2-testing-2021-02-08-1109-pacific-distr... Sebastian Wagner
11:23 AM Bug #49232 (Can't reproduce): standard_init_linux.go:211: exec user process caused "exec format e...
https://pulpito.ceph.com/swagner-2021-02-09_10:28:14-rados:cephadm-wip-swagner2-testing-2021-02-08-1109-pacific-distr... Sebastian Wagner

02/09/2021

10:21 PM Bug #49228: qa/tasks/cephadm.py: file changed as we read it
In pacific... Neha Ojha
10:15 PM Bug #49228 (Resolved): qa/tasks/cephadm.py: file changed as we read it
... Neha Ojha
02:35 PM Bug #49223: unrecognized arguments: --container-init
If I remember correctly, the "--container-init" saga went about like so:
1. in general, there is a need for contai...
Nathan Cutler
10:09 AM Bug #49223: unrecognized arguments: --container-init
looks like https://github.com/ceph/ceph/pull/36822 was broken back then and https://github.com/ceph/ceph/pull/37648 n... Sebastian Wagner
09:51 AM Bug #49223 (Resolved): unrecognized arguments: --container-init
... Sebastian Wagner
10:15 AM Bug #49126 (Fix Under Review): rook: 'ceph orch ls' throws type error
Varsha Rao

02/08/2021

06:47 PM Bug #49143: rados/upgrade/pacific-x/parallel: monclient(hunting): authenticate timed out after 30...
... Neha Ojha
06:30 PM Bug #48142: rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutua...
description: rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade
4-wait distro$/{centos_8....
Deepika Upadhyay
03:52 PM Documentation #49214 (Resolved): Docs: howto Restore MON quorum
https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#removing-monitors-from-an-unhealthy-cluster
Sebastian Wagner
09:38 AM Bug #48068 (Resolved): cephadm: Various properties like 'last_refresh' do not contain timezone
Volker Theile
06:13 AM Bug #49013: cephadm: Service definition causes some container startups to fail
> 1. During upgrade, the new mgr doesn't redeploy the other mgrs (again) to ensure the unit.run file is in sync with ... Gunther Heinrich

02/05/2021

04:06 PM Bug #49013: cephadm: Service definition causes some container startups to fail
Sage Weil wrote:
> This wasn't explicitly stated in the initial ticket, but the only daemons that failed to start ...
Marvin Boothby
03:36 PM Bug #49013 (In Progress): cephadm: Service definition causes some container startups to fail
Sebastian Wagner
02:36 PM Bug #49013: cephadm: Service definition causes some container startups to fail
Okay, summarizing to make sure I understand. We have two problems:
1. During upgrade, the new mgr doesn't redeplo...
Sage Weil
08:24 AM Bug #49013: cephadm: Service definition causes some container startups to fail
@Marvin
Thanks for confirming that this is an issue unrelated to my cluster.
I did another upgrade (this time fro...
Gunther Heinrich
03:50 PM Feature #49171: cephadm: set osd-memory-target
https://pad.ceph.com/p/autotune_memory_target Sebastian Wagner
02:29 PM Bug #49191 (Duplicate): cephadm: service_type: osd: Failed to apply: ''NoneType'' object has no a...
... Sebastian Wagner
11:25 AM Bug #48933: cephadm: EOFError: couldnt load message header, expected 9 bytes, got 0
Unfortunately I already cannot go back to do that. It was no big issue from the beginning since as far as I remember ... Gunther Heinrich

02/04/2021

04:52 PM Bug #48715 (Resolved): docker-mirror: x509: certificate relies on legacy Common Name field, use S...
appears to be fixed! Sage Weil
04:36 PM Bug #48754 (In Progress): "failed xx 'sudo systemctl start ceph-None@rgw.client.1'" in upgrade:oc...
Sage Weil
03:59 PM Feature #49171 (Resolved): cephadm: set osd-memory-target
cpeh-ansible sets @osd memory target@ based on:
https://github.com/ceph/ceph-ansible/blob/71a5e666e39b11cd7945afa2...
Sebastian Wagner
02:25 PM Bug #49013: cephadm: Service definition causes some container startups to fail
Sorry @Gunther I kind of missed the part where you reported that the mgr damon shows the same behaviour I observed. T... Marvin Boothby
01:59 PM Bug #49013: cephadm: Service definition causes some container startups to fail
Hello there,
we are running two identical v15.2.8 clusters who apparently now show the same problem. I also did no...
Marvin Boothby
01:54 PM Feature #49165 (Need More Info): ceph crush class in osd service spec
It would be a nice feature to be able to override a crush class for ceph osd matching a certain drive_group or patter... Kenneth Waegeman
01:31 PM Bug #48157 (Resolved): test_cephadm.sh failure You have reached your pull rate limit. You may inc...
not seeing this issue anymore Deepika Upadhyay
12:17 PM Bug #48933: cephadm: EOFError: couldnt load message header, expected 9 bytes, got 0
Gunther Heinrich wrote:
> Yes, "python3 -V" gives me "Python 3.8.5".
Hm.
>
> Am I correct to assume that th...
Sebastian Wagner
12:04 PM Bug #48933: cephadm: EOFError: couldnt load message header, expected 9 bytes, got 0
Yes, "python3 -V" gives me "Python 3.8.5".
Am I correct to assume that the exception refers to python inside a con...
Gunther Heinrich
09:29 AM Feature #49159 (Resolved): "cephadm ceph-volume activate" does not support cephadm
On 15.2.8, when running cephadm ceph-volume -- lvm activate --all`, I get an error related to dmcrypt:... Kenneth Waegeman
03:46 AM Feature #44055 (Closed): cephadm: make 'ls' faster
PR closed without merge. cephadm exporter merge has made this change less important. Focus needs to be on exploiting ... Paul Cuzner
02:29 AM Feature #48846 (Closed): cephadm bootstrap: add --cluster-network
Paul Cuzner

02/03/2021

07:47 PM Bug #49143 (Resolved): rados/upgrade/pacific-x/parallel: monclient(hunting): authenticate timed o...
... Neha Ojha
05:19 PM Bug #48788 (Duplicate): cephadm bootstrap: monmaptool --create: error writing to '/tmp/monmap': (...
Sebastian Wagner
05:18 PM Bug #45279 (New): cephadm bootstrap: monmaptool --create: error writing to '/tmp/monmap': (21) Is...
Sebastian Wagner
03:25 PM Bug #48164 (Fix Under Review): Orchestrator: failed deployments leave orphaned auth entries
Sebastian Wagner
01:51 PM Bug #49013: cephadm: Service definition causes some container startups to fail
Based on some code analysis in "/src/pybind/mgr/cephadm/upgrade.py" in found... Gunther Heinrich
12:56 PM Tasks #46551 (Fix Under Review): cephadm: Add better a better hint how to add a host
Sebastian Wagner
12:52 PM Bug #47700 (Resolved): during OSD deletion: Module 'cephadm' has failed: Set changed size during ...
Sebastian Wagner
12:50 PM Bug #48510: CEPHADM_REFRESH_FAILED: detail item 0 not a [unicode] string
works for me. if I do that locally, I'm getting... Sebastian Wagner
12:25 PM Bug #48597 (Fix Under Review): pybind/mgr/cephadm: mds_join_fs not cleaned up
Sebastian Wagner
12:23 PM Feature #49127 (New): rook: Add support for service restart
... Varsha Rao
12:12 PM Bug #49126 (Resolved): rook: 'ceph orch ls' throws type error
... Varsha Rao
12:03 PM Bug #48924 (Fix Under Review): cephadm: upgrade process failed to pull target image: not enough v...
fixed by https://github.com/ceph/ceph/pull/39069/commits/d31bed79411ca493ec48eeed4e9cbb7ad92295c3 Sebastian Wagner
12:02 PM Bug #48924: cephadm: upgrade process failed to pull target image: not enough values to unpack (ex...
... Sebastian Wagner
11:59 AM Bug #48933: cephadm: EOFError: couldnt load message header, expected 9 bytes, got 0
Do you have installed /usr/bin/python3 on the remote host? Sebastian Wagner
11:57 AM Bug #48939: Orchestrator removes mon daemon from wrong host when removing host from cluster
At this point in the development,... Sebastian Wagner
11:53 AM Feature #47139 (Pending Backport): Require a minimum version for podman/docker
Sebastian Wagner
11:52 AM Bug #48982 (Resolved): cephadm: ubuntu_18_04: Error: error creating container storage: the contai...
Fixed by https://github.com/ceph/ceph/pull/39003 Sebastian Wagner
11:51 AM Bug #48982: cephadm: ubuntu_18_04: Error: error creating container storage: the container name
We already clean up the storage with
https://github.com/ceph/ceph/blob/faa93b751dc13003b23370f769a8ea252972c3dc/s...
Sebastian Wagner
11:50 AM Bug #48982: cephadm: ubuntu_18_04: Error: error creating container storage: the container name
The error is:... Sebastian Wagner
11:40 AM Bug #49014 (Pending Backport): OSD service specifications ignore "rotational: 0"
Sebastian Wagner
11:37 AM Bug #48930: when removing the iscsi service, the gateway config object remains
depends on https://github.com/ceph/ceph/pull/38883 Sebastian Wagner
11:35 AM Bug #49041 (Fix Under Review): cephadm: update container image tag for pacific
Sebastian Wagner
11:15 AM Bug #49076 (Duplicate): cephadm: Bootstrapping fails: json.decoder.JSONDecodeError: Expecting val...
Sebastian Wagner
11:14 AM Bug #49076 (Resolved): cephadm: Bootstrapping fails: json.decoder.JSONDecodeError: Expecting valu...
fixed by https://github.com/containers/conmon/pull/237 Sebastian Wagner
11:14 AM Bug #48993 (Resolved): cephadm: 'mgr stat' and/or 'pg dump' output truncated
fixed by https://github.com/containers/conmon/pull/237 Sebastian Wagner
11:02 AM Bug #49000 (Duplicate): JSONDecodeError when wait_for_mgr_restart()
Sebastian Wagner
11:01 AM Bug #49000 (New): JSONDecodeError when wait_for_mgr_restart()
Sebastian Wagner
06:33 AM Bug #48916: "File system None does not exist in the map" in upgrade:octopus-x:parallel-master
This issue is related to https://tracker.ceph.com/issues/45595 Varsha Rao

02/01/2021

10:23 PM Bug #48981 (Resolved): cephadm exporter: manager errors out with assertion error
Can't reproduce with current master - assume that Sage's PR https://github.com/ceph/ceph/pull/39097 resolved the issu... Paul Cuzner
05:51 PM Bug #48142: rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutua...
/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smith... Deepika Upadhyay
05:41 PM Bug #48993: cephadm: 'mgr stat' and/or 'pg dump' output truncated
/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smith... Deepika Upadhyay
04:30 PM Bug #49079 (Duplicate): cephadm: slow to clear CEPHADM_FAILED_DAEMON
the health alert is only cleared after a full cluster state update (_refresh_hosts_and_daemons()), but we may find th... Sage Weil
01:29 PM Bug #49076: cephadm: Bootstrapping fails: json.decoder.JSONDecodeError: Expecting value: line 321...
yes Sebastian Wagner
01:09 PM Bug #49076: cephadm: Bootstrapping fails: json.decoder.JSONDecodeError: Expecting value: line 321...
Thanks for the quick reply and the info. FYI and for the record, it seems that this error - as indicated in the podma... Gunther Heinrich
12:51 PM Bug #49076: cephadm: Bootstrapping fails: json.decoder.JSONDecodeError: Expecting value: line 321...
this is due to https://github.com/containers/podman/issues/9096 . Nothing we can do about this right now. Sebastian Wagner
12:42 PM Bug #49076 (Duplicate): cephadm: Bootstrapping fails: json.decoder.JSONDecodeError: Expecting val...
On latest Ubuntu 20.04.1 and Podman 2.2.1...
In relation to Bug #49013 I tried to bootstrap a new cluster for test...
Gunther Heinrich
10:31 AM Bug #49013: cephadm: Service definition causes some container startups to fail
I have an assumption and hope that someone can check if I'm correct.
For the update to 15.2.8 the service definiti...
Gunther Heinrich

01/31/2021

02:55 PM Bug #49032 (Resolved): cephadm: prepare-host fails after podman install on ubuntu
Kefu Chai

01/29/2021

06:09 PM Bug #48951 (Duplicate): teuthology only collected coredumps, not daemon logs
Sebastian Wagner
01:34 PM Bug #49056: faulty behaviour running ceph orch apply mds with missing fsname
removing worked doing 'ceph orch rm label:mds' Kenneth Waegeman
11:27 AM Bug #49056 (Resolved): faulty behaviour running ceph orch apply mds with missing fsname
On 15.2.8 , I accidently ran `ceph orch apply mds label:mds`, so without the <fsname>.
This did not give an error ...
Kenneth Waegeman
11:39 AM Bug #49013: cephadm: Service definition causes some container startups to fail
Short update:
All monitor containers fail to start (alertmanager, grafana, node-exporter and prometheus) because n...
Gunther Heinrich
10:11 AM Bug #48981: cephadm exporter: manager errors out with assertion error
see also: https://github.com/ceph/ceph/pull/39162 Sebastian Wagner
12:28 AM Bug #48916: "File system None does not exist in the map" in upgrade:octopus-x:parallel-master
Looks like a cephadm issue? Brad Hubbard
12:18 AM Bug #48916: "File system None does not exist in the map" in upgrade:octopus-x:parallel-master
I think this is the original issue.... Brad Hubbard

01/28/2021

10:51 PM Bug #49016: find multiple coredumps of conmon
https://pulpito.ceph.com/swagner-2021-01-28_08:47:22-rados:cephadm-wip-swagner2-testing-2021-01-27-1411-octopus-distr... Sebastian Wagner
03:51 PM Bug #46558 (In Progress): cephadm: paths attribute ignored for db_devices/wal_devices via OSD spec
Juan Miguel Olmo Martínez
02:39 PM Bug #49041 (Resolved): cephadm: update container image tag for pacific
Since pacific branch has been created, the latest-master-devel container image tag now refers to ceph@master v17.
...
Dimitri Savineau
12:55 PM Bug #49013: cephadm: Service definition causes some container startups to fail
Thanks for your reply and your suggestion.
I can report that your solution indeed solves the issue. After I commen...
Gunther Heinrich

01/27/2021

06:09 PM Bug #44559 (New): cephadm logs an invalid stat command
(no longer on my plate) Nathan Cutler
05:52 PM Bug #47862 (Resolved): cepham no longer requires apparmor-abstractions on SUSE
Nathan Cutler
03:37 PM Bug #49032 (Fix Under Review): cephadm: prepare-host fails after podman install on ubuntu
Michael Fritch
03:34 PM Bug #49032: cephadm: prepare-host fails after podman install on ubuntu
root cause is using the apt binary for install/update of packages when we should use the lower level apt-get instead:... Michael Fritch
03:33 PM Bug #49032 (Resolved): cephadm: prepare-host fails after podman install on ubuntu
cephadm prepare-host will install podman, but then later fail to detect the installation of podman... Michael Fritch
02:45 PM Bug #49013: cephadm: Service definition causes some container startups to fail
The change to forking is in this commit https://github.com/ceph/ceph/commit/e6792f306ab4d07251588fdca6ed3876ae3a092a
...
Kai Stian Olstad
09:47 AM Bug #49013: cephadm: Service definition causes some container startups to fail
I finally found out that it's the line *"Type=forking"* that's causing those issues.
Here are the steps I took to ...
Gunther Heinrich
11:49 AM Bug #48981: cephadm exporter: manager errors out with assertion error
https://pulpito.ceph.com/sage-2021-01-26_15:14:30-rados:cephadm-wip-sage-testing-2021-01-23-1326-distro-basic-smithi/... Sebastian Wagner

01/26/2021

05:30 PM Bug #49016: find multiple coredumps of conmon
might be fixed by https://github.com/containers/conmon/commit/d9bd8f838830bd507046f488d612f92c1524139a Kefu Chai
03:15 PM Bug #49016 (Resolved): find multiple coredumps of conmon
... Kefu Chai
03:08 PM Bug #48993: cephadm: 'mgr stat' and/or 'pg dump' output truncated
i have the same issue when testing Ubuntu_20.04 + conmon 2.0.24~1
- /a/kchai-2021-01-26_13:24:13-rados:cephadm-wip...
Kefu Chai
12:03 AM Bug #48993: cephadm: 'mgr stat' and/or 'pg dump' output truncated
Filed https://github.com/containers/podman/issues/9096 Sage Weil
02:21 PM Bug #49014: OSD service specifications ignore "rotational: 0"
Fix PR is 39083. Lukas Stockner
02:17 PM Bug #49014 (Resolved): OSD service specifications ignore "rotational: 0"
When applying an OSD service specification like... Lukas Stockner
01:14 PM Bug #49013 (Resolved): cephadm: Service definition causes some container startups to fail
In Bug #48870 I described the problem of failing containers during startup after I updated the cluster from 15.2.5 to... Gunther Heinrich
12:43 PM Bug #48870: cephadm: Several services in error status after upgrade to 15.2.8: unrecognized argum...
I think I found the underlying issue of the container startup problems which is unrelated to the unrecognized options... Gunther Heinrich
07:24 AM Bug #49000 (Duplicate): JSONDecodeError when wait_for_mgr_restart()
... Kefu Chai

01/25/2021

11:53 PM Bug #48993: cephadm: 'mgr stat' and/or 'pg dump' output truncated
The reproducer on that other bug doesn't work for 20.04.. different bug. However, I can reproduce it on ubuntu 18.04... Sage Weil
09:32 PM Bug #48993: cephadm: 'mgr stat' and/or 'pg dump' output truncated
Current theory: common denominator is the ubuntu 18.04 version of podman, which is currently:... Sage Weil
09:29 PM Bug #48993 (Resolved): cephadm: 'mgr stat' and/or 'pg dump' output truncated
Two symptoms:
- cephadm bootstrap 'ceph mgr dump' output is truncated (always at 180224 bytes, oddly)
exampl...
Sage Weil
06:14 PM Bug #48981: cephadm exporter: manager errors out with assertion error
relates to https://github.com/ceph/ceph/pull/39061 Sebastian Wagner
04:55 PM Bug #48981 (Resolved): cephadm exporter: manager errors out with assertion error
... Deepika Upadhyay
04:59 PM Bug #48982 (Resolved): cephadm: ubuntu_18_04: Error: error creating container storage: the contai...
mon dameon stops and dies: ... Deepika Upadhyay
04:45 PM Bug #48142: rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutua...
http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-25_12:19:15-rados-wip-deepika-testing-2021-01-25-1527-distro-bas... Deepika Upadhyay
04:29 PM Feature #48980: orch: add image properties to monitoring spec files
... Sebastian Wagner
04:28 PM Feature #48980 (Closed): orch: add image properties to monitoring spec files
add image properties to monitoring spec files
*There is a downside*: If you use this property, users will loose th...
Sebastian Wagner
04:28 PM Feature #48979 (New): bin/cephadm: add possibilty to query default monitoring images to cephadm
possibily add this to --help???? Sebastian Wagner
04:24 PM Feature #48978 (New): cephadm: show default container images in ceph orch status
cephadm: show default container images in ceph orch status Sebastian Wagner
11:07 AM Bug #48068: cephadm: Various properties like 'last_refresh' do not contain timezone
Backport to Octopus: https://github.com/ceph/ceph/pull/39059 Volker Theile
10:51 AM Documentation #48974 (Rejected): document repo_digest
> cephadm converts the image to the sha256 digest, in order to keep the cluster in a consistent state.
>
> Like, ...
Sebastian Wagner

01/22/2021

11:15 AM Bug #48870: cephadm: Several services in error status after upgrade to 15.2.8: unrecognized argum...
I did some analysis of the node-exporter on the osd node 3 to see what might be happening.
To me it looks as if th...
Gunther Heinrich

01/21/2021

05:53 PM Feature #47139 (Fix Under Review): Require a minimum version for podman/docker
Michael Fritch
03:50 PM Bug #48951: teuthology only collected coredumps, not daemon logs
This is due to the daemons not starting, these coredumps are all from podman:... Josh Durgin
12:28 PM Bug #48951 (Duplicate): teuthology only collected coredumps, not daemon logs
... Deepika Upadhyay
11:00 AM Bug #48594 (Pending Backport): cephadm: too many osd privileges for osd caps
Juan Miguel Olmo Martínez
10:58 AM Bug #48947 (Pending Backport): cephadm: fix rgw osd cap tag
Juan Miguel Olmo Martínez
10:55 AM Bug #48947 (Resolved): cephadm: fix rgw osd cap tag
The syntax is "allow rwx tag rgw ='. Juan Miguel Olmo Martínez
10:50 AM Bug #48157: test_cephadm.sh failure You have reached your pull rate limit. You may increase the l...
@Sebastian seeing this issue still in octopus:
/ceph/teuthology-archive/yuriw-2021-01-18_19:17:40-rados-wip-yuri2-te...
Deepika Upadhyay

01/20/2021

04:34 PM Bug #48939: Orchestrator removes mon daemon from wrong host when removing host from cluster
Quick update to confirm this behavior. I have been able to reproduce this on my personal homelab ceph cluster, also r... Daniël Vos
04:21 PM Bug #48939 (Can't reproduce): Orchestrator removes mon daemon from wrong host when removing host ...
It is as shocking as the subject describes.
To summarize:
Removing host mon1 nukes mon.mon3
Removing host mon3...
Daniël Vos
09:41 AM Bug #48925 (Fix Under Review): cephadm: iscsi missing mgr permissions
Juan Miguel Olmo Martínez
06:54 AM Bug #48933 (Can't reproduce): cephadm: EOFError: couldnt load message header, expected 9 bytes, g...
Found several uncatched exceptions, if I remember correctly, one host was rebooting or updating at that time.... Gunther Heinrich
12:10 AM Bug #48930 (Resolved): when removing the iscsi service, the gateway config object remains
When the first rbd-target-api daemon starts it creates a gateway.conf object. When the iscsi service is removed via "... Paul Cuzner

01/19/2021

10:50 PM Bug #45628 (Fix Under Review): cephadm qa: smoke should verify daemons are actually running
Sage Weil
07:34 PM Bug #46429: cephadm fails bootstrap with new Podman Versions 2.0.1 and 2.0.2
Current cephadm avoids combining --cap-add and --privileged, but older cephadm does not, and some distros still have ... Sage Weil
05:28 PM Bug #48826: cephadm: does not tolerate 15.2.4 upgrade state
if you happen to have a mgr failover from .8 to .5, users might also hit this bug. Sebastian Wagner
03:37 PM Bug #48826 (Won't Fix): cephadm: does not tolerate 15.2.4 upgrade state
I first tried to upgraded to .8, hit a different error (the ceph-volume --filter-batch thing), then switched to .5 in... Sage Weil
05:15 PM Bug #48715: docker-mirror: x509: certificate relies on legacy Common Name field, use SANs or temp...
https://github.com/ceph/ceph-sepia-secrets/pull/595... David Galloway
04:18 PM Support #48630: non-LVM OSD do not start after upgrade from 15.2.4 -> 15.2.7
I think you probably want to migrate to ceph-volume for now. Sebastian Wagner
04:12 PM Bug #47968 (Pending Backport): rook: 'ceph orch rm' throws type error
Sebastian Wagner
04:01 PM Feature #45766: cephadm: Removal: make sure, enough daemons joined the maps
this is related to the scheduler. not a simple addition to ok-to-stop Sebastian Wagner
04:00 PM Feature #44875 (New): mgr/rook: PlacementSpec to K8s POD scheduling conversion
was github pr 35542 Sebastian Wagner
03:55 PM Bug #45973: Adopted MDS daemons are removed by the orchestrator because they're orphans
still open Sebastian Wagner
03:53 PM Documentation #45564 (Duplicate): cephadm: document workaround for accessing the admin socket by ...
Sebastian Wagner
03:52 PM Bug #48510: CEPHADM_REFRESH_FAILED: detail item 0 not a [unicode] string
https://github.com/ceph/ceph/pull/38935/files#r560281346 Sebastian Wagner
03:48 PM Bug #48694 (Fix Under Review): ceph-volume: unrecognized arguments: --filter-for-batch
Sebastian Wagner
02:27 PM Bug #48925 (Resolved): cephadm: iscsi missing mgr permissions
Error after deploying iscsi daemons:... Juan Miguel Olmo Martínez
09:27 AM Bug #48924 (Resolved): cephadm: upgrade process failed to pull target image: not enough values to...
When trying to upgrade a cluster from version 15.2.7 to 15.2.8 the process fails after seeemingly switching over the ... Gunther Heinrich

01/18/2021

09:19 PM Bug #48916: "File system None does not exist in the map" in upgrade:octopus-x:parallel-master
Perhaps, you need to add... Neha Ojha
09:02 PM Bug #48916: "File system None does not exist in the map" in upgrade:octopus-x:parallel-master
This is testing `blogbench`(https://github.com/yuriw/ceph/tree/wip-yuriw-octopus-x-master/qa/suites/upgrade/octopus-x... Yuri Weinstein
08:57 PM Bug #48916 (Duplicate): "File system None does not exist in the map" in upgrade:octopus-x:paralle...
Run: https://pulpito.ceph.com/teuthology-2021-01-16_16:12:35-upgrade:octopus-x:parallel-master-distro-basic-smithi/
...
Yuri Weinstein
05:19 PM Bug #48598: "ceph orch daemon redeploy" fails with [errno 13] RADOS permission denied
bumping up priority on this one Neha Ojha
04:31 PM Bug #48715: docker-mirror: x509: certificate relies on legacy Common Name field, use SANs or temp...
https://pulpito.ceph.com/swagner-2021-01-18_11:41:36-rados:cephadm-wip-swagner-testing-2021-01-15-1448-distro-basic-s... Sebastian Wagner
04:30 PM Bug #48894: cephadm e2e: ceph device monitoring off: Error EINVAL
https://pulpito.ceph.com/swagner-2021-01-18_11:41:36-rados:cephadm-wip-swagner-testing-2021-01-15-1448-distro-basic-s... Sebastian Wagner
01:13 PM Bug #48870: cephadm: Several services in error status after upgrade to 15.2.8: unrecognized argum...
It seems that Ubuntu/Podman is not the cause for this issue because I am running into the same errors after updating ... Gunther Heinrich
07:11 AM Bug #48870: cephadm: Several services in error status after upgrade to 15.2.8: unrecognized argum...
I restarted all hosts but that didn't solve the problem unfortunately:... Gunther Heinrich
11:52 AM Bug #48891: cephadm does not set required dependencies in systemd
Sebastian Wagner wrote:
> Thank you very much for reporting this!
Oh. I'm sorry i've reported this as i did not f...
Michael Wodniok
11:13 AM Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory
This is no longer critical, as it only affects octopus at this point. Sebastian Wagner

01/16/2021

08:47 PM Bug #45279: cephadm bootstrap: monmaptool --create: error writing to '/tmp/monmap': (21) Is a dir...
I am still seeing the same problem:... Ian Yates
11:05 AM Bug #48826: cephadm: does not tolerate 15.2.4 upgrade state
If you're in an intermediate upgrade step and one MGR is already upgraded and the other one is still running with the... Sebastian Wagner
03:24 AM Bug #48826: cephadm: does not tolerate 15.2.4 upgrade state
We're not trying to support downgrade at all, right? Dan Mick
02:32 AM Bug #48826: cephadm: does not tolerate 15.2.4 upgrade state
What was the job config? Dan Mick
01:06 AM Bug #48891 (Duplicate): cephadm does not set required dependencies in systemd
Thank you very much for reporting this! Sebastian Wagner
01:02 AM Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory
it has nothing to do with the image. The issue went away with https://github.com/ceph/ceph/pull/35524 indicating some... Sebastian Wagner
12:54 AM Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory
I've been staring at that last failure for a few days, and I can't figure out what could possibly have caused it. Th... Dan Mick

01/15/2021

05:14 PM Bug #48142: rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutua...
... Sebastian Wagner
03:59 PM Bug #48594: cephadm: too many osd privileges for osd caps
https://github.com/ceph/ceph/pull/38910 also needs backporting Sebastian Wagner
03:57 PM Bug #48120 (Resolved): JSONDecodeErrors unclear during cephadm upgrade
Sebastian Wagner
03:56 PM Bug #48119 (Resolved): JSONDecodeErrors break OSD deploy
Sebastian Wagner
03:56 PM Bug #48118 (Resolved): JSONDecodeErrors break the server loop
Sebastian Wagner
03:56 PM Bug #48205 (Resolved): cephadm: last local ceph image tag can be null
Sebastian Wagner
03:56 PM Bug #48223 (Resolved): host key might not exist during host refresh
Sebastian Wagner
03:56 PM Bug #47684 (Resolved): cephadm: auth get failed: failed to find osd.27 in keyring retval: -2
Sebastian Wagner
03:55 PM Documentation #47651 (Resolved): mgr/cephadm: document custom configuration templates
Sebastian Wagner
02:56 PM Bug #45628: cephadm qa: smoke should verify daemons are actually running
latest output looks like ... Sebastian Wagner
11:53 AM Bug #48870: cephadm: Several services in error status after upgrade to 15.2.8: unrecognized argum...
Ok, I see two distinct issues here:
1. --filter-for-batch which is addressed in https://github.com/ceph/ceph/pull...
Sebastian Wagner
10:54 AM Bug #48870 (In Progress): cephadm: Several services in error status after upgrade to 15.2.8: unre...
Sebastian Wagner
10:54 AM Bug #48870: cephadm: Several services in error status after upgrade to 15.2.8: unrecognized argum...
possible solution: https://github.com/ceph/ceph/pull/38926 Sebastian Wagner
11:20 AM Bug #48894 (Can't reproduce): cephadm e2e: ceph device monitoring off: Error EINVAL
https://pulpito.ceph.com/swagner-2021-01-15_09:42:49-rados:cephadm-wip-swagner-testing-2021-01-14-1551-distro-basic-s... Sebastian Wagner
11:13 AM Bug #48715: docker-mirror: x509: certificate relies on legacy Common Name field, use SANs or temp...
unfortunately, it's still there: https://pulpito.ceph.com/swagner-2021-01-15_09:42:49-rados:cephadm-wip-swagner-testi... Sebastian Wagner
05:55 AM Bug #48891 (Duplicate): cephadm does not set required dependencies in systemd
cephadm does not set the dependency list correctly. This leads to race conditions. Example:
In our environment we ...
Michael Wodniok
 

Also available in: Atom