General

Profile

Sage Weil's activity

From 01/28/2020 to 02/26/2020

02/26/2020

11:13 PM Dashboard Feature #44301 (Resolved): mgr/dashboard: Enhance user create CLI command to force password change
Sage Weil
11:13 PM mgr Cleanup #44306 (Resolved): mgr/modules: outsource certificate creation to `mgr_util.py`
Sage Weil
08:51 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
... Sage Weil
08:50 PM Orchestrator Bug #44313: ceph-volume prepare is not idempotent and may get called twice
similar failure, this time a 'daemon mon add' dup:
/a/sage-2020-02-26_08:10:43-rados-wip-sage2-testing-2020-02-25-...
Sage Weil
08:27 PM Orchestrator Bug #44313: ceph-volume prepare is not idempotent and may get called twice
One possible fix would be to make ceph-volume itself idempotent, so that calling prepare on an already-prepared devic... Sage Weil
08:26 PM Orchestrator Bug #44313 (Resolved): ceph-volume prepare is not idempotent and may get called twice
symptom is a failure like so:... Sage Weil
08:48 PM RADOS Bug #44314 (Resolved): osd-backfill-stats.sh failing intermittently in TEST_backfill_sizeup_out()...
... Sage Weil
08:24 PM Orchestrator Bug #44312 (Duplicate): ceph-volume prepare is not idempotent and may get called twice
Sage Weil
07:47 PM RADOS Bug #43914 (Fix Under Review): nautilus: ceph tell command times out
Sage Weil
06:48 PM RADOS Bug #43914: nautilus: ceph tell command times out
okay yeah, it's because the command wq uses osd_lock... Sage Weil
06:41 PM RADOS Bug #43914: nautilus: ceph tell command times out
so, this was fixed in nautilus, in the sense that https://github.com/ceph/ceph/pull/27696 went into nautilus.
Sage Weil
06:37 PM RADOS Bug #43914: nautilus: ceph tell command times out
The thread (or lock?) is busy with... Sage Weil
07:33 PM RADOS Bug #42328: osd/PrimaryLogPG.cc: 3962: ceph_abort_msg("out of order op")
follow-up fix: https://github.com/ceph/ceph/pull/33559 (typo in original commit) Sage Weil
06:19 PM Ceph Bug #44250 (Resolved): Error: unable to find a name and tag match for ceph-ci/daemon-base in repo...
Sage Weil
06:02 PM mgr Bug #44299 (Need More Info): nautilus: mgr osd communication stalled in mixed cluster
I can't make heads or tails of this. I think we need to reproduce with debug_ms=20 Sage Weil
12:09 AM RADOS Bug #44297 (Fix Under Review): mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymo...
Sage Weil
12:02 AM RADOS Bug #44297: mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous connection")
The command is passed from a nautilus monitor:... Sage Weil

02/25/2020

11:51 PM RADOS Bug #44275 (Resolved): NameError: name 'retval' is not defined
Sage Weil
11:50 PM RADOS Bug #44248 (Pending Backport): Receiving RemoteBackfillReserved in WaitLocalBackfillReserved can ...
Sage Weil
11:43 PM RADOS Bug #44297 (Resolved): mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous conne...
on nautilus->octopus/master upgrade... Sage Weil
11:40 PM Ceph Bug #44078: centos 8.1: ceph-mon: assertion "z >= signedspan::zero()" failed in ceph::to_timespan(
centos 8.1 again
/a/sage-2020-02-25_15:51:04-rados-wip-sage2-testing-2020-02-25-0704-distro-basic-smithi/4801785
Sage Weil
11:39 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
... Sage Weil
01:18 PM mgr Bug #44271 (Resolved): dashboard: unhandled exception ('pwdUpdateRequired',)
Sage Weil

02/24/2020

07:42 PM RADOS Bug #44076 (Pending Backport): mon: update + monmap update triggers spawn loop
Sage Weil
07:29 PM Orchestrator Bug #41746 (Resolved): mgr/rook: `ceph orchestrator device ls` doesn't set `available`
this is working now AFAICS Sage Weil
07:29 PM Orchestrator Bug #43838: cephadm: Forcefully Remove Services (unresponsive hosts)
One option is to have them 'ceph orch host rm $hostname'... Sage Weil
07:27 PM Orchestrator Bug #44121 (Fix Under Review): calling cephadm shell again looses bash history
Sage Weil
07:19 PM Orchestrator Bug #44270 (Triaged): Under certain circumstances, "ceph orch apply" returns success even when no...
i bet the problem is that the drive inventory isn't populated yet immediately after bootstrap. Sage Weil
07:17 PM Orchestrator Bug #44273 (Need More Info): Getting "stray daemon osd.3 on host admin not managed by cephadm" on...
This should have been fixed by 607263224c26... can you reproduce this with debug_mgr = 20 and attach a log? Sage Weil
07:13 PM Orchestrator Feature #43867 (Resolved): cephadm: progress item for upgrade
Sage Weil
03:44 PM mgr Bug #44271 (Fix Under Review): dashboard: unhandled exception ('pwdUpdateRequired',)
Sage Weil
03:36 PM mgr Bug #44271 (Resolved): dashboard: unhandled exception ('pwdUpdateRequired',)
... Sage Weil
03:09 PM Ceph Bug #44078: centos 8.1: ceph-mon: assertion "z >= signedspan::zero()" failed in ceph::to_timespan(
I hit this on centos 8.1... Sage Weil

02/23/2020

09:05 PM RADOS Bug #43861: ceph_test_rados_watch_notify hang
... Sage Weil
09:01 PM Dashboard Bug #44256 (Closed): Test failure: test_create_user_with_pwd_expiration_date (tasks.mgr.dashboard...
... Sage Weil
08:12 PM Orchestrator Feature #44255 (New): cephadm: scheduler should consider other daemons on each node
When choosing a home for a daemon, we should prefer nodes that have fewer daemons, and/or fewer daemons of the same t... Sage Weil
08:11 PM Orchestrator Bug #44254 (Resolved): scheduler should prefer existing daemon locations
If we are placing N daemons, then we should select nodes that already have daemons for the service. (Otherwise, an a... Sage Weil
08:10 PM Orchestrator Bug #44253 (Resolved): _apply_service should move services, not just expand/contract
if placement is based on, e.g., labels, then moving a label should cause us to move services too (add first, then rem... Sage Weil
07:17 PM Orchestrator Bug #44252 (Resolved): cephadm: mgr,mds scale-down should prefer standby daemons
There are three types of daemons:
1. active daemons
2. standby daemons
3. unknown daemons that are (not yet) par...
Sage Weil
02:37 PM Ceph Bug #44242 (Resolved): shaman container build fails
Sage Weil
02:36 PM Orchestrator Bug #44170 (Duplicate): Teuthology is testing unrelated container images
this was because the wip-swagner-testing branch was reused and because #44242 had not been fixed Sage Weil

02/22/2020

09:33 PM Ceph Bug #44250: Error: unable to find a name and tag match for ceph-ci/daemon-base in repotags: no su...
these failed too: https://shaman.ceph.com/repos/ceph/wip-sage2-testing-2020-02-22-1230/a4ce2efef9f5d4ebcff1f0a05bb662... Sage Weil
08:46 PM Ceph Bug #44250 (Resolved): Error: unable to find a name and tag match for ceph-ci/daemon-base in repo...
... Sage Weil
09:32 PM mgr Bug #44245: nautilus: mgr: connection halt
My guess is we are leaking a message ref somewhere... Sage Weil
09:27 PM mgr Bug #44245: nautilus: mgr: connection halt
lots of connections are busy looping, also waiting on the same throttle... Sage Weil
09:24 PM mgr Bug #44245: nautilus: mgr: connection halt
with debug_ms=20 we see, on the mgr,... Sage Weil
06:06 PM mgr Fix #44002 (Resolved): mgr/devicehealth: set default monitoring to 'on'
Sage Weil
06:05 PM Ceph Bug #44196 (Pending Backport): selinux setsched denials for 'fn_anonymous'
Sage Weil
02:16 PM Orchestrator Feature #43675 (Resolved): workflow for using a signed dashboard cert
Sage Weil
03:03 AM Orchestrator Bug #43949 (Resolved): mgr/cephadm: ceph fs volume create: TypeError: %d format: a number is requ...
Sage Weil
03:02 AM Orchestrator Bug #44119 (Resolved): installing cephamd on bionic is painful:
Sage Weil
03:01 AM Orchestrator Bug #44121: calling cephadm shell again looses bash history
we could bind the root .bash_history file to something like /var/lib/ceph/$fsid/.bash_history ? Sage Weil
03:01 AM Orchestrator Bug #44209 (Resolved): qa/workunits/cephadm/test_cephadm.sh: prometheus:latest: Invalid JSON in -
Sage Weil
03:01 AM Orchestrator Bug #44003 (Resolved): cephadm: multiple mgrs scheduled on same host
Sage Weil
01:14 AM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
/a/sage-2020-02-21_21:08:33-rados-wip-sage3-testing-2020-02-21-1218-distro-basic-smithi/4788714... Sage Weil
12:41 AM Ceph Bug #44242: shaman container build fails
motherfucker
wip-sage2-testing-2020-02-21-1459 51.68.95.182 https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x...
Sage Weil
12:30 AM Ceph Bug #44242: shaman container build fails
AAAHAHHHHHAHHHHHH
wip-sage-testing-2020-02-21-1528
51.68.95.182
https://jenkins.ceph.com/job/ceph-dev-new-build/A...
Sage Weil
12:00 AM Ceph Bug #44242: shaman container build fails
wip-sage2-testing-2020-02-21-1459 51.68.84.74 https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABL... Sage Weil

02/21/2020

11:03 PM CephFS Bug #44244 (Fix Under Review): pybind/mgr/volumes: "handle_command module 'volumes' command handl...
Sage Weil
10:12 PM Ceph Bug #44242: shaman container build fails
wip-sage4-testing-2020-02-21-1319 .... 51.68.93.161 ... https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,A... Sage Weil
04:42 PM Ceph Bug #44242: shaman container build fails
Recent instances:
- wip-sage-testing-2020-02-21-0633 ... 51.91.144.35 ... https://jenkins.ceph.com/job/ceph-dev-n...
Sage Weil
04:41 PM Ceph Bug #44242 (Resolved): shaman container build fails
The failure looks like so:... Sage Weil
09:03 PM RADOS Bug #44243 (Can't reproduce): memstore make check test fails
... Sage Weil
08:10 PM mgr Bug #44241 (Resolved): mgr: deadlock w/ register/unregister_client
Sage Weil
03:01 PM mgr Bug #44241 (Fix Under Review): mgr: deadlock w/ register/unregister_client
Sage Weil
02:58 PM mgr Bug #44241 (Resolved): mgr: deadlock w/ register/unregister_client
... Sage Weil
01:21 PM RADOS Bug #42347: nautilus assert during osd shutdown: FAILED ceph_assert((sharded_in_flight_list.back(...
Bastian Mäuser wrote:
> This is still an issue on 14.2.6 (at least the one shipped with proxmox)
It will appear i...
Sage Weil
02:02 AM Dashboard Bug #44201 (Resolved): mgr/dashboard: Teuthology and dashboard backend API test failure: setUpCla...
Sage Weil

02/20/2020

07:32 PM Orchestrator Bug #44231 (Resolved): cephadm: cannot capture core files
At least, I can't figure it out.
On my test box, i set kernel.core_pattern to both a valid hsot and container path...
Sage Weil
07:02 PM RADOS Bug #39525: lz4 compressor corrupts data when buffers are unaligned
ok, the first crash isn't becuase we just got bad data.. it's because we just read bad data off of disk. see:... Sage Weil
04:09 PM RADOS Bug #39525: lz4 compressor corrupts data when buffers are unaligned
Notes from CERN incident:
- identical corruption, different OSDmaps on different OSDs:...
Sage Weil
05:40 PM RADOS Bug #44229 (New): monclient: _check_auth_rotating possible clock skew, rotating keys expired way ...
seems to affect cephadm bootstrap tests
first, the error message doesn't make sense, since the bound 2020-02-20T16...
Sage Weil

02/19/2020

11:42 PM RADOS Bug #44076 (Fix Under Review): mon: update + monmap update triggers spawn loop
Sage Weil
10:47 PM Orchestrator Bug #44169 (Resolved): informative exception eaten
Sage Weil
10:46 PM mgr Bug #44186 (Pending Backport): Module 'pg_autoscaler' has failed: division by zero
Sage Weil
10:45 PM RADOS Bug #44157 (Resolved): cli throws bad exceptoin on control-c
Sage Weil
10:38 PM Orchestrator Feature #43670 (Resolved): teuthology: Add new upgrade/downgrade process
Sage Weil
10:37 PM Orchestrator Feature #43867 (Fix Under Review): cephadm: progress item for upgrade
Sage Weil
10:36 PM Orchestrator Feature #44031 (Fix Under Review): cephadm: Also cache `device ls`.
Sage Weil
10:35 PM Orchestrator Feature #43836 (Fix Under Review): cephadm adopt: also adopt Prometheus and Grafana daemons from ...
Sage Weil
10:35 PM Orchestrator Feature #43695 (Fix Under Review): cephadm: alertmanager
Sage Weil
10:35 PM Orchestrator Feature #43940 (Resolved): orchestrator mgr add and rm
Sage Weil
10:35 PM Orchestrator Feature #43685 (Resolved): host prepare
Sage Weil
07:04 PM mgr Bug #44210 (Pending Backport): Devicehealth scrape fails when smartctl return code is non-zero
Sage Weil
07:00 PM Orchestrator Bug #44165 (Resolved): test_load_data fails
Sage Weil
06:59 PM Orchestrator Bug #44188 (Resolved): Module 'cephadm' has failed: dictionary changed size during iteration
Sage Weil
06:59 PM CephFS Bug #44176 (Resolved): qa: "Error EINVAL: 'Module' object has no attribute 'remove_mds'"
Sage Weil
12:33 PM Orchestrator Bug #44175 (Resolved): cephadm: adopt does not work with filestore OSDs
Sage Weil

02/18/2020

11:57 PM mgr Bug #44186 (Fix Under Review): Module 'pg_autoscaler' has failed: division by zero
Sage Weil
07:51 PM mgr Bug #44186 (Resolved): Module 'pg_autoscaler' has failed: division by zero
... Sage Weil
10:25 PM Orchestrator Bug #44188 (Resolved): Module 'cephadm' has failed: dictionary changed size during iteration
... Sage Weil
08:32 PM Orchestrator Bug #44175 (Fix Under Review): cephadm: adopt does not work with filestore OSDs
Sage Weil
07:55 PM RADOS Bug #43903 (Pending Backport): osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
Sage Weil
07:52 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
... Sage Weil
02:30 PM Orchestrator Bug #44168 (Resolved): qa/tasks/cephadm in ceph_bootstrap: AttributeError: 'NoneType' object has ...
Sage Weil

02/17/2020

11:32 PM Orchestrator Bug #44175 (Resolved): cephadm: adopt does not work with filestore OSDs
it tries to *copy* the data directory, bad bad bad Sage Weil
05:40 PM Orchestrator Feature #43685 (Fix Under Review): host prepare
Sage Weil
02:13 PM Orchestrator Bug #44169 (Resolved): informative exception eaten
... Sage Weil

02/16/2020

03:30 PM RADOS Bug #44156 (Resolved): RenewLease sent to pre-octopus osds during upgrade
Sage Weil

02/15/2020

03:11 PM RADOS Bug #44157 (Fix Under Review): cli throws bad exceptoin on control-c
Sage Weil
02:38 PM Ceph Bug #43737 (Resolved): install-dep.sh fails in Centos-7.6
Sage Weil
02:37 PM RADOS Bug #44041 (Resolved): osd: MLease in stray state -> Crashed
Sage Weil
02:37 PM RADOS Bug #42328 (Resolved): osd/PrimaryLogPG.cc: 3962: ceph_abort_msg("out of order op")
Sage Weil
02:36 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
/a/sage-2020-02-15_04:59:38-rados-wip-sage3-testing-2020-02-14-1951-distro-basic-smithi/4765960 Sage Weil

02/14/2020

05:35 PM RADOS Bug #44157 (Resolved): cli throws bad exceptoin on control-c
... Sage Weil

02/13/2020

01:09 AM Orchestrator Bug #44077 (Resolved): grafana container doesn't start on 18.04
Sage Weil

02/12/2020

12:18 AM RADOS Bug #44062 (Triaged): LibRadosWatchNotify.WatchNotify failure
/a/sage-2020-02-11_20:49:48-rados-wip-sage-testing-2020-02-11-1121-distro-basic-smithi/4755080 Sage Weil

02/11/2020

08:08 PM devops Bug #44065 (Fix Under Review): ceph.spec.in wrong cephadm dependency
Sage Weil
04:43 PM Orchestrator Bug #44018 (Resolved): cephadm: down host kills serve() thread
Sage Weil
02:33 PM Orchestrator Bug #44077 (Resolved): grafana container doesn't start on 18.04
... Sage Weil
02:28 PM bluestore Bug #43904 (Resolved): bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upg...
Sage Weil
02:28 PM RADOS Bug #44067 (Resolved): cephtool/test.sh test fails to scrub all pools
Sage Weil
02:27 PM RADOS Bug #44076 (Resolved): mon: update + monmap update triggers spawn loop
- upgrade monitors from mimic to octopus
- quorum of 2/3 monitors
- enable msgr2
then
- third monitor probes...
Sage Weil
02:57 AM RADOS Bug #44050 (Resolved): mon tell command args don't work
Sage Weil

02/10/2020

11:14 PM Orchestrator Bug #43913 (Resolved): test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): Assertio...
Sage Weil
11:14 PM RADOS Bug #43889 (Pending Backport): expected MON_CLOCK_SKEW but got none
Sage Weil
02:41 PM RADOS Bug #43889 (Fix Under Review): expected MON_CLOCK_SKEW but got none
Sage Weil
06:17 PM Orchestrator Bug #43883 (Resolved): cephadm: Found left-over process 15516 (podman) in control group while sta...
Sage Weil
05:27 PM RADOS Bug #44067 (Fix Under Review): cephtool/test.sh test fails to scrub all pools
Sage Weil
05:14 PM RADOS Bug #44067 (Resolved): cephtool/test.sh test fails to scrub all pools
... Sage Weil
03:08 PM Orchestrator Feature #44005 (Resolved): cephadm: associate addrs to hosts
Sage Weil
02:55 PM RADOS Bug #44052 (Pending Backport): ceph -s does not show >32bit pg states
Sage Weil
02:42 PM RADOS Bug #44062 (Resolved): LibRadosWatchNotify.WatchNotify failure
... Sage Weil
02:37 PM RADOS Bug #43048: nautilus: upgrade/mimic-x/stress-split: failed to recover before timeout expired
/a/sage-2020-02-09_21:18:03-rados-wip-sage2-testing-2020-02-09-1152-distro-basic-smithi/4749175... Sage Weil

02/09/2020

09:34 PM Orchestrator Feature #44055 (Closed): cephadm: make 'ls' faster
For both podman and docker, 'ps' tells you the image name but not its hash.
With podman, you can do:...
Sage Weil
05:40 PM RADOS Bug #43889 (In Progress): expected MON_CLOCK_SKEW but got none
Sage Weil
12:45 AM RADOS Bug #42328 (Fix Under Review): osd/PrimaryLogPG.cc: 3962: ceph_abort_msg("out of order op")
Sage Weil
12:25 AM RADOS Bug #43903 (In Progress): osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
Sage Weil
12:24 AM Orchestrator Bug #43703 (Resolved): selinux vs logrotate
I'm calling this one "fixed", even though for el 8.0 and 8.1 (pre-z-stream) the error is still there.
https://gith...
Sage Weil
12:22 AM Orchestrator Bug #43883 (Fix Under Review): cephadm: Found left-over process 15516 (podman) in control group w...
Sage Weil

02/08/2020

07:33 PM Orchestrator Bug #44018 (Fix Under Review): cephadm: down host kills serve() thread
Sage Weil
06:11 PM RADOS Bug #44052 (Fix Under Review): ceph -s does not show >32bit pg states
Sage Weil
06:07 PM RADOS Bug #44052 (Resolved): ceph -s does not show >32bit pg states
ceph -s does not show newer pg states, like repair_failed Sage Weil
05:15 PM mgr Bug #44051 (New): crash: archive-all slow with lots of crashes
I have ~500 crashes and it is taking multiple minutes to do 'ceph crash archive-all' Sage Weil
03:26 PM RADOS Bug #44050 (Fix Under Review): mon tell command args don't work
Sage Weil
02:37 PM RADOS Bug #44050: mon tell command args don't work
'ceph tell mon.a help' works, but '-h' does not. Sage Weil
02:07 PM RADOS Bug #44050 (Resolved): mon tell command args don't work
Also, 'ceph tell mon.a force-sync --yes-i-really-mean-it' seems to be broken:... Sage Weil
02:11 PM RADOS Feature #42638 (Resolved): Allow specifying pg_autoscale_mode when creating a new pool
Sage Weil
02:08 PM mgr Bug #43313 (Pending Backport): telemetry module can crash on entity name with multiple '.' separa...
Sage Weil
01:55 PM Orchestrator Bug #43913: test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): AssertionError: 22 ...
/a/sage-2020-02-07_23:51:30-rados-wip-sage2-testing-2020-02-07-1439-distro-basic-smithi/4742679 Sage Weil
01:53 PM RADOS Bug #43889: expected MON_CLOCK_SKEW but got none
/a/sage-2020-02-07_23:51:30-rados-wip-sage2-testing-2020-02-07-1439-distro-basic-smithi/4742672 Sage Weil
01:47 PM Dashboard Bug #44049 (Can't reproduce): mgr/dashboard: test_perf_counters_mgr_get failed
... Sage Weil
01:34 PM RADOS Bug #44024 (Resolved): change in utime_t rendering ('T' separator) conflicts with cache tiering h...
Sage Weil

02/07/2020

07:54 PM mgr Bug #43946 (Resolved): mimic: EINVAL on 'osd dump'
Sage Weil
05:50 PM mgr Bug #43946 (Fix Under Review): mimic: EINVAL on 'osd dump'
Sage Weil
01:55 PM mgr Bug #43946: mimic: EINVAL on 'osd dump'
/a/sage-2020-02-06_19:01:25-rados-wip-sage2-testing-2020-02-05-1649-distro-basic-smithi/4738937
Sage Weil
07:38 PM mgr Bug #43963 (Pending Backport): telemetry: crash when posting
Sage Weil
07:27 PM Orchestrator Bug #44029 (Resolved): orchestrator cli: confusing rgw param order
Sage Weil
05:10 PM Orchestrator Feature #43671 (Resolved): cephadm: {add,remove}-repo
Sage Weil
04:34 PM RADOS Bug #44041 (Fix Under Review): osd: MLease in stray state -> Crashed
Sage Weil
04:03 PM RADOS Bug #44041 (Resolved): osd: MLease in stray state -> Crashed
... Sage Weil

02/06/2020

11:53 PM RADOS Bug #44024 (Fix Under Review): change in utime_t rendering ('T' separator) conflicts with cache t...
Sage Weil
11:26 PM RADOS Bug #44024 (Resolved): change in utime_t rendering ('T' separator) conflicts with cache tiering h...
crash like... Sage Weil
10:05 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
... Sage Weil
08:42 PM Orchestrator Feature #43973 (Resolved): There is a need for a wrapper to fetch the journald logs
Sage Weil
03:02 PM Orchestrator Bug #44018 (Resolved): cephadm: down host kills serve() thread
... Sage Weil
03:39 AM Orchestrator Feature #44005 (Fix Under Review): cephadm: associate addrs to hosts
Sage Weil
03:38 AM Orchestrator Feature #44005 (Resolved): cephadm: associate addrs to hosts
Sage Weil
03:38 AM Orchestrator Feature #43940 (Fix Under Review): orchestrator mgr add and rm
Sage Weil

02/05/2020

11:02 PM Orchestrator Bug #44003 (Resolved): cephadm: multiple mgrs scheduled on same host
By default the mgr binds to a bunch of random ports. We shouldn't put more than one on a single host (unless forced ... Sage Weil
10:09 PM Orchestrator Bug #43945 (Resolved): Test failure: test_osd_create (tasks.mgr.test_orchestrator_cli.TestOrchest...
Sage Weil
06:58 PM mgr Bug #43946: mimic: EINVAL on 'osd dump'
/a/sage-2020-02-05_03:10:48-rados-wip-sage2-testing-2020-02-04-1448-distro-basic-smithi/4733167 Sage Weil
05:39 PM RADOS Bug #42347 (Won't Fix): nautilus assert during osd shutdown: FAILED ceph_assert((sharded_in_fligh...
we've backported the osd fast shutdown ( https://github.com/ceph/ceph/pull/32743 ), so this will effectively go away ... Sage Weil
05:00 PM Messengers Bug #43808: segv in MonClient::handle_auth_done
/a/sage-2020-02-05_03:10:48-rados-wip-sage2-testing-2020-02-04-1448-distro-basic-smithi/4733235
ceph-osd.2.log
...
Sage Weil
02:11 PM Orchestrator Feature #43940 (In Progress): orchestrator mgr add and rm
Sage Weil
02:03 PM Ceph Wiki edit: CDM_05-FEB-2020 (#7)
Sage Weil
01:36 PM mgr Bug #43642 (Pending Backport): "ceph telemetry show" shows error: AttributeError: 'NoneType' obje...
Sage Weil

02/04/2020

07:00 PM mgr Bug #43963 (Fix Under Review): telemetry: crash when posting
Sage Weil
06:55 PM mgr Bug #43963 (Resolved): telemetry: crash when posting
... Sage Weil
04:39 PM Ceph Bug #40647 (Can't reproduce): OP_CREATE issued to nautilus OSD
I think this got fixed.. we haven't seen it in 6 months. Sage Weil
04:37 PM mgr Bug #39955 (Resolved): After upgrade to Nautilus 14.2.1 mon DB is growing too fast when state of ...
I think this is okay now? Sage Weil
03:37 PM Ceph Wiki edit: Planning (#119)
Sage Weil
03:14 PM Orchestrator Feature #43671 (Fix Under Review): cephadm: {add,remove}-repo
Sage Weil
04:00 AM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
/a/sage-2020-02-04_03:27:12-rados-wip-sage-testing-2020-02-03-1650-distro-basic-smithi/4730285
centos 7.6. others...
Sage Weil
03:28 AM RADOS Bug #43813 (Pending Backport): objecter doesn't send osd_op
Sage Weil

02/03/2020

10:53 PM Ceph Wiki edit: CDM_05-FEB-2020 (#2)
Sage Weil
05:01 PM Orchestrator Feature #43671 (In Progress): cephadm: {add,remove}-repo
Sage Weil

02/02/2020

03:22 PM mgr Bug #43946 (Triaged): mimic: EINVAL on 'osd dump'
this is actually on the mimic version. shortly after cluster creation, but hte mgr was active and should have provid... Sage Weil
03:18 PM mgr Bug #43946 (Resolved): mimic: EINVAL on 'osd dump'
... Sage Weil
03:17 PM Orchestrator Bug #43945: Test failure: test_osd_create (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli)
/a/sage-2020-02-01_20:59:41-rados-wip-sage-testing-2020-02-01-1055-distro-basic-smithi/4725047 Sage Weil

02/01/2020

04:53 PM RADOS Bug #43861: ceph_test_rados_watch_notify hang
same?
/a/sage-2020-02-01_03:27:35-rados-wip-sage-testing-2020-01-31-1746-distro-basic-smithi/4723146
ceph_test_wa...
Sage Weil
04:50 PM Orchestrator Bug #43945 (Resolved): Test failure: test_osd_create (tasks.mgr.test_orchestrator_cli.TestOrchest...
... Sage Weil

01/31/2020

11:31 PM RADOS Bug #43795 (Pending Backport): Ceph tools utilizing "global_[pre_]init" no longer process "early"...
Sage Weil
11:30 PM mgr Bug #43886 (Pending Backport): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
Sage Weil
10:55 PM mgr Bug #38614: Disable SSLv3 on ceph-mgr?
I tried this against current master/octopus (containerized):... Sage Weil
10:34 PM RADOS Bug #43365 (Closed): Nautilus: Random mon crashes in failed assertion at ceph::time_detail::signe...
FWIW teh two clusters reporting this crash via telemetry are both ubuntu 18.04
closing this as not a ceph issue; l...
Sage Weil
07:49 PM bluestore Bug #43904 (Fix Under Review): bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given...
Sage Weil
06:02 PM RADOS Bug #43813 (Fix Under Review): objecter doesn't send osd_op
Sage Weil
03:50 AM RADOS Bug #43813 (In Progress): objecter doesn't send osd_op
Sage Weil
03:46 AM RADOS Bug #43813: objecter doesn't send osd_op
/a/sage-2020-01-30_22:27:29-rados-wip-sage-testing-2020-01-30-1230-distro-basic-smithi/4719487... Sage Weil
05:10 PM Orchestrator Feature #43682 (Resolved): fix mds groupings
Sage Weil
05:09 PM Orchestrator Feature #43679 (Resolved): run host-check periodically
Sage Weil
05:04 PM Orchestrator Feature #43940 (Resolved): orchestrator mgr add and rm
these are useful, and not having them is incongruous with the other services. Sage Weil
03:56 AM RADOS Bug #42977 (Resolved): mon/Elector.cc: FAILED ceph_assert(m->epoch == get_epoch())
Sage Weil
03:55 AM CephFS Bug #40867 (In Progress): mgr: failover during in qa testing causes unresponsive client warnings
Another one:
/a/sage-2020-01-30_22:27:29-rados-wip-sage-testing-2020-01-30-1230-distro-basic-smithi/4719492
Sage Weil

01/30/2020

06:46 PM bluestore Bug #43904: bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upgrade)
I think the root problem is that the alloc unit was 64k (bluefs_shared_alloc_size=64k) but there was a more granular ... Sage Weil
03:02 PM bluestore Bug #43904: bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upgrade)
... Sage Weil
02:56 PM bluestore Bug #43904 (Resolved): bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upg...
... Sage Weil
06:40 PM mgr Bug #43886 (Fix Under Review): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
Sage Weil
04:43 PM RADOS Bug #43602 (Fix Under Review): Core dumps not collected in standalone tests for distros using sys...
Sage Weil
04:43 PM RADOS Bug #43602 (Resolved): Core dumps not collected in standalone tests for distros using systemd-cor...
Sage Weil
04:41 PM RADOS Bug #43889: expected MON_CLOCK_SKEW but got none
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718332
Sage Weil
04:16 PM RADOS Bug #43889: expected MON_CLOCK_SKEW but got none
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718133 Sage Weil
04:40 PM RADOS Bug #43915 (New): leaked Session (alloc from OSD::ms_handle_authentication)
... Sage Weil
04:37 PM RADOS Bug #43914 (Need More Info): nautilus: ceph tell command times out
see https://github.com/ceph/ceph/pull/32989 Sage Weil
04:35 PM RADOS Bug #43914 (Resolved): nautilus: ceph tell command times out
... Sage Weil
04:17 PM RADOS Bug #43864: osd/repro_long_log.sh failure
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718221 Sage Weil
04:17 PM RADOS Bug #43885: failed to reach quorum size 9 before timeout expired
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718154
description: rados/...
Sage Weil
04:16 PM Orchestrator Bug #43913: test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): AssertionError: 22 ...
https://github.com/ceph/ceph/pull/32881 was part of the test branch... that's maybe the cause? Sage Weil
04:15 PM Orchestrator Bug #43913 (Resolved): test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): Assertio...
... Sage Weil
04:12 PM Orchestrator Bug #43657: test_host_ls test fails
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4717990 Sage Weil
04:12 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4717957 Sage Weil
04:11 PM Dashboard Bug #43912 (Resolved): FAIL: test_safe_to_destroy (tasks.mgr.dashboard.test_osd.OsdTest)
... Sage Weil
03:40 PM Orchestrator Feature #43911 (Resolved): test cephadm rgw deployment
Sage Weil
03:34 PM ceph-volume Bug #43899: cephadm: Remove the clutch between Teuthology and ceph-volume
workaround that we'll need to revert/clean up: https://github.com/ceph/ceph/pull/32972/commits/614c0eb77eb44dd7416561... Sage Weil
03:22 PM bluestore Bug #42823 (Duplicate): crash in BlueStore::Onode destructor
Going to optimistically assume this is a dup of #43147 / #43131
Sage Weil
03:18 PM bluestore Bug #43068 (Need More Info): on disk size (81292) does not match object info size (81237)
was this bluestore or filestore? Sage Weil
03:17 PM bluestore Bug #43183 (Can't reproduce): Segmentation fault in tcmalloc when create osd
Sage Weil
03:11 PM bluestore Bug #43370: OSD crash in function bluefs::_flush_range with ceph_abort_msg "bluefs enospc"
crash sig (for the record):... Sage Weil
03:10 PM bluestore Bug #42010 (Can't reproduce): segv in BlueStore::OnodeSpace::lookup during deletions
Sage Weil
02:51 PM RADOS Bug #43903: osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
the second time,... Sage Weil
02:50 PM RADOS Bug #43903: osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
if i start the osd manually, i can reproduce the same crash:... Sage Weil
02:48 PM RADOS Bug #43903 (Resolved): osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
... Sage Weil
01:18 PM Messengers Bug #43808: segv in MonClient::handle_auth_done
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718136
ceph-osd.4.log
d...
Sage Weil
04:11 AM RADOS Bug #43892 (Pending Backport): mon/PaxosService.cc: 188: FAILED ceph_assert(have_pending) during ...
Sage Weil

01/29/2020

11:48 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
ok, a similar problem:... Sage Weil
05:16 PM Orchestrator Bug #43883 (Need More Info): cephadm: Found left-over process 15516 (podman) in control group whi...
Sage Weil
05:16 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
... Sage Weil
01:09 PM Orchestrator Bug #43883 (Resolved): cephadm: Found left-over process 15516 (podman) in control group while sta...
... Sage Weil
11:18 PM RADOS Bug #43892 (Fix Under Review): mon/PaxosService.cc: 188: FAILED ceph_assert(have_pending) during ...
Sage Weil
11:15 PM RADOS Bug #43892 (Resolved): mon/PaxosService.cc: 188: FAILED ceph_assert(have_pending) during n->o upg...
... Sage Weil
08:54 PM mgr Bug #43886 (In Progress): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
Sage Weil
08:30 PM mgr Bug #43886: FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
pass insights report has... Sage Weil
02:54 PM mgr Bug #43886: FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
about a 50% failure rate. os independent.
http://pulpito.ceph.com/sage-2020-01-29_13:33:59-rados-wip-sage-testing...
Sage Weil
01:33 PM mgr Bug #43886 (Resolved): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
... Sage Weil
07:39 PM RADOS Bug #42977: mon/Elector.cc: FAILED ceph_assert(m->epoch == get_epoch())
on mon.g (3), the epoch is 55 (or looks that way, it just sent these):... Sage Weil
03:55 PM RADOS Bug #43882 (Need More Info): osd to mon connection lost, osd stuck down
adding debug: https://github.com/ceph/ceph/pull/32968 Sage Weil
01:06 PM RADOS Bug #43882 (Can't reproduce): osd to mon connection lost, osd stuck down
This is a similar symptom to #43825, but it does not appear to be related to split/merge.
OSD is marked down, but ...
Sage Weil
02:51 PM Ceph Revision dd1fe6ac (ceph): Merge pull request #32957 from tchaikov/wip-mgr-cephadm
mgr/cephadm: init attrs created by settattr() Sage Weil
01:45 PM RADOS Bug #43889 (Resolved): expected MON_CLOCK_SKEW but got none
description: rados/multimon/{clusters/6.yaml msgr-failures/many.yaml msgr/async.yaml
no_pools.yaml objectstore...
Sage Weil
01:44 PM RADOS Bug #43888: osd/osd-bench.sh 'tell osd.N bench' hang
https://github.com/ceph/ceph/pull/32961 to debug Sage Weil
01:41 PM RADOS Bug #43888 (Resolved): osd/osd-bench.sh 'tell osd.N bench' hang
... Sage Weil
01:36 PM RADOS Bug #43887 (Resolved): ceph_test_rados_delete_pools_parallel failure
... Sage Weil
01:24 PM bluestore Bug #43814 (Pending Backport): common/bl: claim_append() corrupts memory when a bl consecutively ...
Sage Weil
01:23 PM rgw Bug #43661 (Resolved): failure in test_sse_kms_post_object_authenticated_request
Sage Weil
01:23 PM RADOS Bug #43825 (Pending Backport): osd stuck down
Sage Weil
01:14 PM RADOS Bug #43885 (Can't reproduce): failed to reach quorum size 9 before timeout expired
This pops up occasionally. Here is a recent one:... Sage Weil
12:05 AM Orchestrator Feature #43867 (Resolved): cephadm: progress item for upgrade
show a progress item for upgrade based on the number of services and how many are updated to the new container image. Sage Weil

01/28/2020

08:26 PM rgw Bug #43661 (Fix Under Review): failure in test_sse_kms_post_object_authenticated_request
Sage Weil
08:24 PM rgw Bug #43842 (Duplicate): s3-tests failure in rados suite
Sage Weil
12:30 AM rgw Bug #43842 (Duplicate): s3-tests failure in rados suite
... Sage Weil
08:09 PM mgr Bug #43837 (Pending Backport): mgr/devicehealth: fix telemetry stops sending device reports after...
Sage Weil
08:08 PM RADOS Bug #38345 (Pending Backport): mon: segv in MonOpRequest::~MonOpRequest OpHistory::cleanup
Sage Weil
08:07 PM RADOS Bug #43826 (Resolved): osd: leak of from send_lease
Sage Weil
08:07 PM RADOS Bug #43865 (Resolved): osd-scrub-test.sh fails date check
... Sage Weil
08:05 PM CephFS Bug #40867: mgr: failover during in qa testing causes unresponsive client warnings
another instance of this on master,... Sage Weil
08:03 PM RADOS Bug #43864 (Resolved): osd/repro_long_log.sh failure
... Sage Weil
08:01 PM teuthology Bug #43863 (Can't reproduce): mkdir: cannot create directory ‘/home/ubuntu/cephtest/archive/audit...
... Sage Weil
07:59 PM RADOS Bug #43862 (Can't reproduce): mkfs fsck found fatal error: (2) No such file or directory during c...
... Sage Weil
07:45 PM RADOS Bug #43861: ceph_test_rados_watch_notify hang
/a/sage-2020-01-28_03:52:05-rados-wip-sage2-testing-2020-01-27-1839-distro-basic-smithi/4713217 Sage Weil
07:43 PM RADOS Bug #43861 (Resolved): ceph_test_rados_watch_notify hang
... Sage Weil
07:34 PM RADOS Bug #43825 (Fix Under Review): osd stuck down
Sage Weil
07:27 PM RADOS Bug #43825 (In Progress): osd stuck down
we are splitting:... Sage Weil
06:59 PM RADOS Bug #43825: osd stuck down
2020-01-28T14:56:26.155+0000 7fd3ba08d700 20 osd.6 285 identify_splits_and_merges 1.5 e245 to e285 pg_nums {76=28,89=... Sage Weil
06:39 PM RADOS Bug #43825: osd stuck down
... Sage Weil
04:34 PM teuthology Bug #43843 (Resolved): ERROR: paramiko 2.7.1 has requirement cryptography>=2.5, but you'll have c...
Sage Weil
04:34 PM teuthology Bug #43799 (Resolved): Running ceph-ansible task on older releases no longer works with python2.x
Sage Weil
04:33 PM teuthology Bug #43798 (Resolved): ceph-ansible teuthology task fails with AttributeError
Sage Weil
03:17 PM Ceph Bug #40683 (Can't reproduce): selinux allow ceph_t to call sudo
I'm not able to reproduce this (from a cephadm mon container on rhel 8 or centos 7.6), and we're not seeing this deni... Sage Weil
 

Also available in: Atom