Project

General

Profile

Activity

From 03/09/2018 to 04/07/2018

04/07/2018

07:02 AM Bug #23584 (Resolved): mgr: prometheus: 'PG_STATES' still have not all PG_STATES.
... Konstantin Shalygin

04/06/2018

08:39 PM Bug #23017 (Resolved): mgr log spamming about down osds
Nathan Cutler
08:39 PM Backport #23224 (Resolved): luminous: mgr log spamming about down osds
Nathan Cutler
07:27 PM Backport #23224: luminous: mgr log spamming about down osds
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/21053
merged
Yuri Weinstein
08:38 PM Bug #23037 (Resolved): mgr not reporting when ports conflict
Nathan Cutler
08:37 PM Backport #23175 (Resolved): luminous: mgr not reporting when ports conflict
Nathan Cutler
07:24 PM Backport #23175: luminous: mgr not reporting when ports conflict
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/20712
merged
Yuri Weinstein
06:03 PM Bug #23167 (Resolved): mgr: prometheus: ceph_pg metrics reported by prometheus plugin inconsisten...
Nathan Cutler
03:59 PM Bug #23167: mgr: prometheus: ceph_pg metrics reported by prometheus plugin inconsistent with "cep...
John Spray wrote:
> https://github.com/ceph/ceph/pull/20642
merged
Yuri Weinstein
02:39 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
Thanks -- looks like we have a deadlock bug in ceph-mgr itself.
This thread has taken the (shared) Objecter lock a...
John Spray
02:22 PM Feature #23574 (New): Add a HeartbeatMap to ceph-mgr (die on deadlocks)
In issues such as https://tracker.ceph.com/issues/23460, a deadlock can manifest as a completely stuck daemon. We sh... John Spray

04/05/2018

04:58 PM Bug #22457: ceph-mgr dashboard has dependency on python-jinja2
merged https://github.com/ceph/ceph/pull/20748 Yuri Weinstein
06:29 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
I've attached to a hanging mgr process (eating 40% CPU) and the output of GDB is attached.
Things to know about th...
Wido den Hollander

04/04/2018

03:57 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
Wido: so I guess the monclient messages are more of a symptom of something else getting badly stuck -- I suspect that... John Spray
08:52 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
And I also found one cpu core, which was allocated to this osd by cgroup, had 100% us time.
And the osd is still ali...
wei jin
08:32 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
I ran into the same issue from osd, which leads to slow request. (Luminous 12.2.4)
The primary osd's ops were blocke...
wei jin
06:12 AM Bug #22457: ceph-mgr dashboard has dependency on python-jinja2
follow-on fix backported to luminous via https://github.com/ceph/ceph/pull/21233 Nathan Cutler
06:09 AM Bug #22457 (Resolved): ceph-mgr dashboard has dependency on python-jinja2
Nathan Cutler
06:08 AM Bug #22457 (Pending Backport): ceph-mgr dashboard has dependency on python-jinja2
Nathan Cutler

03/30/2018

05:18 AM Bug #22424 (Resolved): balancer should warn about missing requirements
Nathan Cutler
02:27 AM Backport #22983 (Resolved): luminous: balancer should warn about missing requirements
xie xingguo

03/29/2018

01:22 PM Backport #22983: luminous: balancer should warn about missing requirements
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/20359
merged
Yuri Weinstein
08:18 AM Bug #23083 (Resolved): ceph-mgr fails to start after a system reboot on Ubuntu 16.04
Nathan Cutler
08:17 AM Backport #23101 (Resolved): luminous: ceph-mgr fails to start after a system reboot on Ubuntu 16.04
Nathan Cutler

03/28/2018

10:28 PM Backport #23101: luminous: ceph-mgr fails to start after a system reboot on Ubuntu 16.04
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/20604
merged
Yuri Weinstein
04:25 PM Bug #23219 (Resolved): Update mgr/restful documentation
Nathan Cutler
04:25 PM Backport #23230 (Resolved): luminous: Update mgr/restful documentation
Nathan Cutler
03:35 PM Bug #23482: ceph-mgr --help stopped working (regression in master)
If my regression hypothesis is correct, the "culprit" would be one or more of these:... Nathan Cutler
02:39 PM Bug #23482 (Resolved): ceph-mgr --help stopped working (regression in master)
(Opening with increased priority since this appears to be a regression)
With a recent master, ceph-mgr --help stop...
Nathan Cutler
08:31 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
So I set the logging of the active manager to:
- debug_mgr: 20
- debug_auth: 5
After running for over 24 hours...
Wido den Hollander
02:31 AM Backport #23416 (Need More Info): luminous: mgr sends early beacon with no modules reported
File: src/mgr/PyModule.h needs to be added to luminous to backport this PR. Prashant D

03/27/2018

10:32 AM Dashboard Bug #23406 (Resolved): Attempt to set dashboard login credentials causes ceph-mgr to crash in Pyt...
Ricardo Dias
07:18 AM Backport #23224 (In Progress): luminous: mgr log spamming about down osds
https://github.com/ceph/ceph/pull/21053 Prashant D

03/26/2018

02:21 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
Raising priority since this affects a stable release (Luminous) Nathan Cutler
01:52 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
I just found another mgr in this cluster which is "dead", but is eating 300% CPU on 3 cores:... Wido den Hollander
01:39 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
I found another mgr in a cluster which has been down now for 4 days.... Wido den Hollander
11:05 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
I don't think the TZ is related. In all the cases I've seen the Mon and Mgr were running on the same node.
I saw i...
Wido den Hollander
10:50 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
This one's still a mystery to me :-/
Only initial thought is that we're in daylight-savings territory right now, c...
John Spray
08:46 AM Bug #23460 (Resolved): mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expi...
On Luminous v12.2.2 and v12.2.4 clusters running either CentOS or Ubuntu I've seen many Manager going offline with th... Wido den Hollander
10:00 AM Dashboard Bug #23404 (Resolved): dashboard module does not work in Python 3-only environment
Ricardo Dias
08:40 AM Bug #22226 (Rejected): ceph zabbix plugin sends incorrect motinoring info to zabbix server
I this one still active? Otherwise we can close it I think.
Setting it to Rejected for now as I think it is resolv...
Wido den Hollander

03/25/2018

03:19 PM Bug #23205: Blocked requests no longer show details
We continue to get these with no clues as to where to go next:
2018-03-25 06:58:39.693404 7fa0e9ee2700 0 log_chan...
Alex Gorbachev

03/23/2018

02:03 AM Dashboard Bug #23326 (Resolved): mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot ...
Fixed by: https://github.com/ceph/ceph/pull/20986 Patrick Donnelly

03/22/2018

12:21 PM Dashboard Bug #23404 (Fix Under Review): dashboard module does not work in Python 3-only environment
PR: https://github.com/ceph/ceph/pull/21006 Ricardo Dias
08:30 AM Dashboard Bug #23404 (In Progress): dashboard module does not work in Python 3-only environment
Ricardo Dias
12:13 PM Dashboard Bug #23406 (Fix Under Review): Attempt to set dashboard login credentials causes ceph-mgr to cras...
PR: https://github.com/ceph/ceph/pull/21005 Ricardo Dias
12:12 PM Dashboard Bug #23406: Attempt to set dashboard login credentials causes ceph-mgr to crash in Python 3-only ...
The current `handle_pyerror` function implementation relies in the `traceback.format_exception_only` python function ... Ricardo Dias
08:31 AM Dashboard Bug #23406 (In Progress): Attempt to set dashboard login credentials causes ceph-mgr to crash in ...
Ricardo Dias

03/20/2018

11:20 AM Backport #23409 (In Progress): luminous: mgr: fix MSG_MGR_MAP handling
https://github.com/ceph/ceph/pull/20973 Prashant D

03/19/2018

08:29 PM Bug #23418 (Won't Fix): doc: Dashboard account creation
The documentation at http://docs.ceph.com/docs/master/mgr/dashboard/ refers the the following incorrect command:
...
Marc Schöchlin
04:43 PM Backport #23416 (Rejected): luminous: mgr sends early beacon with no modules reported
Nathan Cutler
04:42 PM Backport #23409 (Resolved): luminous: mgr: fix MSG_MGR_MAP handling
https://github.com/ceph/ceph/pull/20973 Nathan Cutler
03:18 PM Dashboard Bug #23406 (Resolved): Attempt to set dashboard login credentials causes ceph-mgr to crash in Pyt...
Environment: SLE-15 (Python 3-only)
Currently running mimic_dev2 but the issue is presumed to be present on master...
Nathan Cutler
02:25 PM Dashboard Bug #23404 (Resolved): dashboard module does not work in Python 3-only environment
Environment: SLE-15 (Python 3-only)
After enabling the dashboard, it is impossible to run "ceph mgr dashboard set-...
Nathan Cutler
01:14 AM Feature #23400 (New): ceph-mgr should raise health alert if OSDs are up but no data being received
John Spray

03/17/2018

10:46 AM Feature #23397 (Resolved): Central "mgr self-test <module>" command
Currently, many of the manager modules implement a "self-test" command (validate that they can fetch and parse their ... John Spray
10:38 AM Bug #21598 (Fix Under Review): Users can do "config-key set" while mgr runs, but it doesn't see s...
This issue will be fixed implicitly when we merge https://github.com/ceph/ceph/pull/20458 John Spray
10:27 AM Feature #21682 (Resolved): mgr should raise health alert when a module throws exceptions (post-load)
John Spray
10:27 AM Dashboard Feature #22522 (Resolved): dashboard: configuration setting browser
John Spray

03/16/2018

02:37 PM Dashboard Bug #23389 (Resolved): dashboard: OSD throughput sparkline graphic appears to show running total
... I suspect it's meant to show historical rates instead. Jason Dillaman
11:14 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
A vstart flag is easier to implement, and we should fix this problem that way in the short term. But we can also thin... Ricardo Dias
03:26 AM Bug #23368 (Pending Backport): mgr: fix MSG_MGR_MAP handling
Kefu Chai
03:24 AM Bug #23378 (Resolved): Test failure: test_perf_counters_mgr_get (tasks.mgr.dashboard_v2.test_perf...
https://github.com/ceph/ceph/pull/20916 Kefu Chai

03/15/2018

07:15 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
John Spray wrote:
> I think what's going on here is that the dashboard used to fail silently (it always required rbd...
Patrick Donnelly
12:40 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
I think what's going on here is that the dashboard used to fail silently (it always required rbd bindings), because v... John Spray
11:14 AM Bug #23378 (Resolved): Test failure: test_perf_counters_mgr_get (tasks.mgr.dashboard_v2.test_perf...
... Kefu Chai
06:21 AM Bug #22918 (Pending Backport): mgr sends early beacon with no modules reported
Kefu Chai
02:10 AM Bug #23368 (Fix Under Review): mgr: fix MSG_MGR_MAP handling
Kefu Chai
01:26 AM Bug #23368 (Resolved): mgr: fix MSG_MGR_MAP handling
ceph config show mgr.x doesn't work
root cause is mgr daemon's mgrc has no chance
to process MSG_MGR_MAP in the mgr...
cory gu

03/14/2018

05:38 AM Backport #23313 (In Progress): luminous: mgr: prometheus: internal server error while new OSDs ar...
https://github.com/ceph/ceph/pull/20891 Prashant D

03/13/2018

04:28 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
I understand the temptation but I'd prefer this be fixed. Taking out RBD compilation has significantly improved compi... Patrick Donnelly
04:21 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
It's not impossible to put some special handling in the dashboard for disabling rbd functionality when it's missing, ... John Spray
07:19 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
Patrick Donnelly wrote:
> Omitting RBD must be why. Is there a way I can avoid this mgr error without building RBD...
Ricardo Dias
12:48 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
Oh, I'm building with:... Patrick Donnelly
12:16 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
Generally that's what you'd get from an incomplete build, perhaps doing the `cephfs_testing` vstart target rather tha... John Spray
04:28 PM Bug #23300 (Duplicate): ceph-mgr returns internal error
This was fixed in master recently and is being backported to luminous here: https://github.com/ceph/ceph/pull/20642
John Spray
01:55 PM Bug #23330: mon command "mgr metadata $name" has inconsistent argument naming
Marking for backport because will want to take it along with fix for 23286 when that's done John Spray
01:55 PM Bug #23330 (Fix Under Review): mon command "mgr metadata $name" has inconsistent argument naming
This command was using `id` where all the other metadata commands were using `who`, so anyone passing `who` is gettin... John Spray
11:33 AM Bug #23330 (Resolved): mon command "mgr metadata $name" has inconsistent argument naming
When running the following mon command:... Jan Fajerski
11:34 AM Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
I have a potential fix (updating the metadata in DaemonServer::got_mgr_map()) but encountered http://tracker.ceph.com... Jan Fajerski
12:47 AM Bug #23276: balancer/osd: segfault in calc_pg_upmaps
Hi Dan,
There is already a pending backport for Luminous, see https://github.com/ceph/ceph/pull/20840
xie xingguo

03/12/2018

11:43 PM Dashboard Bug #23326 (Resolved): mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot ...
When running vstart against master:... Patrick Donnelly
04:08 PM Bug #23276: balancer/osd: segfault in calc_pg_upmaps
Great thanks.
Could someone please add the backports tag for l?
Dan van der Ster
06:42 AM Bug #23276: balancer/osd: segfault in calc_pg_upmaps
https://github.com/ceph/ceph/pull/20655 xie xingguo
09:14 AM Backport #23313 (Resolved): luminous: mgr: prometheus: internal server error while new OSDs are b...
https://github.com/ceph/ceph/pull/21492 Nathan Cutler
03:10 AM Bug #23205: Blocked requests no longer show details
in other words, what is the best way to troubleshoot the following situation:
2018-03-11 22:00:00.000132 mon.roc-v...
Alex Gorbachev

03/11/2018

07:26 PM Bug #23300: ceph-mgr returns internal error
Found it! We had several osds without a device class attached, because we did not want to use them at the moment.
Ad...
Nico Schottelius
07:20 PM Bug #23300: ceph-mgr returns internal error
Fun fact: it used to run fine until we were introducing new crush rules and changing the crush rule for a pool:
<p...
Nico Schottelius
07:17 PM Bug #23300 (Duplicate): ceph-mgr returns internal error
Hello,
after some weeks of running a new ceph cluster, we get the following answer from the mgr:
black3.place6:...
Nico Schottelius
05:10 AM Bug #23205: Blocked requests no longer show details
the one big advantage of having the few slowest OSDs listed in ceph.log or MON log was the ability to go back to trou... Alex Gorbachev

03/09/2018

01:59 PM Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
This bug is a result of how we populate the mgrs into DaemonState from DaemonServer::got_mgr_map without ever reading... John Spray
01:08 PM Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
- Hello, what's your ceph version? -
Oh, sorry. it's "ceph version 13.0.1-2832 .. mimic (dev)".
Chang Liu
10:50 AM Bug #23286 (Resolved): mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
When running a vstart.sh cluster on master at cf52fc5a, @list_servers_python()@ returns:... Sebastian Wagner
02:23 AM Dashboard Bug #23265 (Resolved): FAIL: test_get (tasks.mgr.dashboard_v2.test_cluster_configuration.ClusterC...
Kefu Chai
 

Also available in: Atom