Project

General

Profile

Activity

From 09/04/2017 to 10/03/2017

10/03/2017

01:30 PM Bug #21593: segv in PyList_New from PyFormatter
Spun off https://github.com/ceph/ceph/pull/18093 while trying to get to bottom of this John Spray
12:24 PM Bug #21399: ceph-mgr module(s) inaccessible after a reboot
Sorry for the delay. I have been playing with this a bit. I am not saying it is related to the executable name being ... Boris Ranto
02:59 AM Backport #21659 (Resolved): luminous: Crash in get_metadata_python during MDS restart
https://github.com/ceph/ceph/pull/18412 Nathan Cutler
02:58 AM Backport #21656 (Resolved): luminous: crash on DaemonPerfCounters::update
https://github.com/ceph/ceph/pull/18675
Nathan Cutler
02:58 AM Backport #21648 (Resolved): luminous: mgr[zabbix] float division by zero
https://github.com/ceph/ceph/pull/18734 Nathan Cutler
02:57 AM Backport #21638 (Resolved): luminous: dashboard OSD list has servers and osds in arbitrary order
Nathan Cutler

10/02/2017

03:34 PM Bug #21593 (In Progress): segv in PyList_New from PyFormatter
John Spray
02:34 AM Bug #21518 (Pending Backport): mgr[zabbix] float division by zero
Sage Weil

10/01/2017

08:56 PM Backport #21452 (Resolved): luminous: prometheus module generates invalid output when counter nam...
Sage Weil
08:56 PM Backport #21443 (Resolved): luminous: Prometheus crash when update
Sage Weil
05:17 PM Bug #21612 (New): mgr: thread create assert failure when racing with respawn
Run: http://pulpito.ceph.com/teuthology-2017-09-26_02:30:02-rados-luminous-distro-basic-smithi/
Job: 1673575
Logs: ...
Yuri Weinstein

09/29/2017

11:06 AM Bug #21599 (Fix Under Review): List of filesystems does not get refreshed after a filesystem dele...
https://github.com/ceph/ceph/pull/18039 John Spray
10:37 AM Bug #21599: List of filesystems does not get refreshed after a filesystem deletion
Never mind, this was easy to reproduce locally. When a filesystem is deleted, ceph-mgr is somehow still holding on t... John Spray
10:29 AM Bug #21599: List of filesystems does not get refreshed after a filesystem deletion
Hmm, the list in the menu is supposed to be updated every 5 seconds.
If you leave the console (the right click->in...
John Spray

09/28/2017

10:56 PM Bug #21599 (Resolved): List of filesystems does not get refreshed after a filesystem deletion
Just tested 12.2.1 and was playing with multiple filesystems (CephFS), creating some and deleting some. After I have ... Jean-Charles Lopez
06:30 PM Bug #21598 (Resolved): Users can do "config-key set" while mgr runs, but it doesn't see settings
This is kind of confusing for users and for module developers.
Originally the config-key stuff was meant to just b...
John Spray
04:22 PM Bug #20629: Spurious ceph-mgr failovers during mon elections
Seen here:
http://pulpito.ceph.com/dzafman-2017-09-27_09:14:10-rados-wip-no-lock-distro-basic-smithi/1679370/
David Zafman
02:55 PM Feature #21594 (Fix Under Review): prometheus meta-series describing OSD<->disk mapping
https://github.com/ceph/ceph/pull/18021 John Spray
02:52 PM Feature #21594 (Resolved): prometheus meta-series describing OSD<->disk mapping
John Spray
02:35 PM Bug #21593: segv in PyList_New from PyFormatter
I'm triggering this with github.com/liewegas/ceph wip-balancer,... Sage Weil
02:34 PM Bug #21593 (Resolved): segv in PyList_New from PyFormatter
... Sage Weil
10:06 AM Bug #20420 (Can't reproduce): Crash in MgrClient::send_report->PerfCountersCollection::with_counters
DaemonState locking has been substantially fixed since this happened -- hopefully whatever the issue was got fixed as... John Spray
06:38 AM Dashboard Bug #21572 (Pending Backport): dashboard OSD list has servers and osds in arbitrary order
Kefu Chai

09/27/2017

02:16 PM Dashboard Bug #21570: dashboard barfs on nulls where it expects numbers
Oops, accidentally put this in fs project, sorry for spam. John Spray
01:37 PM Dashboard Bug #21570 (Fix Under Review): dashboard barfs on nulls where it expects numbers
https://github.com/ceph/ceph/pull/17991 John Spray
01:36 PM Dashboard Bug #21570 (Resolved): dashboard barfs on nulls where it expects numbers

In this particular instance it has nulls for the usage on one of the pools for some unknown reason.
The nulls sh...
John Spray
02:15 PM Dashboard Bug #21572 (Fix Under Review): dashboard OSD list has servers and osds in arbitrary order
https://github.com/ceph/ceph/pull/17993 John Spray
02:10 PM Dashboard Bug #21572 (Resolved): dashboard OSD list has servers and osds in arbitrary order
Doesn't tend to be noticeable on vstart clusters but is very apparent on real clusters.
John Spray

09/26/2017

01:36 PM Bug #21197 (Pending Backport): crash on DaemonPerfCounters::update
Sage Weil
01:36 PM Bug #17737 (Pending Backport): Crash in get_metadata_python during MDS restart
Sage Weil

09/25/2017

09:22 PM Backport #21549 (Resolved): luminous: the dashboard uses absolute links for filesystems and clients
Nathan Cutler
09:21 PM Backport #21547 (Resolved): luminous: ceph-mgr gets process called "exe" after respawn
https://github.com/ceph/ceph/pull/18738 Nathan Cutler

09/23/2017

05:23 PM Bug #17737 (Fix Under Review): Crash in get_metadata_python during MDS restart
https://github.com/ceph/ceph/pull/17933 John Spray
05:11 PM Bug #21197 (Fix Under Review): crash on DaemonPerfCounters::update
https://github.com/ceph/ceph/pull/17932
John Spray
03:53 PM Bug #21197 (In Progress): crash on DaemonPerfCounters::update
John Spray
03:49 PM Documentation #20257 (Closed): ceph-mgr doc updates
Closing this -- I'm very willing to add specific information to the docs, but this request is a bit too general for m... John Spray
03:48 PM Documentation #20257: ceph-mgr doc updates
> a) ceph-mgrs keys and bootstrap process and verify properly bootstraped
We have manual setup instructions, they ...
John Spray
03:29 PM Bug #21367 (Fix Under Review): 'ZabbixSender' object has no attribute 'hostname'
John Spray
03:24 PM Bug #20886 (Can't reproduce): Bad perf counters in ceph-mgr for MDS
John Spray
03:22 PM Bug #20568 (Pending Backport): the dashboard uses absolute links for filesystems and clients
John Spray
02:26 PM Bug #21518 (Fix Under Review): mgr[zabbix] float division by zero
By inspection, I'm assuming this was happening in the avg() function due to lack of data (why there was no data is a ... John Spray

09/22/2017

09:09 PM Bug #21404 (Pending Backport): ceph-mgr gets process called "exe" after respawn
Sage Weil
08:34 PM Backport #21524 (Resolved): luminous: DaemonState members accessed outside of locks
https://github.com/ceph/ceph/pull/18675 Nathan Cutler
08:06 PM Bug #21518: mgr[zabbix] float division by zero
After upgrading all OSDs to bluestore, the issue resolved.
One of OSD had filestore format while others had bluest...
Ilja Slepnev
07:47 PM Bug #21518 (Resolved): mgr[zabbix] float division by zero
Enabled Zabbix module, configured as described on http://docs.ceph.com/docs/master/mgr/zabbix/, but no data is receiv... Ilja Slepnev
02:44 PM Feature #21502: Enable mgr modules to report their "runnability"
Right, although now that I think about it, we could probably also catch ImportErrors from when we try to load the mod... John Spray
02:40 PM Feature #21502: Enable mgr modules to report their "runnability"
This them implies that no import of any "foreign module" may be done in the header of a Mgr Module but should always ... Wido den Hollander
02:25 PM Feature #21502 (Resolved): Enable mgr modules to report their "runnability"
If a module has a dependency which is missing, we should be able to learn that on a mgr (even in standby), so that it... John Spray
02:37 PM Bug #20396 (Can't reproduce): segv in python somewhere from restful test
John Spray
04:30 AM Bug #21158 (Pending Backport): DaemonState members accessed outside of locks
Kefu Chai

09/21/2017

07:43 PM Bug #20222 (Duplicate): v12.0.3 Luminous bluestore 'tp_osd_tp thread tp_osd_tp' had timed out aft...
Highly likely this is #21171; please try latest luminous branch or wait for 12.2.1 (out very very soon now!) Sage Weil
11:57 AM Bug #21122: Kraken to luminous upgrade: Error EINVAL: key for mgr.vpm037 exists but cap mds does ...
I have the same error when I migrate my cluster and create mgr Nicolas Drufin
04:15 AM Backport #21479 (In Progress): luminous: Services reported with blank hostname by mgr
Nathan Cutler
04:13 AM Backport #21452 (In Progress): luminous: prometheus module generates invalid output when counter ...
Nathan Cutler
04:11 AM Backport #21443 (In Progress): luminous: Prometheus crash when update
Nathan Cutler
04:09 AM Backport #21320 (In Progress): luminous: Quieten scary RuntimeError from restful module on startup
Nathan Cutler

09/20/2017

06:33 PM Bug #21399: ceph-mgr module(s) inaccessible after a reboot
The 'exe' thing is really just the process naming - even if there is an issue of sockets not getting torn down, that ... John Spray
01:16 PM Bug #21399: ceph-mgr module(s) inaccessible after a reboot
I think it might be related. If there are any issues with the module not being killed properly before respawn then th... Boris Ranto
01:40 PM Backport #21479 (Resolved): luminous: Services reported with blank hostname by mgr
https://github.com/ceph/ceph/pull/17869 Nathan Cutler
03:24 AM Bug #20887 (Pending Backport): Services reported with blank hostname by mgr
Kefu Chai
03:24 AM Bug #20887 (Resolved): Services reported with blank hostname by mgr
https://github.com/ceph/ceph/pull/17138 Kefu Chai

09/19/2017

11:37 AM Backport #21452 (Resolved): luminous: prometheus module generates invalid output when counter nam...
https://github.com/ceph/ceph/pull/17868 Nathan Cutler
11:37 AM Backport #21443 (Resolved): luminous: Prometheus crash when update
https://github.com/ceph/ceph/pull/17867 Nathan Cutler

09/18/2017

11:31 AM Bug #21356: ceph-mgr admin socket starts failing after many attempts to call nonexistent command
I think it was just the same " do_accept error: '(24) Too many open files'" message many times, right? John Spray
06:14 AM Bug #21356 (Need More Info): ceph-mgr admin socket starts failing after many attempts to call non...
Mark, what log messages were filling your disk? could you pastebin or just paste a sample of it? and what debug level... Kefu Chai
10:36 AM Bug #20899 (Pending Backport): prometheus module generates invalid output when counter names cont...
John Spray

09/15/2017

08:35 PM Bug #21260 (Resolved): ceph mgr versions shows active mgr as "Unknown"
Nathan Cutler
08:35 PM Backport #21342 (Resolved): luminous: ceph mgr versions shows active mgr as "Unknown"
Nathan Cutler
04:28 PM Bug #21404 (Fix Under Review): ceph-mgr gets process called "exe" after respawn
https://github.com/ceph/ceph/pull/17756 John Spray
04:21 PM Bug #21404 (Resolved): ceph-mgr gets process called "exe" after respawn

This is identical to http://tracker.ceph.com/issues/19291, as the respawn code was lifted from ceph-mds for ceph-mg...
John Spray
04:22 PM Bug #21399: ceph-mgr module(s) inaccessible after a reboot
Good to know it's reproducible, if I have the log from the existing instance where this is happening then that would ... John Spray
02:46 PM Bug #21399: ceph-mgr module(s) inaccessible after a reboot
John, this is reproducible, you just need to deploy a cluster with ansible and reboot the machine. For some reason, t... Boris Ranto
11:08 AM Bug #21399: ceph-mgr module(s) inaccessible after a reboot
Hmm, I can't see much from that log -- it looks like it's from a single run of the mgr rather than spanning the reboo... John Spray
10:59 AM Bug #21399 (Closed): ceph-mgr module(s) inaccessible after a reboot
I've enabled and secured the RESTful API and verified that I can access its web UI at https://<active-mgr-node-ip-add... Bara Ancincova
11:50 AM Bug #21253 (Pending Backport): Prometheus crash when update
Kefu Chai
07:10 AM Bug #21367: 'ZabbixSender' object has no attribute 'hostname'
*PR*
https://github.com/ceph/ceph/pull/17751
UEST
jiantao zhu

09/12/2017

10:15 AM Bug #20692 (Resolved): mgr: 500 error when attempting to view filesystem data
Nathan Cutler
10:15 AM Backport #21137 (Resolved): luminous: mgr: 500 error when attempting to view filesystem data
Nathan Cutler
09:04 AM Bug #21367: 'ZabbixSender' object has no attribute 'hostname'
You are welcome, I will do it as soon as I can. jiantao zhu
08:42 AM Bug #21367: 'ZabbixSender' object has no attribute 'hostname'
Jiantao: thank you, would it be possible to open a github pull request with your fix? John Spray
07:40 AM Bug #21367: 'ZabbixSender' object has no attribute 'hostname'
The bug is solved by myself.See module.py jiantao zhu
07:21 AM Bug #21367 (Can't reproduce): 'ZabbixSender' object has no attribute 'hostname'
there is a bug in ceph-zabbix and the log in ceph-mgr.lum001.log is:
2017-09-10 23:14:01.259667 7f21c2ada700 20 m...
jiantao zhu
08:32 AM Bug #21197: crash on DaemonPerfCounters::update
Thanks for the debugging -- when the RGW <-> mgr stuff was added we knew that it would require unique daemon names to... John Spray
05:07 AM Bug #21197: crash on DaemonPerfCounters::update
TLDR at EOF
Some more information to my setup:
I'm running a Docker swarm with 15 radosgw instances, the used D...
Katie Holly
01:34 AM Bug #21197: crash on DaemonPerfCounters::update
Same issues here.
The cluster has been upgraded from Jewel to Kraken and then to Luminous on Ubuntu 16.04, ceph-mg...
Katie Holly

09/11/2017

09:10 PM Bug #21157 (Resolved): Crash in MonCommandCompletion
Nathan Cutler
09:10 PM Backport #21183 (Resolved): luminous: Crash in MonCommandCompletion
Nathan Cutler
09:10 PM Bug #20746 (Resolved): dashboard: usage graph is getting more and mor big
Nathan Cutler
09:09 PM Backport #21188 (Resolved): luminous: dashboard: usage graph is getting more and mor big
Nathan Cutler
04:41 PM Bug #21356: ceph-mgr admin socket starts failing after many attempts to call nonexistent command
At the same time as fixing this, let's fix the code in common/admin_socket that spins in AdminSocket::entry when do_a... John Spray
04:39 PM Bug #21356 (Can't reproduce): ceph-mgr admin socket starts failing after many attempts to call no...
Apparently if something is repeatedly calling nonexistent commands on the mgr's admin socket, it ends up with errno 2... John Spray
09:50 AM Backport #21342 (In Progress): luminous: ceph mgr versions shows active mgr as "Unknown"
Nathan Cutler
09:37 AM Backport #21342 (Resolved): luminous: ceph mgr versions shows active mgr as "Unknown"
https://github.com/ceph/ceph/pull/17635 Nathan Cutler
09:21 AM Bug #21340 (New): mgr modules without serve threads cannot receive notifications
Originally any module would receive notifications, but during development of the prometheus module it was causing iss... John Spray
09:09 AM Bug #18994 (Closed): High CPU usage for ceph-mgr daemon v11.2.0
Closing this because Kraken is retired now that Luminous is out.
Please open a fresh ticket if having this issue w...
John Spray
08:50 AM Bug #18994: High CPU usage for ceph-mgr daemon v11.2.0
I'm seeing the same in a recently upgraded 11.2.1 cluster.
top:
3151 ceph 20 0 2153644 1,469g 16992 S ...
Kees Hoekzema
09:07 AM Bug #21225 (Can't reproduce): ceph-mgr: dashboard and zabbix plugin report wrong values
John Spray
08:57 AM Bug #21225: ceph-mgr: dashboard and zabbix plugin report wrong values
It is getting strange now. I restarted my mon/mgr nodes and the problem disappeared. It seems to work now and I canno... Tobias Rehn

09/10/2017

07:20 PM Bug #21260 (Pending Backport): ceph mgr versions shows active mgr as "Unknown"
Sage Weil
04:40 PM Bug #20222: v12.0.3 Luminous bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60
Sage,
Able to reproduce the issue in v12.2.0
Env - 5 node , EC 4+1 , 120 OSD's
----
2017-09-10 15:39:14....
Nokia ceph-users

09/08/2017

08:20 PM Backport #21320 (Resolved): luminous: Quieten scary RuntimeError from restful module on startup
https://github.com/ceph/ceph/pull/17866 Nathan Cutler
03:44 PM Bug #21253 (Fix Under Review): Prometheus crash when update
https://github.com/ceph/ceph/pull/17605 John Spray
02:28 PM Bug #21225: ceph-mgr: dashboard and zabbix plugin report wrong values
How odd...
Please could you add these settings on mon and mgr nodes:...
John Spray
01:37 PM Fix #21292 (Pending Backport): Quieten scary RuntimeError from restful module on startup
John Spray

09/07/2017

02:08 PM Fix #21292 (Fix Under Review): Quieten scary RuntimeError from restful module on startup
https://github.com/ceph/ceph/pull/17573/files John Spray
10:05 AM Fix #21292 (Resolved): Quieten scary RuntimeError from restful module on startup
John Spray
01:48 PM Bug #21260 (Fix Under Review): ceph mgr versions shows active mgr as "Unknown"
https://github.com/ceph/ceph/pull/17571 John Spray

09/06/2017

10:34 AM Bug #21260 (Resolved): ceph mgr versions shows active mgr as "Unknown"
After migrating from Luminous 12.1.4 (rc) to Luminous 12.2.0, I found that 'ceph versions' command showa one mgr daem... Lluis Arasanz
01:24 AM Bug #21253: Prometheus crash when update
when mgr service is running, refresh the prometheus:9283 mutiple times, then it would crash with this error. Ji You
01:21 AM Bug #21253 (Resolved): Prometheus crash when update
... Ji You

09/05/2017

03:31 PM Bug #20444 (Resolved): mon: Hit assert in PaxosService::propose_pending after election
Nathan Cutler
03:31 PM Backport #20640 (Rejected): kraken: mon: Hit assert in PaxosService::propose_pending after election
Kraken is EOL. Nathan Cutler
03:31 PM Bug #19568 (Resolved): mgr: reopen logs on sighup
Nathan Cutler
03:31 PM Backport #19572 (Rejected): kraken: mgr: reopen logs on sighup
Kraken is EOL. Nathan Cutler
10:01 AM Backport #21188 (In Progress): luminous: dashboard: usage graph is getting more and mor big
Nathan Cutler
10:01 AM Bug #20746: dashboard: usage graph is getting more and mor big
*master PR*: https://github.com/ceph/ceph/pull/16857 Nathan Cutler
09:53 AM Backport #21183 (In Progress): luminous: Crash in MonCommandCompletion
Nathan Cutler
09:45 AM Backport #21137 (In Progress): luminous: mgr: 500 error when attempting to view filesystem data
Nathan Cutler

09/04/2017

02:04 PM Bug #21225 (Can't reproduce): ceph-mgr: dashboard and zabbix plugin report wrong values
I have installed a ceph cluster using the latest stable (luminous 12.2.0). I enabled the dashboard and zabbix plugin.... Tobias Rehn
 

Also available in: Atom