Project

General

Profile

Activity

From 01/04/2016 to 02/02/2016

02/02/2016

10:47 PM Support #14614: Add access to Sepia Lab for Joe Mario
I reran sepia/new-client. The new output is:
# sepia/new-client joemario@blazingsaddles
Please submit the foll...
Joe Mario
07:29 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
Per a conversation with Ben England and Mark Nelson, I'd like to run a development feature we're adding to the perf t... Joe Mario
05:52 PM Bug #14612 (Duplicate): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:infe...
Run: http://pulpito.ceph.com/teuthology-2016-02-01_17:10:11-upgrade:infernalis-infernalis-distro-basic-vps/
Job: 151...
Yuri Weinstein
03:20 AM Bug #10174 (Won't Fix): saya037 unresponsive
sayas are currently not used. I'm sure I'll run into this if we end up reusing the systems in the future. David Galloway
03:16 AM Bug #13131 (Resolved): mira078 has a Read-Only FS, boots you out of ssh instantly
Cleaning up the queue.. This system appears to have been running jobs successfully lately, has had drives replaced an... David Galloway
03:11 AM Feature #11785 (Closed): install radian memory NVRAM cards in some machines
Assuming this was already taken care of or is no longer needed.
Please re-open if that's not the case.
David Galloway
03:09 AM Bug #14576 (Resolved): mira112 stuck at reboot
HDD boot order fixed David Galloway
03:05 AM Bug #13836 (Can't reproduce): vpm103.stderr:bash: git: command not found
Due to the nature of VPSes and the age of this ticket, this is near impossible to troubleshoot. The system has been ... David Galloway
02:54 AM Bug #12555 (Resolved): mira083 stuck in raid bios on reboot
Replaced drives and reimaged 7JAN2016 David Galloway
02:45 AM Support #14430 (Resolved): need access to Sepia Lab
Access granted and confirmed David Galloway
12:35 AM Bug #14157 (Resolved): smithi004, smithi005, smithi007, smithi055 NVMe cards bad
David Galloway wrote:
> Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked Super...
David Galloway

02/01/2016

10:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked SuperMicro expedite a shipment... David Galloway
10:31 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
There were files under vpm104's mountpoint consuming space from root. Embarrassing that it took so long to figure th... Dan Mick

01/31/2016

10:14 PM Support #14430: need access to Sepia Lab
That worked! I can now ssh to bengland@teuthology.front.sepia.ceph.com. Thanks for the help logging in. Ben England

01/30/2016

08:07 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
dubia001 can be repurposed from openstack if it hasn't been reused already. I was the one using it for openstack test... Josh Durgin
07:37 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
dubia001 appears to be running openstack. It's unreachable but ethtool via SOL indicates its NIC isn't connected. W... David Galloway
05:47 PM Bug #14576 (Resolved): mira112 stuck at reboot
It was at stale state and stuck at nuke/stale
could not ssl and at the end ot sol power cycle got stuck at:...
Yuri Weinstein
12:22 AM Bug #13282 (In Progress): ssh from my vm to centos vpm machines are slow.
I might have been a bit hasty with tat last update.
ssh to Centos still seems to take a while.
Centos ssh:
</p...
Anonymous

01/29/2016

09:23 PM Fix #11889 (Resolved): ubuntu@tracker.front.sepia.ceph.com lingering git fetch-pack
Redmine and the underlying OS were updated when tracker got moved to DHC.
I'm going to close this ticket but if yo...
David Galloway
09:00 PM Support #13863 (Closed): Ask For Access To Sepia Lab Teuthology Gate
please find us in #sepia on irc.oftc.net to discuss! note that the lab capacity is pretty well utilized, so if you h... Sage Weil
08:35 PM Bug #11865 (Resolved): mira041 bad sdc sdd
Drives replaced 19JAN2016 David Galloway
08:33 PM Bug #11121 (Resolved): mira054 sdb sdc sdd sde sdf bad
This system died. Its disks were moved to mira021 which took mira054's place in the long running cluster. David Galloway
08:30 PM Feature #11496 (Rejected): pulpito-rdu needs a proxy
pulpito-rdu decomissioned David Galloway
07:51 PM Bug #14455 (Resolved): mira112 has bad drives
Replaced this host's drives last night. David Galloway
07:28 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
All smithi nodes that had 7.1 installed have been reinstalled with 7.2
There are 3 machines I still have locked th...
David Galloway
12:38 AM Bug #14541: "No space left on device" when locking vpm097
I can't figure this one out. By all accounts there is only about 15GB used on /, but df shows 628GB used. Normally ... Dan Mick

01/28/2016

11:32 PM Bug #14548 (Can't reproduce): mira052 MCE
... David Galloway
10:28 PM Bug #14546 (Resolved): mira033 kernel panic from MCE
... David Galloway
05:19 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-27_18:49:02-rados-hammer-distro-basic-vps/46886/teutholo... Yuri Weinstein

01/27/2016

09:24 PM Support #14531 (Resolved): RCA: download.ceph.com outage 27JAN2016
Wanted to get this documented in a ticket until we have a better way of documenting outage RCAs.
Over the past cou...
David Galloway
07:50 PM Bug #13282 (Resolved): ssh from my vm to centos vpm machines are slow.
Seems to be better. Anonymous
04:26 PM Cleanup #14528 (Resolved): Track down usage and purpose of mira{123..126} aka dubia{001..004}
A couple of these are currently being used for infrastucture but at least one of them can probably be nuked and added... David Galloway

01/26/2016

08:53 PM Bug #14214 (In Progress): mira115 has bad disks (marked down)
Re-opening this. Still showing up as drives missing. David Galloway
06:20 PM Bug #14214 (Resolved): mira115 has bad disks (marked down)
Drives replaced on 19JAN2016 David Galloway
07:32 PM Bug #14432 (Closed): mira095 got wedged
Without more logs (or bandwidth) to investigate further, not a lot more I can do to say for sure whether this is a ha... David Galloway
07:25 PM Bug #12203 (Closed): create a Ceph profile @ cloudlab.us
Closing this. See http://tracker.ceph.com/issues/12317 David Galloway
07:23 PM Bug #12772 (Resolved): incerta installed with Centos 7.1 don't reboot
Closing this out as part of a bug scrub. If problem persists, you can reopen this issue.
Thanks!
David Galloway
06:59 PM Bug #14453 (In Progress): Reimage smithi to centos 7.2 (instead of centos 7.1)
smithis 003, 008, 015, 016, 017, 020, 026, 027, 028, 029 are running 7.2.
I just locked a few more that I will rei...
David Galloway
06:56 PM Bug #14424 (Resolved): disk replacements for the LRC
To replace a disk (from memory, may be missing a step or have wrong order):
* unweight the OSD if it's still viabl...
Dan Mick
06:51 PM Bug #12556 (Resolved): mira032 failing drives
Drives 1, 3, 5 replaced 7JAN2016 David Galloway
06:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
The defective cards were approved for RMA and delivered to SuperMicro on 12JAN2016.
FedEx tracking: 782146800750
...
David Galloway
06:45 PM Bug #12317 (Rejected): use AARCH64 instances from datacentred.co.uk
No longer needed now that we have our own arm64 boxes to test on: limata{001..004}
<dgalloway> sage, do we need th...
David Galloway
06:43 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
All failing drives were replaced.
The gitbuilders are running smoothly and Zack rebuilt sentry-db.
I'll track s...
David Galloway
06:41 PM Support #14430: need access to Sepia Lab
Ben,
I just run all my OpenVPN tunnels as a background service so I don't have much experience with setting this u...
David Galloway
06:31 PM Support #14502 (Resolved): access to Sepia Lab
Key added to gw
https://github.com/ceph/cookbook-gw/pull/4
Thomas verified connectivity
David Galloway
06:19 PM Bug #13652 (Resolved): mira118 has disk errors
Drives replaced on 20JAN2016 David Galloway
04:10 PM Bug #14514: vps leaked after teuthology run
The leaked vpses were from a variety of hosts:... Josh Durgin
04:03 PM Bug #14514 (Closed): vps leaked after teuthology run
vps failed to unlock long after the test suite completed.
When unlocking them manually it asked for ubuntu@mira043.f...
Orit Wasserman

01/25/2016

10:44 PM Support #14430: need access to Sepia Lab
I'm sorry, I really don't get what I'm supposed to do. I went into GUI, added an OpenVPN connect, importing it from ... Ben England
07:37 PM Support #14430: need access to Sepia Lab
Ben,
You should now have access to the sepia lab. Please restart the openvpn connection on your workstation to ve...
David Galloway
07:23 PM Support #14502 (Resolved): access to Sepia Lab
Accidentally wiped my /etc/ dir when upgrading workstation. Here's new VPN:
tserlin@annarbor DlKe+OWBPcFAQtWMUAH...
Thomas Serlin

01/22/2016

11:18 PM Feature #14479 (Resolved): Backups
Create host with public IP and also connected to LRC for SFTP and backups David Galloway
10:12 PM Support #14430: need access to Sepia Lab
yes that's right Ben. Dan Mick
08:26 PM Support #14430: need access to Sepia Lab
Sorry I don't follow instructions well, it was all there, is this what you want?
[root@bene-laptop sepia]# ./new-c...
Ben England
07:20 PM Support #14430 (In Progress): need access to Sepia Lab
Ben,
Step 4 should probably be clarified but we need the output from 'new-client USER$HOST'
https://ceph.github.i...
David Galloway
09:19 PM Bug #14478 (Resolved): mira089 MCE, bad processor?
ubuntu@teuthology:~$ ssh mira089
ssh: connect to host mira089 port 22: No route to host
Wondering if we have some...
Yuri Weinstein

01/21/2016

05:27 PM Bug #14455 (Resolved): mira112 has bad drives
ubuntu@mira112:~$ sudo /usr/libexec/smart.pl
4 of 8 drives failing/missing |
Drive 1 N.A.
Drive 2 N.A.
Drive 4 (...
Yuri Weinstein
04:12 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
Since we changed supported distros to centos 7.2 we need to bring smithi in line with that, otherwise tests wait for ... Yuri Weinstein
01:35 AM Bug #14432: mira095 got wedged
Nothing more in syslog or kern.log about the fault. Dan Mick
12:50 AM Bug #14432: mira095 got wedged
Maybe a hardware fault?... Dan Mick
01:17 AM Bug #14398 (Resolved): Install Centos 7.2 on some mira
Set up 7 CentOS 7.2 miras
$ tl --brief -a --machine-type mira --os-version 7.2
mira038.front.sepia.ceph.com loc...
David Galloway
12:02 AM Bug #14422 (Resolved): mira078 failing tests has bad disk
Drives replaced David Galloway
12:02 AM Bug #14404 (Resolved): mira117 failing tests has bad disk
Drives replaced David Galloway
12:02 AM Bug #14403 (Resolved): mira088 failing tests has bad disk
Drives replaced David Galloway

01/20/2016

05:30 PM Bug #14424: disk replacements for the LRC
All disks replaced last night David Galloway
02:42 AM Bug #14432 (Closed): mira095 got wedged
It disappeared while running http://pulpito.ceph.com/gregf-2016-01-18_18:01:09-fs-greg-fs-testing-118-1---basic-mira/... Greg Farnum
01:04 AM Support #14430 (Resolved): need access to Sepia Lab
Mark Nelson wants Joe Mario and I to run Ceph CBT and profile using perf "C2C" NUMA profiling tool, because of high-s... Ben England

01/19/2016

06:26 PM Bug #14424 (Resolved): disk replacements for the LRC
mira049/2
mira021/6
mira060/5
mira116/7
mira120/7 missing (not sdg, beware)
mira055/2 still emptying
Dan Mick
04:17 PM Bug #14422 (Resolved): mira078 failing tests has bad disk
ubuntu@mira078:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 335 uncorrect secto...
Yuri Weinstein

01/18/2016

09:45 PM Bug #14403: mira088 failing tests has bad disk
ubuntu@mira088:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 8 (sdh) has 121 uncorrect secto...
Yuri Weinstein
09:43 PM Bug #14403 (Resolved): mira088 failing tests has bad disk
Yuri Weinstein
09:45 PM Bug #14404: mira117 failing tests has bad disk
ubuntu@mira117:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 4 uncorrect sectors...
Yuri Weinstein
09:43 PM Bug #14404 (Resolved): mira117 failing tests has bad disk
Yuri Weinstein
08:55 PM Bug #11315 (Rejected): ceph-deploy suite in hammer (and giant) can no longer run on plana nodes
No longer have plana nodes David Galloway
08:54 PM Bug #12048 (Resolved): mira082 bad disk
System had its drives replaced JAN7 David Galloway
08:52 PM Bug #13147 (Resolved): mira092: SOL stuck, "power reset" has no apparent effect
System's been running jobs fine for a bit now. David Galloway
08:51 PM Bug #13696 (Resolved): mira077 has bad drives
This system had its bad drives replaced on JAN7. David Galloway
05:35 PM Bug #14398 (Resolved): Install Centos 7.2 on some mira
We started running rados on mira and some tests ask for centos 7.2 OS, so we need to reimage ~10% to accommodate this... Yuri Weinstein

01/15/2016

11:53 PM Support #14294: Infrastructure Host Drive replacements
senta02's RAID failed and its filesystem was broken beyond salvaging. We lost the following VMs:
* gitbuilder-hadoo...
David Galloway
10:06 PM Tasks #13085 (Closed): kill gitbuilder-ceph-tarball-precise-amd64-basic
done Dan Mick
10:06 PM Tasks #13086 (Resolved): kill gitbuilder-ceph-rpm-fedora20-amd64-basic
While cleaning up gitbuilders, noticed this one's already gone so closing ticket. David Galloway

01/14/2016

09:05 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml) David Galloway
06:26 PM Bug #14338 (In Progress): Incorrect IPs set in /etc/hosts on mira nodes
David Galloway
05:50 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
*sigh*... David Galloway
05:36 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy ... David Galloway
04:56 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
(Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alf...
Zack Cerza
04:47 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704...
Yuri Weinstein

01/13/2016

05:49 AM Tasks #12705 (Resolved): Abhishek Varshney access to the lab
Account should be added to teuthology machines, mira, and smithi. Please reopen/comment/email if not. Dan Mick
01:49 AM Tasks #12705 (In Progress): Abhishek Varshney access to the lab
OpenVPN access added; not sure if working yet. Adding account info now. Dan Mick
01:44 AM Tasks #11652 (Resolved): smithfarm access to the lab
Added new vpn secret. Nathan confirms a connection. Dan Mick

01/12/2016

05:49 PM Bug #14338 (New): Incorrect IPs set in /etc/hosts on mira nodes
David, can you take a look at mira061 configuration, pls?
And then close/resolve the ticket.
Yuri Weinstein
03:52 PM Bug #14338 (Closed): Incorrect IPs set in /etc/hosts on mira nodes
Samuel Just
12:12 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
This looks like the monitors can't resolve the mira061 address:... Alfredo Deza
01:28 AM Bug #11867 (Resolved): mira089 bad disk
All failed drives replaced David Galloway

01/11/2016

10:45 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
Run: http://pulpito.ceph.com/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/
Jobs: 18887, 18889
...
Yuri Weinstein
08:16 PM Bug #14336 (Resolved): All runs failing in ovh lab
https://github.com/ceph/teuthology/commit/1a74f929430ee1976a7696d837cc3edb407c7f36 Zack Cerza
07:59 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
Zack Cerza
07:51 PM Bug #14336 (Resolved): All runs failing in ovh lab
Zack Cerza
07:43 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
https://github.com/ceph/teuthology/pull/767 Zack Cerza
07:41 PM Bug #14336: All runs failing in ovh lab
... Zack Cerza
07:40 PM Bug #14336 (In Progress): All runs failing in ovh lab
Zack Cerza
05:05 PM Bug #14336 (Resolved): All runs failing in ovh lab
See last 2 days runs, here is an example:
Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-10_18:02:01...
Yuri Weinstein
10:11 AM Tasks #11652: smithfarm access to the lab
Hi Dan:
My previous sepia credentials were lost in global destruction. Could you please renew them?
# requesti...
Nathan Cutler
08:59 AM Tasks #11652 (In Progress): smithfarm access to the lab
@nathan could you please re-iterate the required information here to make it easier for Dan ? See http://ceph.github.... Loïc Dachary

01/08/2016

04:29 AM Support #14294: Infrastructure Host Drive replacements
senta01 rebuild complete
senta02 has problems with md1, investigating through rescue boot
senta03 rebuild complete
Dan Mick
02:50 AM Bug #13696 (In Progress): mira077 has bad drives
David Galloway

01/07/2016

11:03 PM Feature #14296 (Resolved): Create VPSHOST nagios hostgroup
David Galloway
10:55 PM Support #14294: Infrastructure Host Drive replacements
senta01 had drive 2 replaced and its Areca RAID is rebuilding automatically David Galloway
10:45 PM Support #14294: Infrastructure Host Drive replacements
senta03 has had sdb replaced
md1 is rebuilding
md127 needs sdb1 readded still
David Galloway
10:30 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
senta02 needs sdc replaced
senta02 has had sdd replaced and md1 is in process of recovering. md127 needs recovering...
David Galloway
10:29 PM Bug #12723 (Closed): megabug: all the dead disks in the test miras
Most of these have been replaced long ago, adn we have a new set
Dan Mick
09:21 PM Bug #14291 (Resolved): gw redundancy
We used to have a hot standby host, gw2, that was inaccessible but ready to take over gw's IP if an emergency happene... Dan Mick
09:18 PM Bug #14290 (Resolved): Add public IP to yan-zheng; make sure packages up to date, no passwords (a...
Basically restore yan-zheng as a bastion host for developers behind Great Firewall Dan Mick
07:39 AM Tasks #12705: Abhishek Varshney access to the lab
username: abhishekvrshny
email: abhishek.varshney@flipkart.com
SSH key: https://github.com/ceph/keys/blob/master/ss...
Abhishek Varshney
07:02 AM Tasks #12705: Abhishek Varshney access to the lab
It's not a private key; it's a hashed password. The secret stays with you. Here is fine. Note that there are four ... Dan Mick
06:36 AM Tasks #12705: Abhishek Varshney access to the lab
Dan: My key exists on https://github.com/ceph/keys and I have just completed http://ceph.github.io/sepia/adding_users... Abhishek Varshney
07:20 AM Support #14250 (Resolved): Requesting Lab Access
Dan Mick
07:20 AM Support #14250: Requesting Lab Access
I guess I'll close it. Dan Mick

01/06/2016

11:17 PM Bug #13147: mira092: SOL stuck, "power reset" has no apparent effect
Not sure of exact root cause but was able to re-image, nuke, and release on 22DEC2015. David Galloway
10:45 PM Bug #12555 (In Progress): mira083 stuck in raid bios on reboot
The RAID controller firmware hang has been resolved but 3 drives are missing.
I'm installing an OS to identify whi...
David Galloway
10:02 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
Here are today's findings/testing.
I took smithi003 with a known working NVMe card and tested both PCI slots. Con...
David Galloway
06:43 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
seems like it should to me, Ken
Dan Mick
06:33 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
(Should ceph.git's @admin/build-doc@ script be checking for the graphviz package?) Ken Dreyer
06:32 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
Sure, I've installed graphviz now.
Exact steps I did today:
# Log into the DHC OpenVPN (Dan Mick has documentat...
Ken Dreyer
06:12 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
Ken, i see you are using docs nodes to build the docs in https://github.com/ceph/ceph-build/pull/210. i am wondering ... Kefu Chai
12:46 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
if graphviz is installed on the gitbuilder of the doc, the dot should be rendered. Kefu Chai
06:42 PM Tasks #12705: Abhishek Varshney access to the lab
Abhishek: did you supply the information requested in http://ceph.github.io/sepia/adding_users/#requesting-lab-access... Dan Mick
02:33 PM Tasks #12705: Abhishek Varshney access to the lab
@Dan I think Abhishek did what was required of him at http://ceph.github.io/sepia/adding_users/ . I'm escalating this... Loïc Dachary
06:36 PM Bug #14268 (Resolved): logrotate failing on gitbuilders
I noticed a large stdout.log on the ceph-deb-trusty-amd64-basic gitbuilder.
After discussing with dmick it is pres...
David Galloway
05:24 AM Support #14250: Requesting Lab Access
I think this is taken care of; John, please close this when you agree. Dan Mick
02:02 AM Support #14250: Requesting Lab Access
https://github.com/ceph/keys/pull/10
https://github.com/ceph/ceph-sepia-secrets/pull/69
https://github.com/ceph/coo...
Dan Mick
12:45 AM Support #14250 (In Progress): Requesting Lab Access
Dan Mick
12:44 AM Support #14250 (Resolved): Requesting Lab Access
I'm requesting access to help Warren test some Java client code.
The login I would like to use is: jowilkin
SS...
John Wilkins
 

Also available in: Atom