Project

General

Profile

Activity

From 01/13/2016 to 02/11/2016

02/11/2016

11:18 PM Bug #14733: Could not fetch updated apt files
So...the upstream format of the i18n Index files has changed; I'm sure it's an error but we can put a hack into a loc... Dan Mick
04:24 PM Bug #14733: Could not fetch updated apt files
http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-02-11_02:00:01-rados-infernalis-distro-basic-openstack/
http:...
Yuri Weinstein
04:23 PM Bug #14733 (In Progress): Could not fetch updated apt files
... David Galloway
05:31 AM Bug #14733: Could not fetch updated apt files
http://pulpito.ceph.com/loic-2016-02-10_21:25:43-fs-infernalis---basic-multi/5065/ Loïc Dachary
05:03 AM Bug #14733 (Resolved): Could not fetch updated apt files
General failure of some suites today
* http://pulpito.ceph.com/loic-2016-02-10_20:32:58-rados-infernalis-backport...
Loïc Dachary
05:18 PM Bug #12032: activate beanstalkd logs
I set the following:... Zack Cerza
08:32 AM Support #14722: VPN Access to DELL Ceph Benchmark Systems
dmesser@flab mJhwtddvlOI/hkpAGnFr5w 13b92ea100bb995ec71f1d5f6309271c3fca47d20098f0bb0aa9e4906b62f092
Daniel Messer
04:25 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
Many occurrences at http://tracker.ceph.com/issues/14692 Loïc Dachary

02/10/2016

10:02 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
David Galloway
09:03 PM Support #14614: Add access to Sepia Lab for Joe Mario
Hi David:
That was it. Thank you for all your help.
Joe
Joe Mario
08:57 PM Support #14614: Add access to Sepia Lab for Joe Mario
Your user was set up as 'jmario' so you'd:... David Galloway
08:54 PM Support #14614: Add access to Sepia Lab for Joe Mario
Hi David:
I am now able to create the VPM and get to the login prompt for teuthology.front.sepia.ceph.com.
Thank ...
Joe Mario
08:59 PM Support #14704: Rack tala or saya
I think this box had 2x 1Gb uplinks and a 10Gb uplink and that's why we can't reach the actual OSes.
I put in a ti...
David Galloway
03:55 AM Support #14704: Rack tala or saya
tala{001..016} have been re-racked and their IPMI interfaces are accessible. I think some magic will be needed in Co... David Galloway
07:08 PM Support #14722: VPN Access to DELL Ceph Benchmark Systems
Daniel,
I need you to follow the steps to generate a hashed password.
https://ceph.github.io/sepia/adding_users...
David Galloway
01:46 PM Support #14722 (Resolved): VPN Access to DELL Ceph Benchmark Systems
Hi,
I am part of the technical team to support the benchmarking exercise we are doing on the 730xd servers which h...
Daniel Messer

02/09/2016

03:32 PM Bug #10798 (Resolved): mkfs.ext4 fails with "apparently in use by the system"
assuming this is fixed on that mira Sage Weil
12:51 AM Support #14704 (Resolved): Rack tala or saya
For 32-bit ARM testing David Galloway

02/08/2016

11:55 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
mira123 (formerly dubia001) is full of healthy 4TB drives
mira124 (formerly dubia002)'s disks are almost all bad
...
David Galloway
09:57 PM Bug #14548 (Can't reproduce): mira052 MCE
memtest passed. Will monitor host for future failures and troubleshoot further if needed.
Reinstalled and released.
David Galloway
06:27 PM Support #14614: Add access to Sepia Lab for Joe Mario
Joe,
I changed your user string on the server. Try again and let me know how things go.
David Galloway
04:54 PM Feature #14696 (Resolved): Migrate from PowerDNS to (probably) BIND with version control
David Galloway

02/06/2016

03:26 AM Tasks #11597 (Rejected): xinxinsh access to the lab
Loïc Dachary

02/05/2016

11:44 PM Tasks #14683 (In Progress): openSUSE Leap gitbuilder
Dan Mick
11:42 PM Tasks #14683 (Won't Fix): openSUSE Leap gitbuilder
Nathan Cutler
10:22 PM Feature #2057: sepia: serial console service
Propose we set up conserver for this: http://www.conserver.com/ David Galloway
10:19 PM Feature #3356 (Closed): "rescue" network got replaced by serial console access, remove from diagr...
These networks are all gone...alll gonnnne... Dan Mick
10:12 PM Feature #10087 (Duplicate): support provisioning with usernames other than 'ubuntu'
#10655 Zack Cerza
10:12 PM Bug #10472: ccache misconfigured on gitbuilders
maybe so?...
Dan Mick
10:10 PM Bug #10675 (Resolved): failures related to clock skew
We are using more reliable NTP servers now and believe this to be resolved
Zack Cerza
10:09 PM Bug #10751 (Resolved): move setup/build release scripts out of jenkins
Dan Mick
10:09 PM Bug #10760 (Resolved): NTP errors, message from mon.0 was stamped 0.500977s in the future
We are using more reliable NTP servers now and believe this to be resolved Zack Cerza
10:08 PM Tasks #10932 (Won't Fix): set up local ntp server(s) for teuthology clusters
NTP service has been good lately; defer until we have more problems Dan Mick
10:05 PM Bug #11060 (Can't reproduce): ERROR: execute[add release gpg key to apt] ... summary shows wget f...
This is so old and everything's changed that the suggestion is "reopen if it happens again" Dan Mick
10:05 PM Bug #11168 (Resolved): mira018 has bad disks
Drives replaced 28JAN2016
VPSes are in use
David Galloway
09:59 PM Tasks #11597: xinxinsh access to the lab
Loic, is this still needed? It's pretty old Dan Mick
09:58 PM Feature #11646: add apama00[12] to sepia lab ceph cluster 
Reinstall and re-add to cluster David Galloway
09:54 PM Bug #11782 (Can't reproduce): lab clocks not reliably synced
If it gets worse again, we can try adding clock.sync to internal tasks, but it seems to have gotten better somehow. Dan Mick
09:52 PM Feature #12024 (Duplicate): Half of the VPS(virtual machine) nodes can use IPv6
Closing this in favor of http://tracker.ceph.com/issues/14680 David Galloway
09:50 PM Bug #12660 (Resolved): NFS exportfs failing (missing kmod?) on centos7 host
Assuming the PR resolves the issue; please reopen if that's not the case Zack Cerza
09:49 PM Bug #13191 (Won't Fix): CentOS 7 multipath test fail because libdevmapper version must be >= 1.02.89
Dan Mick
09:49 PM Bug #13191: CentOS 7 multipath test fail because libdevmapper version must be >= 1.02.89
7.2 is available and should be used for tests
Dan Mick
09:46 PM Bug #13212: mira121 fails in mkfs -f, perhaps because mpath devices left over
Check mira088 and 103 to see if CentOS install will fail and verify Areca RAID Controller firmware update fixes issue. David Galloway
09:40 PM Bug #13269 (Can't reproduce): GPG key retrieval failed on CentOS 6.5, 7.0 with IPv6
Closing this in favor of http://tracker.ceph.com/issues/14680 David Galloway
09:40 PM Feature #14680 (New): Enable IPv6 support in Sepia lab
David Galloway
09:31 PM Bug #13763 (In Progress): gitbuilder seems to have bad sudo setup
This gitbuilder now has sudo problems; look into fabfile's sudo setup Dan Mick
09:23 PM Bug #14268 (In Progress): logrotate failing on gitbuilders
I had looked at fixing this with a 'user' in logrotate.conf, but I think it has to be permissions, and I'm not sure i... Dan Mick
09:13 PM Bug #14290: Add public IP to yan-zheng; make sure packages up to date, no passwords (and passwd -...
To be created on incoming new infrastructure hardware David Galloway
09:11 PM Bug #14290: Add public IP to yan-zheng; make sure packages up to date, no passwords (and passwd -...
....or something. Developers are using rexNNN from lab.lax.redhat.com for the moment Dan Mick
09:07 PM Bug #14478: mira089 MCE, bad processor?
"reboot and nuke" Dan Mick
09:06 PM Feature #14479: Backups
Asked IT what they can provide as far as off-site backups: INC0359452 David Galloway
09:05 PM Bug #14514 (Closed): vps leaked after teuthology run
This was probably a collision between vmhost maintenance and old stale jobs. If it happens again, reopen or rereport.
Dan Mick
09:03 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
dubia003 is jenkins, and should rename
dubia004 is pulpito and paddles, and should be renamed
Dan Mick
09:01 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
PR is merged Zack Cerza
07:56 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
https://github.com/ceph/ceph-sepia-secrets/pull/77 David Galloway
08:52 PM Bug #14612 (Duplicate): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:infe...
#14639 Zack Cerza
08:51 PM Support #14614 (In Progress): Add access to Sepia Lab for Joe Mario
Dan Mick
08:39 PM Support #14614: Add access to Sepia Lab for Joe Mario
Hi David:
I am getting AUTH_FAILED errors, as appended below.
To restart clean, I've now rerun "new-client".
Th...
Joe Mario
07:27 PM Support #14614: Add access to Sepia Lab for Joe Mario
Hi Joe,
Access has been set up for you to access the Sepia lab.
Please verify you can connect to the VPN and ss...
David Galloway
08:13 PM Feature #3034 (Rejected): sepia: capture & monitor ipmitool sel log (ECC RAM errors etc): vercoi,...
No longer have Dell systems David Galloway
08:03 PM Bug #11528 (Resolved): mira015 has bad disks 1,3,6
Drives were replaced, system reimaged and VPSes are functional David Galloway
08:00 PM Bug #11947 (Resolved): mira029: drives 1,2,3,4 failing (including root)
Drives replaced 28JAN2016. System reimaged and VPSes are functional David Galloway
07:59 PM Support #13344 (Resolved): Request access to sepia for running teuthology
David Galloway
03:56 PM Support #13344: Request access to sepia for running teuthology
Okay, I confirm that I can connect to the VPN and ssh in. Daniel Gryniewicz
07:59 PM Feature #12679 (Resolved): centos 7.1 cloud-init image
CentOS 6.3 thru 7.2 appear to be available through downburst David Galloway
07:24 PM Bug #14627 (Resolved): RCA: 3FEB2016 senta01 kernel panic
senta01's volumes were found in a degraded state last night. I believe the crash of this system was due to drive fai... David Galloway
05:15 PM Bug #14675 (Resolved): mira110 has missing drive
ubuntu@mira110:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 6 (sdf) has 3 pending sectors
Yuri Weinstein
05:12 PM Bug #14674 (Resolved): mira097 has bad drives
ubuntu@mira097:~$ sudo /usr/libexec/smart.pl
4 of 8 drives failing/missing |
Drive 2 (sdb) has 5 pending sectors
...
Yuri Weinstein
04:10 PM Bug #14664 (Won't Fix): valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
let's just wait it out Loïc Dachary
09:00 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
Loïc Dachary
08:59 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
Valgrind fix : https://bugs.kde.org/show_bug.cgi?id=357833 Loïc Dachary
08:59 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
Happens with > 4.4 kernels ( https://lkml.org/lkml/2016/1/25/345 ) Loïc Dachary
08:56 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
See https://bugzilla.redhat.com/show_bug.cgi?id=1301093 for a similar bug Loïc Dachary
07:50 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
* http://pulpito.ceph.com/loic-2016-02-04_00:40:51-rgw-infernalis-backports---basic-multi/6184/
* http://pulpito.cep...
Loïc Dachary
06:23 AM Bug #14664 (Won't Fix): valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
http://pulpito.ceph.com/loic-2016-02-04_00:33:20-rados-infernalis-backports---basic-multi/6141/
CommandFailedError...
Loïc Dachary
01:20 AM Bug #14661: mira104 has missing drive
also something weird going on with its networking; it can't seem to reach my vpn client, but it can reach other miras... Dan Mick
01:08 AM Bug #14661 (Resolved): mira104 has missing drive
ubuntu@mira104:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 91 uncorrect sector
Yuri Weinstein

02/04/2016

10:06 PM Bug #14630 (In Progress): mira020 root drive failure, BMC issues?
Filesystem errors found on /dev/sda1.
Ran fsck on root partition and marked VPSes back up. Drive 4 still needs re...
David Galloway
09:01 PM Bug #14612 (In Progress): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:in...
http://sentry.ceph.com/sepia/teuthology/issues/6/events/ David Galloway
08:23 PM Bug #14648 (Resolved): safely transition hosts to UTC
both teuthology and paddles will need to change. probably also pulpito. Zack Cerza
06:14 PM Bug #14644 (Duplicate): "Could not resolve host: gitbuilder.ceph.com" in rados-jewel-distro-basi...
#14639 Zack Cerza
05:11 PM Bug #14644 (Duplicate): "Could not resolve host: gitbuilder.ceph.com" in rados-jewel-distro-basi...
See several in late runs
Run: http://pulpito.ceph.com/teuthology-2016-02-02_22:00:01-rados-jewel-distro-basic-smithi...
Yuri Weinstein
06:12 PM Bug #14639: failing gitbuilder connections from mira (DNS lookup errors)
I see 8 instances of this as of now:
http://sentry.ceph.com/sepia/teuthology/issues/44/
Dropping priority
Zack Cerza
07:19 AM Bug #14639 (Resolved): failing gitbuilder connections from mira (DNS lookup errors)
http://pulpito.ceph.com/gregf-2016-02-03_09:19:17-fs-greg-fs-testing-23---basic-mira/4628/... Greg Farnum
04:43 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
PR to add a graphviz check to @admin/build-doc@: https://github.com/ceph/ceph/pull/7522 Ken Dreyer
01:31 AM Support #14614 (In Progress): Add access to Sepia Lab for Joe Mario
David Galloway
01:20 AM Support #13344: Request access to sepia for running teuthology
David Galloway
01:19 AM Support #13344: Request access to sepia for running teuthology
Dan,
You've been added to the OpenVPN server and an account has been created on the teuthology host.
Please ver...
David Galloway

02/03/2016

11:18 PM Bug #14152 (Resolved): the dot graph is not rendered in http://docs.ceph.com/
Looks like Ken resolved this. David Galloway
11:10 PM Support #14531 (Resolved): RCA: download.ceph.com outage 27JAN2016
download.ceph.com is now being served by nginx instead of apache2.
SSL certificate security was also improved in t...
David Galloway
05:54 PM Bug #14630 (Resolved): mira020 root drive failure, BMC issues?
mira020's root drive appears to be failing or has failed. I/O errors upon checking BMC KVM.
Marked VPSes and mira...
David Galloway
02:09 PM Bug #14627 (Resolved): RCA: 3FEB2016 senta01 kernel panic
senta01 became unresponsive around 5AM UTC.
SOL was unresponsive and BMC web UI showed Kernel Panic and Read-error...
David Galloway
02:44 AM Bug #14619: mira039 missing drive 5
it's also behaving badly on reboot; still spending a lot of time in RAID BIOS. Maybe the drive is failed in such a w... Dan Mick
01:48 AM Bug #14619: mira039 missing drive 5
The upshot is: mira036 was not involved; mira039 was unresponsive, and on reboot spent a long time determining that d... Dan Mick
01:13 AM Bug #14619 (Resolved): mira039 missing drive 5
I mira039 was listed as 'coluld not nuke' on teh stale nodes list:... Yuri Weinstein

02/02/2016

10:47 PM Support #14614: Add access to Sepia Lab for Joe Mario
I reran sepia/new-client. The new output is:
# sepia/new-client joemario@blazingsaddles
Please submit the foll...
Joe Mario
07:29 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
Per a conversation with Ben England and Mark Nelson, I'd like to run a development feature we're adding to the perf t... Joe Mario
05:52 PM Bug #14612 (Duplicate): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:infe...
Run: http://pulpito.ceph.com/teuthology-2016-02-01_17:10:11-upgrade:infernalis-infernalis-distro-basic-vps/
Job: 151...
Yuri Weinstein
03:20 AM Bug #10174 (Won't Fix): saya037 unresponsive
sayas are currently not used. I'm sure I'll run into this if we end up reusing the systems in the future. David Galloway
03:16 AM Bug #13131 (Resolved): mira078 has a Read-Only FS, boots you out of ssh instantly
Cleaning up the queue.. This system appears to have been running jobs successfully lately, has had drives replaced an... David Galloway
03:11 AM Feature #11785 (Closed): install radian memory NVRAM cards in some machines
Assuming this was already taken care of or is no longer needed.
Please re-open if that's not the case.
David Galloway
03:09 AM Bug #14576 (Resolved): mira112 stuck at reboot
HDD boot order fixed David Galloway
03:05 AM Bug #13836 (Can't reproduce): vpm103.stderr:bash: git: command not found
Due to the nature of VPSes and the age of this ticket, this is near impossible to troubleshoot. The system has been ... David Galloway
02:54 AM Bug #12555 (Resolved): mira083 stuck in raid bios on reboot
Replaced drives and reimaged 7JAN2016 David Galloway
02:45 AM Support #14430 (Resolved): need access to Sepia Lab
Access granted and confirmed David Galloway
12:35 AM Bug #14157 (Resolved): smithi004, smithi005, smithi007, smithi055 NVMe cards bad
David Galloway wrote:
> Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked Super...
David Galloway

02/01/2016

10:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked SuperMicro expedite a shipment... David Galloway
10:31 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
There were files under vpm104's mountpoint consuming space from root. Embarrassing that it took so long to figure th... Dan Mick

01/31/2016

10:14 PM Support #14430: need access to Sepia Lab
That worked! I can now ssh to bengland@teuthology.front.sepia.ceph.com. Thanks for the help logging in. Ben England

01/30/2016

08:07 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
dubia001 can be repurposed from openstack if it hasn't been reused already. I was the one using it for openstack test... Josh Durgin
07:37 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
dubia001 appears to be running openstack. It's unreachable but ethtool via SOL indicates its NIC isn't connected. W... David Galloway
05:47 PM Bug #14576 (Resolved): mira112 stuck at reboot
It was at stale state and stuck at nuke/stale
could not ssl and at the end ot sol power cycle got stuck at:...
Yuri Weinstein
12:22 AM Bug #13282 (In Progress): ssh from my vm to centos vpm machines are slow.
I might have been a bit hasty with tat last update.
ssh to Centos still seems to take a while.
Centos ssh:
</p...
Anonymous

01/29/2016

09:23 PM Fix #11889 (Resolved): ubuntu@tracker.front.sepia.ceph.com lingering git fetch-pack
Redmine and the underlying OS were updated when tracker got moved to DHC.
I'm going to close this ticket but if yo...
David Galloway
09:00 PM Support #13863 (Closed): Ask For Access To Sepia Lab Teuthology Gate
please find us in #sepia on irc.oftc.net to discuss! note that the lab capacity is pretty well utilized, so if you h... Sage Weil
08:35 PM Bug #11865 (Resolved): mira041 bad sdc sdd
Drives replaced 19JAN2016 David Galloway
08:33 PM Bug #11121 (Resolved): mira054 sdb sdc sdd sde sdf bad
This system died. Its disks were moved to mira021 which took mira054's place in the long running cluster. David Galloway
08:30 PM Feature #11496 (Rejected): pulpito-rdu needs a proxy
pulpito-rdu decomissioned David Galloway
07:51 PM Bug #14455 (Resolved): mira112 has bad drives
Replaced this host's drives last night. David Galloway
07:28 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
All smithi nodes that had 7.1 installed have been reinstalled with 7.2
There are 3 machines I still have locked th...
David Galloway
12:38 AM Bug #14541: "No space left on device" when locking vpm097
I can't figure this one out. By all accounts there is only about 15GB used on /, but df shows 628GB used. Normally ... Dan Mick

01/28/2016

11:32 PM Bug #14548 (Can't reproduce): mira052 MCE
... David Galloway
10:28 PM Bug #14546 (Resolved): mira033 kernel panic from MCE
... David Galloway
05:19 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-27_18:49:02-rados-hammer-distro-basic-vps/46886/teutholo... Yuri Weinstein

01/27/2016

09:24 PM Support #14531 (Resolved): RCA: download.ceph.com outage 27JAN2016
Wanted to get this documented in a ticket until we have a better way of documenting outage RCAs.
Over the past cou...
David Galloway
07:50 PM Bug #13282 (Resolved): ssh from my vm to centos vpm machines are slow.
Seems to be better. Anonymous
04:26 PM Cleanup #14528 (Resolved): Track down usage and purpose of mira{123..126} aka dubia{001..004}
A couple of these are currently being used for infrastucture but at least one of them can probably be nuked and added... David Galloway

01/26/2016

08:53 PM Bug #14214 (In Progress): mira115 has bad disks (marked down)
Re-opening this. Still showing up as drives missing. David Galloway
06:20 PM Bug #14214 (Resolved): mira115 has bad disks (marked down)
Drives replaced on 19JAN2016 David Galloway
07:32 PM Bug #14432 (Closed): mira095 got wedged
Without more logs (or bandwidth) to investigate further, not a lot more I can do to say for sure whether this is a ha... David Galloway
07:25 PM Bug #12203 (Closed): create a Ceph profile @ cloudlab.us
Closing this. See http://tracker.ceph.com/issues/12317 David Galloway
07:23 PM Bug #12772 (Resolved): incerta installed with Centos 7.1 don't reboot
Closing this out as part of a bug scrub. If problem persists, you can reopen this issue.
Thanks!
David Galloway
06:59 PM Bug #14453 (In Progress): Reimage smithi to centos 7.2 (instead of centos 7.1)
smithis 003, 008, 015, 016, 017, 020, 026, 027, 028, 029 are running 7.2.
I just locked a few more that I will rei...
David Galloway
06:56 PM Bug #14424 (Resolved): disk replacements for the LRC
To replace a disk (from memory, may be missing a step or have wrong order):
* unweight the OSD if it's still viabl...
Dan Mick
06:51 PM Bug #12556 (Resolved): mira032 failing drives
Drives 1, 3, 5 replaced 7JAN2016 David Galloway
06:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
The defective cards were approved for RMA and delivered to SuperMicro on 12JAN2016.
FedEx tracking: 782146800750
...
David Galloway
06:45 PM Bug #12317 (Rejected): use AARCH64 instances from datacentred.co.uk
No longer needed now that we have our own arm64 boxes to test on: limata{001..004}
<dgalloway> sage, do we need th...
David Galloway
06:43 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
All failing drives were replaced.
The gitbuilders are running smoothly and Zack rebuilt sentry-db.
I'll track s...
David Galloway
06:41 PM Support #14430: need access to Sepia Lab
Ben,
I just run all my OpenVPN tunnels as a background service so I don't have much experience with setting this u...
David Galloway
06:31 PM Support #14502 (Resolved): access to Sepia Lab
Key added to gw
https://github.com/ceph/cookbook-gw/pull/4
Thomas verified connectivity
David Galloway
06:19 PM Bug #13652 (Resolved): mira118 has disk errors
Drives replaced on 20JAN2016 David Galloway
04:10 PM Bug #14514: vps leaked after teuthology run
The leaked vpses were from a variety of hosts:... Josh Durgin
04:03 PM Bug #14514 (Closed): vps leaked after teuthology run
vps failed to unlock long after the test suite completed.
When unlocking them manually it asked for ubuntu@mira043.f...
Orit Wasserman

01/25/2016

10:44 PM Support #14430: need access to Sepia Lab
I'm sorry, I really don't get what I'm supposed to do. I went into GUI, added an OpenVPN connect, importing it from ... Ben England
07:37 PM Support #14430: need access to Sepia Lab
Ben,
You should now have access to the sepia lab. Please restart the openvpn connection on your workstation to ve...
David Galloway
07:23 PM Support #14502 (Resolved): access to Sepia Lab
Accidentally wiped my /etc/ dir when upgrading workstation. Here's new VPN:
tserlin@annarbor DlKe+OWBPcFAQtWMUAH...
Thomas Serlin

01/22/2016

11:18 PM Feature #14479 (Resolved): Backups
Create host with public IP and also connected to LRC for SFTP and backups David Galloway
10:12 PM Support #14430: need access to Sepia Lab
yes that's right Ben. Dan Mick
08:26 PM Support #14430: need access to Sepia Lab
Sorry I don't follow instructions well, it was all there, is this what you want?
[root@bene-laptop sepia]# ./new-c...
Ben England
07:20 PM Support #14430 (In Progress): need access to Sepia Lab
Ben,
Step 4 should probably be clarified but we need the output from 'new-client USER$HOST'
https://ceph.github.i...
David Galloway
09:19 PM Bug #14478 (Resolved): mira089 MCE, bad processor?
ubuntu@teuthology:~$ ssh mira089
ssh: connect to host mira089 port 22: No route to host
Wondering if we have some...
Yuri Weinstein

01/21/2016

05:27 PM Bug #14455 (Resolved): mira112 has bad drives
ubuntu@mira112:~$ sudo /usr/libexec/smart.pl
4 of 8 drives failing/missing |
Drive 1 N.A.
Drive 2 N.A.
Drive 4 (...
Yuri Weinstein
04:12 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
Since we changed supported distros to centos 7.2 we need to bring smithi in line with that, otherwise tests wait for ... Yuri Weinstein
01:35 AM Bug #14432: mira095 got wedged
Nothing more in syslog or kern.log about the fault. Dan Mick
12:50 AM Bug #14432: mira095 got wedged
Maybe a hardware fault?... Dan Mick
01:17 AM Bug #14398 (Resolved): Install Centos 7.2 on some mira
Set up 7 CentOS 7.2 miras
$ tl --brief -a --machine-type mira --os-version 7.2
mira038.front.sepia.ceph.com loc...
David Galloway
12:02 AM Bug #14422 (Resolved): mira078 failing tests has bad disk
Drives replaced David Galloway
12:02 AM Bug #14404 (Resolved): mira117 failing tests has bad disk
Drives replaced David Galloway
12:02 AM Bug #14403 (Resolved): mira088 failing tests has bad disk
Drives replaced David Galloway

01/20/2016

05:30 PM Bug #14424: disk replacements for the LRC
All disks replaced last night David Galloway
02:42 AM Bug #14432 (Closed): mira095 got wedged
It disappeared while running http://pulpito.ceph.com/gregf-2016-01-18_18:01:09-fs-greg-fs-testing-118-1---basic-mira/... Greg Farnum
01:04 AM Support #14430 (Resolved): need access to Sepia Lab
Mark Nelson wants Joe Mario and I to run Ceph CBT and profile using perf "C2C" NUMA profiling tool, because of high-s... Ben England

01/19/2016

06:26 PM Bug #14424 (Resolved): disk replacements for the LRC
mira049/2
mira021/6
mira060/5
mira116/7
mira120/7 missing (not sdg, beware)
mira055/2 still emptying
Dan Mick
04:17 PM Bug #14422 (Resolved): mira078 failing tests has bad disk
ubuntu@mira078:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 335 uncorrect secto...
Yuri Weinstein

01/18/2016

09:45 PM Bug #14403: mira088 failing tests has bad disk
ubuntu@mira088:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 8 (sdh) has 121 uncorrect secto...
Yuri Weinstein
09:43 PM Bug #14403 (Resolved): mira088 failing tests has bad disk
Yuri Weinstein
09:45 PM Bug #14404: mira117 failing tests has bad disk
ubuntu@mira117:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 4 uncorrect sectors...
Yuri Weinstein
09:43 PM Bug #14404 (Resolved): mira117 failing tests has bad disk
Yuri Weinstein
08:55 PM Bug #11315 (Rejected): ceph-deploy suite in hammer (and giant) can no longer run on plana nodes
No longer have plana nodes David Galloway
08:54 PM Bug #12048 (Resolved): mira082 bad disk
System had its drives replaced JAN7 David Galloway
08:52 PM Bug #13147 (Resolved): mira092: SOL stuck, "power reset" has no apparent effect
System's been running jobs fine for a bit now. David Galloway
08:51 PM Bug #13696 (Resolved): mira077 has bad drives
This system had its bad drives replaced on JAN7. David Galloway
05:35 PM Bug #14398 (Resolved): Install Centos 7.2 on some mira
We started running rados on mira and some tests ask for centos 7.2 OS, so we need to reimage ~10% to accommodate this... Yuri Weinstein

01/15/2016

11:53 PM Support #14294: Infrastructure Host Drive replacements
senta02's RAID failed and its filesystem was broken beyond salvaging. We lost the following VMs:
* gitbuilder-hadoo...
David Galloway
10:06 PM Tasks #13085 (Closed): kill gitbuilder-ceph-tarball-precise-amd64-basic
done Dan Mick
10:06 PM Tasks #13086 (Resolved): kill gitbuilder-ceph-rpm-fedora20-amd64-basic
While cleaning up gitbuilders, noticed this one's already gone so closing ticket. David Galloway

01/14/2016

09:05 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml) David Galloway
06:26 PM Bug #14338 (In Progress): Incorrect IPs set in /etc/hosts on mira nodes
David Galloway
05:50 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
*sigh*... David Galloway
05:36 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy ... David Galloway
04:56 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
(Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alf...
Zack Cerza
04:47 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704...
Yuri Weinstein

01/13/2016

05:49 AM Tasks #12705 (Resolved): Abhishek Varshney access to the lab
Account should be added to teuthology machines, mira, and smithi. Please reopen/comment/email if not. Dan Mick
01:49 AM Tasks #12705 (In Progress): Abhishek Varshney access to the lab
OpenVPN access added; not sure if working yet. Adding account info now. Dan Mick
01:44 AM Tasks #11652 (Resolved): smithfarm access to the lab
Added new vpn secret. Nathan confirms a connection. Dan Mick
 

Also available in: Atom