Activity
From 01/13/2016 to 02/11/2016
02/11/2016
- 11:18 PM Bug #14733: Could not fetch updated apt files
- So...the upstream format of the i18n Index files has changed; I'm sure it's an error but we can put a hack into a loc...
- 04:24 PM Bug #14733: Could not fetch updated apt files
- http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-02-11_02:00:01-rados-infernalis-distro-basic-openstack/
http:... - 04:23 PM Bug #14733 (In Progress): Could not fetch updated apt files
- ...
- 05:31 AM Bug #14733: Could not fetch updated apt files
- http://pulpito.ceph.com/loic-2016-02-10_21:25:43-fs-infernalis---basic-multi/5065/
- 05:03 AM Bug #14733 (Resolved): Could not fetch updated apt files
- General failure of some suites today
* http://pulpito.ceph.com/loic-2016-02-10_20:32:58-rados-infernalis-backport... - 05:18 PM Bug #12032: activate beanstalkd logs
- I set the following:...
- 08:32 AM Support #14722: VPN Access to DELL Ceph Benchmark Systems
- dmesser@flab mJhwtddvlOI/hkpAGnFr5w 13b92ea100bb995ec71f1d5f6309271c3fca47d20098f0bb0aa9e4906b62f092
- 04:25 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- Many occurrences at http://tracker.ceph.com/issues/14692
02/10/2016
- 10:02 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
- 09:03 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Hi David:
That was it. Thank you for all your help.
Joe - 08:57 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Your user was set up as 'jmario' so you'd:...
- 08:54 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Hi David:
I am now able to create the VPM and get to the login prompt for teuthology.front.sepia.ceph.com.
Thank ... - 08:59 PM Support #14704: Rack tala or saya
- I think this box had 2x 1Gb uplinks and a 10Gb uplink and that's why we can't reach the actual OSes.
I put in a ti... - 03:55 AM Support #14704: Rack tala or saya
- tala{001..016} have been re-racked and their IPMI interfaces are accessible. I think some magic will be needed in Co...
- 07:08 PM Support #14722: VPN Access to DELL Ceph Benchmark Systems
- Daniel,
I need you to follow the steps to generate a hashed password.
https://ceph.github.io/sepia/adding_users... - 01:46 PM Support #14722 (Resolved): VPN Access to DELL Ceph Benchmark Systems
- Hi,
I am part of the technical team to support the benchmarking exercise we are doing on the 730xd servers which h...
02/09/2016
- 03:32 PM Bug #10798 (Resolved): mkfs.ext4 fails with "apparently in use by the system"
- assuming this is fixed on that mira
- 12:51 AM Support #14704 (Resolved): Rack tala or saya
- For 32-bit ARM testing
02/08/2016
- 11:55 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- mira123 (formerly dubia001) is full of healthy 4TB drives
mira124 (formerly dubia002)'s disks are almost all bad
... - 09:57 PM Bug #14548 (Can't reproduce): mira052 MCE
- memtest passed. Will monitor host for future failures and troubleshoot further if needed.
Reinstalled and released. - 06:27 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Joe,
I changed your user string on the server. Try again and let me know how things go. - 04:54 PM Feature #14696 (Resolved): Migrate from PowerDNS to (probably) BIND with version control
02/06/2016
02/05/2016
- 11:44 PM Tasks #14683 (In Progress): openSUSE Leap gitbuilder
- 11:42 PM Tasks #14683 (Won't Fix): openSUSE Leap gitbuilder
- 10:22 PM Feature #2057: sepia: serial console service
- Propose we set up conserver for this: http://www.conserver.com/
- 10:19 PM Feature #3356 (Closed): "rescue" network got replaced by serial console access, remove from diagr...
- These networks are all gone...alll gonnnne...
- 10:12 PM Feature #10087 (Duplicate): support provisioning with usernames other than 'ubuntu'
- #10655
- 10:12 PM Bug #10472: ccache misconfigured on gitbuilders
- maybe so?...
- 10:10 PM Bug #10675 (Resolved): failures related to clock skew
- We are using more reliable NTP servers now and believe this to be resolved
- 10:09 PM Bug #10751 (Resolved): move setup/build release scripts out of jenkins
- 10:09 PM Bug #10760 (Resolved): NTP errors, message from mon.0 was stamped 0.500977s in the future
- We are using more reliable NTP servers now and believe this to be resolved
- 10:08 PM Tasks #10932 (Won't Fix): set up local ntp server(s) for teuthology clusters
- NTP service has been good lately; defer until we have more problems
- 10:05 PM Bug #11060 (Can't reproduce): ERROR: execute[add release gpg key to apt] ... summary shows wget f...
- This is so old and everything's changed that the suggestion is "reopen if it happens again"
- 10:05 PM Bug #11168 (Resolved): mira018 has bad disks
- Drives replaced 28JAN2016
VPSes are in use - 09:59 PM Tasks #11597: xinxinsh access to the lab
- Loic, is this still needed? It's pretty old
- 09:58 PM Feature #11646: add apama00[12] to sepia lab ceph cluster
- Reinstall and re-add to cluster
- 09:54 PM Bug #11782 (Can't reproduce): lab clocks not reliably synced
- If it gets worse again, we can try adding clock.sync to internal tasks, but it seems to have gotten better somehow.
- 09:52 PM Feature #12024 (Duplicate): Half of the VPS(virtual machine) nodes can use IPv6
- Closing this in favor of http://tracker.ceph.com/issues/14680
- 09:50 PM Bug #12660 (Resolved): NFS exportfs failing (missing kmod?) on centos7 host
- Assuming the PR resolves the issue; please reopen if that's not the case
- 09:49 PM Bug #13191 (Won't Fix): CentOS 7 multipath test fail because libdevmapper version must be >= 1.02.89
- 09:49 PM Bug #13191: CentOS 7 multipath test fail because libdevmapper version must be >= 1.02.89
- 7.2 is available and should be used for tests
- 09:46 PM Bug #13212: mira121 fails in mkfs -f, perhaps because mpath devices left over
- Check mira088 and 103 to see if CentOS install will fail and verify Areca RAID Controller firmware update fixes issue.
- 09:40 PM Bug #13269 (Can't reproduce): GPG key retrieval failed on CentOS 6.5, 7.0 with IPv6
- Closing this in favor of http://tracker.ceph.com/issues/14680
- 09:40 PM Feature #14680 (New): Enable IPv6 support in Sepia lab
- 09:31 PM Bug #13763 (In Progress): gitbuilder seems to have bad sudo setup
- This gitbuilder now has sudo problems; look into fabfile's sudo setup
- 09:23 PM Bug #14268 (In Progress): logrotate failing on gitbuilders
- I had looked at fixing this with a 'user' in logrotate.conf, but I think it has to be permissions, and I'm not sure i...
- 09:13 PM Bug #14290: Add public IP to yan-zheng; make sure packages up to date, no passwords (and passwd -...
- To be created on incoming new infrastructure hardware
- 09:11 PM Bug #14290: Add public IP to yan-zheng; make sure packages up to date, no passwords (and passwd -...
- ....or something. Developers are using rexNNN from lab.lax.redhat.com for the moment
- 09:07 PM Bug #14478: mira089 MCE, bad processor?
- "reboot and nuke"
- 09:06 PM Feature #14479: Backups
- Asked IT what they can provide as far as off-site backups: INC0359452
- 09:05 PM Bug #14514 (Closed): vps leaked after teuthology run
- This was probably a collision between vmhost maintenance and old stale jobs. If it happens again, reopen or rereport.
- 09:03 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- dubia003 is jenkins, and should rename
dubia004 is pulpito and paddles, and should be renamed - 09:01 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- PR is merged
- 07:56 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- https://github.com/ceph/ceph-sepia-secrets/pull/77
- 08:52 PM Bug #14612 (Duplicate): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:infe...
- #14639
- 08:51 PM Support #14614 (In Progress): Add access to Sepia Lab for Joe Mario
- 08:39 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Hi David:
I am getting AUTH_FAILED errors, as appended below.
To restart clean, I've now rerun "new-client".
Th... - 07:27 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Hi Joe,
Access has been set up for you to access the Sepia lab.
Please verify you can connect to the VPN and ss... - 08:13 PM Feature #3034 (Rejected): sepia: capture & monitor ipmitool sel log (ECC RAM errors etc): vercoi,...
- No longer have Dell systems
- 08:03 PM Bug #11528 (Resolved): mira015 has bad disks 1,3,6
- Drives were replaced, system reimaged and VPSes are functional
- 08:00 PM Bug #11947 (Resolved): mira029: drives 1,2,3,4 failing (including root)
- Drives replaced 28JAN2016. System reimaged and VPSes are functional
- 07:59 PM Support #13344 (Resolved): Request access to sepia for running teuthology
- 03:56 PM Support #13344: Request access to sepia for running teuthology
- Okay, I confirm that I can connect to the VPN and ssh in.
- 07:59 PM Feature #12679 (Resolved): centos 7.1 cloud-init image
- CentOS 6.3 thru 7.2 appear to be available through downburst
- 07:24 PM Bug #14627 (Resolved): RCA: 3FEB2016 senta01 kernel panic
- senta01's volumes were found in a degraded state last night. I believe the crash of this system was due to drive fai...
- 05:15 PM Bug #14675 (Resolved): mira110 has missing drive
- ubuntu@mira110:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 6 (sdf) has 3 pending sectors - 05:12 PM Bug #14674 (Resolved): mira097 has bad drives
- ubuntu@mira097:~$ sudo /usr/libexec/smart.pl
4 of 8 drives failing/missing |
Drive 2 (sdb) has 5 pending sectors
... - 04:10 PM Bug #14664 (Won't Fix): valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- let's just wait it out
- 09:00 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- 08:59 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- Valgrind fix : https://bugs.kde.org/show_bug.cgi?id=357833
- 08:59 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- Happens with > 4.4 kernels ( https://lkml.org/lkml/2016/1/25/345 )
- 08:56 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- See https://bugzilla.redhat.com/show_bug.cgi?id=1301093 for a similar bug
- 07:50 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- * http://pulpito.ceph.com/loic-2016-02-04_00:40:51-rgw-infernalis-backports---basic-multi/6184/
* http://pulpito.cep... - 06:23 AM Bug #14664 (Won't Fix): valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- http://pulpito.ceph.com/loic-2016-02-04_00:33:20-rados-infernalis-backports---basic-multi/6141/
CommandFailedError... - 01:20 AM Bug #14661: mira104 has missing drive
- also something weird going on with its networking; it can't seem to reach my vpn client, but it can reach other miras...
- 01:08 AM Bug #14661 (Resolved): mira104 has missing drive
- ubuntu@mira104:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 91 uncorrect sector
02/04/2016
- 10:06 PM Bug #14630 (In Progress): mira020 root drive failure, BMC issues?
- Filesystem errors found on /dev/sda1.
Ran fsck on root partition and marked VPSes back up. Drive 4 still needs re... - 09:01 PM Bug #14612 (In Progress): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:in...
- http://sentry.ceph.com/sepia/teuthology/issues/6/events/
- 08:23 PM Bug #14648 (Resolved): safely transition hosts to UTC
- both teuthology and paddles will need to change. probably also pulpito.
- 06:14 PM Bug #14644 (Duplicate): "Could not resolve host: gitbuilder.ceph.com" in rados-jewel-distro-basi...
- #14639
- 05:11 PM Bug #14644 (Duplicate): "Could not resolve host: gitbuilder.ceph.com" in rados-jewel-distro-basi...
- See several in late runs
Run: http://pulpito.ceph.com/teuthology-2016-02-02_22:00:01-rados-jewel-distro-basic-smithi... - 06:12 PM Bug #14639: failing gitbuilder connections from mira (DNS lookup errors)
- I see 8 instances of this as of now:
http://sentry.ceph.com/sepia/teuthology/issues/44/
Dropping priority - 07:19 AM Bug #14639 (Resolved): failing gitbuilder connections from mira (DNS lookup errors)
- http://pulpito.ceph.com/gregf-2016-02-03_09:19:17-fs-greg-fs-testing-23---basic-mira/4628/...
- 04:43 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- PR to add a graphviz check to @admin/build-doc@: https://github.com/ceph/ceph/pull/7522
- 01:31 AM Support #14614 (In Progress): Add access to Sepia Lab for Joe Mario
- 01:20 AM Support #13344: Request access to sepia for running teuthology
- 01:19 AM Support #13344: Request access to sepia for running teuthology
- Dan,
You've been added to the OpenVPN server and an account has been created on the teuthology host.
Please ver...
02/03/2016
- 11:18 PM Bug #14152 (Resolved): the dot graph is not rendered in http://docs.ceph.com/
- Looks like Ken resolved this.
- 11:10 PM Support #14531 (Resolved): RCA: download.ceph.com outage 27JAN2016
- download.ceph.com is now being served by nginx instead of apache2.
SSL certificate security was also improved in t... - 05:54 PM Bug #14630 (Resolved): mira020 root drive failure, BMC issues?
- mira020's root drive appears to be failing or has failed. I/O errors upon checking BMC KVM.
Marked VPSes and mira... - 02:09 PM Bug #14627 (Resolved): RCA: 3FEB2016 senta01 kernel panic
- senta01 became unresponsive around 5AM UTC.
SOL was unresponsive and BMC web UI showed Kernel Panic and Read-error... - 02:44 AM Bug #14619: mira039 missing drive 5
- it's also behaving badly on reboot; still spending a lot of time in RAID BIOS. Maybe the drive is failed in such a w...
- 01:48 AM Bug #14619: mira039 missing drive 5
- The upshot is: mira036 was not involved; mira039 was unresponsive, and on reboot spent a long time determining that d...
- 01:13 AM Bug #14619 (Resolved): mira039 missing drive 5
- I mira039 was listed as 'coluld not nuke' on teh stale nodes list:...
02/02/2016
- 10:47 PM Support #14614: Add access to Sepia Lab for Joe Mario
- I reran sepia/new-client. The new output is:
# sepia/new-client joemario@blazingsaddles
Please submit the foll... - 07:29 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
- Per a conversation with Ben England and Mark Nelson, I'd like to run a development feature we're adding to the perf t...
- 05:52 PM Bug #14612 (Duplicate): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:infe...
- Run: http://pulpito.ceph.com/teuthology-2016-02-01_17:10:11-upgrade:infernalis-infernalis-distro-basic-vps/
Job: 151... - 03:20 AM Bug #10174 (Won't Fix): saya037 unresponsive
- sayas are currently not used. I'm sure I'll run into this if we end up reusing the systems in the future.
- 03:16 AM Bug #13131 (Resolved): mira078 has a Read-Only FS, boots you out of ssh instantly
- Cleaning up the queue.. This system appears to have been running jobs successfully lately, has had drives replaced an...
- 03:11 AM Feature #11785 (Closed): install radian memory NVRAM cards in some machines
- Assuming this was already taken care of or is no longer needed.
Please re-open if that's not the case. - 03:09 AM Bug #14576 (Resolved): mira112 stuck at reboot
- HDD boot order fixed
- 03:05 AM Bug #13836 (Can't reproduce): vpm103.stderr:bash: git: command not found
- Due to the nature of VPSes and the age of this ticket, this is near impossible to troubleshoot. The system has been ...
- 02:54 AM Bug #12555 (Resolved): mira083 stuck in raid bios on reboot
- Replaced drives and reimaged 7JAN2016
- 02:45 AM Support #14430 (Resolved): need access to Sepia Lab
- Access granted and confirmed
- 12:35 AM Bug #14157 (Resolved): smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- David Galloway wrote:
> Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked Super...
02/01/2016
- 10:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked SuperMicro expedite a shipment...
- 10:31 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
- There were files under vpm104's mountpoint consuming space from root. Embarrassing that it took so long to figure th...
01/31/2016
- 10:14 PM Support #14430: need access to Sepia Lab
- That worked! I can now ssh to bengland@teuthology.front.sepia.ceph.com. Thanks for the help logging in.
01/30/2016
- 08:07 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- dubia001 can be repurposed from openstack if it hasn't been reused already. I was the one using it for openstack test...
- 07:37 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- dubia001 appears to be running openstack. It's unreachable but ethtool via SOL indicates its NIC isn't connected. W...
- 05:47 PM Bug #14576 (Resolved): mira112 stuck at reboot
- It was at stale state and stuck at nuke/stale
could not ssl and at the end ot sol power cycle got stuck at:... - 12:22 AM Bug #13282 (In Progress): ssh from my vm to centos vpm machines are slow.
- I might have been a bit hasty with tat last update.
ssh to Centos still seems to take a while.
Centos ssh:
</p...
01/29/2016
- 09:23 PM Fix #11889 (Resolved): ubuntu@tracker.front.sepia.ceph.com lingering git fetch-pack
- Redmine and the underlying OS were updated when tracker got moved to DHC.
I'm going to close this ticket but if yo... - 09:00 PM Support #13863 (Closed): Ask For Access To Sepia Lab Teuthology Gate
- please find us in #sepia on irc.oftc.net to discuss! note that the lab capacity is pretty well utilized, so if you h...
- 08:35 PM Bug #11865 (Resolved): mira041 bad sdc sdd
- Drives replaced 19JAN2016
- 08:33 PM Bug #11121 (Resolved): mira054 sdb sdc sdd sde sdf bad
- This system died. Its disks were moved to mira021 which took mira054's place in the long running cluster.
- 08:30 PM Feature #11496 (Rejected): pulpito-rdu needs a proxy
- pulpito-rdu decomissioned
- 07:51 PM Bug #14455 (Resolved): mira112 has bad drives
- Replaced this host's drives last night.
- 07:28 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
- All smithi nodes that had 7.1 installed have been reinstalled with 7.2
There are 3 machines I still have locked th... - 12:38 AM Bug #14541: "No space left on device" when locking vpm097
- I can't figure this one out. By all accounts there is only about 15GB used on /, but df shows 628GB used. Normally ...
01/28/2016
- 11:32 PM Bug #14548 (Can't reproduce): mira052 MCE
- ...
- 10:28 PM Bug #14546 (Resolved): mira033 kernel panic from MCE
- ...
- 05:19 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
- Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-27_18:49:02-rados-hammer-distro-basic-vps/46886/teutholo...
01/27/2016
- 09:24 PM Support #14531 (Resolved): RCA: download.ceph.com outage 27JAN2016
- Wanted to get this documented in a ticket until we have a better way of documenting outage RCAs.
Over the past cou... - 07:50 PM Bug #13282 (Resolved): ssh from my vm to centos vpm machines are slow.
- Seems to be better.
- 04:26 PM Cleanup #14528 (Resolved): Track down usage and purpose of mira{123..126} aka dubia{001..004}
- A couple of these are currently being used for infrastucture but at least one of them can probably be nuked and added...
01/26/2016
- 08:53 PM Bug #14214 (In Progress): mira115 has bad disks (marked down)
- Re-opening this. Still showing up as drives missing.
- 06:20 PM Bug #14214 (Resolved): mira115 has bad disks (marked down)
- Drives replaced on 19JAN2016
- 07:32 PM Bug #14432 (Closed): mira095 got wedged
- Without more logs (or bandwidth) to investigate further, not a lot more I can do to say for sure whether this is a ha...
- 07:25 PM Bug #12203 (Closed): create a Ceph profile @ cloudlab.us
- Closing this. See http://tracker.ceph.com/issues/12317
- 07:23 PM Bug #12772 (Resolved): incerta installed with Centos 7.1 don't reboot
- Closing this out as part of a bug scrub. If problem persists, you can reopen this issue.
Thanks! - 06:59 PM Bug #14453 (In Progress): Reimage smithi to centos 7.2 (instead of centos 7.1)
- smithis 003, 008, 015, 016, 017, 020, 026, 027, 028, 029 are running 7.2.
I just locked a few more that I will rei... - 06:56 PM Bug #14424 (Resolved): disk replacements for the LRC
- To replace a disk (from memory, may be missing a step or have wrong order):
* unweight the OSD if it's still viabl... - 06:51 PM Bug #12556 (Resolved): mira032 failing drives
- Drives 1, 3, 5 replaced 7JAN2016
- 06:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- The defective cards were approved for RMA and delivered to SuperMicro on 12JAN2016.
FedEx tracking: 782146800750
... - 06:45 PM Bug #12317 (Rejected): use AARCH64 instances from datacentred.co.uk
- No longer needed now that we have our own arm64 boxes to test on: limata{001..004}
<dgalloway> sage, do we need th... - 06:43 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
- All failing drives were replaced.
The gitbuilders are running smoothly and Zack rebuilt sentry-db.
I'll track s... - 06:41 PM Support #14430: need access to Sepia Lab
- Ben,
I just run all my OpenVPN tunnels as a background service so I don't have much experience with setting this u... - 06:31 PM Support #14502 (Resolved): access to Sepia Lab
- Key added to gw
https://github.com/ceph/cookbook-gw/pull/4
Thomas verified connectivity - 06:19 PM Bug #13652 (Resolved): mira118 has disk errors
- Drives replaced on 20JAN2016
- 04:10 PM Bug #14514: vps leaked after teuthology run
- The leaked vpses were from a variety of hosts:...
- 04:03 PM Bug #14514 (Closed): vps leaked after teuthology run
- vps failed to unlock long after the test suite completed.
When unlocking them manually it asked for ubuntu@mira043.f...
01/25/2016
- 10:44 PM Support #14430: need access to Sepia Lab
- I'm sorry, I really don't get what I'm supposed to do. I went into GUI, added an OpenVPN connect, importing it from ...
- 07:37 PM Support #14430: need access to Sepia Lab
- Ben,
You should now have access to the sepia lab. Please restart the openvpn connection on your workstation to ve... - 07:23 PM Support #14502 (Resolved): access to Sepia Lab
- Accidentally wiped my /etc/ dir when upgrading workstation. Here's new VPN:
tserlin@annarbor DlKe+OWBPcFAQtWMUAH...
01/22/2016
- 11:18 PM Feature #14479 (Resolved): Backups
- Create host with public IP and also connected to LRC for SFTP and backups
- 10:12 PM Support #14430: need access to Sepia Lab
- yes that's right Ben.
- 08:26 PM Support #14430: need access to Sepia Lab
- Sorry I don't follow instructions well, it was all there, is this what you want?
[root@bene-laptop sepia]# ./new-c... - 07:20 PM Support #14430 (In Progress): need access to Sepia Lab
- Ben,
Step 4 should probably be clarified but we need the output from 'new-client USER$HOST'
https://ceph.github.i... - 09:19 PM Bug #14478 (Resolved): mira089 MCE, bad processor?
- ubuntu@teuthology:~$ ssh mira089
ssh: connect to host mira089 port 22: No route to host
Wondering if we have some...
01/21/2016
- 05:27 PM Bug #14455 (Resolved): mira112 has bad drives
- ubuntu@mira112:~$ sudo /usr/libexec/smart.pl
4 of 8 drives failing/missing |
Drive 1 N.A.
Drive 2 N.A.
Drive 4 (... - 04:12 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
- Since we changed supported distros to centos 7.2 we need to bring smithi in line with that, otherwise tests wait for ...
- 01:35 AM Bug #14432: mira095 got wedged
- Nothing more in syslog or kern.log about the fault.
- 12:50 AM Bug #14432: mira095 got wedged
- Maybe a hardware fault?...
- 01:17 AM Bug #14398 (Resolved): Install Centos 7.2 on some mira
- Set up 7 CentOS 7.2 miras
$ tl --brief -a --machine-type mira --os-version 7.2
mira038.front.sepia.ceph.com loc... - 12:02 AM Bug #14422 (Resolved): mira078 failing tests has bad disk
- Drives replaced
- 12:02 AM Bug #14404 (Resolved): mira117 failing tests has bad disk
- Drives replaced
- 12:02 AM Bug #14403 (Resolved): mira088 failing tests has bad disk
- Drives replaced
01/20/2016
- 05:30 PM Bug #14424: disk replacements for the LRC
- All disks replaced last night
- 02:42 AM Bug #14432 (Closed): mira095 got wedged
- It disappeared while running http://pulpito.ceph.com/gregf-2016-01-18_18:01:09-fs-greg-fs-testing-118-1---basic-mira/...
- 01:04 AM Support #14430 (Resolved): need access to Sepia Lab
- Mark Nelson wants Joe Mario and I to run Ceph CBT and profile using perf "C2C" NUMA profiling tool, because of high-s...
01/19/2016
- 06:26 PM Bug #14424 (Resolved): disk replacements for the LRC
- mira049/2
mira021/6
mira060/5
mira116/7
mira120/7 missing (not sdg, beware)
mira055/2 still emptying
- 04:17 PM Bug #14422 (Resolved): mira078 failing tests has bad disk
- ubuntu@mira078:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 335 uncorrect secto...
01/18/2016
- 09:45 PM Bug #14403: mira088 failing tests has bad disk
- ubuntu@mira088:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 8 (sdh) has 121 uncorrect secto... - 09:43 PM Bug #14403 (Resolved): mira088 failing tests has bad disk
- 09:45 PM Bug #14404: mira117 failing tests has bad disk
- ubuntu@mira117:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 4 uncorrect sectors... - 09:43 PM Bug #14404 (Resolved): mira117 failing tests has bad disk
- 08:55 PM Bug #11315 (Rejected): ceph-deploy suite in hammer (and giant) can no longer run on plana nodes
- No longer have plana nodes
- 08:54 PM Bug #12048 (Resolved): mira082 bad disk
- System had its drives replaced JAN7
- 08:52 PM Bug #13147 (Resolved): mira092: SOL stuck, "power reset" has no apparent effect
- System's been running jobs fine for a bit now.
- 08:51 PM Bug #13696 (Resolved): mira077 has bad drives
- This system had its bad drives replaced on JAN7.
- 05:35 PM Bug #14398 (Resolved): Install Centos 7.2 on some mira
- We started running rados on mira and some tests ask for centos 7.2 OS, so we need to reimage ~10% to accommodate this...
01/15/2016
- 11:53 PM Support #14294: Infrastructure Host Drive replacements
- senta02's RAID failed and its filesystem was broken beyond salvaging. We lost the following VMs:
* gitbuilder-hadoo... - 10:06 PM Tasks #13085 (Closed): kill gitbuilder-ceph-tarball-precise-amd64-basic
- done
- 10:06 PM Tasks #13086 (Resolved): kill gitbuilder-ceph-rpm-fedora20-amd64-basic
- While cleaning up gitbuilders, noticed this one's already gone so closing ticket.
01/14/2016
- 09:05 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
- Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml)
- 06:26 PM Bug #14338 (In Progress): Incorrect IPs set in /etc/hosts on mira nodes
- 05:50 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- *sigh*...
- 05:36 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy ...
- 04:56 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- (Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alf... - 04:47 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704...
01/13/2016
- 05:49 AM Tasks #12705 (Resolved): Abhishek Varshney access to the lab
- Account should be added to teuthology machines, mira, and smithi. Please reopen/comment/email if not.
- 01:49 AM Tasks #12705 (In Progress): Abhishek Varshney access to the lab
- OpenVPN access added; not sure if working yet. Adding account info now.
- 01:44 AM Tasks #11652 (Resolved): smithfarm access to the lab
- Added new vpn secret. Nathan confirms a connection.
Also available in: Atom