Activity
From 01/04/2016 to 02/02/2016
02/02/2016
- 10:47 PM Support #14614: Add access to Sepia Lab for Joe Mario
- I reran sepia/new-client. The new output is:
# sepia/new-client joemario@blazingsaddles
Please submit the foll... - 07:29 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
- Per a conversation with Ben England and Mark Nelson, I'd like to run a development feature we're adding to the perf t...
- 05:52 PM Bug #14612 (Duplicate): "Could not resolve host: apt-mirror.front.sepia.ceph.com" in upgrade:infe...
- Run: http://pulpito.ceph.com/teuthology-2016-02-01_17:10:11-upgrade:infernalis-infernalis-distro-basic-vps/
Job: 151... - 03:20 AM Bug #10174 (Won't Fix): saya037 unresponsive
- sayas are currently not used. I'm sure I'll run into this if we end up reusing the systems in the future.
- 03:16 AM Bug #13131 (Resolved): mira078 has a Read-Only FS, boots you out of ssh instantly
- Cleaning up the queue.. This system appears to have been running jobs successfully lately, has had drives replaced an...
- 03:11 AM Feature #11785 (Closed): install radian memory NVRAM cards in some machines
- Assuming this was already taken care of or is no longer needed.
Please re-open if that's not the case. - 03:09 AM Bug #14576 (Resolved): mira112 stuck at reboot
- HDD boot order fixed
- 03:05 AM Bug #13836 (Can't reproduce): vpm103.stderr:bash: git: command not found
- Due to the nature of VPSes and the age of this ticket, this is near impossible to troubleshoot. The system has been ...
- 02:54 AM Bug #12555 (Resolved): mira083 stuck in raid bios on reboot
- Replaced drives and reimaged 7JAN2016
- 02:45 AM Support #14430 (Resolved): need access to Sepia Lab
- Access granted and confirmed
- 12:35 AM Bug #14157 (Resolved): smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- David Galloway wrote:
> Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked Super...
02/01/2016
- 10:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- Replacement NVMe cards arrived but were shipped with low-profile brackets. Have asked SuperMicro expedite a shipment...
- 10:31 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
- There were files under vpm104's mountpoint consuming space from root. Embarrassing that it took so long to figure th...
01/31/2016
- 10:14 PM Support #14430: need access to Sepia Lab
- That worked! I can now ssh to bengland@teuthology.front.sepia.ceph.com. Thanks for the help logging in.
01/30/2016
- 08:07 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- dubia001 can be repurposed from openstack if it hasn't been reused already. I was the one using it for openstack test...
- 07:37 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- dubia001 appears to be running openstack. It's unreachable but ethtool via SOL indicates its NIC isn't connected. W...
- 05:47 PM Bug #14576 (Resolved): mira112 stuck at reboot
- It was at stale state and stuck at nuke/stale
could not ssl and at the end ot sol power cycle got stuck at:... - 12:22 AM Bug #13282 (In Progress): ssh from my vm to centos vpm machines are slow.
- I might have been a bit hasty with tat last update.
ssh to Centos still seems to take a while.
Centos ssh:
</p...
01/29/2016
- 09:23 PM Fix #11889 (Resolved): ubuntu@tracker.front.sepia.ceph.com lingering git fetch-pack
- Redmine and the underlying OS were updated when tracker got moved to DHC.
I'm going to close this ticket but if yo... - 09:00 PM Support #13863 (Closed): Ask For Access To Sepia Lab Teuthology Gate
- please find us in #sepia on irc.oftc.net to discuss! note that the lab capacity is pretty well utilized, so if you h...
- 08:35 PM Bug #11865 (Resolved): mira041 bad sdc sdd
- Drives replaced 19JAN2016
- 08:33 PM Bug #11121 (Resolved): mira054 sdb sdc sdd sde sdf bad
- This system died. Its disks were moved to mira021 which took mira054's place in the long running cluster.
- 08:30 PM Feature #11496 (Rejected): pulpito-rdu needs a proxy
- pulpito-rdu decomissioned
- 07:51 PM Bug #14455 (Resolved): mira112 has bad drives
- Replaced this host's drives last night.
- 07:28 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
- All smithi nodes that had 7.1 installed have been reinstalled with 7.2
There are 3 machines I still have locked th... - 12:38 AM Bug #14541: "No space left on device" when locking vpm097
- I can't figure this one out. By all accounts there is only about 15GB used on /, but df shows 628GB used. Normally ...
01/28/2016
- 11:32 PM Bug #14548 (Can't reproduce): mira052 MCE
- ...
- 10:28 PM Bug #14546 (Resolved): mira033 kernel panic from MCE
- ...
- 05:19 PM Bug #14541 (Resolved): "No space left on device" when locking vpm097
- Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-27_18:49:02-rados-hammer-distro-basic-vps/46886/teutholo...
01/27/2016
- 09:24 PM Support #14531 (Resolved): RCA: download.ceph.com outage 27JAN2016
- Wanted to get this documented in a ticket until we have a better way of documenting outage RCAs.
Over the past cou... - 07:50 PM Bug #13282 (Resolved): ssh from my vm to centos vpm machines are slow.
- Seems to be better.
- 04:26 PM Cleanup #14528 (Resolved): Track down usage and purpose of mira{123..126} aka dubia{001..004}
- A couple of these are currently being used for infrastucture but at least one of them can probably be nuked and added...
01/26/2016
- 08:53 PM Bug #14214 (In Progress): mira115 has bad disks (marked down)
- Re-opening this. Still showing up as drives missing.
- 06:20 PM Bug #14214 (Resolved): mira115 has bad disks (marked down)
- Drives replaced on 19JAN2016
- 07:32 PM Bug #14432 (Closed): mira095 got wedged
- Without more logs (or bandwidth) to investigate further, not a lot more I can do to say for sure whether this is a ha...
- 07:25 PM Bug #12203 (Closed): create a Ceph profile @ cloudlab.us
- Closing this. See http://tracker.ceph.com/issues/12317
- 07:23 PM Bug #12772 (Resolved): incerta installed with Centos 7.1 don't reboot
- Closing this out as part of a bug scrub. If problem persists, you can reopen this issue.
Thanks! - 06:59 PM Bug #14453 (In Progress): Reimage smithi to centos 7.2 (instead of centos 7.1)
- smithis 003, 008, 015, 016, 017, 020, 026, 027, 028, 029 are running 7.2.
I just locked a few more that I will rei... - 06:56 PM Bug #14424 (Resolved): disk replacements for the LRC
- To replace a disk (from memory, may be missing a step or have wrong order):
* unweight the OSD if it's still viabl... - 06:51 PM Bug #12556 (Resolved): mira032 failing drives
- Drives 1, 3, 5 replaced 7JAN2016
- 06:48 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- The defective cards were approved for RMA and delivered to SuperMicro on 12JAN2016.
FedEx tracking: 782146800750
... - 06:45 PM Bug #12317 (Rejected): use AARCH64 instances from datacentred.co.uk
- No longer needed now that we have our own arm64 boxes to test on: limata{001..004}
<dgalloway> sage, do we need th... - 06:43 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
- All failing drives were replaced.
The gitbuilders are running smoothly and Zack rebuilt sentry-db.
I'll track s... - 06:41 PM Support #14430: need access to Sepia Lab
- Ben,
I just run all my OpenVPN tunnels as a background service so I don't have much experience with setting this u... - 06:31 PM Support #14502 (Resolved): access to Sepia Lab
- Key added to gw
https://github.com/ceph/cookbook-gw/pull/4
Thomas verified connectivity - 06:19 PM Bug #13652 (Resolved): mira118 has disk errors
- Drives replaced on 20JAN2016
- 04:10 PM Bug #14514: vps leaked after teuthology run
- The leaked vpses were from a variety of hosts:...
- 04:03 PM Bug #14514 (Closed): vps leaked after teuthology run
- vps failed to unlock long after the test suite completed.
When unlocking them manually it asked for ubuntu@mira043.f...
01/25/2016
- 10:44 PM Support #14430: need access to Sepia Lab
- I'm sorry, I really don't get what I'm supposed to do. I went into GUI, added an OpenVPN connect, importing it from ...
- 07:37 PM Support #14430: need access to Sepia Lab
- Ben,
You should now have access to the sepia lab. Please restart the openvpn connection on your workstation to ve... - 07:23 PM Support #14502 (Resolved): access to Sepia Lab
- Accidentally wiped my /etc/ dir when upgrading workstation. Here's new VPN:
tserlin@annarbor DlKe+OWBPcFAQtWMUAH...
01/22/2016
- 11:18 PM Feature #14479 (Resolved): Backups
- Create host with public IP and also connected to LRC for SFTP and backups
- 10:12 PM Support #14430: need access to Sepia Lab
- yes that's right Ben.
- 08:26 PM Support #14430: need access to Sepia Lab
- Sorry I don't follow instructions well, it was all there, is this what you want?
[root@bene-laptop sepia]# ./new-c... - 07:20 PM Support #14430 (In Progress): need access to Sepia Lab
- Ben,
Step 4 should probably be clarified but we need the output from 'new-client USER$HOST'
https://ceph.github.i... - 09:19 PM Bug #14478 (Resolved): mira089 MCE, bad processor?
- ubuntu@teuthology:~$ ssh mira089
ssh: connect to host mira089 port 22: No route to host
Wondering if we have some...
01/21/2016
- 05:27 PM Bug #14455 (Resolved): mira112 has bad drives
- ubuntu@mira112:~$ sudo /usr/libexec/smart.pl
4 of 8 drives failing/missing |
Drive 1 N.A.
Drive 2 N.A.
Drive 4 (... - 04:12 PM Bug #14453 (Resolved): Reimage smithi to centos 7.2 (instead of centos 7.1)
- Since we changed supported distros to centos 7.2 we need to bring smithi in line with that, otherwise tests wait for ...
- 01:35 AM Bug #14432: mira095 got wedged
- Nothing more in syslog or kern.log about the fault.
- 12:50 AM Bug #14432: mira095 got wedged
- Maybe a hardware fault?...
- 01:17 AM Bug #14398 (Resolved): Install Centos 7.2 on some mira
- Set up 7 CentOS 7.2 miras
$ tl --brief -a --machine-type mira --os-version 7.2
mira038.front.sepia.ceph.com loc... - 12:02 AM Bug #14422 (Resolved): mira078 failing tests has bad disk
- Drives replaced
- 12:02 AM Bug #14404 (Resolved): mira117 failing tests has bad disk
- Drives replaced
- 12:02 AM Bug #14403 (Resolved): mira088 failing tests has bad disk
- Drives replaced
01/20/2016
- 05:30 PM Bug #14424: disk replacements for the LRC
- All disks replaced last night
- 02:42 AM Bug #14432 (Closed): mira095 got wedged
- It disappeared while running http://pulpito.ceph.com/gregf-2016-01-18_18:01:09-fs-greg-fs-testing-118-1---basic-mira/...
- 01:04 AM Support #14430 (Resolved): need access to Sepia Lab
- Mark Nelson wants Joe Mario and I to run Ceph CBT and profile using perf "C2C" NUMA profiling tool, because of high-s...
01/19/2016
- 06:26 PM Bug #14424 (Resolved): disk replacements for the LRC
- mira049/2
mira021/6
mira060/5
mira116/7
mira120/7 missing (not sdg, beware)
mira055/2 still emptying
- 04:17 PM Bug #14422 (Resolved): mira078 failing tests has bad disk
- ubuntu@mira078:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 335 uncorrect secto...
01/18/2016
- 09:45 PM Bug #14403: mira088 failing tests has bad disk
- ubuntu@mira088:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 8 (sdh) has 121 uncorrect secto... - 09:43 PM Bug #14403 (Resolved): mira088 failing tests has bad disk
- 09:45 PM Bug #14404: mira117 failing tests has bad disk
- ubuntu@mira117:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 4 uncorrect sectors... - 09:43 PM Bug #14404 (Resolved): mira117 failing tests has bad disk
- 08:55 PM Bug #11315 (Rejected): ceph-deploy suite in hammer (and giant) can no longer run on plana nodes
- No longer have plana nodes
- 08:54 PM Bug #12048 (Resolved): mira082 bad disk
- System had its drives replaced JAN7
- 08:52 PM Bug #13147 (Resolved): mira092: SOL stuck, "power reset" has no apparent effect
- System's been running jobs fine for a bit now.
- 08:51 PM Bug #13696 (Resolved): mira077 has bad drives
- This system had its bad drives replaced on JAN7.
- 05:35 PM Bug #14398 (Resolved): Install Centos 7.2 on some mira
- We started running rados on mira and some tests ask for centos 7.2 OS, so we need to reimage ~10% to accommodate this...
01/15/2016
- 11:53 PM Support #14294: Infrastructure Host Drive replacements
- senta02's RAID failed and its filesystem was broken beyond salvaging. We lost the following VMs:
* gitbuilder-hadoo... - 10:06 PM Tasks #13085 (Closed): kill gitbuilder-ceph-tarball-precise-amd64-basic
- done
- 10:06 PM Tasks #13086 (Resolved): kill gitbuilder-ceph-rpm-fedora20-amd64-basic
- While cleaning up gitbuilders, noticed this one's already gone so closing ticket.
01/14/2016
- 09:05 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
- Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml)
- 06:26 PM Bug #14338 (In Progress): Incorrect IPs set in /etc/hosts on mira nodes
- 05:50 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- *sigh*...
- 05:36 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy ...
- 04:56 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- (Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alf... - 04:47 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704...
01/13/2016
- 05:49 AM Tasks #12705 (Resolved): Abhishek Varshney access to the lab
- Account should be added to teuthology machines, mira, and smithi. Please reopen/comment/email if not.
- 01:49 AM Tasks #12705 (In Progress): Abhishek Varshney access to the lab
- OpenVPN access added; not sure if working yet. Adding account info now.
- 01:44 AM Tasks #11652 (Resolved): smithfarm access to the lab
- Added new vpn secret. Nathan confirms a connection.
01/12/2016
- 05:49 PM Bug #14338 (New): Incorrect IPs set in /etc/hosts on mira nodes
- David, can you take a look at mira061 configuration, pls?
And then close/resolve the ticket. - 03:52 PM Bug #14338 (Closed): Incorrect IPs set in /etc/hosts on mira nodes
- 12:12 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- This looks like the monitors can't resolve the mira061 address:...
- 01:28 AM Bug #11867 (Resolved): mira089 bad disk
- All failed drives replaced
01/11/2016
- 10:45 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
- Run: http://pulpito.ceph.com/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/
Jobs: 18887, 18889
... - 08:16 PM Bug #14336 (Resolved): All runs failing in ovh lab
- https://github.com/ceph/teuthology/commit/1a74f929430ee1976a7696d837cc3edb407c7f36
- 07:59 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
- 07:51 PM Bug #14336 (Resolved): All runs failing in ovh lab
- 07:43 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
- https://github.com/ceph/teuthology/pull/767
- 07:41 PM Bug #14336: All runs failing in ovh lab
- ...
- 07:40 PM Bug #14336 (In Progress): All runs failing in ovh lab
- 05:05 PM Bug #14336 (Resolved): All runs failing in ovh lab
- See last 2 days runs, here is an example:
Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-10_18:02:01... - 10:11 AM Tasks #11652: smithfarm access to the lab
- Hi Dan:
My previous sepia credentials were lost in global destruction. Could you please renew them?
# requesti... - 08:59 AM Tasks #11652 (In Progress): smithfarm access to the lab
- @nathan could you please re-iterate the required information here to make it easier for Dan ? See http://ceph.github....
01/08/2016
- 04:29 AM Support #14294: Infrastructure Host Drive replacements
- senta01 rebuild complete
senta02 has problems with md1, investigating through rescue boot
senta03 rebuild complete
- 02:50 AM Bug #13696 (In Progress): mira077 has bad drives
01/07/2016
- 11:03 PM Feature #14296 (Resolved): Create VPSHOST nagios hostgroup
- 10:55 PM Support #14294: Infrastructure Host Drive replacements
- senta01 had drive 2 replaced and its Areca RAID is rebuilding automatically
- 10:45 PM Support #14294: Infrastructure Host Drive replacements
- senta03 has had sdb replaced
md1 is rebuilding
md127 needs sdb1 readded still - 10:30 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
- senta02 needs sdc replaced
senta02 has had sdd replaced and md1 is in process of recovering. md127 needs recovering... - 10:29 PM Bug #12723 (Closed): megabug: all the dead disks in the test miras
- Most of these have been replaced long ago, adn we have a new set
- 09:21 PM Bug #14291 (Resolved): gw redundancy
- We used to have a hot standby host, gw2, that was inaccessible but ready to take over gw's IP if an emergency happene...
- 09:18 PM Bug #14290 (Resolved): Add public IP to yan-zheng; make sure packages up to date, no passwords (a...
- Basically restore yan-zheng as a bastion host for developers behind Great Firewall
- 07:39 AM Tasks #12705: Abhishek Varshney access to the lab
- username: abhishekvrshny
email: abhishek.varshney@flipkart.com
SSH key: https://github.com/ceph/keys/blob/master/ss... - 07:02 AM Tasks #12705: Abhishek Varshney access to the lab
- It's not a private key; it's a hashed password. The secret stays with you. Here is fine. Note that there are four ...
- 06:36 AM Tasks #12705: Abhishek Varshney access to the lab
- Dan: My key exists on https://github.com/ceph/keys and I have just completed http://ceph.github.io/sepia/adding_users...
- 07:20 AM Support #14250 (Resolved): Requesting Lab Access
- 07:20 AM Support #14250: Requesting Lab Access
- I guess I'll close it.
01/06/2016
- 11:17 PM Bug #13147: mira092: SOL stuck, "power reset" has no apparent effect
- Not sure of exact root cause but was able to re-image, nuke, and release on 22DEC2015.
- 10:45 PM Bug #12555 (In Progress): mira083 stuck in raid bios on reboot
- The RAID controller firmware hang has been resolved but 3 drives are missing.
I'm installing an OS to identify whi... - 10:02 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- Here are today's findings/testing.
I took smithi003 with a known working NVMe card and tested both PCI slots. Con... - 06:43 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- seems like it should to me, Ken
- 06:33 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- (Should ceph.git's @admin/build-doc@ script be checking for the graphviz package?)
- 06:32 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- Sure, I've installed graphviz now.
Exact steps I did today:
# Log into the DHC OpenVPN (Dan Mick has documentat... - 06:12 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- Ken, i see you are using docs nodes to build the docs in https://github.com/ceph/ceph-build/pull/210. i am wondering ...
- 12:46 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- if graphviz is installed on the gitbuilder of the doc, the dot should be rendered.
- 06:42 PM Tasks #12705: Abhishek Varshney access to the lab
- Abhishek: did you supply the information requested in http://ceph.github.io/sepia/adding_users/#requesting-lab-access...
- 02:33 PM Tasks #12705: Abhishek Varshney access to the lab
- @Dan I think Abhishek did what was required of him at http://ceph.github.io/sepia/adding_users/ . I'm escalating this...
- 06:36 PM Bug #14268 (Resolved): logrotate failing on gitbuilders
- I noticed a large stdout.log on the ceph-deb-trusty-amd64-basic gitbuilder.
After discussing with dmick it is pres... - 05:24 AM Support #14250: Requesting Lab Access
- I think this is taken care of; John, please close this when you agree.
- 02:02 AM Support #14250: Requesting Lab Access
- https://github.com/ceph/keys/pull/10
https://github.com/ceph/ceph-sepia-secrets/pull/69
https://github.com/ceph/coo... - 12:45 AM Support #14250 (In Progress): Requesting Lab Access
- 12:44 AM Support #14250 (Resolved): Requesting Lab Access
- I'm requesting access to help Warren test some Java client code.
The login I would like to use is: jowilkin
SS...
Also available in: Atom