Activity
From 02/08/2016 to 03/08/2016
03/08/2016
- 10:11 PM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
- This (second) outage was due to Dreamhost moving the Etherpad service to another server.
The A record for pad.ceph... - 06:27 AM Bug #14836: http://pad.ceph.com/ Unable to connect
- re-opening to keep track of the frequency of the problem
- 09:40 PM Bug #14993 (Resolved): centos6 gitbuilder broken
- Once all the yum repo issues were cleared, up hammer successfully built.
- 05:19 PM Bug #14993: centos6 gitbuilder broken
- Fixed yum errors by updating repo file. The Centos Base repofile got mangled somehow *and* there was a new *.rpmnew ...
- 02:44 AM Bug #14993 (In Progress): centos6 gitbuilder broken
- 02:44 AM Bug #14993: centos6 gitbuilder broken
- Ha, much better now, thanks !
It still display errorrs that turns gitbuilder red. Have you encoutered them before ... - 07:03 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
- mira114's network was incorrectly configured. It hadn't been updated since before the lab move. I reran the testnodes...
- 06:05 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
- http://pulpito.ceph.com/gregf-2016-03-07_23:12:27-fs-greg-fs-testing-3-7-safe---basic-mira/46170/
It looks like it... - 05:28 PM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
- I don't have a login for the wordpress portion of the site but would imagine the link should point to http://docs.cep...
03/07/2016
- 06:58 PM Cleanup #15001 (Resolved): Migrate ceph.com zone from Dreamhost to Red Hat external DNS
- 06:52 PM Bug #14993: centos6 gitbuilder broken
- I added ...
- 03:43 AM Bug #14993: centos6 gitbuilder broken
- ...
- 03:42 AM Bug #14993 (Resolved): centos6 gitbuilder broken
- Every build fail with...
03/05/2016
- 01:06 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
- thanks for the hard work dmick !...
03/04/2016
- 11:23 PM Bug #14290 (In Progress): Add public IP to yan-zheng; make sure packages up to date, no passwords...
- This is in progress. I have a VM set up in RHEV ready for this purpose. It's just of figuring out how to get it acc...
- 11:12 PM Bug #13392 (Resolved): mira074 has a bad root disk
- Drive got replaced on 28JAN2016 and host was reimaged
- 11:09 PM Bug #14768 (Resolved): mira009 dead
- DIMMs replaced. Reinstalled host with Ubuntu 14.04 and set back up as fresh VPSHOST while I was at it.
- 10:42 PM Bug #14619: mira039 missing drive 5
- Something's up with this machine. I started a reimage on 5 miras at the same time. The other four are done and this...
- 08:04 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
- Thanks for the workaround.
A few scripts have that path encoded though and they will keep breaking. A fix would b... - 06:54 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
- @Loic: Leave out the "git/" in the path:...
- 03:34 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
- things change a bit, here it is for the record, from teuthology.front.sepia.ceph.com...
- 01:18 AM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
03/03/2016
- 05:12 PM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
- This is also causing the load on teuthology.front to hover around 70 because downburst is doing *something* insane. I...
- 04:08 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
- It happens on teuthology.front.sepia.ceph.com
03/02/2016
- 08:28 AM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
- I would open a PR to fix this, but this website page doesn't seem to be in the source tree?
03/01/2016
- 10:39 PM Bug #14943 (Closed): 404 link on webpage http://ceph.com/resources/development/
- On http://ceph.com/resources/development/, the paragraph ...
- 08:11 PM Bug #14619 (In Progress): mira039 missing drive 5
- Updated RAID controller firmware from V1.49 2011-08-24 to V1.52 2015-11-20 and system boots considerably faster durin...
- 04:08 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
- Thanks for the update David (and the resolution), much appreciated.
02/29/2016
- 10:43 PM Bug #14768 (In Progress): mira009 dead
- Lots of logs in syslog indicating DIMM has gone bad...
- 10:38 PM Bug #14915: unable to access logs from http://qa-proxy.ceph.com/teuthology/
- This was a result of the teuthology host locking up on Saturday. The system locked up due to a runaway vi process un...
- 10:36 PM Bug #14840 (Resolved): mira091 is not accessible
- I ran memtest on this machine without issue.
I updated its BIOS and set the VPSes back up. If the issue persists,... - 10:23 PM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
- Tracker had a few mysqld instanced causing CPU soft lockups. I've updated all packages on the system and restarted m...
- 06:12 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
- (virtualenv)loic@teuthology:~$ time curl http://tracker.ceph.com/issues/14919
real 0m24.707s
(virtualenv)loic@teuth... - 05:56 AM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
- From the teuthology.front machine ...
- 10:13 PM Bug #14859: xiaoxi access to the lab
- Can you enable more verbose output and provide the IP you're attempting to connect from please? I don't even see you...
- 04:42 PM Bug #14926 (Can't reproduce): jenkins.ceph.com signed-off-by false negative
- https://jenkins.ceph.com/job/ceph-pr-commits/2768/console
- 03:56 AM Bug #14841 (Resolved): sepia/zyan: Access to the sepia lab
- It works
02/27/2016
- 09:00 PM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
- Cannot access any teuthology log from qa-proxy.cpeh.com/teuthology
504 Gateway Time-out
nginx/1.6.3 - 02:30 PM Bug #14859: xiaoxi access to the lab
- Hi David,
I cannot connect the OpenVPN due to auth error,
2016-02-27 22:25:02 VERIFY EKU OK
2016-02-27 ...
02/26/2016
- 03:20 AM Bug #14841: sepia/zyan: Access to the sepia lab
- Zheng, I put the new password hash into place; can you try again please
- 02:31 AM Bug #14841 (In Progress): sepia/zyan: Access to the sepia lab
- 02:30 AM Bug #14841: sepia/zyan: Access to the sepia lab
- sorry, I ruined my OpenVPN secret
please change it to:
./new-client zyan@redhat
Please submit the following ...
02/25/2016
- 11:19 PM Support #14704 (Resolved): Rack tala or saya
- Finally got around to replacing the uplink with a 10Gb cable. Resolving.
- 05:37 PM Bug #14841: sepia/zyan: Access to the sepia lab
- Hi,
Thanks for your patience and understanding my wanting to verify your identity.
I've added your hashed VPN p... - 05:17 PM Bug #14859: xiaoxi access to the lab
- Hi Xiaoxi,
You should now be able to access the lab and schedule tests.
Please verify you can connect to our Op... - 05:22 AM Bug #14859: xiaoxi access to the lab
- Username:
xiaoxichen
SSH key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTP... - 05:00 AM Bug #14859 (Closed): xiaoxi access to the lab
- Xiaoxi Chen aka xiaoxi is working on http://tracker.ceph.com/projects/ceph-releases he needs access to the lab to:
... - 04:57 PM Cleanup #14868 (Resolved): Audit user list
- - Verify all users with OpenVPN access and ssh pubkeys still need access
- Mixture of "X left the company" and e-m...
02/24/2016
- 03:11 AM Bug #14841: sepia/zyan: Access to the sepia lab
- please replace the existing public key and OpenVPN password.
thanks
02/23/2016
- 04:41 PM Bug #14841 (Need More Info): sepia/zyan: Access to the sepia lab
- Should this replace your existing pubkey and OpenVPN password or is this in addition to the existing one?
- 02:50 AM Bug #14841 (Resolved): sepia/zyan: Access to the sepia lab
- I need to access sepia lab to schedule tests.
user: zyan
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9... - 07:07 AM Bug #14836: http://pad.ceph.com/ Unable to connect
- thanks david !
02/22/2016
- 10:53 PM Bug #14840 (Closed): mira091 is not accessible
- via ssh and/or ipmitool
- 06:35 PM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
- The httpd process acting as the reverse proxy failed to restart after rsyslogd ran its daily cycle....
- 07:02 AM Bug #14836: http://pad.ceph.com/ Unable to connect
- there are informations in pad.ceph.com that I need to implement key management in ceph-disk. If it's gone that's not ...
- 06:02 AM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
02/16/2016
- 11:33 PM Bug #14778 (Resolved): mira090 3 of 8 drives failing/missing
- ubuntu@mira090:~$ sudo /usr/libexec/smart.pl
3 of 8 drives failing/missing |
Drive 3 N.A.
Drive 4 (sdd) has 364 r... - 04:39 PM Bug #14749 (Resolved): smithi049 and smithi041 rebooting constantly
- This was a strange one to debug.
I stayed logged in to both machines tailing @/var/log/{auth.log,syslog}@ and noti...
02/15/2016
- 11:46 PM Bug #14763 (Resolved): teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
- Timeouts are still intermittent but mostly resolved. Lingering network issues are being tracked in http://tracker.ce...
- 02:35 PM Bug #14763: teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
- and gitbuilder is not able to access github now, see http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-deb-trusty-amd6...
- 06:50 AM Bug #14763: teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
- ...
- 05:34 AM Bug #14763 (Resolved): teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
- ...
- 11:46 PM Bug #14767 (Resolved): Massive packet loss to/from gw.sepia.ceph.com
- Timeouts are still intermittent but mostly resolved. Lingering network issues are still being investigated by IT.
... - 11:44 PM Bug #14767 (In Progress): Massive packet loss to/from gw.sepia.ceph.com
- gw to google now...
- 05:39 PM Bug #14767: Massive packet loss to/from gw.sepia.ceph.com
- Yet, it's not entirely broken:...
- 05:36 PM Bug #14767: Massive packet loss to/from gw.sepia.ceph.com
- From gw to google.com:...
- 05:35 PM Bug #14767: Massive packet loss to/from gw.sepia.ceph.com
- From my laptop in CO to gw:...
- 05:34 PM Bug #14767 (Resolved): Massive packet loss to/from gw.sepia.ceph.com
- 06:27 PM Bug #14768 (Resolved): mira009 dead
- 04:47 PM Bug #14749: smithi049 and smithi041 rebooting constantly
- I marked smithi041 and smithi049 down. Yuri provided these logs just now:
http://qa-proxy.ceph.com/teuthology/sage...
02/12/2016
- 09:33 PM Bug #14749: smithi049 and smithi041 rebooting constantly
- smithi041 currently responds to apt-get update with
E: dpkg was interrupted, you must manually run 'sudo dpkg --co... - 09:25 PM Bug #14749: smithi049 and smithi041 rebooting constantly
- see also
smithi049 smithi041 - 07:12 PM Bug #14749: smithi049 and smithi041 rebooting constantly
- From http://qa-proxy.ceph.com/teuthology/sage-2016-02-12_05:26:12-rados-wip-sage-testing---basic-smithi/7136/teutholo...
- 06:31 PM Bug #14749: smithi049 and smithi041 rebooting constantly
- @teuthology.log@:...
- 06:24 PM Bug #14749: smithi049 and smithi041 rebooting constantly
- As of right now, mira049's job history, most recent first:
running http://pulpito.front.sepia.ceph.com/sage-2016-02-... - 05:51 PM Bug #14749 (Resolved): smithi049 and smithi041 rebooting constantly
- Majority of nodes that failed during @nuke/stale@ operation fail with ...
- 08:33 PM Support #14704: Rack tala or saya
- These are accessible on their front ifaces now. I'd still like to replace the connection with a 10Gb cable once IT c...
- 08:31 PM Support #14722 (Resolved): VPN Access to DELL Ceph Benchmark Systems
- VPN set up and working
https://github.com/ceph/cookbook-gw/pull/8 - 04:17 AM Bug #14733 (Resolved): Could not fetch updated apt files
- 04:16 AM Bug #14733: Could not fetch updated apt files
- See also https://bugs.launchpad.net/launchpad/+bug/1544806
- 12:58 AM Cleanup #14744 (Resolved): Clean up epel and rhel synced repos on gitbuilder-archive
- Commenting out repomirror.sh from cron
Seems unnecessary
Backup of script is in /home/backup/
02/11/2016
- 11:18 PM Bug #14733: Could not fetch updated apt files
- So...the upstream format of the i18n Index files has changed; I'm sure it's an error but we can put a hack into a loc...
- 04:24 PM Bug #14733: Could not fetch updated apt files
- http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-02-11_02:00:01-rados-infernalis-distro-basic-openstack/
http:... - 04:23 PM Bug #14733 (In Progress): Could not fetch updated apt files
- ...
- 05:31 AM Bug #14733: Could not fetch updated apt files
- http://pulpito.ceph.com/loic-2016-02-10_21:25:43-fs-infernalis---basic-multi/5065/
- 05:03 AM Bug #14733 (Resolved): Could not fetch updated apt files
- General failure of some suites today
* http://pulpito.ceph.com/loic-2016-02-10_20:32:58-rados-infernalis-backport... - 05:18 PM Bug #12032: activate beanstalkd logs
- I set the following:...
- 08:32 AM Support #14722: VPN Access to DELL Ceph Benchmark Systems
- dmesser@flab mJhwtddvlOI/hkpAGnFr5w 13b92ea100bb995ec71f1d5f6309271c3fca47d20098f0bb0aa9e4906b62f092
- 04:25 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
- Many occurrences at http://tracker.ceph.com/issues/14692
02/10/2016
- 10:02 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
- 09:03 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Hi David:
That was it. Thank you for all your help.
Joe - 08:57 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Your user was set up as 'jmario' so you'd:...
- 08:54 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Hi David:
I am now able to create the VPM and get to the login prompt for teuthology.front.sepia.ceph.com.
Thank ... - 08:59 PM Support #14704: Rack tala or saya
- I think this box had 2x 1Gb uplinks and a 10Gb uplink and that's why we can't reach the actual OSes.
I put in a ti... - 03:55 AM Support #14704: Rack tala or saya
- tala{001..016} have been re-racked and their IPMI interfaces are accessible. I think some magic will be needed in Co...
- 07:08 PM Support #14722: VPN Access to DELL Ceph Benchmark Systems
- Daniel,
I need you to follow the steps to generate a hashed password.
https://ceph.github.io/sepia/adding_users... - 01:46 PM Support #14722 (Resolved): VPN Access to DELL Ceph Benchmark Systems
- Hi,
I am part of the technical team to support the benchmarking exercise we are doing on the 730xd servers which h...
02/09/2016
- 03:32 PM Bug #10798 (Resolved): mkfs.ext4 fails with "apparently in use by the system"
- assuming this is fixed on that mira
- 12:51 AM Support #14704 (Resolved): Rack tala or saya
- For 32-bit ARM testing
02/08/2016
- 11:55 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
- mira123 (formerly dubia001) is full of healthy 4TB drives
mira124 (formerly dubia002)'s disks are almost all bad
... - 09:57 PM Bug #14548 (Can't reproduce): mira052 MCE
- memtest passed. Will monitor host for future failures and troubleshoot further if needed.
Reinstalled and released. - 06:27 PM Support #14614: Add access to Sepia Lab for Joe Mario
- Joe,
I changed your user string on the server. Try again and let me know how things go. - 04:54 PM Feature #14696 (Resolved): Migrate from PowerDNS to (probably) BIND with version control
Also available in: Atom