Project

General

Profile

Activity

From 02/08/2016 to 03/08/2016

03/08/2016

10:11 PM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
This (second) outage was due to Dreamhost moving the Etherpad service to another server.
The A record for pad.ceph...
David Galloway
06:27 AM Bug #14836: http://pad.ceph.com/ Unable to connect
re-opening to keep track of the frequency of the problem Loïc Dachary
09:40 PM Bug #14993 (Resolved): centos6 gitbuilder broken
Once all the yum repo issues were cleared, up hammer successfully built. David Galloway
05:19 PM Bug #14993: centos6 gitbuilder broken
Fixed yum errors by updating repo file. The Centos Base repofile got mangled somehow *and* there was a new *.rpmnew ... David Galloway
02:44 AM Bug #14993 (In Progress): centos6 gitbuilder broken
Loïc Dachary
02:44 AM Bug #14993: centos6 gitbuilder broken
Ha, much better now, thanks !
It still display errorrs that turns gitbuilder red. Have you encoutered them before ...
Loïc Dachary
07:03 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
mira114's network was incorrectly configured. It hadn't been updated since before the lab move. I reran the testnodes... Zack Cerza
06:05 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
http://pulpito.ceph.com/gregf-2016-03-07_23:12:27-fs-greg-fs-testing-3-7-safe---basic-mira/46170/
It looks like it...
Greg Farnum
05:28 PM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
I don't have a login for the wordpress portion of the site but would imagine the link should point to http://docs.cep... David Galloway

03/07/2016

06:58 PM Cleanup #15001 (Resolved): Migrate ceph.com zone from Dreamhost to Red Hat external DNS
David Galloway
06:52 PM Bug #14993: centos6 gitbuilder broken
I added ... David Galloway
03:43 AM Bug #14993: centos6 gitbuilder broken
... Loïc Dachary
03:42 AM Bug #14993 (Resolved): centos6 gitbuilder broken
Every build fail with... Loïc Dachary

03/05/2016

01:06 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
thanks for the hard work dmick !... Loïc Dachary

03/04/2016

11:23 PM Bug #14290 (In Progress): Add public IP to yan-zheng; make sure packages up to date, no passwords...
This is in progress. I have a VM set up in RHEV ready for this purpose. It's just of figuring out how to get it acc... David Galloway
11:12 PM Bug #13392 (Resolved): mira074 has a bad root disk
Drive got replaced on 28JAN2016 and host was reimaged David Galloway
11:09 PM Bug #14768 (Resolved): mira009 dead
DIMMs replaced. Reinstalled host with Ubuntu 14.04 and set back up as fresh VPSHOST while I was at it. David Galloway
10:42 PM Bug #14619: mira039 missing drive 5
Something's up with this machine. I started a reimage on 5 miras at the same time. The other four are done and this... David Galloway
08:04 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
Thanks for the workaround.
A few scripts have that path encoded though and they will keep breaking. A fix would b...
Loïc Dachary
06:54 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
@Loic: Leave out the "git/" in the path:... Nathan Cutler
03:34 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
things change a bit, here it is for the record, from teuthology.front.sepia.ceph.com... Loïc Dachary
01:18 AM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
David Galloway

03/03/2016

05:12 PM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
This is also causing the load on teuthology.front to hover around 70 because downburst is doing *something* insane. I... Zack Cerza
04:08 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
It happens on teuthology.front.sepia.ceph.com
Loïc Dachary

03/02/2016

08:28 AM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
I would open a PR to fix this, but this website page doesn't seem to be in the source tree? Nathan Cutler

03/01/2016

10:39 PM Bug #14943 (Closed): 404 link on webpage http://ceph.com/resources/development/
On http://ceph.com/resources/development/, the paragraph ... Aaron T
08:11 PM Bug #14619 (In Progress): mira039 missing drive 5
Updated RAID controller firmware from V1.49 2011-08-24 to V1.52 2015-11-20 and system boots considerably faster durin... David Galloway
04:08 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
Thanks for the update David (and the resolution), much appreciated. Loïc Dachary

02/29/2016

10:43 PM Bug #14768 (In Progress): mira009 dead
Lots of logs in syslog indicating DIMM has gone bad... David Galloway
10:38 PM Bug #14915: unable to access logs from http://qa-proxy.ceph.com/teuthology/
This was a result of the teuthology host locking up on Saturday. The system locked up due to a runaway vi process un... David Galloway
10:36 PM Bug #14840 (Resolved): mira091 is not accessible
I ran memtest on this machine without issue.
I updated its BIOS and set the VPSes back up. If the issue persists,...
David Galloway
10:23 PM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
Tracker had a few mysqld instanced causing CPU soft lockups. I've updated all packages on the system and restarted m... David Galloway
06:12 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
(virtualenv)loic@teuthology:~$ time curl http://tracker.ceph.com/issues/14919
real 0m24.707s
(virtualenv)loic@teuth...
Loïc Dachary
05:56 AM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
From the teuthology.front machine ... Loïc Dachary
10:13 PM Bug #14859: xiaoxi access to the lab
Can you enable more verbose output and provide the IP you're attempting to connect from please? I don't even see you... David Galloway
04:42 PM Bug #14926 (Can't reproduce): jenkins.ceph.com signed-off-by false negative
https://jenkins.ceph.com/job/ceph-pr-commits/2768/console Loïc Dachary
03:56 AM Bug #14841 (Resolved): sepia/zyan: Access to the sepia lab
It works Zheng Yan

02/27/2016

09:00 PM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
Cannot access any teuthology log from qa-proxy.cpeh.com/teuthology
504 Gateway Time-out
nginx/1.6.3
Vasu Kulkarni
02:30 PM Bug #14859: xiaoxi access to the lab
Hi David,
I cannot connect the OpenVPN due to auth error,
2016-02-27 22:25:02 VERIFY EKU OK
2016-02-27 ...
Xiaoxi Chen

02/26/2016

03:20 AM Bug #14841: sepia/zyan: Access to the sepia lab
Zheng, I put the new password hash into place; can you try again please Dan Mick
02:31 AM Bug #14841 (In Progress): sepia/zyan: Access to the sepia lab
Zheng Yan
02:30 AM Bug #14841: sepia/zyan: Access to the sepia lab
sorry, I ruined my OpenVPN secret
please change it to:
./new-client zyan@redhat
Please submit the following ...
Zheng Yan

02/25/2016

11:19 PM Support #14704 (Resolved): Rack tala or saya
Finally got around to replacing the uplink with a 10Gb cable. Resolving. David Galloway
05:37 PM Bug #14841: sepia/zyan: Access to the sepia lab
Hi,
Thanks for your patience and understanding my wanting to verify your identity.
I've added your hashed VPN p...
David Galloway
05:17 PM Bug #14859: xiaoxi access to the lab
Hi Xiaoxi,
You should now be able to access the lab and schedule tests.
Please verify you can connect to our Op...
David Galloway
05:22 AM Bug #14859: xiaoxi access to the lab
Username:
xiaoxichen
SSH key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTP...
Xiaoxi Chen
05:00 AM Bug #14859 (Closed): xiaoxi access to the lab
Xiaoxi Chen aka xiaoxi is working on http://tracker.ceph.com/projects/ceph-releases he needs access to the lab to:
...
Loïc Dachary
04:57 PM Cleanup #14868 (Resolved): Audit user list
- Verify all users with OpenVPN access and ssh pubkeys still need access
- Mixture of "X left the company" and e-m...
David Galloway

02/24/2016

03:11 AM Bug #14841: sepia/zyan: Access to the sepia lab
please replace the existing public key and OpenVPN password.
thanks
Zheng Yan

02/23/2016

04:41 PM Bug #14841 (Need More Info): sepia/zyan: Access to the sepia lab
Should this replace your existing pubkey and OpenVPN password or is this in addition to the existing one? David Galloway
02:50 AM Bug #14841 (Resolved): sepia/zyan: Access to the sepia lab
I need to access sepia lab to schedule tests.
user: zyan
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9...
Zheng Yan
07:07 AM Bug #14836: http://pad.ceph.com/ Unable to connect
thanks david ! Loïc Dachary

02/22/2016

10:53 PM Bug #14840 (Closed): mira091 is not accessible
via ssh and/or ipmitool Yuri Weinstein
06:35 PM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
The httpd process acting as the reverse proxy failed to restart after rsyslogd ran its daily cycle.... David Galloway
07:02 AM Bug #14836: http://pad.ceph.com/ Unable to connect
there are informations in pad.ceph.com that I need to implement key management in ceph-disk. If it's gone that's not ... Loïc Dachary
06:02 AM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
Loïc Dachary

02/16/2016

11:33 PM Bug #14778 (Resolved): mira090 3 of 8 drives failing/missing
ubuntu@mira090:~$ sudo /usr/libexec/smart.pl
3 of 8 drives failing/missing |
Drive 3 N.A.
Drive 4 (sdd) has 364 r...
Yuri Weinstein
04:39 PM Bug #14749 (Resolved): smithi049 and smithi041 rebooting constantly
This was a strange one to debug.
I stayed logged in to both machines tailing @/var/log/{auth.log,syslog}@ and noti...
Zack Cerza

02/15/2016

11:46 PM Bug #14763 (Resolved): teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
Timeouts are still intermittent but mostly resolved. Lingering network issues are being tracked in http://tracker.ce... David Galloway
02:35 PM Bug #14763: teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
and gitbuilder is not able to access github now, see http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-deb-trusty-amd6... Kefu Chai
06:50 AM Bug #14763: teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
... Kefu Chai
05:34 AM Bug #14763 (Resolved): teuthology.front.sepia.ceph.com: unable to connect to git.ceph.com
... Loïc Dachary
11:46 PM Bug #14767 (Resolved): Massive packet loss to/from gw.sepia.ceph.com
Timeouts are still intermittent but mostly resolved. Lingering network issues are still being investigated by IT.
...
David Galloway
11:44 PM Bug #14767 (In Progress): Massive packet loss to/from gw.sepia.ceph.com
gw to google now... David Galloway
05:39 PM Bug #14767: Massive packet loss to/from gw.sepia.ceph.com
Yet, it's not entirely broken:... Zack Cerza
05:36 PM Bug #14767: Massive packet loss to/from gw.sepia.ceph.com
From gw to google.com:... Zack Cerza
05:35 PM Bug #14767: Massive packet loss to/from gw.sepia.ceph.com
From my laptop in CO to gw:... Zack Cerza
05:34 PM Bug #14767 (Resolved): Massive packet loss to/from gw.sepia.ceph.com
Zack Cerza
06:27 PM Bug #14768 (Resolved): mira009 dead
David Galloway
04:47 PM Bug #14749: smithi049 and smithi041 rebooting constantly
I marked smithi041 and smithi049 down. Yuri provided these logs just now:
http://qa-proxy.ceph.com/teuthology/sage...
Zack Cerza

02/12/2016

09:33 PM Bug #14749: smithi049 and smithi041 rebooting constantly
smithi041 currently responds to apt-get update with
E: dpkg was interrupted, you must manually run 'sudo dpkg --co...
Dan Mick
09:25 PM Bug #14749: smithi049 and smithi041 rebooting constantly
see also
smithi049 smithi041
Yuri Weinstein
07:12 PM Bug #14749: smithi049 and smithi041 rebooting constantly
From http://qa-proxy.ceph.com/teuthology/sage-2016-02-12_05:26:12-rados-wip-sage-testing---basic-smithi/7136/teutholo... Zack Cerza
06:31 PM Bug #14749: smithi049 and smithi041 rebooting constantly
@teuthology.log@:... Zack Cerza
06:24 PM Bug #14749: smithi049 and smithi041 rebooting constantly
As of right now, mira049's job history, most recent first:
running http://pulpito.front.sepia.ceph.com/sage-2016-02-...
Zack Cerza
05:51 PM Bug #14749 (Resolved): smithi049 and smithi041 rebooting constantly
Majority of nodes that failed during @nuke/stale@ operation fail with ... Yuri Weinstein
08:33 PM Support #14704: Rack tala or saya
These are accessible on their front ifaces now. I'd still like to replace the connection with a 10Gb cable once IT c... David Galloway
08:31 PM Support #14722 (Resolved): VPN Access to DELL Ceph Benchmark Systems
VPN set up and working
https://github.com/ceph/cookbook-gw/pull/8
David Galloway
04:17 AM Bug #14733 (Resolved): Could not fetch updated apt files
Dan Mick
04:16 AM Bug #14733: Could not fetch updated apt files
See also https://bugs.launchpad.net/launchpad/+bug/1544806
Dan Mick
12:58 AM Cleanup #14744 (Resolved): Clean up epel and rhel synced repos on gitbuilder-archive
Commenting out repomirror.sh from cron
Seems unnecessary
Backup of script is in /home/backup/
David Galloway

02/11/2016

11:18 PM Bug #14733: Could not fetch updated apt files
So...the upstream format of the i18n Index files has changed; I'm sure it's an error but we can put a hack into a loc... Dan Mick
04:24 PM Bug #14733: Could not fetch updated apt files
http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-02-11_02:00:01-rados-infernalis-distro-basic-openstack/
http:...
Yuri Weinstein
04:23 PM Bug #14733 (In Progress): Could not fetch updated apt files
... David Galloway
05:31 AM Bug #14733: Could not fetch updated apt files
http://pulpito.ceph.com/loic-2016-02-10_21:25:43-fs-infernalis---basic-multi/5065/ Loïc Dachary
05:03 AM Bug #14733 (Resolved): Could not fetch updated apt files
General failure of some suites today
* http://pulpito.ceph.com/loic-2016-02-10_20:32:58-rados-infernalis-backport...
Loïc Dachary
05:18 PM Bug #12032: activate beanstalkd logs
I set the following:... Zack Cerza
08:32 AM Support #14722: VPN Access to DELL Ceph Benchmark Systems
dmesser@flab mJhwtddvlOI/hkpAGnFr5w 13b92ea100bb995ec71f1d5f6309271c3fca47d20098f0bb0aa9e4906b62f092
Daniel Messer
04:25 AM Bug #14664: valgrind: mmap(...) failed in UME with error 12 (Cannot allocate memory).
Many occurrences at http://tracker.ceph.com/issues/14692 Loïc Dachary

02/10/2016

10:02 PM Support #14614 (Resolved): Add access to Sepia Lab for Joe Mario
David Galloway
09:03 PM Support #14614: Add access to Sepia Lab for Joe Mario
Hi David:
That was it. Thank you for all your help.
Joe
Joe Mario
08:57 PM Support #14614: Add access to Sepia Lab for Joe Mario
Your user was set up as 'jmario' so you'd:... David Galloway
08:54 PM Support #14614: Add access to Sepia Lab for Joe Mario
Hi David:
I am now able to create the VPM and get to the login prompt for teuthology.front.sepia.ceph.com.
Thank ...
Joe Mario
08:59 PM Support #14704: Rack tala or saya
I think this box had 2x 1Gb uplinks and a 10Gb uplink and that's why we can't reach the actual OSes.
I put in a ti...
David Galloway
03:55 AM Support #14704: Rack tala or saya
tala{001..016} have been re-racked and their IPMI interfaces are accessible. I think some magic will be needed in Co... David Galloway
07:08 PM Support #14722: VPN Access to DELL Ceph Benchmark Systems
Daniel,
I need you to follow the steps to generate a hashed password.
https://ceph.github.io/sepia/adding_users...
David Galloway
01:46 PM Support #14722 (Resolved): VPN Access to DELL Ceph Benchmark Systems
Hi,
I am part of the technical team to support the benchmarking exercise we are doing on the 730xd servers which h...
Daniel Messer

02/09/2016

03:32 PM Bug #10798 (Resolved): mkfs.ext4 fails with "apparently in use by the system"
assuming this is fixed on that mira Sage Weil
12:51 AM Support #14704 (Resolved): Rack tala or saya
For 32-bit ARM testing David Galloway

02/08/2016

11:55 PM Cleanup #14528: Track down usage and purpose of mira{123..126} aka dubia{001..004}
mira123 (formerly dubia001) is full of healthy 4TB drives
mira124 (formerly dubia002)'s disks are almost all bad
...
David Galloway
09:57 PM Bug #14548 (Can't reproduce): mira052 MCE
memtest passed. Will monitor host for future failures and troubleshoot further if needed.
Reinstalled and released.
David Galloway
06:27 PM Support #14614: Add access to Sepia Lab for Joe Mario
Joe,
I changed your user string on the server. Try again and let me know how things go.
David Galloway
04:54 PM Feature #14696 (Resolved): Migrate from PowerDNS to (probably) BIND with version control
David Galloway
 

Also available in: Atom