Project

General

Profile

Activity

From 02/24/2016 to 03/24/2016

03/24/2016

11:12 PM Support #14722: VPN Access to DELL Ceph Benchmark Systems
Hey David - it seems my account stopped working, several ovpn clients state that the password is rejected. Did it exp... Daniel Messer
09:44 PM Bug #15215 (Need More Info): smithfarm cannot ssh to test nodes, and therefore cannot teuthology-...
Nathan,
Is this the key in your pubkey file or do you need me to add another one?
https://github.com/ceph/keys/...
David Galloway
09:38 PM Bug #15215 (In Progress): smithfarm cannot ssh to test nodes, and therefore cannot teuthology-kill
David Galloway
09:34 PM Support #15237: Getting access to Sepia
Erwan,
You should be all set. Would you please verify you can connect to the VPN and ssh as 'erwan' to teuthology...
David Galloway
04:14 PM Bug #15166 (Closed): broken sudoers file?
David Galloway
04:10 PM Bug #15208 (Resolved): Reimage smithi022
Reimaged David Galloway

03/23/2016

08:17 AM Support #15237: Getting access to Sepia
I forgot to add the sha1sum of my rsa key : 165bb9de2a949be36c1c0aaf652222e52b7e9e03 Anonymous
08:16 AM Support #15237: Getting access to Sepia
Username: erwan or evelu
I'm adding my rsa key in attachment.
My VPN HASH is :
erwan@r1 nw4KtNSFj1/TbIy95RE2q...
Anonymous

03/22/2016

06:59 PM Cleanup #15001 (Resolved): Migrate ceph.com zone from Dreamhost to Red Hat external DNS
Completed 18MAR2016 David Galloway
05:22 PM Bug #15229: smithi004 ansible failure on yum
Reimaging. See http://tracker.ceph.com/issues/15214 David Galloway
05:21 PM Bug #15229 (In Progress): smithi004 ansible failure on yum
David Galloway
04:48 PM Support #15237: Getting access to Sepia
Erwan,
Can you go through this process please? Ping me if you have any questions.
https://ceph.github.io/sepia...
David Galloway
04:05 PM Support #15237 (Resolved): Getting access to Sepia
I'm Erwan Velu, just join Greg's Team and as per Yuri Weinstein request, I'm opening a ticket to get access to Sepia.... Anonymous

03/21/2016

08:53 PM Bug #15229 (Resolved): smithi004 ansible failure on yum
AnsibleFailedError: {'smithi004.front.sepia.ceph.com': {'item': 'boost-random,boost-program-options,leveldb,xmlstarle... Samuel Just
08:38 PM Bug #15228 (Resolved): mira121 has 2 failing drives
Not marked down... Yuri Weinstein

03/19/2016

05:06 PM Bug #15215 (Resolved): smithfarm cannot ssh to test nodes, and therefore cannot teuthology-kill
I can schedule suites, but I cannot kill them. When I run, for example:... Nathan Cutler

03/18/2016

11:55 PM Bug #15172 (Resolved): 2 smithis need reimage
Reimaged and released David Galloway
09:48 PM Bug #15172 (In Progress): 2 smithis need reimage
David Galloway
09:45 PM Support #15191 (Resolved): download.ceph.com does not work via IPv6 Only
Thank you for bringing this to our attention.
The DNS record has been updated and server has been configured to li...
David Galloway
05:11 AM Support #15191 (Resolved): download.ceph.com does not work via IPv6 Only
Posted this to users@ceph.com, but didn't get a response.. give I think its a bug.. i'll raise it here.
I’m trying...
David LeVene
09:15 PM Support #15174 (Resolved): Requesting lab access
User access established.
Verified with Frank via IRC.
David Galloway
06:58 PM Support #15174 (In Progress): Requesting lab access
David Galloway
08:06 PM Bug #15208: Reimage smithi022
smithi022 was just reimaged yesterday by me. Perhaps this would be a good opportunity to investigate what is corrupt... David Galloway
07:58 PM Bug #15208 (Resolved): Reimage smithi022
looks like rpmdb corruption
http://qa-proxy.ceph.com/teuthology/teuthology-2016-03-18_08:14:44-powercycle-jewel-te...
Yuri Weinstein
03:08 PM Bug #15199 (Resolved): git.ceph.com[0: 67.205.20.229]: errno=Connection timed out
External DNS for the ceph.com domain was transferred over to Red Hat IT's DNS last night.
The record for git.ceph....
David Galloway
02:46 PM Bug #15199: git.ceph.com[0: 67.205.20.229]: errno=Connection timed out
DNS is wrong. It should point to 66.33.193.78 but is going to 67.205.20.229 instead. The new zone file just needs t... Sage Weil
01:07 PM Bug #15199 (Resolved): git.ceph.com[0: 67.205.20.229]: errno=Connection timed out
from teuthology.sepia... Loïc Dachary

03/17/2016

05:43 PM Support #15174 (Resolved): Requesting lab access
Hello,
I have recently joined the Red Hat Ceph team and will be working with teuthology, I expect to need to sched...
Frank Filz
12:52 PM Bug #15172 (Resolved): 2 smithis need reimage
smithi028
smithi013
Sage Weil
05:33 AM Bug #15166: broken sudoers file?
So, yeah, this is basically an archival ticket as it seems to be working now. But in case it comes up again, here's o... Greg Farnum
05:32 AM Bug #15166 (Closed): broken sudoers file?
http://qa-proxy.ceph.com/teuthology/gregf-2016-03-16_13:01:47-fs-greg-fs-testing-316---basic-mira/66399/... Greg Farnum

03/16/2016

07:19 PM Bug #15158 (Resolved): reimaged smithi have nrpe enabled?
Sage Weil wrote:
> i did this
>
> for f in 004 025 002 029 023 018 018 020 023 024 025 020 002 004 017 001 008 ;...
David Galloway
03:20 PM Bug #15158: reimaged smithi have nrpe enabled?
i did this
for f in 004 025 002 029 023 018 018 020 023 024 025 020 002 004 017 001 008 ; do echo $f ; ssh smithi...
Sage Weil
01:44 PM Bug #15158 (Resolved): reimaged smithi have nrpe enabled?
I'm getting selinux denials:
type=AVC msg=audit(1458134560.409:4128): avc: denied { getattr } for pid=18680 comm="...
Sage Weil
02:15 AM Bug #15120 (Resolved): smithis with bad rpmdbs
Issue with 2 hour DHCP lease expiring after time zone had been changed 4 hours difference.
We changed our DHCP lea...
David Galloway
12:04 AM Bug #15126 (Won't Fix): Rename NIC1 to eth0 on CentOS testnodes
I was mistaken. This is not the case.
What was happening was our DHCP server was handing out a 2 hour lease.
O...
David Galloway

03/15/2016

07:40 PM Bug #15147 (Resolved): mira095 RAID6 degraded
1 ARC-1222-VOL#000 Raid Set # 000 Raid6 80.0GB 00/00/00 Degraded David Galloway

03/14/2016

10:39 PM Bug #15126 (Won't Fix): Rename NIC1 to eth0 on CentOS testnodes
Freshly installed testnodes (mira and smithi) are left with their primary NIC in a down state at the end of an ansibl... David Galloway
09:20 PM Bug #15120: smithis with bad rpmdbs
smithi004 - http://pulpito.ceph.com/sage-2016-03-14_14:05:31-rados-wip-sage-testing---basic-smithi/59112
smithi025 -...
Sage Weil
09:16 PM Bug #15120: smithis with bad rpmdbs
smithi020 - http://pulpito.ceph.com/sage-2016-03-14_14:05:31-rados-wip-sage-testing---basic-smithi/59108 (not part o... Sage Weil
09:15 PM Bug #15120: smithis with bad rpmdbs
smithi001
smithi018
smithi023
failed on their first runs out of the gate.
Sage Weil
06:41 PM Bug #15120: smithis with bad rpmdbs
Systems reimaged David Galloway
06:40 PM Bug #15120 (Resolved): smithis with bad rpmdbs
Reimaged David Galloway
05:39 PM Bug #15120 (In Progress): smithis with bad rpmdbs
Reimaging David Galloway
05:23 PM Bug #15120 (Resolved): smithis with bad rpmdbs
smithi001
smithi002
smithi004
smithi008
smithi018
smithi020
smithi023
smithi024
smithi025
need reimage?
Sage Weil
09:01 PM Bug #15118: need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
I will run some tests that use centos on mira, I can update the other ticket if I dont find any issues and the origin... Vasu Kulkarni
08:57 PM Bug #15118: need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
I should mention.. the 6 machines I reimaged did have their Areca firmwares updated prior to rebooting. I'm surprise... David Galloway
08:44 PM Bug #15118 (Resolved): need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
David Galloway
08:30 PM Bug #15118: need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
David,
That is sufficient for now, Can you also release them into free pool. Thanks.
Vasu Kulkarni
08:15 PM Bug #15118 (Need More Info): need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
We have 70 mira that are up and used as standalone testnodes.
I just provisioned 6 as CentOS testnodes....
David Galloway
04:41 PM Bug #15118 (Resolved): need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
Right now only 5 /126 mira's are on centos and this is not sufficient for many tests, I think 15 to 20 nodes on cento... Vasu Kulkarni
07:04 PM Bug #14926: jenkins.ceph.com signed-off-by false negative
It is not possible to debug this any longer. Please copy and paste the relevant log output you may find (along with t... Alfredo Deza
06:55 PM Bug #14290 (Resolved): Add public IP to yan-zheng; make sure packages up to date, no passwords (a...
Set up circle.front for this purpose.
Automatic security updates are enabled.
Will document further on internal...
David Galloway
06:40 PM Bug #15083 (Resolved): smithi024 marked down needs reimage
Reimaged David Galloway
05:49 PM Bug #15116 (Resolved): mira007 unreachable
Nothing in syslog or kern.log from the time the system hung until reboot.
I flashed the latest BIOS for the mainbo...
David Galloway
05:42 PM Bug #15116 (In Progress): mira007 unreachable
Zack Cerza wrote:
> Do we not have nagios set up on vm hosts?
We do. A notification went out on Saturday that th...
David Galloway
04:59 PM Bug #15116: mira007 unreachable
Do we not have nagios set up on vm hosts? Zack Cerza
04:56 PM Bug #15116: mira007 unreachable
rebooting mira007 Zack Cerza
04:53 PM Bug #15116: mira007 unreachable
Marked VMs down David Galloway
01:23 PM Bug #15116 (Resolved): mira007 unreachable
Seeing this in multiple runs, e.g. http://qa-proxy.ceph.com/teuthology/teuthology-2016-03-13_17:10:11-upgrade:inferna... Nathan Cutler

03/12/2016

04:47 PM Bug #15105: smithi004's rpm database is broken
just logon to smithi004 and rebuild the rpmdb.
then ...
Kefu Chai
04:36 PM Bug #15105 (Resolved): smithi004's rpm database is broken
... Kefu Chai

03/11/2016

10:45 PM Bug #14840: mira091 is not accessible
cycled power and it came back up; syslog doesn't seem to have anything very useful in it, nor does kern.log.
I do...
Dan Mick
09:53 PM Bug #14840 (New): mira091 is not accessible
Yuri Weinstein
09:52 PM Bug #14840: mira091 is not accessible
Reopening this as:... Yuri Weinstein
05:08 PM Bug #15083 (Resolved): smithi024 marked down needs reimage
[ubuntu@smithi024 ~]$ sudo apt-get install -f
ubuntu is not in the sudoers file. This incident will be reported.
[...
Yuri Weinstein
03:32 AM Bug #14859: xiaoxi access to the lab
My Public IP for this attemp is 211.97.128.224 Xiaoxi Chen
03:30 AM Bug #14859: xiaoxi access to the lab
*Tunnelblick: OS X 10.11.3; Tunnelblick 3.5.7 (build 4270.4517); Admin user
Configuration client
"Sanitized" co...
Xiaoxi Chen

03/10/2016

11:11 PM Bug #15052: Rebalance vps instances to allocate more RAM
I manually enabled @sar@ and told it to sample every second for these tests. Bear in mind that @teuthology.front@ use... Zack Cerza
10:56 PM Bug #15052: Rebalance vps instances to allocate more RAM
3/10/16
Issue #14985 has been updated by Yuri Weinstein.
The job below produces 'ceph::buffer::bad_alloc' error...
Zack Cerza
10:55 PM Bug #15052: Rebalance vps instances to allocate more RAM
3/10/16
Issue #14985 has been updated by Yuri Weinstein.
> Zack Cerza wrote:
>> Yuri Weinstein wrote:
>> For te...
Zack Cerza
10:54 PM Bug #15052: Rebalance vps instances to allocate more RAM
3/10/16
Issue #14985 has been updated by Yuri Weinstein.
Another good job for low memory testing:...
Zack Cerza
10:53 PM Bug #15052: Rebalance vps instances to allocate more RAM
3/10/16
Issue #14985 has been updated by Yuri Weinstein.
For testing - this job seems to be reliably hanging on...
Zack Cerza
10:53 PM Bug #15052: Rebalance vps instances to allocate more RAM
3/10/16
Issue #14985 has been updated by Yuri Weinstein.
Note: list of jobs with 'bad_alloc' error on vps
<pre...
Zack Cerza
10:51 PM Bug #15052: Rebalance vps instances to allocate more RAM
3/8/16
Issue #14985 has been updated by Tamilarasi muthamizhan.
hi Yuri, brought this topic up in the weekly le...
Zack Cerza
10:50 PM Bug #15052 (Resolved): Rebalance vps instances to allocate more RAM
I accidentally deleted the original ticket (#14985) so I'll have to recreate this from my email history. The original... Zack Cerza
12:16 PM Bug #15044 (Resolved): Read-only filesystem on mira096
On this run http://pulpito.ceph.com/loic-2016-03-07_21:25:36-rgw-hammer-backports---basic-multi/ in job 45886
"rgw/m...
Nathan Cutler

03/09/2016

09:12 PM Bug #13763: gitbuilder seems to have bad sudo setup
FYI: Manually fixed centos6-5 gitbuilder due to this problem. See http://tracker.ceph.com/issues/14993 David Galloway
08:31 PM Bug #14630 (Resolved): mira020 root drive failure, BMC issues?
Drive 4 replaced David Galloway
03:15 AM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
@David - AFAICT the version actually being served is still the same (I tried reloading). Nathan Cutler
12:07 AM Bug #14836: http://pad.ceph.com/ Unable to connect
Great, that gives us all hope we're good for some time now :-) Loïc Dachary
12:04 AM Bug #14993: centos6 gitbuilder broken
Thanks David ! Loïc Dachary

03/08/2016

10:11 PM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
This (second) outage was due to Dreamhost moving the Etherpad service to another server.
The A record for pad.ceph...
David Galloway
06:27 AM Bug #14836: http://pad.ceph.com/ Unable to connect
re-opening to keep track of the frequency of the problem Loïc Dachary
09:40 PM Bug #14993 (Resolved): centos6 gitbuilder broken
Once all the yum repo issues were cleared, up hammer successfully built. David Galloway
05:19 PM Bug #14993: centos6 gitbuilder broken
Fixed yum errors by updating repo file. The Centos Base repofile got mangled somehow *and* there was a new *.rpmnew ... David Galloway
02:44 AM Bug #14993 (In Progress): centos6 gitbuilder broken
Loïc Dachary
02:44 AM Bug #14993: centos6 gitbuilder broken
Ha, much better now, thanks !
It still display errorrs that turns gitbuilder red. Have you encoutered them before ...
Loïc Dachary
07:03 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
mira114's network was incorrectly configured. It hadn't been updated since before the lab move. I reran the testnodes... Zack Cerza
06:05 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
http://pulpito.ceph.com/gregf-2016-03-07_23:12:27-fs-greg-fs-testing-3-7-safe---basic-mira/46170/
It looks like it...
Greg Farnum
05:28 PM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
I don't have a login for the wordpress portion of the site but would imagine the link should point to http://docs.cep... David Galloway

03/07/2016

06:58 PM Cleanup #15001 (Resolved): Migrate ceph.com zone from Dreamhost to Red Hat external DNS
David Galloway
06:52 PM Bug #14993: centos6 gitbuilder broken
I added ... David Galloway
03:43 AM Bug #14993: centos6 gitbuilder broken
... Loïc Dachary
03:42 AM Bug #14993 (Resolved): centos6 gitbuilder broken
Every build fail with... Loïc Dachary

03/05/2016

01:06 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
thanks for the hard work dmick !... Loïc Dachary

03/04/2016

11:23 PM Bug #14290 (In Progress): Add public IP to yan-zheng; make sure packages up to date, no passwords...
This is in progress. I have a VM set up in RHEV ready for this purpose. It's just of figuring out how to get it acc... David Galloway
11:12 PM Bug #13392 (Resolved): mira074 has a bad root disk
Drive got replaced on 28JAN2016 and host was reimaged David Galloway
11:09 PM Bug #14768 (Resolved): mira009 dead
DIMMs replaced. Reinstalled host with Ubuntu 14.04 and set back up as fresh VPSHOST while I was at it. David Galloway
10:42 PM Bug #14619: mira039 missing drive 5
Something's up with this machine. I started a reimage on 5 miras at the same time. The other four are done and this... David Galloway
08:04 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
Thanks for the workaround.
A few scripts have that path encoded though and they will keep breaking. A fix would b...
Loïc Dachary
06:54 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
@Loic: Leave out the "git/" in the path:... Nathan Cutler
03:34 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
things change a bit, here it is for the record, from teuthology.front.sepia.ceph.com... Loïc Dachary
01:18 AM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
David Galloway

03/03/2016

05:12 PM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
This is also causing the load on teuthology.front to hover around 70 because downburst is doing *something* insane. I... Zack Cerza
04:08 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
It happens on teuthology.front.sepia.ceph.com
Loïc Dachary

03/02/2016

08:28 AM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
I would open a PR to fix this, but this website page doesn't seem to be in the source tree? Nathan Cutler

03/01/2016

10:39 PM Bug #14943 (Closed): 404 link on webpage http://ceph.com/resources/development/
On http://ceph.com/resources/development/, the paragraph ... Aaron T
08:11 PM Bug #14619 (In Progress): mira039 missing drive 5
Updated RAID controller firmware from V1.49 2011-08-24 to V1.52 2015-11-20 and system boots considerably faster durin... David Galloway
04:08 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
Thanks for the update David (and the resolution), much appreciated. Loïc Dachary

02/29/2016

10:43 PM Bug #14768 (In Progress): mira009 dead
Lots of logs in syslog indicating DIMM has gone bad... David Galloway
10:38 PM Bug #14915: unable to access logs from http://qa-proxy.ceph.com/teuthology/
This was a result of the teuthology host locking up on Saturday. The system locked up due to a runaway vi process un... David Galloway
10:36 PM Bug #14840 (Resolved): mira091 is not accessible
I ran memtest on this machine without issue.
I updated its BIOS and set the VPSes back up. If the issue persists,...
David Galloway
10:23 PM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
Tracker had a few mysqld instanced causing CPU soft lockups. I've updated all packages on the system and restarted m... David Galloway
06:12 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
(virtualenv)loic@teuthology:~$ time curl http://tracker.ceph.com/issues/14919
real 0m24.707s
(virtualenv)loic@teuth...
Loïc Dachary
05:56 AM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
From the teuthology.front machine ... Loïc Dachary
10:13 PM Bug #14859: xiaoxi access to the lab
Can you enable more verbose output and provide the IP you're attempting to connect from please? I don't even see you... David Galloway
04:42 PM Bug #14926 (Can't reproduce): jenkins.ceph.com signed-off-by false negative
https://jenkins.ceph.com/job/ceph-pr-commits/2768/console Loïc Dachary
03:56 AM Bug #14841 (Resolved): sepia/zyan: Access to the sepia lab
It works Zheng Yan

02/27/2016

09:00 PM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
Cannot access any teuthology log from qa-proxy.cpeh.com/teuthology
504 Gateway Time-out
nginx/1.6.3
Vasu Kulkarni
02:30 PM Bug #14859: xiaoxi access to the lab
Hi David,
I cannot connect the OpenVPN due to auth error,
2016-02-27 22:25:02 VERIFY EKU OK
2016-02-27 ...
Xiaoxi Chen

02/26/2016

03:20 AM Bug #14841: sepia/zyan: Access to the sepia lab
Zheng, I put the new password hash into place; can you try again please Dan Mick
02:31 AM Bug #14841 (In Progress): sepia/zyan: Access to the sepia lab
Zheng Yan
02:30 AM Bug #14841: sepia/zyan: Access to the sepia lab
sorry, I ruined my OpenVPN secret
please change it to:
./new-client zyan@redhat
Please submit the following ...
Zheng Yan

02/25/2016

11:19 PM Support #14704 (Resolved): Rack tala or saya
Finally got around to replacing the uplink with a 10Gb cable. Resolving. David Galloway
05:37 PM Bug #14841: sepia/zyan: Access to the sepia lab
Hi,
Thanks for your patience and understanding my wanting to verify your identity.
I've added your hashed VPN p...
David Galloway
05:17 PM Bug #14859: xiaoxi access to the lab
Hi Xiaoxi,
You should now be able to access the lab and schedule tests.
Please verify you can connect to our Op...
David Galloway
05:22 AM Bug #14859: xiaoxi access to the lab
Username:
xiaoxichen
SSH key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTP...
Xiaoxi Chen
05:00 AM Bug #14859 (Closed): xiaoxi access to the lab
Xiaoxi Chen aka xiaoxi is working on http://tracker.ceph.com/projects/ceph-releases he needs access to the lab to:
...
Loïc Dachary
04:57 PM Cleanup #14868 (Resolved): Audit user list
- Verify all users with OpenVPN access and ssh pubkeys still need access
- Mixture of "X left the company" and e-m...
David Galloway

02/24/2016

03:11 AM Bug #14841: sepia/zyan: Access to the sepia lab
please replace the existing public key and OpenVPN password.
thanks
Zheng Yan
 

Also available in: Atom