Activity
From 02/24/2016 to 03/24/2016
03/24/2016
- 11:12 PM Support #14722: VPN Access to DELL Ceph Benchmark Systems
- Hey David - it seems my account stopped working, several ovpn clients state that the password is rejected. Did it exp...
- 09:44 PM Bug #15215 (Need More Info): smithfarm cannot ssh to test nodes, and therefore cannot teuthology-...
- Nathan,
Is this the key in your pubkey file or do you need me to add another one?
https://github.com/ceph/keys/... - 09:38 PM Bug #15215 (In Progress): smithfarm cannot ssh to test nodes, and therefore cannot teuthology-kill
- 09:34 PM Support #15237: Getting access to Sepia
- Erwan,
You should be all set. Would you please verify you can connect to the VPN and ssh as 'erwan' to teuthology... - 04:14 PM Bug #15166 (Closed): broken sudoers file?
- 04:10 PM Bug #15208 (Resolved): Reimage smithi022
- Reimaged
03/23/2016
- 08:17 AM Support #15237: Getting access to Sepia
- I forgot to add the sha1sum of my rsa key : 165bb9de2a949be36c1c0aaf652222e52b7e9e03
- 08:16 AM Support #15237: Getting access to Sepia
- Username: erwan or evelu
I'm adding my rsa key in attachment.
My VPN HASH is :
erwan@r1 nw4KtNSFj1/TbIy95RE2q...
03/22/2016
- 06:59 PM Cleanup #15001 (Resolved): Migrate ceph.com zone from Dreamhost to Red Hat external DNS
- Completed 18MAR2016
- 05:22 PM Bug #15229: smithi004 ansible failure on yum
- Reimaging. See http://tracker.ceph.com/issues/15214
- 05:21 PM Bug #15229 (In Progress): smithi004 ansible failure on yum
- 04:48 PM Support #15237: Getting access to Sepia
- Erwan,
Can you go through this process please? Ping me if you have any questions.
https://ceph.github.io/sepia... - 04:05 PM Support #15237 (Resolved): Getting access to Sepia
- I'm Erwan Velu, just join Greg's Team and as per Yuri Weinstein request, I'm opening a ticket to get access to Sepia....
03/21/2016
- 08:53 PM Bug #15229 (Resolved): smithi004 ansible failure on yum
- AnsibleFailedError: {'smithi004.front.sepia.ceph.com': {'item': 'boost-random,boost-program-options,leveldb,xmlstarle...
- 08:38 PM Bug #15228 (Resolved): mira121 has 2 failing drives
- Not marked down...
03/19/2016
- 05:06 PM Bug #15215 (Resolved): smithfarm cannot ssh to test nodes, and therefore cannot teuthology-kill
- I can schedule suites, but I cannot kill them. When I run, for example:...
03/18/2016
- 11:55 PM Bug #15172 (Resolved): 2 smithis need reimage
- Reimaged and released
- 09:48 PM Bug #15172 (In Progress): 2 smithis need reimage
- 09:45 PM Support #15191 (Resolved): download.ceph.com does not work via IPv6 Only
- Thank you for bringing this to our attention.
The DNS record has been updated and server has been configured to li... - 05:11 AM Support #15191 (Resolved): download.ceph.com does not work via IPv6 Only
- Posted this to users@ceph.com, but didn't get a response.. give I think its a bug.. i'll raise it here.
I’m trying... - 09:15 PM Support #15174 (Resolved): Requesting lab access
- User access established.
Verified with Frank via IRC. - 06:58 PM Support #15174 (In Progress): Requesting lab access
- 08:06 PM Bug #15208: Reimage smithi022
- smithi022 was just reimaged yesterday by me. Perhaps this would be a good opportunity to investigate what is corrupt...
- 07:58 PM Bug #15208 (Resolved): Reimage smithi022
- looks like rpmdb corruption
http://qa-proxy.ceph.com/teuthology/teuthology-2016-03-18_08:14:44-powercycle-jewel-te... - 03:08 PM Bug #15199 (Resolved): git.ceph.com[0: 67.205.20.229]: errno=Connection timed out
- External DNS for the ceph.com domain was transferred over to Red Hat IT's DNS last night.
The record for git.ceph.... - 02:46 PM Bug #15199: git.ceph.com[0: 67.205.20.229]: errno=Connection timed out
- DNS is wrong. It should point to 66.33.193.78 but is going to 67.205.20.229 instead. The new zone file just needs t...
- 01:07 PM Bug #15199 (Resolved): git.ceph.com[0: 67.205.20.229]: errno=Connection timed out
- from teuthology.sepia...
03/17/2016
- 05:43 PM Support #15174 (Resolved): Requesting lab access
- Hello,
I have recently joined the Red Hat Ceph team and will be working with teuthology, I expect to need to sched... - 12:52 PM Bug #15172 (Resolved): 2 smithis need reimage
- smithi028
smithi013 - 05:33 AM Bug #15166: broken sudoers file?
- So, yeah, this is basically an archival ticket as it seems to be working now. But in case it comes up again, here's o...
- 05:32 AM Bug #15166 (Closed): broken sudoers file?
- http://qa-proxy.ceph.com/teuthology/gregf-2016-03-16_13:01:47-fs-greg-fs-testing-316---basic-mira/66399/...
03/16/2016
- 07:19 PM Bug #15158 (Resolved): reimaged smithi have nrpe enabled?
- Sage Weil wrote:
> i did this
>
> for f in 004 025 002 029 023 018 018 020 023 024 025 020 002 004 017 001 008 ;... - 03:20 PM Bug #15158: reimaged smithi have nrpe enabled?
- i did this
for f in 004 025 002 029 023 018 018 020 023 024 025 020 002 004 017 001 008 ; do echo $f ; ssh smithi... - 01:44 PM Bug #15158 (Resolved): reimaged smithi have nrpe enabled?
- I'm getting selinux denials:
type=AVC msg=audit(1458134560.409:4128): avc: denied { getattr } for pid=18680 comm="... - 02:15 AM Bug #15120 (Resolved): smithis with bad rpmdbs
- Issue with 2 hour DHCP lease expiring after time zone had been changed 4 hours difference.
We changed our DHCP lea... - 12:04 AM Bug #15126 (Won't Fix): Rename NIC1 to eth0 on CentOS testnodes
- I was mistaken. This is not the case.
What was happening was our DHCP server was handing out a 2 hour lease.
O...
03/15/2016
- 07:40 PM Bug #15147 (Resolved): mira095 RAID6 degraded
- 1 ARC-1222-VOL#000 Raid Set # 000 Raid6 80.0GB 00/00/00 Degraded
03/14/2016
- 10:39 PM Bug #15126 (Won't Fix): Rename NIC1 to eth0 on CentOS testnodes
- Freshly installed testnodes (mira and smithi) are left with their primary NIC in a down state at the end of an ansibl...
- 09:20 PM Bug #15120: smithis with bad rpmdbs
- smithi004 - http://pulpito.ceph.com/sage-2016-03-14_14:05:31-rados-wip-sage-testing---basic-smithi/59112
smithi025 -... - 09:16 PM Bug #15120: smithis with bad rpmdbs
- smithi020 - http://pulpito.ceph.com/sage-2016-03-14_14:05:31-rados-wip-sage-testing---basic-smithi/59108 (not part o...
- 09:15 PM Bug #15120: smithis with bad rpmdbs
- smithi001
smithi018
smithi023
failed on their first runs out of the gate. - 06:41 PM Bug #15120: smithis with bad rpmdbs
- Systems reimaged
- 06:40 PM Bug #15120 (Resolved): smithis with bad rpmdbs
- Reimaged
- 05:39 PM Bug #15120 (In Progress): smithis with bad rpmdbs
- Reimaging
- 05:23 PM Bug #15120 (Resolved): smithis with bad rpmdbs
- smithi001
smithi002
smithi004
smithi008
smithi018
smithi020
smithi023
smithi024
smithi025
need reimage? - 09:01 PM Bug #15118: need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
- I will run some tests that use centos on mira, I can update the other ticket if I dont find any issues and the origin...
- 08:57 PM Bug #15118: need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
- I should mention.. the 6 machines I reimaged did have their Areca firmwares updated prior to rebooting. I'm surprise...
- 08:44 PM Bug #15118 (Resolved): need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
- 08:30 PM Bug #15118: need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
- David,
That is sufficient for now, Can you also release them into free pool. Thanks. - 08:15 PM Bug #15118 (Need More Info): need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
- We have 70 mira that are up and used as standalone testnodes.
I just provisioned 6 as CentOS testnodes.... - 04:41 PM Bug #15118 (Resolved): need atleast 15 to 20 mira's to be on centos 7.1 or 7.2
- Right now only 5 /126 mira's are on centos and this is not sufficient for many tests, I think 15 to 20 nodes on cento...
- 07:04 PM Bug #14926: jenkins.ceph.com signed-off-by false negative
- It is not possible to debug this any longer. Please copy and paste the relevant log output you may find (along with t...
- 06:55 PM Bug #14290 (Resolved): Add public IP to yan-zheng; make sure packages up to date, no passwords (a...
- Set up circle.front for this purpose.
Automatic security updates are enabled.
Will document further on internal... - 06:40 PM Bug #15083 (Resolved): smithi024 marked down needs reimage
- Reimaged
- 05:49 PM Bug #15116 (Resolved): mira007 unreachable
- Nothing in syslog or kern.log from the time the system hung until reboot.
I flashed the latest BIOS for the mainbo... - 05:42 PM Bug #15116 (In Progress): mira007 unreachable
- Zack Cerza wrote:
> Do we not have nagios set up on vm hosts?
We do. A notification went out on Saturday that th... - 04:59 PM Bug #15116: mira007 unreachable
- Do we not have nagios set up on vm hosts?
- 04:56 PM Bug #15116: mira007 unreachable
- rebooting mira007
- 04:53 PM Bug #15116: mira007 unreachable
- Marked VMs down
- 01:23 PM Bug #15116 (Resolved): mira007 unreachable
- Seeing this in multiple runs, e.g. http://qa-proxy.ceph.com/teuthology/teuthology-2016-03-13_17:10:11-upgrade:inferna...
03/12/2016
- 04:47 PM Bug #15105: smithi004's rpm database is broken
- just logon to smithi004 and rebuild the rpmdb.
then ... - 04:36 PM Bug #15105 (Resolved): smithi004's rpm database is broken
- ...
03/11/2016
- 10:45 PM Bug #14840: mira091 is not accessible
- cycled power and it came back up; syslog doesn't seem to have anything very useful in it, nor does kern.log.
I do... - 09:53 PM Bug #14840 (New): mira091 is not accessible
- 09:52 PM Bug #14840: mira091 is not accessible
- Reopening this as:...
- 05:08 PM Bug #15083 (Resolved): smithi024 marked down needs reimage
- [ubuntu@smithi024 ~]$ sudo apt-get install -f
ubuntu is not in the sudoers file. This incident will be reported.
[... - 03:32 AM Bug #14859: xiaoxi access to the lab
- My Public IP for this attemp is 211.97.128.224
- 03:30 AM Bug #14859: xiaoxi access to the lab
- *Tunnelblick: OS X 10.11.3; Tunnelblick 3.5.7 (build 4270.4517); Admin user
Configuration client
"Sanitized" co...
03/10/2016
- 11:11 PM Bug #15052: Rebalance vps instances to allocate more RAM
- I manually enabled @sar@ and told it to sample every second for these tests. Bear in mind that @teuthology.front@ use...
- 10:56 PM Bug #15052: Rebalance vps instances to allocate more RAM
- 3/10/16
Issue #14985 has been updated by Yuri Weinstein.
The job below produces 'ceph::buffer::bad_alloc' error... - 10:55 PM Bug #15052: Rebalance vps instances to allocate more RAM
- 3/10/16
Issue #14985 has been updated by Yuri Weinstein.
> Zack Cerza wrote:
>> Yuri Weinstein wrote:
>> For te... - 10:54 PM Bug #15052: Rebalance vps instances to allocate more RAM
- 3/10/16
Issue #14985 has been updated by Yuri Weinstein.
Another good job for low memory testing:... - 10:53 PM Bug #15052: Rebalance vps instances to allocate more RAM
- 3/10/16
Issue #14985 has been updated by Yuri Weinstein.
For testing - this job seems to be reliably hanging on... - 10:53 PM Bug #15052: Rebalance vps instances to allocate more RAM
- 3/10/16
Issue #14985 has been updated by Yuri Weinstein.
Note: list of jobs with 'bad_alloc' error on vps
<pre... - 10:51 PM Bug #15052: Rebalance vps instances to allocate more RAM
- 3/8/16
Issue #14985 has been updated by Tamilarasi muthamizhan.
hi Yuri, brought this topic up in the weekly le... - 10:50 PM Bug #15052 (Resolved): Rebalance vps instances to allocate more RAM
- I accidentally deleted the original ticket (#14985) so I'll have to recreate this from my email history. The original...
- 12:16 PM Bug #15044 (Resolved): Read-only filesystem on mira096
- On this run http://pulpito.ceph.com/loic-2016-03-07_21:25:36-rgw-hammer-backports---basic-multi/ in job 45886
"rgw/m...
03/09/2016
- 09:12 PM Bug #13763: gitbuilder seems to have bad sudo setup
- FYI: Manually fixed centos6-5 gitbuilder due to this problem. See http://tracker.ceph.com/issues/14993
- 08:31 PM Bug #14630 (Resolved): mira020 root drive failure, BMC issues?
- Drive 4 replaced
- 03:15 AM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
- @David - AFAICT the version actually being served is still the same (I tried reloading).
- 12:07 AM Bug #14836: http://pad.ceph.com/ Unable to connect
- Great, that gives us all hope we're good for some time now :-)
- 12:04 AM Bug #14993: centos6 gitbuilder broken
- Thanks David !
03/08/2016
- 10:11 PM Bug #14836 (Resolved): http://pad.ceph.com/ Unable to connect
- This (second) outage was due to Dreamhost moving the Etherpad service to another server.
The A record for pad.ceph... - 06:27 AM Bug #14836: http://pad.ceph.com/ Unable to connect
- re-opening to keep track of the frequency of the problem
- 09:40 PM Bug #14993 (Resolved): centos6 gitbuilder broken
- Once all the yum repo issues were cleared, up hammer successfully built.
- 05:19 PM Bug #14993: centos6 gitbuilder broken
- Fixed yum errors by updating repo file. The Centos Base repofile got mangled somehow *and* there was a new *.rpmnew ...
- 02:44 AM Bug #14993 (In Progress): centos6 gitbuilder broken
- 02:44 AM Bug #14993: centos6 gitbuilder broken
- Ha, much better now, thanks !
It still display errorrs that turns gitbuilder red. Have you encoutered them before ... - 07:03 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
- mira114's network was incorrectly configured. It hadn't been updated since before the lab move. I reran the testnodes...
- 06:05 PM Bug #15013 (Resolved): ansible: key install failures causing extreme slowness
- http://pulpito.ceph.com/gregf-2016-03-07_23:12:27-fs-greg-fs-testing-3-7-safe---basic-mira/46170/
It looks like it... - 05:28 PM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
- I don't have a login for the wordpress portion of the site but would imagine the link should point to http://docs.cep...
03/07/2016
- 06:58 PM Cleanup #15001 (Resolved): Migrate ceph.com zone from Dreamhost to Red Hat external DNS
- 06:52 PM Bug #14993: centos6 gitbuilder broken
- I added ...
- 03:43 AM Bug #14993: centos6 gitbuilder broken
- ...
- 03:42 AM Bug #14993 (Resolved): centos6 gitbuilder broken
- Every build fail with...
03/05/2016
- 01:06 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
- thanks for the hard work dmick !...
03/04/2016
- 11:23 PM Bug #14290 (In Progress): Add public IP to yan-zheng; make sure packages up to date, no passwords...
- This is in progress. I have a VM set up in RHEV ready for this purpose. It's just of figuring out how to get it acc...
- 11:12 PM Bug #13392 (Resolved): mira074 has a bad root disk
- Drive got replaced on 28JAN2016 and host was reimaged
- 11:09 PM Bug #14768 (Resolved): mira009 dead
- DIMMs replaced. Reinstalled host with Ubuntu 14.04 and set back up as fresh VPSHOST while I was at it.
- 10:42 PM Bug #14619: mira039 missing drive 5
- Something's up with this machine. I started a reimage on 5 miras at the same time. The other four are done and this...
- 08:04 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
- Thanks for the workaround.
A few scripts have that path encoded though and they will keep breaking. A fix would b... - 06:54 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
- @Loic: Leave out the "git/" in the path:...
- 03:34 AM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
- things change a bit, here it is for the record, from teuthology.front.sepia.ceph.com...
- 01:18 AM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
03/03/2016
- 05:12 PM Bug #14959: git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read error: Connec...
- This is also causing the load on teuthology.front to hover around 70 because downburst is doing *something* insane. I...
- 04:08 AM Bug #14959 (Resolved): git ls-remote git://git.ceph.com/git/ceph-qa-suite hammer : fatal: read er...
- It happens on teuthology.front.sepia.ceph.com
03/02/2016
- 08:28 AM Bug #14943: 404 link on webpage http://ceph.com/resources/development/
- I would open a PR to fix this, but this website page doesn't seem to be in the source tree?
03/01/2016
- 10:39 PM Bug #14943 (Closed): 404 link on webpage http://ceph.com/resources/development/
- On http://ceph.com/resources/development/, the paragraph ...
- 08:11 PM Bug #14619 (In Progress): mira039 missing drive 5
- Updated RAID controller firmware from V1.49 2011-08-24 to V1.52 2015-11-20 and system boots considerably faster durin...
- 04:08 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
- Thanks for the update David (and the resolution), much appreciated.
02/29/2016
- 10:43 PM Bug #14768 (In Progress): mira009 dead
- Lots of logs in syslog indicating DIMM has gone bad...
- 10:38 PM Bug #14915: unable to access logs from http://qa-proxy.ceph.com/teuthology/
- This was a result of the teuthology host locking up on Saturday. The system locked up due to a runaway vi process un...
- 10:36 PM Bug #14840 (Resolved): mira091 is not accessible
- I ran memtest on this machine without issue.
I updated its BIOS and set the VPSes back up. If the issue persists,... - 10:23 PM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
- Tracker had a few mysqld instanced causing CPU soft lockups. I've updated all packages on the system and restarted m...
- 06:12 AM Bug #14919: http://tracker.ceph.com/ very slow / timesout
- (virtualenv)loic@teuthology:~$ time curl http://tracker.ceph.com/issues/14919
real 0m24.707s
(virtualenv)loic@teuth... - 05:56 AM Bug #14919 (Resolved): http://tracker.ceph.com/ very slow / timesout
- From the teuthology.front machine ...
- 10:13 PM Bug #14859: xiaoxi access to the lab
- Can you enable more verbose output and provide the IP you're attempting to connect from please? I don't even see you...
- 04:42 PM Bug #14926 (Can't reproduce): jenkins.ceph.com signed-off-by false negative
- https://jenkins.ceph.com/job/ceph-pr-commits/2768/console
- 03:56 AM Bug #14841 (Resolved): sepia/zyan: Access to the sepia lab
- It works
02/27/2016
- 09:00 PM Bug #14915 (Resolved): unable to access logs from http://qa-proxy.ceph.com/teuthology/
- Cannot access any teuthology log from qa-proxy.cpeh.com/teuthology
504 Gateway Time-out
nginx/1.6.3 - 02:30 PM Bug #14859: xiaoxi access to the lab
- Hi David,
I cannot connect the OpenVPN due to auth error,
2016-02-27 22:25:02 VERIFY EKU OK
2016-02-27 ...
02/26/2016
- 03:20 AM Bug #14841: sepia/zyan: Access to the sepia lab
- Zheng, I put the new password hash into place; can you try again please
- 02:31 AM Bug #14841 (In Progress): sepia/zyan: Access to the sepia lab
- 02:30 AM Bug #14841: sepia/zyan: Access to the sepia lab
- sorry, I ruined my OpenVPN secret
please change it to:
./new-client zyan@redhat
Please submit the following ...
02/25/2016
- 11:19 PM Support #14704 (Resolved): Rack tala or saya
- Finally got around to replacing the uplink with a 10Gb cable. Resolving.
- 05:37 PM Bug #14841: sepia/zyan: Access to the sepia lab
- Hi,
Thanks for your patience and understanding my wanting to verify your identity.
I've added your hashed VPN p... - 05:17 PM Bug #14859: xiaoxi access to the lab
- Hi Xiaoxi,
You should now be able to access the lab and schedule tests.
Please verify you can connect to our Op... - 05:22 AM Bug #14859: xiaoxi access to the lab
- Username:
xiaoxichen
SSH key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTP... - 05:00 AM Bug #14859 (Closed): xiaoxi access to the lab
- Xiaoxi Chen aka xiaoxi is working on http://tracker.ceph.com/projects/ceph-releases he needs access to the lab to:
... - 04:57 PM Cleanup #14868 (Resolved): Audit user list
- - Verify all users with OpenVPN access and ssh pubkeys still need access
- Mixture of "X left the company" and e-m...
02/24/2016
- 03:11 AM Bug #14841: sepia/zyan: Access to the sepia lab
- please replace the existing public key and OpenVPN password.
thanks
Also available in: Atom