Project

General

Profile

Activity

From 12/22/2015 to 01/20/2016

01/20/2016

05:30 PM Bug #14424: disk replacements for the LRC
All disks replaced last night David Galloway
02:42 AM Bug #14432 (Closed): mira095 got wedged
It disappeared while running http://pulpito.ceph.com/gregf-2016-01-18_18:01:09-fs-greg-fs-testing-118-1---basic-mira/... Greg Farnum
01:04 AM Support #14430 (Resolved): need access to Sepia Lab
Mark Nelson wants Joe Mario and I to run Ceph CBT and profile using perf "C2C" NUMA profiling tool, because of high-s... Ben England

01/19/2016

06:26 PM Bug #14424 (Resolved): disk replacements for the LRC
mira049/2
mira021/6
mira060/5
mira116/7
mira120/7 missing (not sdg, beware)
mira055/2 still emptying
Dan Mick
04:17 PM Bug #14422 (Resolved): mira078 failing tests has bad disk
ubuntu@mira078:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 335 uncorrect secto...
Yuri Weinstein

01/18/2016

09:45 PM Bug #14403: mira088 failing tests has bad disk
ubuntu@mira088:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 8 (sdh) has 121 uncorrect secto...
Yuri Weinstein
09:43 PM Bug #14403 (Resolved): mira088 failing tests has bad disk
Yuri Weinstein
09:45 PM Bug #14404: mira117 failing tests has bad disk
ubuntu@mira117:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 4 uncorrect sectors...
Yuri Weinstein
09:43 PM Bug #14404 (Resolved): mira117 failing tests has bad disk
Yuri Weinstein
08:55 PM Bug #11315 (Rejected): ceph-deploy suite in hammer (and giant) can no longer run on plana nodes
No longer have plana nodes David Galloway
08:54 PM Bug #12048 (Resolved): mira082 bad disk
System had its drives replaced JAN7 David Galloway
08:52 PM Bug #13147 (Resolved): mira092: SOL stuck, "power reset" has no apparent effect
System's been running jobs fine for a bit now. David Galloway
08:51 PM Bug #13696 (Resolved): mira077 has bad drives
This system had its bad drives replaced on JAN7. David Galloway
05:35 PM Bug #14398 (Resolved): Install Centos 7.2 on some mira
We started running rados on mira and some tests ask for centos 7.2 OS, so we need to reimage ~10% to accommodate this... Yuri Weinstein

01/15/2016

11:53 PM Support #14294: Infrastructure Host Drive replacements
senta02's RAID failed and its filesystem was broken beyond salvaging. We lost the following VMs:
* gitbuilder-hadoo...
David Galloway
10:06 PM Tasks #13085 (Closed): kill gitbuilder-ceph-tarball-precise-amd64-basic
done Dan Mick
10:06 PM Tasks #13086 (Resolved): kill gitbuilder-ceph-rpm-fedora20-amd64-basic
While cleaning up gitbuilders, noticed this one's already gone so closing ticket. David Galloway

01/14/2016

09:05 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml) David Galloway
06:26 PM Bug #14338 (In Progress): Incorrect IPs set in /etc/hosts on mira nodes
David Galloway
05:50 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
*sigh*... David Galloway
05:36 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy ... David Galloway
04:56 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
(Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alf...
Zack Cerza
04:47 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704...
Yuri Weinstein

01/13/2016

05:49 AM Tasks #12705 (Resolved): Abhishek Varshney access to the lab
Account should be added to teuthology machines, mira, and smithi. Please reopen/comment/email if not. Dan Mick
01:49 AM Tasks #12705 (In Progress): Abhishek Varshney access to the lab
OpenVPN access added; not sure if working yet. Adding account info now. Dan Mick
01:44 AM Tasks #11652 (Resolved): smithfarm access to the lab
Added new vpn secret. Nathan confirms a connection. Dan Mick

01/12/2016

05:49 PM Bug #14338 (New): Incorrect IPs set in /etc/hosts on mira nodes
David, can you take a look at mira061 configuration, pls?
And then close/resolve the ticket.
Yuri Weinstein
03:52 PM Bug #14338 (Closed): Incorrect IPs set in /etc/hosts on mira nodes
Samuel Just
12:12 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
This looks like the monitors can't resolve the mira061 address:... Alfredo Deza
01:28 AM Bug #11867 (Resolved): mira089 bad disk
All failed drives replaced David Galloway

01/11/2016

10:45 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
Run: http://pulpito.ceph.com/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/
Jobs: 18887, 18889
...
Yuri Weinstein
08:16 PM Bug #14336 (Resolved): All runs failing in ovh lab
https://github.com/ceph/teuthology/commit/1a74f929430ee1976a7696d837cc3edb407c7f36 Zack Cerza
07:59 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
Zack Cerza
07:51 PM Bug #14336 (Resolved): All runs failing in ovh lab
Zack Cerza
07:43 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
https://github.com/ceph/teuthology/pull/767 Zack Cerza
07:41 PM Bug #14336: All runs failing in ovh lab
... Zack Cerza
07:40 PM Bug #14336 (In Progress): All runs failing in ovh lab
Zack Cerza
05:05 PM Bug #14336 (Resolved): All runs failing in ovh lab
See last 2 days runs, here is an example:
Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-10_18:02:01...
Yuri Weinstein
10:11 AM Tasks #11652: smithfarm access to the lab
Hi Dan:
My previous sepia credentials were lost in global destruction. Could you please renew them?
# requesti...
Nathan Cutler
08:59 AM Tasks #11652 (In Progress): smithfarm access to the lab
@nathan could you please re-iterate the required information here to make it easier for Dan ? See http://ceph.github.... Loïc Dachary

01/08/2016

04:29 AM Support #14294: Infrastructure Host Drive replacements
senta01 rebuild complete
senta02 has problems with md1, investigating through rescue boot
senta03 rebuild complete
Dan Mick
02:50 AM Bug #13696 (In Progress): mira077 has bad drives
David Galloway

01/07/2016

11:03 PM Feature #14296 (Resolved): Create VPSHOST nagios hostgroup
David Galloway
10:55 PM Support #14294: Infrastructure Host Drive replacements
senta01 had drive 2 replaced and its Areca RAID is rebuilding automatically David Galloway
10:45 PM Support #14294: Infrastructure Host Drive replacements
senta03 has had sdb replaced
md1 is rebuilding
md127 needs sdb1 readded still
David Galloway
10:30 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
senta02 needs sdc replaced
senta02 has had sdd replaced and md1 is in process of recovering. md127 needs recovering...
David Galloway
10:29 PM Bug #12723 (Closed): megabug: all the dead disks in the test miras
Most of these have been replaced long ago, adn we have a new set
Dan Mick
09:21 PM Bug #14291 (Resolved): gw redundancy
We used to have a hot standby host, gw2, that was inaccessible but ready to take over gw's IP if an emergency happene... Dan Mick
09:18 PM Bug #14290 (Resolved): Add public IP to yan-zheng; make sure packages up to date, no passwords (a...
Basically restore yan-zheng as a bastion host for developers behind Great Firewall Dan Mick
07:39 AM Tasks #12705: Abhishek Varshney access to the lab
username: abhishekvrshny
email: abhishek.varshney@flipkart.com
SSH key: https://github.com/ceph/keys/blob/master/ss...
Abhishek Varshney
07:02 AM Tasks #12705: Abhishek Varshney access to the lab
It's not a private key; it's a hashed password. The secret stays with you. Here is fine. Note that there are four ... Dan Mick
06:36 AM Tasks #12705: Abhishek Varshney access to the lab
Dan: My key exists on https://github.com/ceph/keys and I have just completed http://ceph.github.io/sepia/adding_users... Abhishek Varshney
07:20 AM Support #14250 (Resolved): Requesting Lab Access
Dan Mick
07:20 AM Support #14250: Requesting Lab Access
I guess I'll close it. Dan Mick

01/06/2016

11:17 PM Bug #13147: mira092: SOL stuck, "power reset" has no apparent effect
Not sure of exact root cause but was able to re-image, nuke, and release on 22DEC2015. David Galloway
10:45 PM Bug #12555 (In Progress): mira083 stuck in raid bios on reboot
The RAID controller firmware hang has been resolved but 3 drives are missing.
I'm installing an OS to identify whi...
David Galloway
10:02 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
Here are today's findings/testing.
I took smithi003 with a known working NVMe card and tested both PCI slots. Con...
David Galloway
06:43 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
seems like it should to me, Ken
Dan Mick
06:33 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
(Should ceph.git's @admin/build-doc@ script be checking for the graphviz package?) Ken Dreyer
06:32 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
Sure, I've installed graphviz now.
Exact steps I did today:
# Log into the DHC OpenVPN (Dan Mick has documentat...
Ken Dreyer
06:12 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
Ken, i see you are using docs nodes to build the docs in https://github.com/ceph/ceph-build/pull/210. i am wondering ... Kefu Chai
12:46 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
if graphviz is installed on the gitbuilder of the doc, the dot should be rendered. Kefu Chai
06:42 PM Tasks #12705: Abhishek Varshney access to the lab
Abhishek: did you supply the information requested in http://ceph.github.io/sepia/adding_users/#requesting-lab-access... Dan Mick
02:33 PM Tasks #12705: Abhishek Varshney access to the lab
@Dan I think Abhishek did what was required of him at http://ceph.github.io/sepia/adding_users/ . I'm escalating this... Loïc Dachary
06:36 PM Bug #14268 (Resolved): logrotate failing on gitbuilders
I noticed a large stdout.log on the ceph-deb-trusty-amd64-basic gitbuilder.
After discussing with dmick it is pres...
David Galloway
05:24 AM Support #14250: Requesting Lab Access
I think this is taken care of; John, please close this when you agree. Dan Mick
02:02 AM Support #14250: Requesting Lab Access
https://github.com/ceph/keys/pull/10
https://github.com/ceph/ceph-sepia-secrets/pull/69
https://github.com/ceph/coo...
Dan Mick
12:45 AM Support #14250 (In Progress): Requesting Lab Access
Dan Mick
12:44 AM Support #14250 (Resolved): Requesting Lab Access
I'm requesting access to help Warren test some Java client code.
The login I would like to use is: jowilkin
SS...
John Wilkins

01/02/2016

11:47 PM Bug #14214 (Resolved): mira115 has bad disks (marked down)
ubuntu@mira115:~$ /usr/libexec/smart.pl
3 of 8 drives failing/missing |
Drive 2 N.A.
Drive 5 (sde) has 465 reallo...
Yuri Weinstein

12/23/2015

08:31 PM Bug #11664 (Closed): some gitbuilder-ceph-tarbal-* are down
Closing old ticket. All three tarball gitbuilders are up now David Galloway
08:25 PM Bug #13282 (Need More Info): ssh from my vm to centos vpm machines are slow.
Warren,
Has this improved at all since the move to RDU2?
If not, could you provide the output of "ssh -vvv" to ...
David Galloway
06:58 PM Bug #13148 (Resolved): mira101: can't find block device
Checked disks, reimaged, nuked, and released system yesterday. David Galloway
06:56 PM Bug #11111 (Resolved): mira005 has bad disks
Drives were replaced, system reimaged, nuked, and released. David Galloway
06:38 PM Bug #11498 (Can't reproduce): dead machines without useful ipmiconsole
plana and burnupi systems are retired David Galloway
06:36 PM Bug #11739 (Closed): gitbuilder-cdep-deb-cloud-precise-amd64-basic connectivity problem
gitbuilder is retired David Galloway
06:34 PM Bug #11497 (Rejected): handle email to teuthworker@typica002.front.sepia.ceph.com
typicas are no more David Galloway
06:26 PM Bug #13839 (Won't Fix): http://file.rdu.redhat.com/ not reachable from vpm
file.rdu.redhat.com is an internal host behind Red Hat's firewall. Octo will be able to access it but not Sepia. David Galloway
06:00 PM Bug #10298 (Resolved): "ImportError (No module named realistic)" in upgrade:dumpling-firefly-x:pa...
Looking through the s3-tests repo, I think it's safe to assume this was fixed by https://github.com/ceph/s3-tests/com... David Galloway

12/22/2015

05:59 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
dmidecode indicates the PCI slot is not populated. Will check on these next time I'm at the DC. David Galloway
04:41 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
Examples:
http://pulpito.ceph.com/teuthology-2015-12-21_19:00:01-rados-jewel-distro-basic-smithi/1582
http://pulpit...
David Galloway
04:39 PM Bug #14157 (Resolved): smithi004, smithi005, smithi007, smithi055 NVMe cards bad
Runs are failing due to not being able to partition the NVMe device. Upon further inspection, the entire device appe... David Galloway
09:28 AM Bug #14152 (Resolved): the dot graph is not rendered in http://docs.ceph.com/
for example, http://docs.ceph.com/docs/master/dev/peering/#state-model
and the same applies to @http://docs.ceph.com...
Kefu Chai
 

Also available in: Atom