Activity
From 12/22/2015 to 01/20/2016
01/20/2016
- 05:30 PM Bug #14424: disk replacements for the LRC
- All disks replaced last night
- 02:42 AM Bug #14432 (Closed): mira095 got wedged
- It disappeared while running http://pulpito.ceph.com/gregf-2016-01-18_18:01:09-fs-greg-fs-testing-118-1---basic-mira/...
- 01:04 AM Support #14430 (Resolved): need access to Sepia Lab
- Mark Nelson wants Joe Mario and I to run Ceph CBT and profile using perf "C2C" NUMA profiling tool, because of high-s...
01/19/2016
- 06:26 PM Bug #14424 (Resolved): disk replacements for the LRC
- mira049/2
mira021/6
mira060/5
mira116/7
mira120/7 missing (not sdg, beware)
mira055/2 still emptying
- 04:17 PM Bug #14422 (Resolved): mira078 failing tests has bad disk
- ubuntu@mira078:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 335 uncorrect secto...
01/18/2016
- 09:45 PM Bug #14403: mira088 failing tests has bad disk
- ubuntu@mira088:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 8 (sdh) has 121 uncorrect secto... - 09:43 PM Bug #14403 (Resolved): mira088 failing tests has bad disk
- 09:45 PM Bug #14404: mira117 failing tests has bad disk
- ubuntu@mira117:~$ sudo /usr/libexec/smart.pl
1 of 8 drives failing/missing |
Drive 7 (sdg) has 4 uncorrect sectors... - 09:43 PM Bug #14404 (Resolved): mira117 failing tests has bad disk
- 08:55 PM Bug #11315 (Rejected): ceph-deploy suite in hammer (and giant) can no longer run on plana nodes
- No longer have plana nodes
- 08:54 PM Bug #12048 (Resolved): mira082 bad disk
- System had its drives replaced JAN7
- 08:52 PM Bug #13147 (Resolved): mira092: SOL stuck, "power reset" has no apparent effect
- System's been running jobs fine for a bit now.
- 08:51 PM Bug #13696 (Resolved): mira077 has bad drives
- This system had its bad drives replaced on JAN7.
- 05:35 PM Bug #14398 (Resolved): Install Centos 7.2 on some mira
- We started running rados on mira and some tests ask for centos 7.2 OS, so we need to reimage ~10% to accommodate this...
01/15/2016
- 11:53 PM Support #14294: Infrastructure Host Drive replacements
- senta02's RAID failed and its filesystem was broken beyond salvaging. We lost the following VMs:
* gitbuilder-hadoo... - 10:06 PM Tasks #13085 (Closed): kill gitbuilder-ceph-tarball-precise-amd64-basic
- done
- 10:06 PM Tasks #13086 (Resolved): kill gitbuilder-ceph-rpm-fedora20-amd64-basic
- While cleaning up gitbuilders, noticed this one's already gone so closing ticket.
01/14/2016
- 09:05 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
- Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml)
- 06:26 PM Bug #14338 (In Progress): Incorrect IPs set in /etc/hosts on mira nodes
- 05:50 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- *sigh*...
- 05:36 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy ...
- 04:56 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- (Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alf... - 04:47 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704...
01/13/2016
- 05:49 AM Tasks #12705 (Resolved): Abhishek Varshney access to the lab
- Account should be added to teuthology machines, mira, and smithi. Please reopen/comment/email if not.
- 01:49 AM Tasks #12705 (In Progress): Abhishek Varshney access to the lab
- OpenVPN access added; not sure if working yet. Adding account info now.
- 01:44 AM Tasks #11652 (Resolved): smithfarm access to the lab
- Added new vpn secret. Nathan confirms a connection.
01/12/2016
- 05:49 PM Bug #14338 (New): Incorrect IPs set in /etc/hosts on mira nodes
- David, can you take a look at mira061 configuration, pls?
And then close/resolve the ticket. - 03:52 PM Bug #14338 (Closed): Incorrect IPs set in /etc/hosts on mira nodes
- 12:12 PM Bug #14338: Incorrect IPs set in /etc/hosts on mira nodes
- This looks like the monitors can't resolve the mira061 address:...
- 01:28 AM Bug #11867 (Resolved): mira089 bad disk
- All failed drives replaced
01/11/2016
- 10:45 PM Bug #14338 (Resolved): Incorrect IPs set in /etc/hosts on mira nodes
- Run: http://pulpito.ceph.com/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/
Jobs: 18887, 18889
... - 08:16 PM Bug #14336 (Resolved): All runs failing in ovh lab
- https://github.com/ceph/teuthology/commit/1a74f929430ee1976a7696d837cc3edb407c7f36
- 07:59 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
- 07:51 PM Bug #14336 (Resolved): All runs failing in ovh lab
- 07:43 PM Bug #14336 (Fix Under Review): All runs failing in ovh lab
- https://github.com/ceph/teuthology/pull/767
- 07:41 PM Bug #14336: All runs failing in ovh lab
- ...
- 07:40 PM Bug #14336 (In Progress): All runs failing in ovh lab
- 05:05 PM Bug #14336 (Resolved): All runs failing in ovh lab
- See last 2 days runs, here is an example:
Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-10_18:02:01... - 10:11 AM Tasks #11652: smithfarm access to the lab
- Hi Dan:
My previous sepia credentials were lost in global destruction. Could you please renew them?
# requesti... - 08:59 AM Tasks #11652 (In Progress): smithfarm access to the lab
- @nathan could you please re-iterate the required information here to make it easier for Dan ? See http://ceph.github....
01/08/2016
- 04:29 AM Support #14294: Infrastructure Host Drive replacements
- senta01 rebuild complete
senta02 has problems with md1, investigating through rescue boot
senta03 rebuild complete
- 02:50 AM Bug #13696 (In Progress): mira077 has bad drives
01/07/2016
- 11:03 PM Feature #14296 (Resolved): Create VPSHOST nagios hostgroup
- 10:55 PM Support #14294: Infrastructure Host Drive replacements
- senta01 had drive 2 replaced and its Areca RAID is rebuilding automatically
- 10:45 PM Support #14294: Infrastructure Host Drive replacements
- senta03 has had sdb replaced
md1 is rebuilding
md127 needs sdb1 readded still - 10:30 PM Support #14294 (Resolved): Infrastructure Host Drive replacements
- senta02 needs sdc replaced
senta02 has had sdd replaced and md1 is in process of recovering. md127 needs recovering... - 10:29 PM Bug #12723 (Closed): megabug: all the dead disks in the test miras
- Most of these have been replaced long ago, adn we have a new set
- 09:21 PM Bug #14291 (Resolved): gw redundancy
- We used to have a hot standby host, gw2, that was inaccessible but ready to take over gw's IP if an emergency happene...
- 09:18 PM Bug #14290 (Resolved): Add public IP to yan-zheng; make sure packages up to date, no passwords (a...
- Basically restore yan-zheng as a bastion host for developers behind Great Firewall
- 07:39 AM Tasks #12705: Abhishek Varshney access to the lab
- username: abhishekvrshny
email: abhishek.varshney@flipkart.com
SSH key: https://github.com/ceph/keys/blob/master/ss... - 07:02 AM Tasks #12705: Abhishek Varshney access to the lab
- It's not a private key; it's a hashed password. The secret stays with you. Here is fine. Note that there are four ...
- 06:36 AM Tasks #12705: Abhishek Varshney access to the lab
- Dan: My key exists on https://github.com/ceph/keys and I have just completed http://ceph.github.io/sepia/adding_users...
- 07:20 AM Support #14250 (Resolved): Requesting Lab Access
- 07:20 AM Support #14250: Requesting Lab Access
- I guess I'll close it.
01/06/2016
- 11:17 PM Bug #13147: mira092: SOL stuck, "power reset" has no apparent effect
- Not sure of exact root cause but was able to re-image, nuke, and release on 22DEC2015.
- 10:45 PM Bug #12555 (In Progress): mira083 stuck in raid bios on reboot
- The RAID controller firmware hang has been resolved but 3 drives are missing.
I'm installing an OS to identify whi... - 10:02 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- Here are today's findings/testing.
I took smithi003 with a known working NVMe card and tested both PCI slots. Con... - 06:43 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- seems like it should to me, Ken
- 06:33 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- (Should ceph.git's @admin/build-doc@ script be checking for the graphviz package?)
- 06:32 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- Sure, I've installed graphviz now.
Exact steps I did today:
# Log into the DHC OpenVPN (Dan Mick has documentat... - 06:12 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- Ken, i see you are using docs nodes to build the docs in https://github.com/ceph/ceph-build/pull/210. i am wondering ...
- 12:46 PM Bug #14152: the dot graph is not rendered in http://docs.ceph.com/
- if graphviz is installed on the gitbuilder of the doc, the dot should be rendered.
- 06:42 PM Tasks #12705: Abhishek Varshney access to the lab
- Abhishek: did you supply the information requested in http://ceph.github.io/sepia/adding_users/#requesting-lab-access...
- 02:33 PM Tasks #12705: Abhishek Varshney access to the lab
- @Dan I think Abhishek did what was required of him at http://ceph.github.io/sepia/adding_users/ . I'm escalating this...
- 06:36 PM Bug #14268 (Resolved): logrotate failing on gitbuilders
- I noticed a large stdout.log on the ceph-deb-trusty-amd64-basic gitbuilder.
After discussing with dmick it is pres... - 05:24 AM Support #14250: Requesting Lab Access
- I think this is taken care of; John, please close this when you agree.
- 02:02 AM Support #14250: Requesting Lab Access
- https://github.com/ceph/keys/pull/10
https://github.com/ceph/ceph-sepia-secrets/pull/69
https://github.com/ceph/coo... - 12:45 AM Support #14250 (In Progress): Requesting Lab Access
- 12:44 AM Support #14250 (Resolved): Requesting Lab Access
- I'm requesting access to help Warren test some Java client code.
The login I would like to use is: jowilkin
SS...
01/02/2016
- 11:47 PM Bug #14214 (Resolved): mira115 has bad disks (marked down)
- ubuntu@mira115:~$ /usr/libexec/smart.pl
3 of 8 drives failing/missing |
Drive 2 N.A.
Drive 5 (sde) has 465 reallo...
12/23/2015
- 08:31 PM Bug #11664 (Closed): some gitbuilder-ceph-tarbal-* are down
- Closing old ticket. All three tarball gitbuilders are up now
- 08:25 PM Bug #13282 (Need More Info): ssh from my vm to centos vpm machines are slow.
- Warren,
Has this improved at all since the move to RDU2?
If not, could you provide the output of "ssh -vvv" to ... - 06:58 PM Bug #13148 (Resolved): mira101: can't find block device
- Checked disks, reimaged, nuked, and released system yesterday.
- 06:56 PM Bug #11111 (Resolved): mira005 has bad disks
- Drives were replaced, system reimaged, nuked, and released.
- 06:38 PM Bug #11498 (Can't reproduce): dead machines without useful ipmiconsole
- plana and burnupi systems are retired
- 06:36 PM Bug #11739 (Closed): gitbuilder-cdep-deb-cloud-precise-amd64-basic connectivity problem
- gitbuilder is retired
- 06:34 PM Bug #11497 (Rejected): handle email to teuthworker@typica002.front.sepia.ceph.com
- typicas are no more
- 06:26 PM Bug #13839 (Won't Fix): http://file.rdu.redhat.com/ not reachable from vpm
- file.rdu.redhat.com is an internal host behind Red Hat's firewall. Octo will be able to access it but not Sepia.
- 06:00 PM Bug #10298 (Resolved): "ImportError (No module named realistic)" in upgrade:dumpling-firefly-x:pa...
- Looking through the s3-tests repo, I think it's safe to assume this was fixed by https://github.com/ceph/s3-tests/com...
12/22/2015
- 05:59 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- dmidecode indicates the PCI slot is not populated. Will check on these next time I'm at the DC.
- 04:41 PM Bug #14157: smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- Examples:
http://pulpito.ceph.com/teuthology-2015-12-21_19:00:01-rados-jewel-distro-basic-smithi/1582
http://pulpit... - 04:39 PM Bug #14157 (Resolved): smithi004, smithi005, smithi007, smithi055 NVMe cards bad
- Runs are failing due to not being able to partition the NVMe device. Upon further inspection, the entire device appe...
- 09:28 AM Bug #14152 (Resolved): the dot graph is not rendered in http://docs.ceph.com/
- for example, http://docs.ceph.com/docs/master/dev/peering/#state-model
and the same applies to @http://docs.ceph.com...
Also available in: Atom