Project

General

Profile

Bug #12203

create a Ceph profile @ cloudlab.us

Added by Loïc Dachary over 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

For cloudlab.us users to use Ceph, a profile must be created ( http://docs.cloudlab.us/creating-profiles.html ). The primary focus should be to make it so Ceph is built for the ARMv8 available ( http://docs.cloudlab.us/hardware.html ).

Ceph has been approved as a Cloudlab project and developers can join at https://www.cloudlab.us/signup.php?pid=Ceph


Related issues

Related to sepia - Bug #12317: use AARCH64 instances from datacentred.co.uk Rejected 07/13/2015

History

#1 Updated by Noah Watkins over 8 years ago

I have ARMv8 builds that I have tested on the Utah CloudLab systems (there are other x86 clusters, too). The builds I had are hosted in http://ceph.com/noah. I was hesitant to start referencing these builds in a CloudLab profile. Do we have an ARMv8 gitbuilder?

#2 Updated by Loïc Dachary over 8 years ago

We don't have an ARMv8 gitbuilder yet. How did you generate the packages at http://ceph.com/noah/ ?

#3 Updated by Noah Watkins over 8 years ago

I built them on CloudLab nodes. One option would be to build Ceph as part of the profile instantiation, but at first perhaps we can start with a profile based on pre-built packages and author point releases. How does that sound?

One issue that I ran into when using Ceph on CloudLab was that the cluster with ARMv8 has a single SSD in each node and no other disks. The last time I checked with CloudLab admins the SSD couldn't be partitioned, so Ceph had to run in pre-made directories. This doesn't affect the ability to create a profile and let users play around, but it is a slight annoyance.

However, there are two other CloudLab clusters that are x86 with varying configurations of disks. It is probably easier to get started building profiles for the x86 clusters while we figure out how to handle ARMv8 releases. Thoughts?

#4 Updated by Loïc Dachary over 8 years ago

I built them on CloudLab nodes.

Do you remember the specific set of commands you used to do so ? Was it just git clone + dpkg-buildpackage ? Some other sequence ? I'd like to capture that as a sequence of commands known to work, even if we're not going to use it as is.

#5 Updated by Loïc Dachary over 8 years ago

It is probably easier to get started building profiles for the x86 clusters

This makes a lot of sense. I have everything to learn about how to create a profile and use it. Did you create a profile already ? If you did I'll start with http://docs.cloudlab.us/creating-profiles.html and ask questions :-)

#6 Updated by Noah Watkins over 8 years ago

Loic Dachary wrote:

I built them on CloudLab nodes.

Do you remember the specific set of commands you used to do so ? Was it just git clone + dpkg-buildpackage ? Some other sequence ? I'd like to capture that as a sequence of commands known to work, even if we're not going to use it as is.

Yes, that is how I built the packages. I built Ceph first without dpkg-buildpackage to make sure there weren't any changes that were required. IIRC the build went quickly, but `make check` or some other set of unit tests run during dpkg-buildpackage took an extraordinary amount of time (e.g. many, many hours).

There will also likely be an issue with `libgoogle-perftools-dev`. When I built Ceph that package wasn't available for ARM in the Ubuntu repositories. I grabbed a copy from Debian testing and things worked just fine.

I'll run through the steps again this evening and document stuff as well.

#7 Updated by Noah Watkins over 8 years ago

Loic Dachary wrote:

It is probably easier to get started building profiles for the x86 clusters

This makes a lot of sense. I have everything to learn about how to create a profile and use it. Did you create a profile already ? If you did I'll start with http://docs.cloudlab.us/creating-profiles.html and ask questions :-)

I've only modified existing profiles to do simple things like add more nodes or network adapters. My understanding is that anyone can make profiles and then choose to make them public (e.g. I have a bunch of private profiles for different setups). I don't know much about official profiles or how the process of creating them works. My guess is that we can work on profiles privately and then once it's solid make it public.

I think that arbitrary scripts can be added to the profiles, so using something like ceph-deploy may be the simplest. There are also openstack profiles that you can examine which might be useful references.

#8 Updated by Loïc Dachary over 8 years ago

Sent an update to cloudlab.us to let them know we're still interested although nothing has been done so far.

#9 Updated by Loïc Dachary over 8 years ago

  • Status changed from In Progress to 12
  • Assignee deleted (Loïc Dachary)

No time to devote to this, unassigning myself.

#10 Updated by David Galloway about 8 years ago

  • Status changed from 12 to Closed

#11 Updated by Loïc Dachary about 8 years ago

  • Status changed from Closed to In Progress
  • Assignee set to Loïc Dachary

resuming work on this

#12 Updated by Loïc Dachary about 8 years ago

  • arm64 hardware are of type m400 in the Utah datacenter https://www.cloudlab.us/hardware.php#utah
  • when using the OpenStack profile at https://www.cloudlab.us/show-profile.php?uuid=c4d601c3-e175-11e5-95f1-020cbce80001 set the machine type to m400 to get an arm64 cluster
  • ssh to the ctl node (see the list view to get the ssh line including the machine name)
  • copy /root/setup/admin-openrc.sh from ctl node (the one downloaded from the horizon dashboard does not work)
  • append unset OS_REGION_NAME and unset OS_TENANT_ID at the end otherwise values set from other openrc.sh will interfere
    export OS_PROJECT_DOMAIN_ID=default
    export OS_USER_DOMAIN_ID=default
    export OS_PROJECT_NAME=admin
    export OS_TENANT_NAME=admin
    export OS_USERNAME=adminapi
    export OS_PASSWORD=d8ec9184c9b96b3bdf75
    export OS_AUTH_URL=http://ctl:35357/v3
    export OS_IDENTITY_API_VERSION=3
    unset OS_REGION_NAME
    unset OS_TENANT_ID
    
  • add 128.110.xxx.xxx ctl to /etc/hosts because OS_AUTH_URL has ctl as the hostname
  • create a 48GB RAM flavor with 8 vcpus to be used for ceph builds

#13 Updated by Loïc Dachary about 8 years ago

Investigating a failure to use the pristine aarch64 14.04 Ubuntu image at https://groups.google.com/forum/#!topic/cloudlab-users/EN6YASWXwZU. In the meantime, manually renaming the default provided image to teuthology-ubuntu-14.04-arm64 is a viable workaround.

#14 Updated by Loïc Dachary about 8 years ago

To get an idea of how long it actually takes to make check on ARMv8, using a compute node:

  • sudo /etc/init.d/nova-compute stop
  • allocate the rest of /dev/sda to /dev/sda2
  • sudo mkfs.ext4 /dev/sda2
  • sudo mount /dev/sda2 /mnt
  • sudo chown dachary /mnt
  • cd /mnt
  • git clone -b wip-virtualenv-jewel http://github.com/dachary/ceph
  • cd ceph
  • NPROC=8 time ./run-make-check.sh

About 2h30, a good part of which is spent verifying erasure code corpus

#15 Updated by Loïc Dachary about 8 years ago

The compute node only have 20GB disk locally which is not enough to build ceph packages. Copy the OpenStack profile at https://www.cloudlab.us/show-profile.php?uuid=c4d601c3-e175-11e5-95f1-020cbce80001 to try and configure the compute node to use the remaining 90GB of the local disk for /var/lib/nova/instances at build time. The documentation is at http://docs.cloudlab.us/creating-profiles.html.

commit c17914caac0db0d10b078d251445ec41e0c40a4e
Author: Loic Dachary <loic@dachary.org>
Date:   Mon Mar 28 15:22:20 2016 +0200

    setup-compute: more disk for instances

    Use the rest of /dev/sda (~90GB on m400) for local instances disks.  The
    default root file system only has 16GB and they are quickly exhausted.

    Signed-off-by: Loic Dachary <loic@dachary.org>

diff --git a/setup-compute.sh b/setup-compute.sh
index 77bafa8..3962ad2 100755
--- a/setup-compute.sh
+++ b/setup-compute.sh
@@ -38,6 +38,15 @@ if [ $OSVERSION -ge $OSKILO ]; then
     patch -d / -p0 < $DIRNAME/etc/oslo_service-liberty-sig-MAINLOOP.patch
 fi

+if ! blkid -p -s TYPE /dev/sda2 ; then
+    sgdisk --largest-new=2 /dev/sda
+    partprobe /dev/sda
+    mkfs.ext4 /dev/sda2
+    mkdir -p /var/lib/nova/instances
+    mount /dev/sda2 /var/lib/nova/instances
+    chmod 777 /var/lib/nova/instances # will be restriced below
+fi
+
 maybe_install_packages nova-compute sysfsutils
 maybe_install_packages libguestfs-tools libguestfs0 python-guestfs

@@ -161,6 +170,10 @@ if [ ${OSCODENAME} = "juno" ]; then
     patch -d / -p0 < $DIRNAME/etc/nova-juno-root-device-name.patch
 fi

+# in case it's mounted on a virgin file system
+chown nova:nova /var/lib/nova/instances
+chmod 755 /var/lib/nova/instances
+
 service_restart nova-compute
 service_enable nova-compute
 service_restart libvirt-bin

Asked at https://groups.google.com/forum/#!topic/cloudlab-users/GuvCxnhMQgk how to contribute that patch back to the OpenStack profile.

Also available in: Atom PDF