Infrastructure » sepia

I have ARMv8 builds that I have tested on the Utah CloudLab systems (there are other x86 clusters, too). The builds I had are hosted in http://ceph.com/noah. I was hesitant to start referencing these builds in a CloudLab profile. Do we have an ARMv8 gitbuilder?

We don't have an ARMv8 gitbuilder yet. How did you generate the packages at http://ceph.com/noah/ ?

I built them on CloudLab nodes. One option would be to build Ceph as part of the profile instantiation, but at first perhaps we can start with a profile based on pre-built packages and author point releases. How does that sound?

One issue that I ran into when using Ceph on CloudLab was that the cluster with ARMv8 has a single SSD in each node and no other disks. The last time I checked with CloudLab admins the SSD couldn't be partitioned, so Ceph had to run in pre-made directories. This doesn't affect the ability to create a profile and let users play around, but it is a slight annoyance.

However, there are two other CloudLab clusters that are x86 with varying configurations of disks. It is probably easier to get started building profiles for the x86 clusters while we figure out how to handle ARMv8 releases. Thoughts?

I built them on CloudLab nodes.

Do you remember the specific set of commands you used to do so ? Was it just git clone + dpkg-buildpackage ? Some other sequence ? I'd like to capture that as a sequence of commands known to work, even if we're not going to use it as is.

It is probably easier to get started building profiles for the x86 clusters

This makes a lot of sense. I have everything to learn about how to create a profile and use it. Did you create a profile already ? If you did I'll start with http://docs.cloudlab.us/creating-profiles.html and ask questions :-)

Loic Dachary wrote:

I built them on CloudLab nodes.

Do you remember the specific set of commands you used to do so ? Was it just git clone + dpkg-buildpackage ? Some other sequence ? I'd like to capture that as a sequence of commands known to work, even if we're not going to use it as is.

Yes, that is how I built the packages. I built Ceph first without dpkg-buildpackage to make sure there weren't any changes that were required. IIRC the build went quickly, but `make check` or some other set of unit tests run during dpkg-buildpackage took an extraordinary amount of time (e.g. many, many hours).

There will also likely be an issue with `libgoogle-perftools-dev`. When I built Ceph that package wasn't available for ARM in the Ubuntu repositories. I grabbed a copy from Debian testing and things worked just fine.

I'll run through the steps again this evening and document stuff as well.

Loic Dachary wrote:

It is probably easier to get started building profiles for the x86 clusters

This makes a lot of sense. I have everything to learn about how to create a profile and use it. Did you create a profile already ? If you did I'll start with http://docs.cloudlab.us/creating-profiles.html and ask questions :-)

I've only modified existing profiles to do simple things like add more nodes or network adapters. My understanding is that anyone can make profiles and then choose to make them public (e.g. I have a bunch of private profiles for different setups). I don't know much about official profiles or how the process of creating them works. My guess is that we can work on profiles privately and then once it's solid make it public.

I think that arbitrary scripts can be added to the profiles, so using something like ceph-deploy may be the simplest. There are also openstack profiles that you can examine which might be useful references.

Sent an update to cloudlab.us to let them know we're still interested although nothing has been done so far.

• arm64 hardware are of type m400 in the Utah datacenter https://www.cloudlab.us/hardware.php#utah
• when using the OpenStack profile at https://www.cloudlab.us/show-profile.php?uuid=c4d601c3-e175-11e5-95f1-020cbce80001 set the machine type to m400 to get an arm64 cluster
• ssh to the ctl node (see the list view to get the ssh line including the machine name)
• copy /root/setup/admin-openrc.sh from ctl node (the one downloaded from the horizon dashboard does not work)
• append unset OS_REGION_NAME and unset OS_TENANT_ID at the end otherwise values set from other openrc.sh will interfere
  

  export OS_PROJECT_DOMAIN_ID=default
  export OS_USER_DOMAIN_ID=default
  export OS_PROJECT_NAME=admin
  export OS_TENANT_NAME=admin
  export OS_USERNAME=adminapi
  export OS_PASSWORD=d8ec9184c9b96b3bdf75
  export OS_AUTH_URL=http://ctl:35357/v3
  export OS_IDENTITY_API_VERSION=3
  unset OS_REGION_NAME
  unset OS_TENANT_ID
  

• add 128.110.xxx.xxx ctl to /etc/hosts because OS_AUTH_URL has ctl as the hostname
• create a 48GB RAM flavor with 8 vcpus to be used for ceph builds

Investigating a failure to use the pristine aarch64 14.04 Ubuntu image at https://groups.google.com/forum/#!topic/cloudlab-users/EN6YASWXwZU. In the meantime, manually renaming the default provided image to teuthology-ubuntu-14.04-arm64 is a viable workaround.

To get an idea of how long it actually takes to make check on ARMv8, using a compute node:

• sudo /etc/init.d/nova-compute stop
• allocate the rest of /dev/sda to /dev/sda2
• sudo mkfs.ext4 /dev/sda2
• sudo mount /dev/sda2 /mnt
• sudo chown dachary /mnt
• cd /mnt
• git clone -b wip-virtualenv-jewel http://github.com/dachary/ceph
• cd ceph
• NPROC=8 time ./run-make-check.sh

About 2h30, a good part of which is spent verifying erasure code corpus

The compute node only have 20GB disk locally which is not enough to build ceph packages. Copy the OpenStack profile at https://www.cloudlab.us/show-profile.php?uuid=c4d601c3-e175-11e5-95f1-020cbce80001 to try and configure the compute node to use the remaining 90GB of the local disk for /var/lib/nova/instances at build time. The documentation is at http://docs.cloudlab.us/creating-profiles.html.

• Copy / paste in a file the "source" (js link) at https://www.cloudlab.us/show-profile.php?uuid=c4d601c3-e175-11e5-95f1-020cbce80001
• Download the tarbal which has the actual scripts used to install all OpenStack nodes from wget http://www.emulab.net/downloads/openstack-setup-v26.tar.gz
• The setup-driver.sh script is the top level script
• The environment variables used in setup-driver.sh are set in the class Parameters(RSpec.Resource): class of the profile source

commit c17914caac0db0d10b078d251445ec41e0c40a4e
Author: Loic Dachary <loic@dachary.org>
Date:   Mon Mar 28 15:22:20 2016 +0200

    setup-compute: more disk for instances

    Use the rest of /dev/sda (~90GB on m400) for local instances disks.  The
    default root file system only has 16GB and they are quickly exhausted.

    Signed-off-by: Loic Dachary <loic@dachary.org>

diff --git a/setup-compute.sh b/setup-compute.sh
index 77bafa8..3962ad2 100755
--- a/setup-compute.sh
+++ b/setup-compute.sh
@@ -38,6 +38,15 @@ if [ $OSVERSION -ge $OSKILO ]; then
     patch -d / -p0 < $DIRNAME/etc/oslo_service-liberty-sig-MAINLOOP.patch
 fi

+if ! blkid -p -s TYPE /dev/sda2 ; then
+    sgdisk --largest-new=2 /dev/sda
+    partprobe /dev/sda
+    mkfs.ext4 /dev/sda2
+    mkdir -p /var/lib/nova/instances
+    mount /dev/sda2 /var/lib/nova/instances
+    chmod 777 /var/lib/nova/instances # will be restriced below
+fi
+
 maybe_install_packages nova-compute sysfsutils
 maybe_install_packages libguestfs-tools libguestfs0 python-guestfs

@@ -161,6 +170,10 @@ if [ ${OSCODENAME} = "juno" ]; then
     patch -d / -p0 < $DIRNAME/etc/nova-juno-root-device-name.patch
 fi

+# in case it's mounted on a virgin file system
+chown nova:nova /var/lib/nova/instances
+chmod 755 /var/lib/nova/instances
+
 service_restart nova-compute
 service_enable nova-compute
 service_restart libvirt-bin

Asked at https://groups.google.com/forum/#!topic/cloudlab-users/GuvCxnhMQgk how to contribute that patch back to the OpenStack profile.

https://groups.google.com/forum/#!topic/cloudlab-users/S2uurdVybqc

passed rados smoke suite http://teuthology-logs.public.ceph.com/ubuntu-2016-04-09_16:13:05-smoke-v10.1.1---basic-openstack/