Integrating Ceph Storage with OpenStack – A Step by Step Guide

Approximately remain in this minutes read.

Integrating Ceph Storage with OpenStack – A Step by Step Guide

Integrating Ceph Storage with OpenStack – A Step by Step Guide
Written by

ceph icon

In our first post, Introduction to Ceph and our second post Deploying Ceph with OpenStack – Challenges and Solutions, we discussed how Ceph can act as  storage backend for OpenStack Block Storage (Cinder), OpenStack Image service (Glance), and OpenStack File Share service (Manila). In addition, Ceph Storage can be a drop-in replacement for OpenStack Object Storage (Swift).

This article provides instructions for integrating the existing OpenStack (Glance, Cinder and Nova) with the existing Ceph cluster. We will use the OpenStack Mitaka release and Ceph 0.94 Hammer, the latest long term stable (LTS) release. At the end of the article you will find examples of configuration files.

Prerequisites

Make sure that the existing OpenStack cloud is in working order. We suggest ensuring that the existing OpenStack deployment is functional, to isolate OpenStack deployment issues from issues that may arise with the Ceph integration. The default Glance and Cinder backends may be used for this test. At a bare minimum, you should be able to launch a new instance.

Also make sure that your environment is configured and you can use OpenStack command line tools, for example:

$ openstack service list
+----------------------------------+----------------+----------------+
| ID                               | Name           | Type           |
+----------------------------------+----------------+----------------+
| 002be70676c54d12bdbd2ef4f1fe787a | swift          | object-store   |
| 3b9a89cff19246559f80cfc1c491d555 | neutron        | network        |
| 4928e8dd8a5046ecb301a9abdc5cf60b | glance         | image          |
| 650074657e334ed9b84e6eb52158eaee | heat-cfn       | cloudformation |
| 6c28fd3251bc4e71ad94dba09250a382 | heat           | orchestration  |
| 7c712600a67f4076968154080238dcb0 | cinderv2       | volumev2       |
| 82f17f7e46a049eab6b17f608d3b10ed | keystone       | identity       |
| 83c7c10aaaaf4efea398db809de97048 | cinderv3       | volumev3       |
| 89248d18f623418a91674614b9e947c4 | glare          | artifact       |
| b36741feccbf4bd48308b10842fec3e0 | nova           | compute        |
| b3b6459ac35a4b978074c5994a8afbbc | cinder         | volume         |
| b557fa0a138c46e2a648bfb704ce6242 | compute_legacy | compute_legacy |
+----------------------------------+----------------+----------------+

Note that services’ IDs from the output above are randomly generated and you should expect different IDs in your case.

$ openstack service show glance
+-------------+----------------------------------+
| Field       | Value                           |
+-------------+----------------------------------+
| description | OpenStack Image Service          |
| enabled     | True                             |
| id          | 4928e8dd8a5046ecb301a9abdc5cf60b |
| name        | glance                           |
| type        | image                            |
+-------------+----------------------------------+

Ensure that the Ceph cluster is fully operational.

To verify, you can execute the following command on a Ceph Storage node or a Ceph Monitor node:

$ ceph health
HEALTH_OK

You need to install a Ceph client (command line tools and libraries required to connect to the Ceph cluster) on the OpenStack nodes. If your Ceph cluster is deployed via the ceph-deploy tool, then you can use the following command for each OpenStack node:

$ ceph-deploy install --cli <node>
$ ceph-deploy config push <node>

Alternatively, you can install the Ceph client packages that are available for your GNU/Linux distribution and copy the /etc/ceph/ceph.conf file manually to each OpenStack node.

Discover the best tools for OpenStack deployments here in this infographic.

Integrating Ceph with Glance

Glance supports several storage backends, such as a locally mounted filesystem, Amazon S3, OpenStack Swift, and Ceph RBD. Use the following steps to use Ceph RBD as a Glance storage backend:

Note: It is good practice to use a dedicated Ceph pool for Glance. As we learned in the post “Introduction to Ceph”, a Ceph pool is a logical partition for storing objects. Ceph allows you to have separate settings, such as permissions, number of replicas, CRUSH rules and so on per pool. By default, this pool will share all the available Ceph OSDs with other pools; however, for example, you can configure Ceph to use dedicated Ceph OSDs (disks) for this pool.

The following command will create a new Ceph pool, where ‘images’ is the name and ‘64’ is the number of placement groups for this pool:

$ ceph osd pool create images 64
pool 'images' created

Note that you can use any valid name for the pool instead of ‘images’. In this case, use your pool’s name in the following commands. Also, the number of placement groups should be appropriate for your deployment (read more about configuring placement groups for Ceph in the post “Deploying Ceph with OpenStack – Challenges and Solutions”).

You will need to create a new Ceph user for Glance and set the appropriate permissions for the user and pool:

$ ceph auth get-or-create images \
  mon ‘allow r’ \
  osd ‘allow class-read object_prefix rbd_children, \
  allow allow rwx pool=images’ \
  -o /ceph.client.images.keyring 

Copy the keyring file from the Ceph node to the nodes where Glance services (specifically glance-api and glance-registry) are deployed:

$ cat /etc/ceph/ceph.client.images.keyring | \
  ssh <node> "sudo tee /etc/ceph/ceph.client.images.keyring"

On that OpenStack node, set the appropriate permissions for the keyring file:

$ sudo chgrp glance /etc/ceph/ceph.client.images.keyring
$ sudo chmod 0640 /etc/ceph/ceph.client.images.keyring

On that OpenStack node, edit /etc/ceph/ceph.conf and add the following lines:

[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

On that OpenStack node, edit the file /etc/glance/glance-api.conf.

In the [glance_store] section set the following parameters, making sure to change the existing parameters and adding new ones:

stores = glance.store.rbd.Store,glance.store.http.Store
default_store = rbd
rbd_store_pool = images
rbd_store_user = images
rbd_store_ceph_conf = /etc/ceph/ceph.conf

Then restart glance-api service and verify that the integration with Ceph works:

$ sudo service glance-api restart

Download a test image:

$ wget \
http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

The downloaded image is in QCOW2 format. Consider converting the image’s format to “raw”.  This format is simply the disk’s raw data, as is, and therefore the image will generally consume more disk space and take longer to upload. However, if the image is uploaded as “raw”, boot disks can later be created from the image using copy-on-write snapshots. To convert the image to RAW format we will use qemu-img tool from qemu-utils package:

$ qemu-img convert \
  cirros-0.3.4-x86_64-disk.img \
  cirros-0.3.4-x86_64-disk.raw

Upload the image to Glance:

$ openstack image create \
  --file cirros-0.3.4-x86_64-disk.raw \
  --container-format bare \
  --disk-format raw \
  --public \
  cirros-0.3.4

Check that the image exists in Glance:

$ openstack image list
+--------------------------------------+--------------+--------+
| ID                                   | Name         | Status |
+--------------------------------------+--------------+--------+
| b47824a5-89f3-4b1c-8b39-60f0afa5bd70 | cirros-0.3.4 | active |
+--------------------------------------+--------------+--------+

The image ID will be different in your case, make sure you use the ID from your output in the following commands.

Check that the image exists in Ceph:

$ rbd -p images ls
b47824a5-89f3-4b1c-8b39-60f0afa5bd70

Use the image ID from the output above to see the details:

$ rbd -p images info b47824a5-89f3-4b1c-8b39-60f0afa5bd70
rbd image 'b47824a5-89f3-4b1c-8b39-60f0afa5bd70':
    size 40162 kB in 5 objects
    order 23 (8192 kB objects)
    block_name_prefix: rbd_data.6eb03f96e580
    format: 2
    features: layering, striping
    flags: 
    stripe unit: 8192 kB
    stripe count: 1

Integrating Ceph with Cinder

To begin integrating Ceph with Cinder, first create a dedicated Ceph pool for Cinder (‘volumes’ is the name, 64 is the number of placement groups):

$ ceph osd pool create volumes 64
pool 'volumes' created

Note that you can use any valid name for the pool instead of ‘volumes’. In this case, use your pool’s name in the following commands. Also the number of placement groups should be appropriate for your deployment.

 Learn how to manage OpenStack Cinder backends here. 

Create a new Ceph user for Cinder and set the appropriate permissions for the user and pools:

$ ceph auth get-or-create volumes \
  mon ‘allow r’ \
  osd ‘allow class-read object_prefix rbd_children, \
  allow allow rwx pool=volumes, \
  allow rx pool=images’ \
  -o /ceph.client.volumes.keyring 

Copy the keyring file from Ceph node to the OpenStack node, where the Cinder services (specifically cinder-api and cinder-volume) are deployed:

$ cat /etc/ceph/ceph.client.volumes.keyring | \
  ssh <node> "sudo tee /etc/ceph/ceph.client.volumes.keyring"

On that OpenStack node, set the appropriate permissions for the keyring file:

$ sudo chgrp cinder /etc/ceph/ceph.client.volumes.keyring
$ sudo chmod 0640 /etc/ceph/ceph.client.volumes.keyring

On that OpenStack node, edit /etc/ceph/ceph.conf and add the following lines:

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

Generate a UUID (universally unique identifier) that we will need for the Ceph integration with libvirt:

$ uuidgen
a5d0dd94-57c4-ae55-ffe0-7e3732a24455

On that OpenStack node, edit the file /etc/cinder/cinder.conf  and set the following parameters in the corresponding sections, making sure to change the existing parameters and adding new ones:

[DEFAULT]
enabled_backends = RBD-backend

[RBD-backend]
volume_backend_name=RBD-backend

rbd_pool=volumes
rbd_user=volumes
rbd_secret_uuid=a5d0dd94-57c4-ae55-ffe0-7e3732a24455
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_ceph_conf=/etc/ceph/ceph.conf

Discover here how to deploy Cinder in production environments.

Note that we used the generated UUID for the rbd_secret_uuid parameter.

On that OpenStack node, restart cinder-api and cinder-volume services:

$ sudo service cinder-api restart
$ sudo service cinder-volume restart

On a Ceph node, create the key file and copy it to the hypervisor nodes (where nova-compute is deployed):

$ ceph auth get-key client.volumes | \
  ssh <node> “tee client.volumes.key”

On the hypervisor nodes, create a new temporary file ceph.xml:

<secret ephemeral="no" private="no">
<uuid>a5d0dd94-57c4-ae55-ffe0-7e3732a24455</uuid>
<usage type="ceph">
<name>client.volumes secret</name>
</usage>
</secret>

Note that we used the generated UUID for the uuid element. Now define a new secret for libvirt on the hypervisor node:

$ virsh secret-define --file ceph.xml
Secret a5d0dd94-57c4-ae55-ffe0-7e3732a24455 created
$ virsh secret-set-value \
  --secret a5d0dd94-57c4-ae55-ffe0-7e3732a24455 \ 
  --base64 $(cat client.volumes.key)

Delete the temporary files:

$ rm client.volumes.key ceph.xml

Verify that that the integration with Ceph works. Create a new volume:

$ openstack volume create --size 1 vol1

Check that the volume exists in Cinder:

$ openstack volume list
+------------------+--------------+-----------+------+-------------+
| ID               | Display Name | Status    | Size | Attached to |
+------------------+--------------+-----------+------+-------------+
| ...-8d3914468ec7 | vol1         | available |    1 |             |
+------------------+--------------+-----------+------+-------------+

Check that the volume exists in Ceph:

$ rbd -p volumes ls
volume-...-8d3914468ec7

Use the ID from the output above to see the volume details.

$ rbd -p volumes info volume-...-8d3914468ec7
rbd image 'volume-...-8d3914468ec7':
    size 1024 MB in 256 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.641e2dc70192
    format: 2
    features: layering, striping
    flags: 
    stripe unit: 4096 kB
    stripe count: 1

Integrating Ceph with Nova

Here we will describe two methods for deploying VMs with boot disks residing on Ceph:

  • Ephemeral disks:, which are deleted when the VM is shut down
  • Persistent disks: which are regular Cinder volumes and are deleted only upon request

To boot virtual machines ephemeral disks residing on Ceph, you must configure the ephemeral backend for Nova. Similar to the previous examples, let’s create a new Ceph pool for Nova ephemeral disks:

$ ceph osd pool create compute 64
pool 'compute' created

Note that you can use any valid name for the pool instead of ‘compute’. In this case, use your pool’s name in the following commands. Also the number of placement groups should be appropriate for your deployment.

You will need to create a new Ceph user for Nova and set the appropriate permissions for the user and pools:

$ ceph auth get-or-create compute \
  mon ‘allow r’ \
  osd ‘allow class-read object_prefix rbd_children, \
  allow allow rwx pool=compute, \
  allow allow rwx pool=volumes, \
  allow rx pool=images’ \
  -o /ceph.client.compute.keyring 

On a Ceph node, copy the keyring file from the Ceph node to the hypervisor nodes (where the nova-compute services are deployed):

$ cat /etc/ceph/ceph.client.compute.keyring | \
  ssh <node> "sudo tee /etc/ceph/ceph.client.compute.keyring"

$ ceph auth get-key client.compute | \
  ssh <node> “tee client.compute.key”

On the hypervisor node, set the appropriate permissions for the keyring file:

$ sudo chgrp nova /etc/ceph/ceph.client.compute.keyring
$ sudo chmod 0640 /etc/ceph/ceph.client.compute.keyring

On the hypervisor node, edit /etc/ceph/ceph.conf and add the following lines:

[client.compute]
keyring = /etc/ceph/ceph.client.compute.keyring

Generate a UUID that we will need for Ceph integration with libvirt:

$ uuidgen
5b67401f-dc5e-496a-8456-9a5dc40e7d3c

On the hypervisor node, edit  /etc/nova/nova.conf file, set the following parameters in the corresponding sections, making sure to change the existing parameters and add new ones:

[libvirt]
images_rbd_pool=compute
images_type=rbd
rbd_secret_uuid=5b67401f-dc5e-496a-8456-9a5dc40e7d3c
rbd_user=compute

Note that we used the generated UUID for the rbd_secret_uuid parameter.

On the hypervisor node, restart nova-compute service:

$ sudo service nova-compute restart

On the hypervisor node, create a new temporary file ceph.xml:

<secret ephemeral="no" private="no">
<uuid>5b67401f-dc5e-496a-8456-9a5dc40e7d3c</uuid>
<usage type="ceph">
<name>client.compute secret</name>
</usage>
</secret>

Note that we used the generated UUID for the uuid element. Now define a new secret for libvirt on the hypervisor node:

$ sudo virsh secret-define --file ceph.xml
Secret 5b67401f-dc5e-496a-8456-9a5dc40e7d3c created

$ sudo virsh secret-set-value \
  --secret 5b67401f-dc5e-496a-8456-9a5dc40e7d3c \ 
  --base64 $(cat client.compute.key)

Delete the temporary files:

$ rm client.compute.key ceph.xml

Now let’s verify that that the Nova integration with Ceph works. List the existing networks:

$ nova net-list
+--------------------------------------+--------------------+------+
| ID                                   | Label              | CIDR |
+--------------------------------------+--------------------+------+
| a2612069-44a5-4421-aeb2-9b296cad2d44 | admin_floating_net | None |
| c8a9e13e-c95d-466f-96c0-27898818aec3 | admin_internal_net | None |
+--------------------------------------+--------------------+------+

Use the network ID from the output above and the cirros-0.3.4 image we uploaded early to boot a new VM using ‘Test’ as its name:

$ nova boot --flavor m1.tiny --image cirros-0.3.4 \
  Test --nic net-id=c8a9e13e-c95d-466f-96c0-27898818aec3

Check that the VM exists in Nova:

$ nova list 
+---------------+------+--------+------------+-------------+---------------+
| ID            | Name | Status | Task State | Power State | Networks      |
+---------------+------+--------+------------+-------------+---------------+
| c0f90bd2-9f8a | Test | ACTIVE | -          | Running     | 192.168.111.3 |
+---------------+------+--------+------------+-------------+---------------+

Check that VM’s ephemeral volume exists in Ceph:

$ rbd -p compute ls
c0f90bd2-9f8a_disk
c0f90bd2-9f8a_disk.config

Use the ID from the output above to see the details:

$ rbd -p compute info c0f90bd2-9f8a_disk
rbd image 'c0f90bd2-9f8a_disk':
    size 1024 MB in 128 objects
    order 23 (8192 kB objects)
    block_name_prefix: rbd_data.6eb87110b48c
    format: 2
    features: layering, striping
    flags: 
    parent: images/b47824a5-89f3-4b1c-8b39-60f0afa5bd70@snap
    overlap: 40162 kB
    stripe unit: 8192 kB
    stripe count: 1

The ‘parent’ in the output above means that the ephemeral volume is a copy-on-write clone of snapshot of the cirros-0.3.4 image that we have uploaded to Glance (read more about Ceph snapshots). The following command shows that there is a new snapshot of the uploaded image (use the ID from the “Integrating Ceph with Glance” section):

$ rbd snap ls images/b47824a5-89f3-4b1c-8b39-60f0afa5bd70
SNAPID NAME     SIZE 
     1 snap 40162 kB

The following command shows that there is one child volume (clone) of the specified snapshot:

$ rbd children images/b47824a5-89f3-4b1c-8b39-60f0afa5bd70@snap
compute/c0f90bd2-9f8a_disk

Now let’s create a new bootable persistent volume from the existing cirros-0.3.4 image. We use ‘bootable-cirros-0.3.4’ as the volume name (you can use any other name) and 1 GB as the volume size (the last parameter):

$ cinder create --image-id b47824a5-89f3-4b1c-8b39-60f0afa5bd70 \
  --display_name=bootable-cirros-0.3.4 1

Check that the new volume exists in Cinder:

$ openstack volume list
+------------------+-----------------------+-----------+------+-------------+
| ID               | Display Name          | Status    | Size | Attached to |
+------------------+-----------------------+-----------+------+-------------+
| ...-c2f272bb8594 | bootable-cirros-0.3.4 | available |    1 |             |
| ...-8d3914468ec7 | vol1                  | available |    1 |             |
+------------------+-----------------------+-----------+------+-------------+

Check that the new volume exists in Ceph:

$ rbd -p volumes ls
volume-...-c2f272bb8594
volume-...-8d3914468ec7

Use the ID from the output above to see the volume details:

$ rbd -p volumes info volume-...-c2f272bb8594
rbd image 'volume-...-c2f272bb8594':
    size 1024 MB in 256 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.740369fa6d55
    format: 2
    features: layering, striping
    flags: 
    parent: images/b47824a5-89f3-4b1c-8b39-60f0afa5bd70@snap
    overlap: 40162 kB
    stripe unit: 4096 kB
    stripe count: 1

The ‘parent’ in the output above means that the new volume is a copy-on-write clone of the snapshot of the cirros-0.3.4 image we have uploaded to Cinder. Let’s boot a new VM from the volume:

$ nova boot --flavor m1.tiny \
  --block-device \
  source=volume,id=...-c2f272bb8594,\
  dest=volume,size=1,shutdown=preserve,bootindex=0 Test2 \
  --nic net-id=c8a9e13e-c95d-466f-96c0-27898818aec3

In the command above we used the same network ID that we used to boot a VM from image. In addition we used ‘–block-device’ parameter to specify the volume ID and that we want to preserve the volume if a VM is deleted. Let’s check that the VM is running:

$ nova list
+---------------+-------+--------+------------+-------------+---------------+
| ID            | Name  | Status | Task State | Power State | Networks      |
+---------------+-------+--------+------------+-------------+---------------+
| c0f90bd2-9f8a | Test  | ACTIVE | -          | Running     | 192.168.111.3 |
| b40c5c63-6150 | Test2 | ACTIVE | -          | Running     | 192.168.111.4 |
+---------------+-------+--------+------------+-------------+---------------+

Check that there is no ephemeral volume in Ceph for that VM, because we have booted it from a volume:

$ rbd -p compute ls
c0f90bd2-9f8a_disk
c0f90bd2-9f8a_disk.config
b40c5c63-6150_disk.config

Check the VM details:

$ nova show Test2
+----------------------+-------------------------------------------------+
| Property             | Value                                           |
...
| id                   | b40c5c63-6150                                   |
| image                | Attempt to boot from volume - no image supplied |
| name                 | Test2                                           |
| os-extended-volumes: | [{"id": "b40c5c63-6150",                        |
  volumes_attached     |   "delete_on_termination": false}]              |
...
+----------------------+-------------------------------------------------+

The output above shows that we have booted the VM from the volume, which is a copy-on-write clone of the snapshot of the Glance image.

Examples of configuration files

  1. /etc/ceph/ceph.conf
  2. /etc/glance/glance-api.conf
  3. /etc/cinder/cinder.conf
  4. /etc/nova/nova.conf

 

Test Drive Stratoscale for Free!

Finally, build, run and scale cloud-native applications on-prem

Sign up for Stratoscale’s fully-featured sandbox environment - No Download Needed!

 

We respect your privacy


July 6, 2016

Simple Share Buttons
Simple Share Buttons