OpenStack is becoming the de-facto solution for the private cloud in many organizations. As this software stack grows, it has become crucial to maintain a standard level of availability for all its moving parts, of which there are many in a standard OpenStack deployment. Customers and end-users must be assured that the services they are consuming are available, healthy and ready for their use.
A variety of infrastructure and application monitoring and telemetry solutions are available; today we are going to focus on a particular solution. Sensu is an open-source monitoring framework that is designed to help solve monitoring challenges in cloud environments. It is not a SaaS solution, but is instead installed at the infrastructure level monitoring servers, services, application health and also providing custom analysis metrics.
Sensu relies on number of open source components and is quite simple to install. The documentation is quite extensive and easy to follow (Install Sensu in 5 minutes). Once Sensu is installed you can manage and view your alerts with another Open source product. Uchiwa aggregates the results of one or multiple Sensu API’s and displays these in a real-time dashboard, with customizable filters.
Sensu has a modern design, relying on local agents to run checks and push results to an AMQP broker. You can write your checks in almost any programming language you like, Sensu is very flexible. A number of enterprises are using Sensu at an extremely large scale.
To assure that your environment is healthy, we recommend performing monitoring at three different layers which could be compared to peeling the layers off an onion. The three layers are covered in each of the following sections:
- API Availability
- Basic API Flows
- Full Stack Development
Note that Full Stack Development will be discussed in Part 2 of this post.
Each and every service within OpenStack has an API endpoint. The purpose of these endpoints are to provide an entry point into the service behind it. It is very easy to see what the endpoints are, provided you have provided the appropriate credentials before hand all you have to do is run the following command:
keystone endpoint-list: the output will be a list of your endpoints, for this post we will choose one – OpenStack Keystone. You will find a public URL similar to this example, but the IP address will be appropriate for your own environment: http://192.168.16.100:5000/v2.0
The first level of monitoring checks to see if the port is actually available. For an example of a Sensu check for Keystone port availability here’s the link on github.
The script above polls the API listening on port 5000 for Keystone and will return a status code based on the result of the request.
But this does not mean that Keystone is actually functioning properly – it just means that the API is up. Which brings us to our next layer.
Basic API Flows
Keystone provides Identity, Token, Catalog and Policy services for use specifically by projects in the OpenStack family. What better way to test if Keystone is functioning in proper manner than by actually asking it to what it is supposed to do – authenticate and supply a token. Here’s a link to a bash script, showing how easy it is to customize your these checks.
This script will try and receive a token – i.e. test that the Keystone API is actually doing its job. If a token is not acquired in a timely manner – or is not acquired at all – then a warning or an alert will be triggered (respectively).
The final layer to be discussed, Full Stack Development, will be covered in Part 2 of this post.