Synchronizing Data Between AWS and OpenStack

Approximately remain in this minutes read.

Synchronizing Data Between AWS and OpenStack

Synchronizing Data Between AWS and OpenStack
Written by

The reality of running your applications on premises and in the cloud is not a dream many organizations have spread their workloads across their own data centers and at least one cloud provider.

Having a single point of failure in a cloud deployment is a major risk, be it an instance, a rack, a data center, or even a cloud provider. It is becoming increasingly common to see workloads spread not only over multiple data centers, but also over multiple cloud providers. Moving application logic across IT environments is relatively simple in comparison with moving data, where maintaining data consistency presents great challenges, in particular when synchronizing information between totally different IT infrastructures.

In this article, we discuss the technical challenges and practices when looking to sync information between the Amazon cloud, the most popular public cloud today, and OpenStack, one of the leading private cloud solutions.

Discover how to size your cloud accurately.

Technical Challenges

Synchronizing data is not a trivial task, regardless of whether within your own data center or between data centers in multiple locations. Not surprisingly, synchronizing between completely different clouds comes with its own challenges.

Different APIs

AWS and OpenStack do not use the same APIs. If you were to find a way to actually sync the information between instances, you would need the APIs to receive the information to connect to those instances. Because the APIs are completely different, you would need to manage two separate sets of queries and tools one for each cloud. This translates into additional operational overhead and maintenance.

AWS APIs versus OpenStack APIs – learn all about the differences here.


Network connectivity between instances in your data center is usually quite trivial. The instances are usually behind the same corporate firewall, and much easier to control and manage.

Although not an unsurmountable problem, communication becomes more complicated when your instances are in the cloud (private or public). All communication in and out of the instances is handled through security groups, which are defined through the API and allow the correct flow of traffic.

To synchronize information between clouds, you have to treat each of them as external entities.

Consider the following example:


An instance in one cloud needs to talk to an instance in another cloud. Communication will be across the open internet and will require an external Public IP address on each of the instances with the correct security group rules in place, allowing the appropriate IP ranges or addresses. (AWS publishes its IP ranges here. This actually opens up communication to your network from the whole of AWS, which can be problematic from a security perspective).

Look at this slideshare to learn how to build a network with OpenStack Neutron.

Shared Storage

In your data center, your instances probably have access to a shared storage device, using a network protocol (such as SMB or NFS) to access the mount points. This is doable between different locations, however the network latency typically makes this solution difficult to maintain and manage, not to mention the need to set up and manage the shared network storage device on your own.

Possible Solutions

There are ways to actually synchronize data between clouds. Here are some examples.

Object Storage

Object storage (also known as object-based storage) is a storage architecture that manages data as objects, as opposed to other storage architectures like file systems (such as NFS or SMB, which manage data as a file hierarchy) and block storage (such as iSCSI or FC which manage data as blocks within sectors and tracks).

AWS offers S3 and OpenStack uses Swift. Completely different APIs but the same architectural concepts and building blocks. Data is stored in blobs called Objects, and can be uploaded and downloaded  via API.

Here is a conceptual diagram:


Several libraries are available today in several languages (for example nodejs), which will allow you to monitor a bucket for changes. By using a library to monitor a change, you can use the bucket as a file system that can be updated across multiple instances, locations, and even clouds.

One such example is s3fs-fuse which is a FUSE-based file system backed by Amazon S3. That means that each instance can mount the same S3 bucket as a local filesystem, and use it regardless of what platform it is running on – be it AWS or OpenStack.

Another option for such a filesystem using the same idea is riofs. These are all open source projects which are available for you to use.

To clarify, S3 is what is called an eventually-consistent storage. That means that it could take some time until all the nodes that are accessing the same S3 bucket actually see that a change has been made. This could cause interesting edge cases – where some nodes think that an update has been applied to the underlying filesystem and others still do not see that anything has changed.

If your application requires real-time, up to date information, you will need to incorporate some application logic that polls S3 on a regular basis, to check for any changes. The examples of a mounted S3 bucket as a local file system would serve better in an Active/Passive model, where not each and every node needs to have real-time, correct data.


Source control (Git)

You can use Git as a conduit to share information across instances.

For those of you not yet acquainted with Git, it is a version control system that is used for software development but also for other version control tasks. It is a distributed revision control system aimed at speed, data integrity, and support for distributed, non-linear workflows.

As an example, let’s take a web front-end that has has content that is updated regularly. In order to adapt to the heightened amount of traffic that your site has been experiencing lately, you need to scale out the number of Web servers. To manage your risk, you decide to distribute the load across a number of cloud providers.

You can use Git as your single source of truth to serve the same content regardless of which provider you are using to run your workloads.

On your web server, you will run a daemon that continuously polls a Git repository. (This can be a public repo on Github or a private self-hosted repo; the concept remains the same.) The daemon polls the Git repo for changes on a regular schedule. (There are more efficient ways of doing this, such as webhooks that trigger the synchronization process.) As soon as a change is detected, a new version of your content is downloaded to the web server, and is served to the public. This works regardless of which cloud provider you are using because this is a process which is decoupled from your different cloud infrastructures.

Here is a diagram of the flow:


The difference between this solution and the one above is that there is no actual filesystem mounted to each instance – the data stays replicated to each local instance when a change is detected, and pulls the files to a local filesystem. There is less of an issue here of data consistency between nodes. because they all point to a sole and single point of truth and reference.


When one thinks about sharing information between instances in the cloud, there is of course another aspect and that is sharing a point in time between instances.

This is usually used when you want to lock a file on one side so that nobody can update that file on the other. This can be for CRUD operations on a database or just plain old making sure that no-one is updating the same configuration file that you are currently working on.

There are numerous applications that require session state management, such as Tomcat. This can also be accomplished across clouds and AWS have such a service available – DynamoDB.

It goes without saying that whatever Database vendor you choose as – it will obviously need to be available to the outside world. AWS DynamoDB is classic example of such a service – that could (depending on your needs) also fit in the AWS free tier offering – and therefore – would not cost you anything.

Setting up the same thing on OpenStack – is not something that comes with a Vanilla OpenStack installation – and is not something that is currently possible today.


The available solutions for where to run your applications is continually growing. Planning applications to allow portability between environments should be an essential factor in your design process and in determining the operational practices for your cloud workloads and the data synchronization and storage solutions that best meet your needs. This is not a trivial task or endeavor; there are many risks and substantial operational overhead involved. Take the time to plan carefully and reap the benefits in the long run.

Public Cloud Experience in House
October 20, 2016

Simple Share Buttons
Simple Share Buttons