Integration Tests Issue 1: Docker

von Patrick Bédat, 24. Juli 2016

I think there are two types of integration tests suites:

  1. Test Suites that have become a bloated, arcane, a non maintainable collection of code wizardry with endless lines of setup code and crazy bash scripts. They test your actually very well crafted applications in a way nobody really understands. They do a good job, because they saved your asses multiple times. You hate them because the only way to make tests pass again is to comment out assertions.
  2. And the test suites written by people who created a test suite of the previous type before.

Requirements to Integration Test Suites

Integration tests are a wonderful instrument, to get as much as possible covered when you are not „test driven“ and to provide low level documentation. But what makes a good integration test suite?

  • Performance: It has to be fast. This applies to all kind of automated tests. Nothing eats up your development time, like waiting for slow tests. If you want to motivate developers to write more tests, don’t make it a burden
  • Close to production: Running a service, that depends on ghostscript on ubuntu 12.04 may give you very different results when running it on 14.04
  • Minimal invasive: So you want to test services like on a production environment. „Ok install those 100 dependencies and make sure they always match the versions like on prod. What you installed mono 4.x – oops now you have to downgrade to 2.4….“ Thinking about continuous integration, we want to have the tests running on almost naked installations. Everywhere.
  • Debuggable: Period.
  • Robustness: When you have to fire up a dozen service for testing, robustness is crucial. You don’t want to log into the build server and killing your services by hand whenever a build fails.
  • Well crafted: Invest as much love and quality into test code as into the rest of the application. Maybe even more.

Docker To The Rescue

Even though we don’t use docker for all applications on production (soon we will), they are all dockerized in our development and testing environment. Back in the days we used VMs to get our applications running locally, we were passing around gigabytes of copied VMs, because they got out of sync from time to time. We are not looking back…

The word docker still is a real „eye-brow-raiser“. Many have heard of it, knowing that is doing virtualization, but don’t really understand what’s the point about docker. Let me give you an example:

When it comes to testing in general, you always want to test on a fresh environment, because the key to predictable test execution is statelessness (docker containers are often called ephemeral). Here is the use case of „running apps on a fresh environment“ looked at from the VM and from the Docker perspective.

VM
1. Create a VM
2. Install a OS (or take snapshot, that you probably want to update)
3. Install dependencies (e.g. zip and ghostscript)
4. Make configurations
5. Deploy binaries
6. Run applications

The get a fresh environment, you have to repeat (or automate) those steps. While you can take snapshots, it will be a resource and time consuming endeavor. You can try to keep the state of the system in sync, with the actual environment, but synchronization can be a nasty thing.

Docker

  1. Create a Dockerfile
    FROM ubuntu:14.04
    MAINAINER Patrick Bédat 
    RUN apt-get update && apt-get install zip ghostscript
    
  2. Write configuration into the Dockerfile
    RUN echo "127.0.0.1 my-awesome-host" >> /etc/hosts
  3. Build an image. After an initial download, further builds will be incredebly fast (seconds)
    docker build -t my-awesome-image .

And here comes the big difference: We wont run all the applications in one docker container. We host one container per app:

docker run --volume ./bin:/opt/my-awesome-app1 my-awesome-image /opt/my-awesome-app1/app1.exe
docker run --volume ./bin:/opt/my-awesome-app2 my-awesome-image /opt/my-awesome-app2/app2.exe

Note that instead of deploying binaries, we just mapped the container directory /opt/my-awesome-appx into our hosts filesystem.
Instead of waiting for VM to boot up or a snapshot be restored, we have a container running in an instant! That is because docker does not virtualize low level stuff like hardware or an operating system. Instead it does kernel virtualization and even reuses libs and executables from the host!

container_vs_vm

Now check that Dockerfile into your repository and have your build server run those apps exactly like you did!

I hope the point, that you can spin up a complete production like environment in seconds, gives you at least clue about the power of docker. I think the hardest part to understand about docker is the ephemeral nature of containers. You don’t log into containers, they usually dont have a gui or a terminal – they just run one application and exist usually only as long as the application is running. I don’t want to dig any deeper, because docker has so much to offer and I want to focus on it’s utilization for testing.

Test Suite Setup

Back to our test suite. Before you can perform tests against real services, you need to spin them up. There are different workflows on how and when to get them running. I think the most common are:

  • Same containers per test
  • Same containers per fixture
  • Same containers per suite

The list is ordered from cleanest to the most practical and fastest solution. While starting a docker container happens in the blink of an eye, running DB setup scripts or waiting for HTTP services to be ready does not. This is our solution:

Workflow

We are using docker-compose to orchestrate the startup of a dozen containers (you know services can depend on each other). There is one docker-compose.yml file for the entire test suite. It works for us, because almost all of the services are stateless. The database is being reset with a dynamically generated list of TRUNCATE TABLE statements. And if you really need a fresh and clean container, you simply restart or recreate them. To ease things up, we created a wrapper for the docker-compose cli in C#.

This is how a docker-compose.yml looks like:

service_a:
  build: ./dir/to/service_a_dockerfile/
    link:
      - mysql
    ports:
     - "8080:80"
mysql:
  image: mysql:latest
    ports:
     - "33066:3306"

Among many other useful features, docker compose builds images and starts containers in the correct order for you.

What basically happens, when you run the nunit test suite:

  1. There is a SetupFixture, where existing containers are cleaned up first and the containers are then brought up with „docker-compose up -d“.
  2. Then we wait for all the services to be usable. E.g. we try to connect to a HTTP service and try to make a GET request.
  3. We setup the structure of the database

Then in each TestFixtureSetup or Setup, which runs either before the tests in a fixture or before each test, we truncate the whole database. Depending on how the fixture is structured, we setup a scenario in the setup or in the tests (or in both actually).

Usually you want to run tests, while you are developing. When you are developing, the services are usually running. But how can you run a HTTP service listening to port 80 run twice on the same machine, so that your test suite can talk to it? You can’t. Luckily docker allows you to remap the ports exposed by the containers, to different ports on their hosts. Hence you can run the service with the same configuration in your tests by simpliy remapping them.

Gotchas

In the first version of the test suite all exposed ports were mapped to a „test suite port“:

service_a:
  ports:
      - "8080:80"
mysql:
  ports:
      - "33066:3306"

This was very practicable while debugging, because when you halted the test after the debug, you knew, that you would be able to connect to the database with port 33066 to check, wether the scenario was correctly setup. But it also led to problems, when the suite was executed on the build server on different branches. On the one hand, because the tests couldn’t be executed in parallel (ports were already occupied) and on the other because when the test run on one branch crashed, it left services running an therefore left the ports occupied.

Our solution was to template the docker-compose file:

service_a:
  ports:
      - {{ ports.service_a }}
mysql:
  ports:
     - {{ ports.mysql }}

Now it was possible to map the ports in the development environment („8080:80“) and simply expose them on the build server („80“). When you expose ports in docker, which means you make the port accessable by the host, docker maps a random free port from the host.
But how do the clients in the integration tests get informed about how the port was mapped? Luckily docker-compose offers a „port“ command:

docker-compose port mysql
0.0.0.0:32673 -> 3306

After integrating this command to the docker-compose wrapper and tieing the clients to it, robustness of our tests multiplied.

Conclusion

Having a test suite which tests against the services as if they are running in production is a huge benefit. Making it easily executable and debuggable for developers is priceless. But it wasn’t for free. It was a long journey that beared many lessons, with a build server often remaining red for days (the team consists of 1.5 people…). But damn it was absolutely worth it.

Dealing with Azure’s StorageException: Object reference not set… using mono

von Patrick Bédat, 17. April 2016

TL;DR

This article will show you
– how to utilize docker to make a bug reproducible
– how to solve the problems with the Azure Storage Client library

The bug

When our client told me we had to export some files to the Azure Storage of some of his clients I thought: Awesome! I finally get in touch with Azure. Getting in touch with cloud services is almost everytime a good chance to get some new impressions on how to tailor APIs for the web. But it turned out different this time…

So I added the WindowsAzure.Storage package from NuGet and was thrilled how easy the implementation was:

var account = new CloudStorageAccount (new StorageCredentials (accountName, accountKey), true);

var blobClient = account.CreateCloudBlobClient();

var container = blobClient.GetContainerReference(containerId);
container.CreateIfNotExists();

var blob = container.GetBlockBlobReference(Path.GetFileName(file));
blob.UploadFromFile (file);

Charming isn’t it? And it was until files got bigger. When working for Media Carrier we are often pushing gigabytes of data over the wire and that’s where the problem started:

Microsoft.WindowsAzure.Storage.StorageException: Object reference not set to an instance of an object
---> System.NullReferenceException: Object reference not set to an instance of an object
  at System.Net.WebConnectionStream.EndRead (IAsyncResult r) <0x41fc7860 + 0x0009e> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Core.ByteCountingStream.EndRead (IAsyncResult asyncResult) <0x41fbc1c0 + 0x00024> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Core.Util.AsyncStreamCopier`1[T].ProcessEndRead () <0x41fd7f10 + 0x0003b> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Core.Util.AsyncStreamCopier`1[T].EndOperation (IAsyncResult res) <0x41fd71c0 + 0x00067> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Core.Util.AsyncStreamCopier`1[T].EndOpWithCatch (IAsyncResult res) <0x41fd6e80 + 0x00073> in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Flush () <0x41fdd180 + 0x0007b> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Commit () <0x41fdcf40 + 0x00023> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Dispose (Boolean disposing) <0x41fdced0 + 0x00043> in <filename unknown>:0 
  at System.IO.Stream.Close () <0x7f1e247c53d0 + 0x00019> in <filename unknown>:0 
  at System.IO.Stream.Dispose () <0x7f1e247c5400 + 0x00013> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromStreamHelper (System.IO.Stream source, Nullable`1 length, Microsoft.WindowsAzure.Storage.AccessCondition accessCondition, Microsoft.WindowsAzure.Storage.Blob.BlobRequestOptions options, Microsoft.WindowsAzure.Storage.OperationContext operationContext) <0x41fc9e10 + 0x009e0> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromStream (System.IO.Stream source, Microsoft.WindowsAzure.Storage.AccessCondition accessCondition, Microsoft.WindowsAzure.Storage.Blob.BlobRequestOptions options, Microsoft.WindowsAzure.Storage.OperationContext operationContext) <0x41fc9db0 + 0x0004b> in <filename unknown>:0 
  at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromFile (System.String path, Microsoft.WindowsAzure.Storage.AccessCondition accessCondition, Microsoft.WindowsAzure.Storage.Blob.BlobRequestOptions options, Microsoft.WindowsAzure.Storage.OperationContext operationContext) <0x41fc9cc0 + 0x00097> in <filename unknown>:0 
  at windowsstoragebug.MainClass.Main (System.String[] args) <0x41f1cd60 + 0x004f0> in <filename unknown>:0 
Request Information
RequestID:c4dedc47-0001-0038-734d-962acc000000
RequestDate:Thu, 14 Apr 2016 12:56:26 GMT
StatusMessage:Created

And my first thought was: „What did I do wrong?“. So I began messing around with the client settings:

blobClient.DefaultRequestOptions.ServerTimeout = new TimeSpan (1, 0, 0);
blobClient.DefaultRequestOptions.MaximumExecutionTime = new TimeSpan (1, 0, 0);
blobClient.DefaultRequestOptions.SingleBlobUploadThresholdInBytes = 67108864; //64M

Nock luck. So I googled: https://github.com/Azure/azure-storage-net/issues/202. It is a similar issue, but the proposed workarounds didn’t help.
It was time to dive a bit deeper into the Azure Storage service. How does it work?

Azure Storage and PutBlock

Azure storage (similar to S3 in AWS) can be used to store files. The files are stored as blobs – block blobs in this case. Block blobs are organized in containers (would be buckets in S3).
You can either upload a BlockBlob in a single transaction – when the block blob is < 64M, or upload the blob separated in blocks, each < 4M with a maximum of 50000 blocks. Each block gets a unique block id in natural order. When all blocks have been transmitted to azure you commit the transaction by sending the whole list of block ids.

See https://msdn.microsoft.com/de-de/library/azure/ee691974.aspx

I felt lucky, when I saw, that the azure client offered a PutBlock and a PutBlockList method. So I tried to upload the file in chunks:

using(var stream = File.OpenRead(file))
{
    int position = 0;
    const int BLOCK_SIZE = 4 * 1024 * 1024;
    int currentBlockSize = BLOCK_SIZE;

    var blockIds = new List<string>();
    var blockId = 0;

    while(currentBlockSize == BLOCK_SIZE)
    {
        if ((position + currentBlockSize) > stream.Length)
            currentBlockSize = (int)stream.Length - position;

        if(currentBlockSize == 0)
            continue;

        byte[] chunk = new byte[currentBlockSize];
        stream.Read (chunk, 0, currentBlockSize);

        var base64BlockId = Convert.ToBase64String(System.Text.Encoding.Default.GetBytes(blockId.ToString("d5")));

        using(var memoryStream = new MemoryStream(chunk))
        {
            memoryStream.Position = 0;                  
            blob.PutBlock(base64BlockId, memoryStream, null);
        }

        blockIds.Add(base64BlockId);

        position += currentBlockSize;
        blockId++;

    }

    blob.PutBlockList(blockIds);
}

I ran it on my machine and started to burst into dancing. Not for long though… After deploying it, uploading suddenly stuck. Then on my maching (and later on the staging server) it suddenly gave me those lines:

_wapi_handle_ref: Attempting to ref unused handle 0x4af
_wapi_handle_unref_full: Attempting to unref unused handle 0x4af

which means that something is very very wrong…

Challenge accepted

When a 3rd party library isn’t working correctly, I’m usually trying to root out possible errors on my side. So after 2 a.m. I did some funny things with memory streams, that I really don’t want to show you. So after my PutBlock experiment didn’t work out my motivation started to multiply. I cannot rest when some 3rd party library, which is supposed to just work, simply doesn’t bend to my will.

My next plan was to implement parts of the storage client against the azure REST api.

Being a proud and arrogant developer, I believed, that I could hack that WebRequest code for azure together in minutes… Behold the Authorization header. What a pain in the ass seriously. Just take a look at this: https://msdn.microsoft.com/de-de/library/azure/dd179428.aspx

Then I came across this beatiful article.

With the help of the article above I was finally able to write my own implementation of PutBlock (I snatched CreateRESTRequest from the article above):

var request = CreateRESTRequest ("PUT", $"{container.Name}/{blobId}?comp=block&blockid={base64BlockId}", chunk);    

var response = request.GetResponse () as HttpWebResponse;

using (var responseStream = new StreamReader(response.GetResponseStream ()))
{
    var output = responseStream.ReadToEnd ();
    if (!string.IsNullOrEmpty (output))
        yield return output;
}

AND IT WORKED LIKE A CHARM. And still does.

Making the bug reproducible

The journey doesn’t end here.

The azure storage lib is open source and whenever you are using open source software you should give something back from time to time. Either through contributions, money or by being a good bug reporter.
Filing an issue on github is easy, but if you want to have it fixed, make sure it is easily reproducible. This saves a lot of time for the hard working open source contributors.

So how do we get there? The bug happened on my Ubuntu linux 14.04 running mono Stable 4.2.3.4/832de4b. Telling somebody to setup a VM, checkout my sample code, compile it and run it, would be a lot to ask.
That’s where Docker comes into play.

To run the desired environment, we have to write a Dockerfile:

# Define the base image
FROM ubuntu:14.04
MAINTAINER Patrick Bédat <patrick.bedat@ixts.de>

# Add the mono sources
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
RUN echo "deb http://download.mono-project.com/repo/debian wheezy main" | tee /etc/apt/sources.list.d/mono-xamarin.list

# Install mono and git
RUN apt-get update && \
    apt-get install -y \
    git mono-complete

RUN mozroots --import --sync

# Clone the sample project and build it
RUN git clone git://github.com/pbedat/azure-mono-bug.git && \
    cd azure-mono-bug && \
    mono NuGet.exe restore && \
    xbuild azure-storage-bug.sln

# This tells docker what to do when we run this image
ENTRYPOINT ["mono", "/azure-mono-bug/azure-storage-bug/bin/Debug/azure-storage-bug.exe"]

From this dockerfile you’re able to build an image:

docker build -t azure-mono-bug ./azure-mono-bug

Then you can run containers based on this image

docker run azure-mono-bug <account-name> <account-key>

The application then
– creates a 500 MB file
– tries to upload it with UploadFromFile to azure
– tries again with the PutBlock method

Conclusion

I’ve pushed the image to the docker hub. Now anybody using docker, can run the sample app by typing

pbedat/azure-mono-bug <account-name> <account-key>

Nuff said.