Docker clock drift on MacBooks

by Chris Hickman

Like many other developer teams, we have standardized on using MacBook Pros for doing our development work. And we use Docker to containerize our (micro)services, making it easy for any team member to run our services locally with the same setup as production.

However, using Docker on MacBook Pros is not without its issues. Once such hiccup is dealing with host VM clock drift. This is where the Docker host VM internal clock gets out of sync with the actual system time.

It appears the primary reason for clock drift with the Docker host VM is due to hibernate cycles. When your MacBook comes out of sleep mode, if the system does not have access to an NTP server, the Docker host VM clock may get considerably out of sync.

This clock skew can cause various problems. For us, the tell-tale sign of clock drift is when AWS API calls start failing due to expired signatures (if the clock drift is greater than 5 minutes from actual time, the AWS signature will be deemed invalid). Here's an example error we might see in our logs (generated by Python code using Boto for making AWS calls):

botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the Decrypt operation: Signature expired: 20160406T191109Z is now earlier than 20160406T191613Z (20160406T192113Z - 5 min.)

When you discover that you have clock drift with your host VM, you can fix this problem by simply forcing a clock sync for your host VM. Setting the clock in any container sets it for the underlying VM. So once you force the clock reset, all containers will see the new time and you should be back to regular operation.

How to force the clock sync depends on whether you are using Docker for Mac or docker-toolbox for your host VM.

Docker for Mac

With Docker for Mac, the Docker engine is running in an Alpine Linux distribution on top of an xhyve Virtual Machine. The VM clock can be manually reset by running the following command:

$ docker run --rm --privileged alpine hwclock -s

docker-toolbox

With docker-machine, you can force a resync by restarting the Docker host VM. However, this can be painfully slow.

A much quicker and easier technique is to shell into the Docker host VM and use ntpclient to force a clock sync. Just run the following command from an OS X terminal:

$ docker-machine ssh dev \
  "date; sudo ntpclient -s -h time.nist.gov; date"

Simple, quick and easy!