Import qcow2 images into AWS

When running multi-cloud applications, sometimes you may want to move an disk image or snapshot from a qemu-based virtualization environment into a public cloud such as Amazon Web Services (AWS).

qcow2 is the most common and also the native format of the disk image used by qmeu. Unfortunately, qcow2 is not a format that the AWS import-image tool can import directly - the tool only supports VMDK, VHD, and RAW formats at the time of writing. Therefore, additional steps need be taken to convert the image from qcow2 into raw for AWS to import.

The rest of this post describes how to import qcow2 images into AWS as a snapshot. Once the image is imported as a snapshot, an Amazon Machine Image (AMI) could be created from the snapshot and used to launch new instances. This procedure requires a Linux host running Ubuntu and access to AWS S3 service. The Linux host would be preferably running on AWS for faster data transfer to and from S3.

  • Prerequisites:
    • Assume the QCOW2 image to be converted/imported is located at ~/example.qcow2
    • The Linux host shall have enough disk space to hold the expanded RAW image, which would be as large as the virtual size of the image (see step 2 below for how to find out the virtual size of an image).
  1. Install qemu-utils package to get the qemu-img tool

     $ sudo apt-get install qemu-utils
    
  2. Find out the virtual size of the disk image and ensure you have enough space to hold the expanded RAW image:

     $ qemu-img info example.qcow2
     image: example.qcow2
     file format: qcow2
     virtual size: 39M (41126400 bytes)
     disk size: 13M
     ...
    
  3. Use qemu-img to convert the image into RAW format. The expanded RAW image is about 39M for this specific image

     $ qemu-img convert example.qcow2 example.raw
    
  4. Install AWS CLI:

     $ sudo pip install awscli --ignore-installed six
    
  5. Put your AWS credentials under ~/.aws/config, see AWS CLI configuration reference for details

     [default]
     aws_access_key_id=FOO
     aws_secret_access_key=BAR
    
  6. Copy the RAW image into S3, assume you’ve already had an S3 bucket named raw-images

     $ aws s3 cp example.raw s3://raw-images
    
  7. Create vmimport role and assign proper policy for the S3 bucket by following the AWS procedure here:

     $ vim trust-policy.json
     $ aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json
     $ vim role-policy.json
     $ aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json
    
  8. Create a JSON file container.json with the following content:

     {
         "Description": "Example image originally in QCOW2 format",
         "Format": "raw",
         "Url": "https://s3-us-west-2.amazonaws.com/raw-images/example.raw"
     }
    
  9. Import the image:

     $ aws ec2 import-snapshot --description "example image" --disk-container file://container.json
    
  10. Finally, if you would like to launch instance with the image, follow these AWS instructions to create an AMI from the snapshot on AWS console.

Using Packet Filter (pf) firewall to block outgoing traffic on macOS

Recently I found out that PyCharm from JetBrains, despite being a wonderful IDE for Python, is continuously broadcasting my username to 230.230.230.230 for license check, see here for a report of the same problem for another Jetbrains’ product, which has the same underpinning IDE as PyCharm.

Naturally, I want a firewall to block the outgoing traffic to avoid leaking my private information to any network I might connect to.

The OS X application firewall (see Apple notes) can block incoming traffic on per-application basis and prevent applications from listening on network ports, but unfortunately it cannot be configured to block outgoing traffic.

The application firewall, however, is indeed implemented with Packet Filter from OpenBSD project. Remember Mac OS X is part of the BSD family? PF has been shipped with recent releases of Mac OS X since Lion, including macOS since Sierra.

There are a number of third-party applications/firewalls on the market such as murus. But they basically provide the user a GUI to configure PF on macOS - of course sometimes with other useful features as well - and they are mostly paid application (although murus does have a lite version that is free).

But if you’re comfortable with command line as I am, all GUI applications are overkill for the problem in hand. It’s possible to configure PF to block outgoing traffic in several easy step and less than a few minutes!

  1. First, create a new anchor file named /etc/pf.anchors/jetbrains with the following PF rule to block traffic on interface en0 for any traffic sent to IP multicast address 230.230.230.230:

     $ cat /etc/pf.anchors/jetbrains
     block drop log quick on en0 from any to 230.230.230.230
    

    You would need sudo privillege to create file under /etc/pf.anchors. An anchor file is used to hold a sub-ruleset, which we will attach to the main PF ruleset in the next step. quick asks PF to stop further processing should a packet matches the rule. See PF filter reference on the syntax of the rules for more details.

  2. Then add the jetbrains anchor to the default PF configuration file /etc/pf.conf. This allows the anchor and the rules to be active whenever you activate the macOS firewall without interfering with any application firewall rule you might have defined through GUI.

     $ cat /etc/pf.conf
     #
     # Default PF configuration file.
     ...
     # See pf.conf(5) for syntax.
     #
    
     #
     # com.apple anchor point
     #
     scrub-anchor "com.apple/*"
     nat-anchor "com.apple/*"
     rdr-anchor "com.apple/*"
     dummynet-anchor "com.apple/*"
     anchor "com.apple/*"
     load anchor "com.apple" from "/etc/pf.anchors/com.apple"
    
     #
     # jetbrains anchor point
     #
     anchor "jetbrains"
     load anchor "jetbrains" from "/etc/pf.anchors/jetbrains"
    
  3. Last, start the firewall from System PreferencesSecurity & PrivacyFirewall.

Tested with the following software versions:

  • macOS 10.12 (16A323)

Other useful resource for PF on macOS:

  • Using pf on OS X Mountain Lion

    This article is mostly still relevant for macOS Sierra, although I believe there is no need to create a launchd item should you put the anchor into the default pf configuration file /etc/pf.conf as shown here.

  • PF on Mac OS X

    This is a detailed wiki about PF and its command line pfctl and pflog etc. Good read if you’d like to see more example usages of these tools.

  • macOS Security and Privacy Guide

    This is a great guide with discussion of security and privacy on macOS to a broad extent. It also touches packet filter and discusses options for third-party firewalls (including options that may not use PF).

Running docker with docker-machine and xhyve on macOS

xhyve is a lightweight OS X virtualization solution and can run on OS X 10.10 Yosemite and higher. xhyve currently supports FreeBSD and Linux distributions as guest systems. xhyve also has a docker-machine driver that allows you to use docker-machine to run docker containers in side a VM and easily manage the lifecycle of the docker container and the VM.

These three components combined is a nice lightweight docker solution on macOS, and all of them can be installed and updated by Homebrew. It is a much better fit with a developer’s console-based workflow - No more VirtualBox and its annoying updates!

Assuming you’ve already have Homebrew installed, here is how to install xhyve and docker-machine-driver-xhyve:

$ brew update
$ brew install --HEAD xhyve
$ brew install docker-machine-driver-xhyve

The last command will also install docker-machine as a dependency.

Then make sure you change the docker-machine-driver-xhyve binary for proper permissions (due to Homebrew policy this could not be automated):

$ sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

Now that you have everything installed, start a helloworld docker:

$ docker-machine create -d xhyve helloworld

Tested with the following software versions:

  • macOS 10.12 (16A323)
  • Homebrew 1.0.5
  • xhyve: stable 0.2.0
  • docker-machine: stable 0.8.2
  • docker-machine-driver-xhyve: stable 0.2.3

Securing email of your domain against spam and phishing

Securing email of your domain against spam and phishing has two aspects:

  1. You need an authentication key to sign all outgoing emails
  2. You need publish via DNS records the public key of the authentication key, along with policies about who can send for your domain and what others should do if they receive unauthenticated emails from your domain.

Hosting email of your domain on Google Apps makes things pretty easy to authenticate your emails, prevent spammers and phishing. But you’ll need a good DNS provider as well to support provisioning a number of DNS records.

Below are three articles from Google Apps that covers what you need do - they are not limited to Google Apps hosted emails and could be very informative in general:

  • Authenticate email with DKIM. This tells you how to enable Google Apps email authentication and publish the public key in a DNS TXT record for DKIM

  • Identify spam messages with SPF records. This is about how to create a DNS TXT record for SPF policy to help receiver identify spammers from your authorized sender or email gateway.

  • Prevent outgoing spam with DMARC. This describes the DNS TXT record for DMARC that publishes your desired policy of how the receiver shall deal with unauthenticated emails from your domain if the email does not pass SPF and DKIM check.

Finally, when you’ve done all your settings, use Google Apps Toolbox - Check MX to validate your domain’s MX records.

Switched to jekyll and CloudFlare

Dreamhost has been a great host for many years but there are other options for hosting a plain blog like this one these days, which makes paying out ~$50 for two years’ hosting start feeling too much.

So I finally converted to jekyll+github-pages solution and uses a free plan from CloudFlare to front the blog with HTTPS. In order to do its job, CloudFlare also becomes my domain DNS server.

CloudFlare for this site now runs in Full SSL mode, which means SSL is run between visitors and CloudFlare CDN, as well as between CloudFlare and github-pages.

I cannot run Full (strict) mode, which would ask CloudFlare to validate its connection to github-pages with a server-side certificate for my domain, because github-pages only serve HTTPS with a certificate for github.com/github.io.


Contents on this site are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Creative Commons License