13 Nov 2016
When running multi-cloud applications, sometimes you may want to move an disk
image or snapshot from a qemu-based virtualization
environment into a public cloud such as Amazon Web Services
is the most common and also the native format of the disk image used by qmeu.
Unfortunately, qcow2 is not a format that the AWS
import-image tool can
import directly - the tool only supports VMDK, VHD, and RAW formats at the time
of writing. Therefore, additional steps need be taken to convert the image from
raw for AWS to import.
The rest of this post describes how to import qcow2 images into AWS as a
snapshot. Once the image is imported as a snapshot, an Amazon Machine Image
(AMI) could be created from the snapshot and used to launch new instances.
This procedure requires a Linux host running Ubuntu and access to AWS S3
service. The Linux host would be preferably
running on AWS for faster data transfer to and from S3.
- Assume the QCOW2 image to be converted/imported is located at
- The Linux host shall have enough disk space to hold the expanded RAW
image, which would be as large as the
virtual size of the image (see
step 2 below for how to find out the
virtual size of an image).
qemu-utils package to get the
$ sudo apt-get install qemu-utils
Find out the
virtual size of the disk image and ensure you have enough
space to hold the expanded RAW image:
$ qemu-img info example.qcow2
file format: qcow2
virtual size: 39M (41126400 bytes)
disk size: 13M
qemu-img to convert the image into RAW format. The expanded RAW image
is about 39M for this specific image
$ qemu-img convert example.qcow2 example.raw
Install AWS CLI:
$ sudo pip install awscli --ignore-installed six
Put your AWS credentials under
~/.aws/config, see AWS
CLI configuration reference for
Copy the RAW image into S3, assume you’ve already had an S3 bucket named
$ aws s3 cp example.raw s3://raw-images
vmimport role and assign proper policy for the S3 bucket by
following the AWS procedure
$ vim trust-policy.json
$ aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json
$ vim role-policy.json
$ aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json
Create a JSON file
container.json with the following content:
"Description": "Example image originally in QCOW2 format",
Import the image:
$ aws ec2 import-snapshot --description "example image" --disk-container file://container.json
Finally, if you would like to launch instance with the image, follow these
to create an AMI from the snapshot on AWS console.
09 Nov 2016
Recently I found out that PyCharm from
JetBrains, despite being a wonderful IDE for
Python, is continuously broadcasting my username to
license check, see
for a report of the same problem for another Jetbrains’ product, which has the
same underpinning IDE as PyCharm.
Naturally, I want a firewall to block the outgoing traffic to avoid leaking my
private information to any network I might connect to.
The OS X application firewall (see Apple
notes) can block incoming traffic on
per-application basis and prevent applications from listening on network ports,
but unfortunately it cannot be configured to block outgoing traffic.
The application firewall, however, is indeed implemented with Packet
OpenBSD project. Remember Mac OS X is part of the
BSD family? PF has been shipped with recent releases of Mac OS X since Lion,
including macOS since Sierra.
There are a number of third-party applications/firewalls on the market such as
murus. But they basically provide the user a
GUI to configure PF on macOS - of course sometimes with other useful features
as well - and they are mostly paid application (although murus does have a
lite version that is free).
But if you’re comfortable with command line as I am, all GUI applications are
overkill for the problem in hand. It’s possible to configure PF to block
outgoing traffic in several easy step and less than a few minutes!
First, create a new anchor file named
/etc/pf.anchors/jetbrains with the
following PF rule to block traffic on interface
en0 for any traffic sent to
IP multicast address
$ cat /etc/pf.anchors/jetbrains
block drop log quick on en0 from any to 18.104.22.168
You would need sudo privillege to create file under
anchor file is used to hold a sub-ruleset, which we will attach to the main PF
ruleset in the next step.
quick asks PF to stop further processing should a
packet matches the rule. See PF filter reference on the syntax of the
rules for more details.
Then add the
jetbrains anchor to the default PF configuration file
/etc/pf.conf. This allows the anchor and the rules to be active whenever you
activate the macOS firewall without interfering with any application firewall
rule you might have defined through GUI.
$ cat /etc/pf.conf
# Default PF configuration file.
# See pf.conf(5) for syntax.
# com.apple anchor point
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
# jetbrains anchor point
load anchor "jetbrains" from "/etc/pf.anchors/jetbrains"
Last, start the firewall from
Security & Privacy
Tested with the following software versions:
Other useful resource for PF on macOS:
Using pf on OS X Mountain
This article is mostly still relevant for macOS Sierra, although I believe
there is no need to create a launchd item should you put the anchor into the
default pf configuration file
/etc/pf.conf as shown here.
PF on Mac OS X
This is a detailed wiki about PF and its command line
etc. Good read if you’d like to see more example usages of these tools.
macOS Security and Privacy Guide
This is a great guide with discussion of security and privacy on macOS to a
broad extent. It also touches packet filter and discusses options for
third-party firewalls (including options that may not use PF).
30 Sep 2016
xhyve is a lightweight OS X virtualization
solution and can run on OS X 10.10 Yosemite and higher. xhyve currently
supports FreeBSD and Linux distributions as guest systems. xhyve also has a
that allows you to use docker-machine to
run docker containers in side a VM and easily manage the lifecycle of the
docker container and the VM.
These three components combined is a nice lightweight docker solution on macOS,
and all of them can be installed and updated by Homebrew. It
is a much better fit with a developer’s console-based workflow - No more
VirtualBox and its annoying updates!
Assuming you’ve already have Homebrew installed, here is how to install xhyve
$ brew update
$ brew install --HEAD xhyve
$ brew install docker-machine-driver-xhyve
The last command will also install docker-machine as a dependency.
Then make sure you change the docker-machine-driver-xhyve binary for proper
permissions (due to Homebrew policy this could not be automated):
$ sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
Now that you have everything installed, start a helloworld docker:
$ docker-machine create -d xhyve helloworld
Tested with the following software versions:
- macOS 10.12 (16A323)
- Homebrew 1.0.5
- xhyve: stable 0.2.0
- docker-machine: stable 0.8.2
- docker-machine-driver-xhyve: stable 0.2.3
09 Sep 2016
Securing email of your domain against spam and phishing has two aspects:
- You need an authentication key to sign all outgoing emails
- You need publish via DNS records the public key of the authentication key,
along with policies about who can send for your domain and what others should
do if they receive unauthenticated emails from your domain.
Hosting email of your domain on Google Apps makes things pretty easy to
authenticate your emails, prevent spammers and phishing. But you’ll need a good
DNS provider as well to support provisioning a number of DNS records.
Below are three articles from Google Apps that covers what you need do - they
are not limited to Google Apps hosted emails and could be very informative in
Authenticate email with DKIM.
This tells you how to enable Google Apps email authentication and publish the
public key in a DNS TXT record for DKIM
Identify spam messages with SPF
records. This is about how to
create a DNS TXT record for SPF policy to help receiver identify spammers from
your authorized sender or email gateway.
Prevent outgoing spam with
DMARC. This describes the DNS
TXT record for DMARC that publishes your desired policy of how the receiver
shall deal with unauthenticated emails from your domain if the email does not
pass SPF and DKIM check.
Finally, when you’ve done all your settings, use Google Apps Toolbox - Check
MX to validate your domain’s MX
08 Sep 2016
Dreamhost has been a great host for many years
but there are other options for hosting a plain blog like this one these days,
which makes paying out ~$50 for two years’ hosting start feeling too much.
So I finally converted to
solution and uses a free plan from CloudFlare to
front the blog with HTTPS. In order to do its job, CloudFlare also becomes my
domain DNS server.
CloudFlare for this site now runs in Full SSL mode, which means SSL is run
between visitors and CloudFlare CDN, as well as between CloudFlare and
I cannot run Full (strict) mode, which would ask CloudFlare to validate its
connection to github-pages with a server-side certificate for my domain,
because github-pages only serve HTTPS with a certificate for