Adventures in GovCloud
Many of our contracts here at Truss are for the federal government. Anyone who has done similar work knows that many clients in this space have requirements like FedRAMP, FISMA, or DoD IL levels that mean if you’re building in AWS, you need to use GovCloud. These regions in AWS fulfill the additional compliance requirements that these certifications require -- but as a result, they lack some of the services and capabilities that commercial AWS regions have, and some of the services they do have work differently than you may be used to.
My last project for Truss was both the first project where we extensively used AWS Organizations and the first where we really had to dig into GovCloud and how it interacts with commercial AWS. In the course of doing so, I found some of the edge cases where GovCloud throws you some curveballs and tried to come up with patterns that make working with it easier. You can find most of this in our Engineering Playbook, but I thought I’d try to write something to make this all a little more digestible.
Getting Started
The first thing to know about GovCloud is that commercial accounts work only in commercial AWS regions, and GovCloud accounts work only in GovCloud regions, and never the twain shall meet. There’s basically no interaction between GovCloud and commercial AWS, except for billing. You can’t assume roles between the two or allow IAM roles in commercial AWS to access GovCloud resources or vice versa. This means that if we’re using AWS Organizations in GovCloud, we’ll need to duplicate the structure that we have in commercial AWS (at least for the basics, like the “-org-root”, “-id”, and “-infra” accounts). If you want to know more about how we handle AWS Organizations here at Truss, we’ve got a recent blog post that describes the patterns we use.
In order to get started then, we need to create a GovCloud account. You need to have an existing commercial AWS account already, because that’s how billing is handled (all your GovCloud resources will get billed to the corresponding commercial account). You will have to log in as the root user in order to do this. If you’re using AWS Organizations, I recommend doing this from the master account for the organization (especially since you probably don’t have root users for the subsidiary accounts). If you think you will want to get rid of the GovCloud accounts but keep the commercial ones around, however, you might want to create a new account specifically to be the placeholder account for your GovCloud deployment, because the only way to delete a GovCloud account is to delete the corresponding commercial AWS account, but I’m not sure how well that would actually work with the rest of our patterns.
To create the GovCloud account, log in as the root user and scroll down on the “My Account” page, and you’ll see a button labelled “Sign Up for AWS GovCloud”. You’ll need to sign an addendum to the AWS user agreement and then you’ll get an email telling you how to log in to this new account. You can name it whatever you want; when I made it, I just added a suffix to the organization prefix we were using for our accounts in commercial AWS; so if our commercial master account was “spacecats-org-root”, the new account would be “spacecats-gov-org-root”.
You’ll notice that your new GovCloud account does not have a “root” user that you can use. Instead, it has an “Administrator” IAM user that has full administrative privileges that for most things functions similarly. However, sometimes, this can bite you. At Truss, we regularly set our password expirations for commercial AWS users to 1 day because we don’t want people using passwords to log in to the console. When we did this in GovCloud, it locked us out of being able to log in as Administrator. Luckily, we had one person who had created their individual IAM user who was able to fix this, but you should be careful (especially with the Administrator user of your org-root account).
Making Your GovCloud Organization
So now you’ve got a GovCloud account and you’re all ready to get started. Here at Truss, we like to do everything with Terraform to keep our infrastructure as code. So your first step is to set up the new organization in GovCloud and add all the skeletal bits you need for that, up to the point where you are ready to start creating new accounts.
For our commercial AWS organizations, we just use Terraform to create all the subsidiary accounts we’re going to use. Unfortunately, you can’t do this in GovCloud. Every GovCloud account needs to be linked to a commercial account, so to create a new GovCloud account you have to do it through commercial AWS. There’s no easy way to do this with Terraform either, so what we ended up doing was basically:
Use the AWS CLI to create the new account via the “aws organizations create-gov-cloud-account” command with credentials from your commercial org-root account. This creates a new account in your commercial AWS organization and a new account with the same alias in GovCloud.
Use credentials from your GovCloud org-root account to assume the OrganizationAccountAccessRole in the new account (yes, this exists even though the new GovCloud account isn’t actually in the GovCloud organization if you look at the GovCloud organization in the org-root account).
Once you’ve got access to the new account, you can invite the account to your GovCloud organization using the org-root account, and then accept the invitation using your new account. The exact process for this is detailed in our Engineering Playbook.
Once you’re done creating your organization, you’ll have a bunch of extra placeholder accounts in your commercial AWS organization. We recommend not using these for any real purpose; instead, add them to the “suspended” OU that prevents them from actually doing anything. This prevents them from being an added management headache, and if you want to get rid of the GovCloud account in the future, you can do it without affecting any of your other infrastructure.
If you think this sounds annoying and convoluted, well, you’re not wrong, and it’s not especially well-explained in the AWS documentation. Knowing the exact process will save you half a day of back and forth with AWS support, though.
Things Are Different Here
Once you have your accounts and your organization created in GovCloud, it’s time to start building out your AWS resources. Here’s where all those little differences between commercial AWS and GovCloud start to become more obvious. You will probably want to take a look at the GovCloud Services in Scope page and the AWS Region Table to know what you can use where. Most services are available in some form, at least in us-gov-west-1, but many don’t work quite the same; and us-gov-east-1 does not have as many available services as us-gov-west-1. We have discovered that some services don’t work exactly as described in AWS documentation -- keep that in mind as you’re exploring.
The biggest difference, and the one that will probably bite you the hardest, is that ARNs in GovCloud have a different format than ARNs in commercial AWS. While commercial AWS ARNs look like this:
arn:aws:iam::123456789000:user/testuser
GovCloud ARNs look like this:
arn:aws-us-gov:iam::123456789000:user/testuser
If you have hardcoded ARNs or patterns that assume the commercial format in your Terraform modules, this will break them. When we began deploying Terraform in earnest to GovCloud, we ended up needing to make a lot of revisions to our modules in order to solve this. Key to this is the Terraform aws_partition data source, which will use the correct ARN element based on the region your code is executing against. You can see how to use this in most of our Terraform modules now.
AWS has a number of security and compliance products that you will probably want to use in GovCloud as well. We use AWS Cloudtrail and AWS Config for this sort of thing in commercial AWS all the time, and we’ve started to use AWS GuardDuty. These are also available in GovCloud and both Config and GuardDuty seem to work the same way in GovCloud that they do in commercial AWS.
CloudTrail, on the other hand, does not work the same way in GovCloud. Organization CloudTrails are not available in GovCloud (so you cannot add “cloudtrail.amazonaws.com” as a service principal in your organization definition). In your first GovCloud account, you will get a S3 bucket named “cloudtrail-<random hex string>” (but no actual CloudTrail); all subsequent accounts will have an automatically created CloudTrail and matching S3 bucket named “CloudTrail-<random hex string>”. Our practice has been to delete these trails and buckets and create a new trail that writes to the S3 bucket where we keep our other AWS logs, just to put things where people expect. If you prefer though, you can import these resources into your Terraform configuration and use them as-is.
There are a number of other complications with AWS services we’ve uncovered in our explorations as well; we have tried to document that in our Engineering Playbook to prevent these kinds of headaches in the future. One of the most annoying examples we’ve encountered is that you cannot add public DNS via Route53 in GovCloud, so if you have externally-facing services in GovCloud, you’ll need to add DNS entries in your commercial AWS environment that point to your services in GovCloud.
It’s Not That Bad
In total, our experience with GovCloud has been that it isn’t that bad, once you are able to shine the light in the dark corners. Most AWS services are available there (although they may not have been approved for the compliance level you are building for) and most good infrastructure patterns from commercial AWS can be applied in GovCloud without too much adjustment. The hardest part is actually figuring out what is in those dark corners, and hopefully sharing our experiences with them will help you avoid the same pitfalls.