About Me

My photo
JHC Technology is a Service Disabled, Veteran-Owned, Small Business based in the Washington, DC Metro area. Our primary focus is to offer customized solutions and IT consulting to our Commercial and Government clients. Our experts have a broad experience delivering and managing Microsoft Enterprise applications and Cloud and Virtualization Solutions, as well as mobilizing Enterprise data.

Monday, September 22, 2014

High Availability (HA) for NAT Instances

While working with a project that utilized an Amazon Linux NAT (network address translation) instance for outbound connections in redundant availability zones, I realized that a single NAT making egress requests for two Availability Zones (AZs) introduces a single point of failure.

A whitepaper written by Jinesh Varia outlines the steps required to implement a two-way monitoring high-availability (HA) failover NAT solution. He provides a script with a guide on how to replace necessary variables to give each NAT instance visibility of each other.

However, I wanted to provide insight into an issue I faced while testing this configuration. I found that when I would stop a NAT instance, it would not restart. The next steps were to see if any intended routing failover was occurring, which in fact was. Using the script below I was able to see the logs of nat_monitor.sh :

tail /tmp/nat_monitor.log

Troubleshooting led me to find out that the active NAT instance was unable to see the downed instance’s state due to two reasons.

The first is that the instances only had public IPs, not Elastic IPs (EIPs). EIPs stick to the instance when the instances are turned off and are visible to the API, so when you are making calls to a box that is turned off, you are still able to communicate with it.

The second is specified in a notation Jinesh made in his whitepaper (in Step 7). He makes it clear that the script works with tools version 1.6.12.2 2013-10-15.  He points out that if NAT_STATE isn’t updating, then to change "print $4;" on line 77 to "print $5;". It's because different versions of the tools output the ec2-describe-instances differently. Here’s the original Line 77:

NAT_STATE=`/opt/aws/bin/ec2-describe-instances $NAT_ID -U $EC2_URL | grep INSTANCE | awk '{print $4;}'`

This article was written January 30, 2014, and the tools have since been upgraded. While opening up a ticket with AWS to assist in troubleshooting the script’s output, the support engineer recommended to change “print $5”; to “print $6”; and the change produced the outcome I’d been seeking.

The script uses the API to see if the NAT box is "stopped". If it is, then it will start it. If it's not stopped, it will try to stop it and then loop back to the previous attempt to start it.

You will be able to successfully test this functionality by stopping an instance within the console and observe it restarting automatically after the threshold in the nat_monitor.sh configuration is met.


Rory Vaughan is a Cloud Engineer with JHC Technology. He can be reached at rvaughan (at) jhctechnology.com or connect with him on LinkedIn.

Tuesday, September 2, 2014

Security OF the Cloud vs. Security IN the Cloud

It’s been another fun day of “blame the cloud” around the media universe, and only very few of those media groups are smart enough to understand what they’re actually looking at.  Word has spread that a hacker, or group of hackers, was able to crack user accounts in Apple’s iCloud and gain access to intimate photos of various celebrities.

The headline of the Washington Post indicates that this raises “more questions around the security of the cloud”.  What the Washington Post doesn’t get is that it’s not the security OF the cloud, it’s the security IN the cloud.  According to most reports, it seems that hackers were able to gather email addresses and passwords, or use tricks to leverage the email addresses in enabling the hacker to reset the passwords.  Another apparent way in was through hacking a service of Apple’s that helped open up a door to the user data on iCloud.

Let’s be very clear that none of these methods means that “the cloud” was compromised.  Whether your data is in a cloud, on a server under your desk, or in your corporate datacenter, if a malicious user gains access to your user name and/or password, they’re going to be able to exploit your account(s).  If a user gains access to a service such as “Find My iPhone” that has connectivity to your data, but has a security flaw, they’ll be able to exploit that.  Again, this has no bearing on where your data rests, cloud or otherwise.  A key sentence from this story by DataCenter Knowledge?:  “Cloud … is only as safe as the services that rest upon [it].”

Cloud infrastructure operates mainly with a shared responsibility model.  This means that the cloud provider is generally responsible for the security of its systems up to the servers on which your data resides. However, beyond that, from the operating system on up, the user or company is responsible for that security.  As an example, an infrastructure (cloud) provider such as Amazon Web Services will provide the servers on which you can run your website or host your files. It, generally, isn’t responsible for what you use that server for. If you don’t bother to (or don’t know to) put in the necessary security firewalls on that server to limit access, you’re running the risk of your data being available.  If you don’t bother to (or don’t know to) limit access to certain ports for traffic to your server, you’re opening major holes for exploitation.  That’s not a fault of the cloud provider, that’s user error.

Cloud and application providers have taken steps over the years to try and increase security not only of their own infrastructure and data, but to help users protect themselves.  Some of these methods include multi-factor authentication (MFA) and rotating passwords.  It also includes some services in which you have to rotate passwords on a regular basis, without using previous ones again.  While seemingly inconvenient to the end user, it provides an important step in trying to stay ahead of the game.  Users should take advantage of these components. 

I suggest, and adhere to when offered to me, utilizing MFA for all accounts.  For those unfamiliar with MFA, examples include setting your email provider or Twitter accounts to text you a code that you enter before you can log into an account.  Despite the overly ominous headline, this article from Entrepreneur offers the same advice:  take advantage of MFA.

The breach of iCloud is not a testament to cloud security.  It is more a testament to vulnerabilities of the applications or end user that has access to data stored on the cloud.  It is incumbent on us to take advantage of the security measures offered so we can all do our part.


Matt Jordan is the Cloud Services Manager for JHC Technology. He can be reached at mjordan(at)jhctechology.com, @matt_jhc, or connect with him on LinkedIn.