High Availability (HA) for NAT Instances

While working with a project that utilized an Amazon Linux NAT (network address translation) instance for outbound connections in redundant availability zones, I realized that a single NAT making egress requests for two Availability Zones (AZs) introduces a single point of failure.

A whitepaper written by Jinesh Varia outlines the steps required to implement a two-way monitoring high-availability (HA) failover NAT solution. He provides a script with a guide on how to replace necessary variables to give each NAT instance visibility of each other.

However, I wanted to provide insight into an issue I faced while testing this configuration. I found that when I would stop a NAT instance, it would not restart. The next steps were to see if any intended routing failover was occurring, which in fact was. Using the script below I was able to see the logs of nat_monitor.sh :

tail /tmp/nat_monitor.log

Troubleshooting led me to find out that the active NAT instance was unable to see the downed instance’s state due to two reasons.

The first is that the instances only had public IPs, not Elastic IPs (EIPs). EIPs stick to the instance when the instances are turned off and are visible to the API, so when you are making calls to a box that is turned off, you are still able to communicate with it.

The second is specified in a notation Jinesh made in his whitepaper (in Step 7). He makes it clear that the script works with tools version 1.6.12.2 2013-10-15.  He points out that if NAT_STATE isn’t updating, then to change "print $4;" on line 77 to "print $5;". It's because different versions of the tools output the ec2-describe-instances differently. Here’s the original Line 77:

NAT_STATE=`/opt/aws/bin/ec2-describe-instances $NAT_ID -U $EC2_URL | grep INSTANCE | awk '{print $4;}'`

This article was written January 30, 2014, and the tools have since been upgraded. While opening up a ticket with AWS to assist in troubleshooting the script’s output, the support engineer recommended to change “print $5”; to “print $6”; and the change produced the outcome I’d been seeking.

The script uses the API to see if the NAT box is "stopped". If it is, then it will start it. If it's not stopped, it will try to stop it and then loop back to the previous attempt to start it.

You will be able to successfully test this functionality by stopping an instance within the console and observe it restarting automatically after the threshold in the nat_monitor.sh configuration is met.

Rory Vaughan is a Cloud Engineer with JHC Technology. He can be reached at rvaughan (at) jhctechnology.com or connect with him on LinkedIn.

Why use the Configuration Logging feature in XenApp 6.5?

The Configuration Logging feature allows you to keep track of administrative changes made to your server farm environment. By generating the reports that this feature makes available, you can determine what changes were made to your server farm, when they were made, and which administrators made them. This is especially useful when multiple administrators are modifying the configuration of your server farm. It also facilitates the identification and, if necessary, reversion of administrative changes that may be causing problems for the server farm.

Below I have provided step by step instructions on how to configure this feature in XenApp 6.5.

Step 1: Left click on "Start (push button)" in "Start"

Step 2: Left click on "Citrix AppCenter (menu item)" in "Start menu" Step 3: Left click on "SOC Farm (outline item)" in "Citrix AppCenter"  Step 4: Right click on "SOC Farm (outline item)" in "Citrix AppCenter"

Step 5: Left click on "Farm properties (menu item)"

Step 6: Left click on "Configuration Logging (outline item)" in "SOC Farm - Farm Properties"

Step 7: Left click on "Configure Database... (push button)" in "SOC Farm - Farm Properties"

Step 8: Left click on "Server name: (This will be the name of your SQL server)" in "Configuration Logging Database"

Step 9: The username specified must have db_owner permissions over the database in SQL.

Citrix recommended to use Windows authentication as it’s more secure than SQL authentication.

Step 10: Left click on "password (editable text)" in "Configuration Logging Database"

Step 11: Left click on "password (editable text)" in "Configuration Logging Database"

Step 12: Enter the password for the Windows account.

 Step 13: Left click on "Next > (push button)" in "Configuration Logging Database"

Step 14: Left click on "specify the database (editable text)" in "Configuration Logging Database"

Step 15: Specify the database created in SQL

Step 16: Left click on "Next > (push button)" in "Configuration Logging Database"

Step 17: Left click on "Open (push button)" in "Configuration Logging Database"

Step 18: left click on "No (list item)"

 Step 19: left click on "Next > (push button)" in "Configuration Logging Database"

Step 20: left click on "Test Database Connection (push button)" in "Configuration Logging Database"

Step 21: left click on "OK (push button)" in "AppCenter"

Step 22: left click on "Finish (push button)" in "Configuration Logging Database (4/4)" 

Step 23: left click on "Apply (push button)" in "SOC Farm - Farm Properties"

Step 24: left click on "OK (push button)" in "SOC Farm - Farm Properties"

Step 25: left click on "History (outline item)" in "Citrix AppCenter"

Step 26: User left click on "History (outline item)" in "Citrix AppCenter"

Step 27: User left click on "Yes (push button)" in "No Filters Specified"

Step 28: Once History is selected, choose “Get Log” under the Action column on the right hand side of the window.

Step 29: As you can see, now that Configuration Logging has been enabled changes made within the AppCenter, such as the publishing of Apps or changes to the permissions of applications or policies will be logged under the History option located within your Farm under XenApp.

David Cuevas is a Jr. Citrix Engineer for JHC Technology.  He can be reached at dcuevas (at) jhctechnology.com.

DIY, Cloud Style

Unless you’re actively working in the cloud technology space, you’re probably relying on those that are to provide you the cloud services you’ve come to depend on.  One of the biggest pieces of the move to cloud is the use of the cloud for backing up your data.  I know many people that either utilize their own external storage drives or pay a yearly/monthly subscription to backup info to an online vendor.

While an external drive is good, it’s not always there when you need it, and while an online vendor is an easy choice, what do you really know about them?  Can you trust the storage?  Can you trust that they’ll be in business?  Can you trust that they won’t make a mistake and delete your files?

For me, I decided that I wasn’t going to pay for an external drive, and there’s no reason for me to utilize an online vendor.  With Amazon Web Services and the free tier, I pay next to nothing for my small backup account.  This post will walk through the ease of using AWS’s Simple Storage Service (S3) to back up your own information and cut out the expensive hardware and expensive middle man.

If you haven’t done so already, create an AWS account by going to http://aws.amazon.com and clicking “Sign Up”.

1. Once you have your account created, go to the AWS Management Console.  Select S3.  
    
2. From the S3 Dashboard, select Create Bucket.  (Note:  an S3 Bucket is simply a container for your files.  Within Buckets you can have Folders to help organize information.)  
    
3. Once you’ve created the bucket, you will be prompted to enter a name.  Keep in mind that this name is not simply unique to your account, which means you won’t be able to go generic.  You may have to enter a couple of names to find one that works.  I ended up with “demo-test-1” as my bucket name, as an FYI.  As you can note, you also get to select the region in which to store your data.  
   
    
4. Once you’ve named your Bucket, click Create (You can also set up logging for the bucket, if you’d like).
5. You will now have your bucket listed on the S3 dashboard.  
   
     
6. Access the bucket by clicking the Bucket name.
7. You can create folders by clicking on the Actions dropdown and selecting Create Folder.  Simply type the folder name in the line that appears.
8. To upload a file or files, click Actions, then select Upload.  
   
     
9. You will then be prompted to select your files.  Click Start Upload when finished selecting your files.  
   
     

By default, the S3 bucket is accessible only to the user account that created the bucket.  In order to make the bucket available to others, permissions will have to be assigned via AWS Identity and Access Management users or through other permissions that open availability to the public.

Matt Jordan is the Cloud Services Manager for JHC Technology.  He can be reached at mjordan@jhctechology.com, @matt_jhc, or connect with him on LinkedIn.

Excel Calculation Services Won’t Start? Here's a How-To on Making it Work

Recently I was working with a client and they had an issue with Excel Calculation Services. The user had an Excel Web Access web part to display an Excel spreadsheet chart on their site. The web part was throwing a ‘Unable to process the request … Please try again …’ error. 

First, I performed the following tasks to ensure that I wasn’t missing anything:

1. Open SharePoint Central Admin
2. Open “Services on server”
3. Make sure everything looked good there
4. Attempted to stop/restart “Excel Calculation Services” – but to no avail – Same error was still showing on the web part
5. I then decided to look through the MOSS ULS logs as my last and final resort and some interesting errors stood out to me. 

1. ExcelServerApp.ClearTempPath: Failed to create the Ecs temp path.
2. EngineInterop::LoadDll: Failed to load our engine (xlsrvol.dll)

To find these, I looked for “Excel Services Application service is starting” and the errors would directly follow.  

I ended up fixing this issue by performing the following steps:

1. Go to IIS Manager (Start, Administrative Tools, Internet Information Services (IIS) Manager) on the server running Excel Calculation Services
2. Expand the server and Sites nodes
3. Select the site running ECS (SharePoint Web Services in my case)
4. Choose the Authentication icon in the IIS area
5. Choose ASP.NET Impersonation in the list and disable it

Gary Arrington is the Cloud Consultant & SharePoint SME for JHC Technology.  He can be reached at garrington(at)jhctechnology.com or connect with him on LinkedIn.

Deleted Emails? One More Step For Hope

Since my last blog, we reviewed several ways to recover deleted email items. From recovering files from the deleted items folder within MS Outlook to searching the dumpster on an exchange server. However, there is one more way to retrieve deleted email items. And that is through your system backups.

All businesses maintain some sort of disaster recovery model that includes regular backups of their email systems. Using a 3rd party tool like Symantec Netbackup and Microsoft Exchange, an entire email database containing user mailbox data can be restored. The information can then be imported back into the user’s mailbox. The steps are simple and as follows:

1. You want to obtain as much information as possible regarding the user’s missing emails. Particularly which email folders are missing mail (i.e. Inbox, Sent Items) and the dates the emails are missing from. This is very important because the restore point selected will be based on the date the exchange server was backed up.

2. You will need to create a Recovery database (RDB) in preparation for the restore process on your MS Exchange server. This recovery database will house an email database and log file information for the mailbox that you want to restore. The database must be dismounted and ready to be overwritten by a restore. These are options that can be selected once the RDB is created.

3. You need to use your third party backup tool to display a timeline of the backup jobs that were performed on the exchange server. You want to select a date prior to that of the missing email and one that will contain the latest backup information. Once obtained, you will start what’s known as a restore to process to pull the email database information from its stored location (backup tapes, disk space, etc.) which is usually rather large file so make sure that you have enough drive space in which to populate the data. The data is placed into the RDB based on a selected path that’s predetermined that tells the Netbackup where to place the restored files.

4. After the restore is completed, using MS Exchange Management Powershell, you can run a simple script to populate the missing emails directly into the users’ mailboxes with no user intervention required.

It’s that simple. You have now gone through the process of restoring missing email files from a recovery database.

Jeronna Freeman  is the Cloud Administrator for JHC Technology.  She can be reached atjfreeman (at) jhctechnology.com or connect with her on LinkedIn.