Sometimes government policies on location of personal data could be a pain in the unmentionables. Here is a problem of such a case. There is a website that services three regions, US, Europe, and Japan. Though the website content is hosted in US, the databases run off each the Regions(AWS) off RDS(Relational Database Service) instances. During normal times(that is when there is no excitement) the website runs two web servers, behind an Elastic Load Balancer(ELB). Its been running fine this way. Imagine now there needs to be AutoScaling Group configured for these Web Servers, in which case how do we go and configure the access to these RDS instances for the newly launched Web Servers. Yeah! Big pain.
There are at least two mechanisms that I can think of; but there might be many more.
One configure AutoScaling with SNS notification enabled, run a HTTP service that gets notified everytime, by SNS, a Web Server stretch(scaling up) or shrink(scaling down) happens. For a stretch, the HTTP service scans the AutoScaling Group for new instances, gets their IP details and configures each RDS’s security groups. Similarly for a shrink, the HTTP service scans the AutoScaling Group for removed instances, and removes theirs IP’s(/32) from each RDS’s security groups. This is kinda good solution, but complex to implement. If the spikes and dips are two frequent, the service can get a little messed up in negotiating the Security Group policies, to accommodate the stretches and shrinks.
Two, configure HAProxy to do TCP routing to the RDS instances, on say ports 3306 and 3307. This way all traffic to the RDS instances passes through HAProxy instance; everything- all reads and all writes. This way we could avoid the pain of Security Group Manipulations.
Two Continued… If we could setup HAProxy on VPC, we could have a Private IP mapped to an RDS in the HAProxy configuration. That way we could bind 3306 to each Private IP. But, we need to map Elastic IP each Private IP. And the Elastic IP bound to the Primary Private IP would need to have an Ingress on the Security Group of each RDS instance. Though this is the simplest, we can clearly see that the HAProxy is a single point of failure and we need to run a fairly large instance to accommodate all traffic through it. Also these is always the one extra hop to the RDS.
Anyways these choices are always skewed depending on two things, the development bandwidth against the agility of solution.