Auto-Scale Ghost Instances (AWS EC2)

Over the past week I have been testing Amazon’s AWS EC2 Auto-scaling with a few custom AMIs. Early today as I was cleaning up the auto-scaling configuration I encountered a bit of a conundrum – the auto-scaling API was returning a few instances that did not exist and was stopping me from removing an auto-scaling-group. After hitting Google I wasn’t any wiser, but it seemed that I did not dismantle my test setup properly, which was most likely causing the problem.

Here’s the problem I stumbled upon:

#~: as-delete-auto-scaling-group groupname
Are you sure you want to delete this AutoScalingGroup? [Ny]y
as-delete-auto-scaling-group:  Service error: You cannot delete an AutoScalingGroup while there are instances or pending Spot instance request(s) still in the group.

There shouldn’t be any instances in the group, yet running the following command:

#~: as-describe-auto-scaling-instances

Lists the following instances:

INSTANCE  i-xxxxxxxx  webhive  eu-west-1b  InService  HEALTHY  groupname
INSTANCE  i-xxxxxxxx  webhive  eu-west-1b  InService  HEALTHY  groupname

According to the EC2 Management Console and API calls, these instances did not exist anywhere!

Remember earlier I mentioned that I did not dismantle the test setup properly? What did I do? I simply suspended the auto-scale policy and manually terminated the instances that had been invoked during the testing.

What should I have done?

Updated the auto-scaling-group and set the min-size and max-size to 0 (this would have automatically terminated the instances)
Suspended the polices
Cleaned up my testing configuration

Here are the commands for the above steps:

#~: as-update-auto-scaling-group groupname --min-size 0 --max-size 0
OK-Updated AutoScalingGroup
#~: as-resume-processes groupname
OK-Processes Resumed
#~: as-describe-auto-scaling-instances
Testing
No instances found

I know, I should have read the documentation thoroughly! However this is what testing is for, and I hope that this little blog entry helps someone out there like me.

P.S. AWS EC2 Auto-Scaling is fantastic!

Mike Mead