Tune RabbitMQ System Limits

[nuthanmunaiah]

Description

When onboarding mysql/mysql-server (in service of https://github.com/samaritan/infrastructure/issues/8), RabbitMQ failed midway through the loading of the project. The cause of failure was determined to be the exhaustion of file descriptors and socket descriptors. The default number of file descriptors and socket descriptors is 1,024 and 829, respectively. The default number is clearly too small in relation to the number of open file handles recommended by RabbitMQ Production Checklist.

[nuthanmunaiah]

Apparently, the tuning of system limits (via ulimit) for containers is not supported by Kubernetes (See kubernetes/kubernetes#3595) or AWS EKS Fargate. According to aws/containers-roadmap#1013 (comment), a resolution for this limitation is unlikely to be available in 2020.

Our alternatives at this point are (1) reduce the number of running containers so as to not exceed the limits on the RabbitMQ container, (2) use RabbitMQ deployed on our RIT workstation, or (3) use Amazon MQ. At this point, I have chosen to use the RabbitMQ deployed on our RIT workstation.

@beefstew What do you think?

[beefstew]

It looks like we don't have a choice to work around the AWS limitation.
One additional option: run Rabbit on EC2
I'm mostly concerned about throughput.

[nuthanmunaiah]

I caved on cost and setup RabbitMQ on Amazon MQ. I started with mq.m5.large (at $0.288 per hour) but soon exhausted the 8 GiB of memory when loading mysql/mysql-server. I then switched to mq.m5.4xlarge (at $1.152 per hour) to have enough room to scale the services out without having RabbitMQ be the limiting factor. Fortunately, with durable queues, switching instance types at runtime does not result in existing messages being lost.

We have a working solution at this point so this issue is going to be put back onto the backburner until a time we determine that it is cheaper to run a cluster of RabbitMQ instances inside of EKS to achieve the throughput requirements than to use RabbitMQ on Amazon MQ.