I am running an Indico server with extremely-low traffic (at least for 6 months in a year) on AWS EC2. Therefore, to keep the cost down, I am trying to reduce the memory consumption of Indico.
The first strategy I used is to move the postgres database to an AWS RDS instance, which is working well enough. The second strategy I could try is to reduce the number of processes in uwsgi.ini file, which seems to work too.
However, I identified that the processes that consume the most memory on my server are the Indico celery processes.
So, are there some strategies/configurations I could adopted to lower the Indico celery’s memory usage? (I am okay with lower-performance/higher-latency end-result)
Also, if anyone has further tips on the matter, please share!
I haven’t tested any of these options in production. But I would suggest trying:
Using AWS SQS as the celery broker (probably not impactful since you mentioned the celery processes and not the broker itself)
Configure your workers
The --autoscale option allows Celery workers to decrease and increase the amount of forked processes. This decreases CPU and memory usage in periods of inactivity. For example, you can set --autoscale=10,3 , meaning Celery will always keep at least three processes around, but will scale up to ten if needed.
The --max-memory-per-child option is very useful when running on less powerful machines. For example, Heroku standard dynos have a limited amount of memory and Heroku will start killing your dyno if it consumes too much memory for extended periods of time. You can use this option to allow Celery to restart forked processes when they exceed a certain amount of memory.