Helm chart for deploying Indico on Kubernetes

I see that there is a repo for the containerization of Indico. Before I run off and start translating the Docker Compose example deployment into a Kubernetes Helm chart, is there already a Helm chart available (even an unofficial one)?

None I’m aware of. If you write one I think it’s be pretty nice to publish it somewhere on GitHub. :slight_smile:

1 Like

Most of the pieces are in place, but I’m confused about the indico-static service. I do not see how this connects to anything else. There is a static-files volume mounted to the indico-web service but this is not shared with the indico-static service.

I think I’ve figured out that I need some combination of the “static” Dockerfile and the “nginx” Dockerfile, with modified base paths to the static assets (css, js, etc). I did build this custom image, but the file path references in the getindico/indico:3.1 image, for example main.6fc6b3a9.css, do not exactly match the file paths in the nginx image with the static files. How do I ensure that the static files I download in the nginx image build are the same version as the getindico/indico:3.1 image?

After more debugging I noticed that the https://github.com/indico/indico-containers repo has not been substantially updated in almost three years :slightly_frowning_face: However, I assume that the Indico sandbox running at https://sandbox.getindico.io/ is containerized. Should I continue adapting and updating the Dockerfiles from that repo on my own, or are there active but unpublished Dockerfiles someone could share?

No, we don’t use containerized versions ourselves. Hence the somewhat poor state of maintenance in that repo…

We publish images built from the indico worker Dockerfile in that repo on the Docker hub, but the rest in that repo is not very up to date… IIRC the idea was for the static container to extract the dist folder from the indico Python wheel and serve it, but I don’t think anyone tested this recently.

So if you just want to use a simple containerized Indico setup it may be best to just use the Dockerfile linked above.

Actually, I need to check with my colleague who built the 3.1.1 image what Dockerfile he used, because the Dockerfile up there references Python 2.7 which is clearly no longer used… probably it’s just a matter of using the Python 3.9 image instead and he forgot to push the change.

We are trying to get it work. We use a traefik proxy on the top so we had modified the some part of the project but now it seems to work. I need only to understand how to customize because my old customizations doesn’t work with the v3.1

If you share a link to a repo containing your old customizations I can have a look and possibly tell you what you need to change.

Thank you for your responses. Apologies for missing the notification email about your replies or I would have responded sooner. Now that I plan to resume working to deploy Indico on Kubernetes, I will try to synthesize what I can to create a Helm chart and will share the results for review.

You can checkout this pr for a more recent docker compose setup :wink:

Guys from Canonical also wrote a Juju charm for deploying Indico on K8s: repo

1 Like

I cannot post a respond because it keeps saying I’m spamming the forum. Maybe I’ll make several small posts.

JuJu looks interesting, but since I am not familiar with that framework I’ll try to avoid it for this one deployment. Perhaps I can still borrow some configuration from

@troun Thanks for the tip! I was able to build the worker image from tag 3.2.3 as shown in the PR and use the Docker Compose config to run Indico locally. I also built the NGINX image from the same PR source (https://github.com/indico/indico-containers/tree/c4d0447bd42fae9a1c0f4e16d705e6ead198688b) you referenced.

I thought it would be straightforward to adapt our Helm chart (https://gitlab.com/nsf-muses/deployment/kubernetes/-/tree/ca1afb90df768c1bb00d4d0a6845acfd63e12de1/charts/indico) to match the Docker Compose config and use the newly built image pushed to our own registry (since I do not see the 3.2.3 tag on Docker Hub (https://hub.docker.com/r/getindico/indico/tags) yet), but I am stuck on the following error:

The three containers in the Indico pod all initialize without error (app, celery, nginx), but when I attempt to visit the site, the redirect to /bootstrap results in a 502 Bad Gateway (served from the NGINX container itself). The nginx and app container logs are shown below. If you have any idea what could be wrong, I’d love to hear your ideas.

NGINX container log

2023-03-31T14:27:10-05:00 10.42.80.1 - - [31/Mar/2023:19:27:10 +0000] "GET / HTTP/1.1" 302 207 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
2023-03-31T14:27:10-05:00 2023/03/31 19:27:10 [error] 13#13: *33 upstream prematurely closed connection while reading response header from upstream, client: 10.42.80.1, server: events.musesframework.io, request: "GET /bootstrap HTTP/1.1", upstream: "http://10.43.47.142:59999/bootstrap", host: "events.musesframework.io"
2023-03-31T14:27:10-05:00 2023/03/31 19:27:10 [error] 13#13: *33 upstream prematurely closed connection while reading response header from upstream, client: 10.42.80.1, server: events.musesframework.io, request: "GET /bootstrap HTTP/1.1", upstream: "http://10.43.47.142:59999/bootstrap", host: "events.musesframework.io"
2023-03-31T14:27:10-05:00 10.42.80.1 - - [31/Mar/2023:19:27:10 +0000] "GET /bootstrap HTTP/1.1" 502 559 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"

App container log

2023-03-31T19:27:10.930563447Z --- Logging error ---
2023-03-31T19:27:10.930661896Z Traceback (most recent call last):
2023-03-31T19:27:10.930708175Z   File "/opt/indico/.venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
2023-03-31T19:27:10.930720110Z     rv = self.dispatch_request()
2023-03-31T19:27:10.930723457Z   File "/opt/indico/.venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
2023-03-31T19:27:10.930726327Z     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2023-03-31T19:27:10.930729004Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/web/flask/util.py", line 79, in wrapper
2023-03-31T19:27:10.930731636Z     return obj().process()
2023-03-31T19:27:10.930733864Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/web/rh.py", line 285, in process
2023-03-31T19:27:10.930736147Z     res = self._do_process()
2023-03-31T19:27:10.930738252Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/web/rh.py", line 256, in _do_process
2023-03-31T19:27:10.930740497Z     rv = self._process()
2023-03-31T19:27:10.930742765Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/modules/bootstrap/controllers.py", line 40, in _process
2023-03-31T19:27:10.930745065Z     return render_template('bootstrap/bootstrap.html',
2023-03-31T19:27:10.930747214Z   File "/opt/indico/.venv/lib/python3.10/site-packages/flask/templating.py", line 147, in render_template
2023-03-31T19:27:10.930749471Z     return _render(app, template, context)
2023-03-31T19:27:10.930751723Z   File "/opt/indico/.venv/lib/python3.10/site-packages/flask/templating.py", line 130, in _render
2023-03-31T19:27:10.930753937Z     rv = template.render(context)
2023-03-31T19:27:10.930756030Z   File "/opt/indico/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
2023-03-31T19:27:10.930758265Z     self.environment.handle_exception()
2023-03-31T19:27:10.930760538Z   File "/opt/indico/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
2023-03-31T19:27:10.930763050Z     raise rewrite_traceback_stack(source=source)
2023-03-31T19:27:10.930765367Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/modules/bootstrap/templates/bootstrap.html", line 3, in top-level template code
2023-03-31T19:27:10.930767723Z     {% from 'forms/_form.html' import form_header, form_row, form_rows, form_fieldset, form_footer %}
2023-03-31T19:27:10.930769926Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/web/templates/base.html", line 14, in top-level template code
2023-03-31T19:27:10.930772253Z     {{ webpack['exports.js'] }}
2023-03-31T19:27:10.930774631Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/web/flask/templating.py", line 315, in getitem
2023-03-31T19:27:10.930776878Z     rv = super().getitem(obj, argument)
2023-03-31T19:27:10.930786547Z   File "/opt/indico/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 466, in getitem
2023-03-31T19:27:10.930789342Z     return obj[argument]
2023-03-31T19:27:10.930791555Z   File "/opt/indico/.venv/lib/python3.10/site-packages/werkzeug/local.py", line 316, in __get__
2023-03-31T19:27:10.930793800Z     obj = instance._get_current_object()
2023-03-31T19:27:10.930795941Z   File "/opt/indico/.venv/lib/python3.10/site-packages/werkzeug/local.py", line 520, in _get_current_object
2023-03-31T19:27:10.930798183Z     return get_name(local())  # type: ignore
2023-03-31T19:27:10.930800365Z   File "/opt/indico/.venv/lib/python3.10/site-packages/flask_webpackext/proxies.py", line 22, in <lambda>
2023-03-31T19:27:10.930802943Z     lambda: current_app.extensions['flask-webpackext'].manifest)
2023-03-31T19:27:10.930805146Z   File "/opt/indico/.venv/lib/python3.10/site-packages/flask_webpackext/ext.py", line 87, in manifest
2023-03-31T19:27:10.930807420Z     return self.manifest_loader().load(
2023-03-31T19:27:10.930809601Z   File "/opt/indico/.venv/lib/python3.10/site-packages/indico/core/webpack.py", line 25, in load
2023-03-31T19:27:10.930811840Z     key = (filepath, os.path.getmtime(filepath))
2023-03-31T19:27:10.930813954Z   File "/usr/local/lib/python3.10/genericpath.py", line 55, in getmtime
2023-03-31T19:27:10.930816164Z     return os.stat(filename).st_mtime
2023-03-31T19:27:10.930818375Z FileNotFoundError: [Errno 2] No such file or directory: '/opt/indico/.venv/lib/python3.10/site-packages/indico/web/static/dist/manifest.json'
2023-03-31T19:27:10.930820702Z 
2023-03-31T19:27:10.930822794Z During handling of the above exception, another exception occurred:
2023-03-31T19:27:10.930824964Z 
2023-03-31T19:27:10.930827081Z Traceback (most recent call last):
2023-03-31T19:27:10.930829178Z   File "/usr/local/lib/python3.10/logging/handlers.py", line 1061, in emit
2023-03-31T19:27:10.930831383Z     smtp = smtplib.SMTP(self.mailhost, port, timeout=self.timeout)
2023-03-31T19:27:10.930833553Z   File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__
2023-03-31T19:27:10.930835859Z     (code, msg) = self.connect(host, port)
2023-03-31T19:27:10.930837968Z   File "/usr/local/lib/python3.10/smtplib.py", line 341, in connect
2023-03-31T19:27:10.930840258Z     self.sock = self._get_socket(host, port, self.timeout)
2023-03-31T19:27:10.930842359Z   File "/usr/local/lib/python3.10/smtplib.py", line 312, in _get_socket
2023-03-31T19:27:10.930844562Z     return socket.create_connection((host, port), timeout,
2023-03-31T19:27:10.930846671Z   File "/usr/local/lib/python3.10/socket.py", line 845, in create_connection
2023-03-31T19:27:10.930848891Z     raise err
2023-03-31T19:27:10.930850939Z   File "/usr/local/lib/python3.10/socket.py", line 833, in create_connection
2023-03-31T19:27:10.930855144Z     sock.connect(sa)
2023-03-31T19:27:10.930857353Z OSError: [Errno 101] Network is unreachable
...

I released your quarantined posts and raised your trust level so you hopefully no longer have problems posting. Having many small posts with links in them triggered the automated spam filter.

Finally after a hundred iterations I got our Indico instance online! Thanks to everyone for their help (including this helpful post about Keycloak configuration). For reference, our current Helm chart being deployed is here, which uses images from the slightly modified container image build script here.

I encountered the same logout redirect bug discussed in this post, and I hacked around it in a horrible way in the init script I execute at the beginning of the app and celery container launcher scripts:

sed -i "s/{'post_logout_redirect_uri': return_url}/{}/g" \
    /opt/indico/.venv/lib/python3.10/site-packages/flask_multipass/providers/authlib.py

The original reason I created that init script was because I wanted to build the Python-language config file from environment variables, because this is the most natural way to get secret values into an app config in Kubernetes, as well making it easier to override individual config parameters via Helm and ArgoCD hierarchically.

envsubst < "${INDICO_CONFIG}" > /tmp/indico.conf
cp /tmp/indico.conf "${INDICO_CONFIG}"

If someone is willing to review our chart and overall deployment strategy and provide feedback, it would be welcomed.

1 Like

There was a bug that @ThiefMaster just helped me resolve, and so for reference, the latest working Helm chart is found at this commit.

At some point I plan to integrate Indico into my DecentCI pet project as a supported Helm chart, as it is a popular and familiar tool for research collaborations.