Translation workflow?

Hello all,

It seems natural to check translations not only by reviewing them on Transifex but also by deploying them on an Indico instance. However, I am worried about how to manage the translations in sync between my instance and Transifex.

I know Transifex has importing feature from .po file, but .po in the code base is one file per programming language while each .po file is split into several files on Transifex so I am not sure whether importing from local .po file is possible without pitfalls.
My current plan is downloading .po files using Transifex API with cURL and merging locally, executed periodically by cron or so on. Thus, editing translation will be always done on Transifex and my instance will just serve the latest data.

However, I feel that it may not be the optimal workflow. How do you manage translation?

2 Likes

Yes, this would definitely be helpful to have a procedure (tool?) to test translations on a dev instance!

1 Like

Another possible approach is that https://sandbox.getindico.io/ imports all the languages with any translated entries periodically.

Pros: Everyone can test them without having an staging instance.
Cons: Some UIs, administration interface for instance, can not be tested.

Indeed we need to prepare development instances for some components anyway, but it would be great if a typical user could easily try work-in-progress translated UIs and report problems to translation groups.

1 Like

Now, my instance downloads translation data from Transifex weekly. I’d share my recipe here. I hope it will help you. Please do not hesitate to ask if anything is not clear.

My instance runs on Ubuntu 16.04 LTS on Oracle VirtualBox VM, for reference.


Preparation

First of all, please define your favorite language as an environment variable. In this instructions, I choose Japanese (ja_JP) for instance.

INDICO_TARGET_LANGUAGE=ja_JP
export INDICO_TARGET_LANGUAGE

Timer settings

cat > /etc/systemd/system/update-indico-translation.service <<'EOF'
[Unit]
Description=Indico translation update script
RefuseManualStart=no
RefuseManualStop=yes

[Service]
Type=oneshot
ExecStart=/opt/indico/etc/update-indico-translation 2.7 ${INDICO_TARGET_LANGUAGE}
User=indico
Group=nginx
EOF

cat > /etc/systemd/system/update-indico-translation.timer <<'EOF'
[Unit]
Description=Update Indico translation data weekly

[Timer]
OnCalendar=weekly
Persistent=true
Unit=update-indico-translation.service

[Install]
WantedBy=timers.target
EOF

Then, let’s enable the timer.

systemctl start update-indico-translation.timer
systemcrl enable update-indico-translation.timer

Please check if your timer is activated.

systemctl list-timers

Then, please continue with indico user previlege.

su - indico

Obtain your Transifex API token

Please log in to Transifex, open user settings, and get your API token.

Then, please save the token as /opt/indico/.tx-api-token.

# You definitely want to change the access permissions of it, don't you?
chmod 600 /opt/indico/.tx-api-token

The script

Do not use cat here, since variables are expanded on the fly if you do so. Please use your favorite editor.

#!/bin/sh

python_version="$1"
lang="$2"
api_token="$(cat /opt/indico/.tx-api-token)"

work_dir="${TMPDIR}/$(mktemp -d)"
cd "${work_dir}"

echo "Updating Indico (Python ${python_version}) translation resources for ${lang}..."

fetch_language_resource() {
  package="$1"
  component="$2"

  package_dir="/opt/indico/.venv/lib/python${python_version}/site-packages/${package}/"
  if [ -d "${package_dir}" ]; then
    echo "The ${package} package is found. Downloading translation resources for ${lang}..."

    locale_dir="${package_dir}/translations/${lang}/LC_MESSAGES/"
    mkdir -p "${locale_dir}"

    echo "Downloading messages.po..."
    py_resource_url="https://www.transifex.com/api/2/project/indico/resource/${component}-messages/translation/${lang}/?file"
    py_status_code=$(curl -s -L --user "api:${api_token}" -X GET "${py_resource_url}" -o "messages.po" -w "%{http_code}")

    if [ $? = 0 ] && [ "${py_status_code}" = "200" ]; then
      msgfmt messages.po
      mv messages.po messages.mo "${locale_dir}"
    else
      echo "Failed to download ${lang} messages.po for ${package}."
      if [ -f messages.po ]; then
        rm -f messages.po
      fi
    fi

    echo "Downloading messages-js.po..."
    js_resource_url="https://www.transifex.com/api/2/project/indico/resource/${component}-messages-js/translation/${lang}/?file"
    js_status_code=$(curl -s -L --user "api:${api_token}" -X GET "${js_resource_url}" -o "messages-js.po" -w "%{http_code}")

    if [ $? = 0 ] && [ "${js_status_code}" = "200" ]; then
      mv messages-js.po "${locale_dir}"
    else
      echo "Failed to download ${lang} messages-js.po for ${package}."
      if [ -f messages-js.po ]; then
        rm -f messages-js.po
      fi
    fi
  fi
}

fetch_language_resource indico core

fetch_language_resource indico_chat chat
fetch_language_resource indico_importer importer
fetch_language_resource indico_importer_invenio importer-invenio
fetch_language_resource indico_livesync livesync
fetch_language_resource indico_livesync_invenio livesync-invenio
fetch_language_resource indico_payment_manual payment-manual
fetch_language_resource indico_payment_paypal payment-paypal
fetch_language_resource indico_piwik piwik
fetch_language_resource indico_search search
fetch_language_resource indico_vc_vidyo vc-vidyo

echo "Reloading Indico..."
touch /opt/indico/web/indico.wsgi

cd
rm -rf "${work_dir}"

echo "Finished updating Indico translation resources."

I assume that you saved it at /opt/indico/etc/update-indico-translation in the description below. Please do not forget to add an executable bit to the script.

chmod +x /opt/indico/etc/update-indico-translation

Done!

Let’s update translations on Transifex and reload the page next week!

I found that user service and timer are not needed here since we can specify user and group with system service and timer. I will change my setup then update instructions here. I have updated the instructions above.

Hi there @maogino,
Sorry that I’m late to this conversation, but I was on holidays at the time of the discussion.
It would be great to have the sandbox (or even better, a separate server) doing some sort of CI of the translations in different languages.
Coincidentally, we have been working quite hard in creating docker images for Indico that can be easily run on any container-based platform. So, my idea is: why not create a Docker-based setup that uses our image and runs your script (on a cronjob or using webhooks…)? We could then even deploy it here at CERN if needed.

Your Dockerfile would just need to do FROM getindico/indico:latest, etc…

Thanks for the great work you’ve been doing!

1 Like

Sounds fantastic! I will take a look into the repo.

1 Like

Cool! I’m also CCing @kolodzie, who is our I18N manager.

1 Like

Hey, just tried to set this up; however, the download fails with 401 Unauthorized someone has to accept me into the indico transifex project for de_DE … @maogino can you do that please?

Hey, sorry for the delay but I never receive notifications from Transifex about new join requests. Your request has been accepted.

Regards,
Michal

Hey @maogino,

there was “core-messages-react” added to indico/transifex.
Could you update your script to reflect that addition?

hi gurus,

can u tell me how to add my language into the selection? Translation is almost complete on Transifex.com image

thanks

Check these two links in the docs:

4. Install the translations

Navigate to ~/dev/indico/src (assuming you used the standard locations from the dev setup guide).

Run tx pull -f -l <language_code>.

these commands as indico user right?

yes, run those as the indico user.

1 Like

hi,

I cant find remaining strings

They are probably from obsolete plugins for which we disabled translations so people do not waste their time on translating them.

1 Like

i dont have this folder ~/dev/indico/src

Did you setup an indico dev environment locally? The documentation for this is here, only important thing to note is to use the 2.3-maintenance branch or v2.3.4 release tag instead of master.

i followed production manual at Apache — Indico 2.3.4 documentation and it is not local it is online on production server.