Errors using the S3 storage plugin with and without rclone

Administrators
1

Hey lovely folks!

I have been setting up the S3 Storage plugin, using it with Vultr’s S3-compatible object storage. We run in Kubernetes.

I have:

  1. Set up the credentials to connect in the ~/.aws/ folder
  2. Enabled the storage_s3 plugin
  3. Configured an extra storage backed in the indico.conf file
  4. Tested that the credentials are valid and that Indico can connect to the bucket (it can)

So far all of the above is working.

I then attempt to run the migrate copy command in a test instance with some basic events and demos, roughly like this:

indico s3 migrate copy -B indico -S indico -e '$VULTR_ENDPOINT' -p s3 test.json

This fails and I get the following output:

Error processing <RegistrationData(6, 55): None>: 'str' object has no attribute 'decode'
Processing RegistrationData (3 rows)  [############------------------------]  1/3   33%
Error processing <RegistrationData(7, 55): None>: 'str' object has no attribute 'decode'
Processing RegistrationData (3 rows)  [########################------------]  2/3   66%
Error processing <RegistrationData(22, 170): None>: 'str' object has no attribute 'decode'
Processing RegistrationData (3 rows)  [####################################]  3/3  100%
Processing AttachmentFile (4 rows)    [------------------------------------]  0/4    0%
Error processing <AttachmentFile(1, 1, [small]REDACTED.jpg, image/jpeg)>: Could not open "event/1/event/1-1-REDACTED.jpg": [Errno 2] No such file or directory: '/opt/indico/archive/event/1/event/1-1-REDACTED.jpg'
Processing AttachmentFile (4 rows)    [#########---------------------------]  1/4   25%
Error processing <AttachmentFile(2, 2, [small]REDACTED.jpg, image/jpeg)>: Could not open "event/1/event/2-2-REDACTED.jpg": [Errno 2] No such file or directory: '/opt/indico/archive/event/1/event/2-2-REDACTED.jpg'
Processing AttachmentFile (4 rows)    [##################------------------]  2/4   50%
Error processing <AttachmentFile(4, 4, REDACTED, image/jpeg)>: 'str' object has no attribute 'decode'
Processing AttachmentFile (4 rows)    [###########################---------]  3/4   75%
Error processing <AttachmentFile(5, 5, REDACTED, image/avif)>: 'str' object has no attribute 'decode'
Processing AttachmentFile (4 rows)    [####################################]  4/4  100%
All done!

Note that the redactions in the output are manual by me. Some of these attachments actually don’t exist (possibly some past failed cleanups?). The registration materials exist. After this process is completed, there are no files on the bucket either from registrations or attachments.

I am not an expert, but to me it looks like this is where it pops. I am not sure indeed why that .decode()is there, but hey, I’m an SRE, not a snake-tamer.

I noticed however that path is only taken if you don’t use rclone. So let’s use rclone, as it seems recommended anyway, and see where that gets us? I installed it, added it to the path, and set up a remote. I then tested that listing the remotes works as expected. It does.

Running the above again, with this minor change, we have:

indico s3 migrate copy -B indico -S indico -e '$VULTR_ENDPOINT' -p s3 -r s3 test.json

This fails… differently.

Processing RegistrationData (3 rows)  [------------------------------------]  0/3    0%
Checking bucket indico
Error processing <RegistrationData(6, 55): None>: a bytes-like object is required, not 'str'
Processing RegistrationData (3 rows)  [############------------------------]  1/3   33%
Error processing <RegistrationData(7, 55): None>: a bytes-like object is required, not 'str'
Processing RegistrationData (3 rows)  [########################------------]  2/3   66%
Error processing <RegistrationData(22, 170): None>: a bytes-like object is required, not 'str'
File not found on disk: /opt/indico/archive/event/1/event/1-1-REDACTED.jpg

It then hangs indefinitely at the first file not found. In this case, I check the bucket and see that despite the errors, the 3 registration files are uploaded successfully.

Can I get some insight here? I’m quite stuck, I can create the missing files to get over the rclone issue, but it seems like a subpar approach, especially for a larger instance, and I’d prefer to resolve this cleanly.

Either way, thanks for your work ^-^

2

Files on disk are not supposed to be missing when they are still in the database… So that’s not a case the code is expected to handle (but of course it could fail more gracefully and/or have an option to skip those).

The broken decode seems to be due to an incorrect import - most likely this should be imported from base64 instead of sqlalchemy.util.compat.

Could you try replacing from sqlalchemy.util.compat import b64encode with from base64 import b64encode in indico_storage_s3/migrate.py and see if it works?

3

Thanks for the prompt help! After applying this fix:

  1. I am still getting the error, for all uploaded objects: a bytes-like object is required, not ‘str’
  2. The other 2 previously failed files are correctly uploaded
4

From what I see, the output file is also empty, which makes it impossible to proceed to applying the migration script. I’ll look into patching the file to catch the exceptions and see if that fixes it

5

So, I’ve logged the full stack trace, and I’m seeing that the error is not coming from the base64 side anymore, this error is popped when we try to write to the changelog file:

Error processing <RegistrationData(6, 55): None>: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico_storage_s3/migrate.py", line 144, in run
    self.process_obj(obj)
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico_storage_s3/migrate.py", line 171, in process_obj
    self.emit_update(obj, backend, new_storage_path, new_filename)
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico_storage_s3/migrate.py", line 320, in emit_update
    json.dump(data, self.output_file)
  File "/usr/local/lib/python3.12/json/__init__.py", line 180, in dump
    fp.write(chunk)
TypeError: a bytes-like object is required, not 'str'

Line numbers are slightly changed due to my modifications, but the error happens here. I’m working on figuring out the actual cause

6

Figured it out, turns out we open the file as binary here. Removing the b from wb makes it work again. I’ll put up a PR once I’m done testing the migration

7

Catching the exceptions is most likely a bad idea.

Try replacing @click.argument('output', metavar='PATH', type=click.File('wb')) with @click.argument('output', metavar='PATH', type=click.File('w')) - this should avoid the error. Writing json to a file opened in bytes mode doesn’t really make sense.

I’ll also see if I can reproduce your issues locally since that would make fixing them much easier…

8

Thanks! I have now successfully completed the migration (both copying and applying). I have also tested I can successfully upload files via the UI and all is working well.

The two changes I needed to get the S3 upload working:

  1. The base64 import fix
  2. The file opening fix (that you also mentioned above)

In the current state, the rclone upload is not working for me, it hangs indefinitely. If you’d like, I can test some solutions with that as well!