After migration, lost registration form and data

yaky · September 4, 2018, 2:46pm

Dear Indico Team,

We have upgraded our indico instance to 2.1 of version. As the documentaion shown, our upgrade procedure were the following steps:

upgrade to 2.0
running the migration script
db sync
upgrade to 2.1

The upgrade procedure is succesfully, but there are some problems

there is no registration form for some events (but not all events), and not show the registrants.
there is no evaluation form for some events
all answer-lists of all questions in the feedback forms seem to be changed into random sequences instead of always from best excellent to worst

Here is the migration log:

× [global_pre] No valid IPs found
! [users] Skipping 100 - not activated
! [users] Skipping 101 - not activated
! [users] Skipping 102 - not activated
! [users] Skipping 103 - not activated
! [users] Skipping 104 - not activated
! [users] Skipping 105 - not activated
! [users] Skipping 106 - not activated
! [users] Skipping 109 - not activated
! [users] Skipping 110 - not activated
! [users] Skipping 113 - not activated
! [users] Skipping 114 - not activated
! [users] Skipping 116 - not activated
! [users] Skipping 117 - not activated
! [users] Skipping 118 - not activated
! [users] Skipping 120 - not activated
! [users] Skipping 122 - not activated
! [users] Skipping 123 - not activated
! [users] Skipping 125 - not activated
! [users] Skipping 127 - not activated
! [users] Skipping 128 - not activated
! [users] Skipping 129 - not activated
! [users] Skipping 131 - not activated
! [users] Skipping 133 - not activated
! [users] Skipping 134 - not activated
! [users] Skipping 136 - not activated
! [users] Skipping 137 - not activated
! [users] Skipping 138 - not activated
! [users] Skipping 139 - not activated
! [users] Skipping 140 - not activated
! [users] Skipping 141 - not activated
! [users] Skipping 142 - not activated
! [users] Skipping 143 - not activated
! [users] Skipping 144 - not activated
! [users] Skipping 146 - not activated
! [users] Skipping 148 - not activated
! [users] Skipping 149 - not activated
! [users] Skipping 150 - not activated
! [users] Skipping 151 - not activated
! [users] Skipping 152 - not activated
! [users] Skipping 153 - not activated
! [users] Skipping 154 - not activated
! [users] Skipping 155 - not activated
! [users] Skipping 156 - not activated
! [users] Skipping 157 - not activated
! [users] Skipping 158 - not activated
! [users] Skipping 159 - not activated
! [users] Skipping 160 - not activated
! [users] Skipping 161 - not activated
! [users] Skipping 162 - not activated
! [users] Skipping 163 - not activated
! [users] Skipping 164 - not activated
! [users] Skipping 165 - not activated
! [users] Skipping 166 - not activated
! [users] Skipping 167 - not activated
! [users] Skipping 168 - not activated
! [users] Skipping 169 - not activated
! [users] Skipping 170 - not activated
! [users] Skipping 171 - not activated
! [users] Skipping 172 - not activated
! [users] Skipping 173 - not activated
! [users] Skipping 174 - not activated
! [users] Skipping 175 - not activated
! [users] Skipping 176 - not activated
! [users] Skipping 177 - not activated
! [users] Skipping 178 - not activated
! [users] Skipping 179 - not activated
! [users] Skipping 180 - not activated
! [users] Skipping 181 - not activated
! [users] Skipping 182 - not activated
! [users] Skipping 183 - not activated
! [users] Skipping 184 - not activated
! [users] Skipping 185 - not activated
! [users] Skipping 186 - not activated
! [users] Skipping 187 - not activated
! [users] Skipping 188 - not activated
! [users] Skipping 189 - not activated
! [users] Skipping 190 - not activated
! [users] Skipping 191 - not activated
! [users] Skipping 192 - not activated
! [users] Skipping 193 - not activated
! [users] Skipping 194 - not activated
! [users] Skipping 195 - not activated
! [users] Skipping 196 - not activated
! [users] Skipping 197 - not activated
! [users] Skipping 198 - not activated
! [users] Skipping 200 - not activated
! [users] Skipping 202 - not activated
! [users] Skipping 203 - not activated
! [users] Skipping 204 - not activated
! [users] Skipping 205 - not activated
! [users] Skipping 212 - not activated
! [users] Skipping 48 - not activated
! [users] Skipping 49 - not activated
! [users] Skipping 58 - not activated
! [users] Skipping 62 - not activated
! [users] Skipping 63 - not activated
! [users] Skipping 77 - not activated
! [users] Skipping 79 - not activated
! [users] Skipping 86 - not activated
! [users] Skipping 89 - not activated
! [users] Skipping 90 - not activated
! [users] Skipping 95 - not activated
! [users] Skipping 98 - not activated
! [users] Skipping 99 - not activated
✓ [users] Added new system user: <User(249, None): "Indico System">
! [categories] Raising visibility from 'invisible' to 'category-only' [9]
!    150  shorturl    Shorturl .https://www.hpc2n.umu.se/prace2013/information is invalid: url
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 0)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 1)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 10)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 11)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 12)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 13)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 14)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 15)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 16)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 17)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 18)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 19)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 2)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 20)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 21)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 22)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 23)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 24)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 25)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 26)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 27)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 28)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 29)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 3)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 30)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 31)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 32)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 33)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 34)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 35)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 36)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 37)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 38)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 39)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 4)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 40)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 41)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 42)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 43)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 44)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 45)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 46)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 47)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 48)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 49)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 5)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 50)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 51)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 52)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 53)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 54)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 55)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 56)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 57)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 6)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 7)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 8)
!    197  regform     Skipping invalid data (field id mismatch) for obsolete version of "Thursday" (registrant 9)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 0)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 1)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 10)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 2)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 3)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 4)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 5)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 6)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 7)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 8)
!    199  regform     Skipping invalid data (field id mismatch) for obsolete version of "Friday" (registrant 9)
!    386  regform     Duplicate email konstantinos.krestenitis@durham.ac.uk; using konstantinos.krestenitis+1@durham.ac.uk instead
!    537  regform     Skipping deleted status Did not attend (No-show)
!    537  regform     Skipping deleted status Did not attend (Cancelled)
!    537  regform     Skipping deleted status Did not attend (No-show)
!    537  regform     Skipping deleted status Did not attend (Cancelled)
!    537  regform     Skipping deleted status Did not attend (No-show)
!    537  regform     Skipping deleted status Did not attend (No-show)
!    537  regform     Skipping deleted status Did not attend (No-show)
!    537  regform     Skipping deleted status Did not attend (No-show)
!    537  regform     Skipping deleted status Did not attend (Cancelled)
!    537  regform     Skipping deleted status Did not attend (No-show)
!    539  regform     Skipping deleted status Did not attend (No-show)
!    539  regform     Skipping deleted status Did not attend (No-show)
!    539  regform     Skipping deleted status Did not attend (No-show)
!    539  regform     Skipping deleted status Did not attend (No-show)
!    539  regform     Skipping deleted status Did not attend (No-show)
!    590  regform     Skipping deleted status Reference received (Yes)
!    590  regform     Skipping deleted status Reference received (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status Did not attend (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status Cancelled (Yes)
!    597  regform     Skipping deleted status No show (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    638  regform     Skipping deleted status Absent (Yes)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (Cancelled)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (No-show)
!    650  regform     Skipping deleted status Did not attend (Cancelled)
! [series] Skipping single-event series: [79]

Apache2 indico.conf:

<VirtualHost *:443>
    ServerName my_indico.domain.hu
    DocumentRoot "/var/empty/apache"

    SSLEngine             on
    SSLCertificateFile    cert.crt
    SSLCertificateKeyFile key.key
    SSLCACertificateFile  CA.crt

    SSLProtocol           all -SSLv2 -SSLv3
    SSLCipherSuite        ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    SSLHonorCipherOrder   on

    XSendFile on
    XSendFilePath /opt/indico
    CustomLog /opt/indico/log/apache/access.log combined
    ErrorLog /opt/indico/log/apache/error.log
    LogLevel error
    ServerSignature Off

    AliasMatch "^/static/assets/(core|(?:plugin|theme)-[^/]+)/(.*)$" "/opt/indico/assets/$1/$2"
    AliasMatch "^/(ihelp|css|images|js|static(?!/plugins|/assets|/themes|/custom))(/.*)$" "/opt/indico/web/htdocs/$1$2"
    Alias /robots.txt /opt/indico/web/htdocs/robots.txt

    SetEnv UWSGI_SCHEME https
    ProxyPass / uwsgi://127.0.0.1:8008/

    <Directory /opt/indico>
        AllowOverride None
        Require all granted
    </Directory>
</VirtualHost>

Apache2 indico-sslredir.conf:

<VirtualHost *:80>
    ServerName my_indico.domain.hu
    RewriteEngine On
    RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>

indico.conf:

# General settings
SQLALCHEMY_DATABASE_URI = 'postgresql:///indico'
SECRET_KEY = 'My_secret_key'
BASE_URL = 'https://my_indico.domain.hu'
CELERY_BROKER = 'redis://127.0.0.1:6379/0'
REDIS_CACHE_URL = 'redis://127.0.0.1:6379/1'
CACHE_BACKEND = 'redis'
DEFAULT_TIMEZONE = 'Europe/Budapest'
DEFAULT_LOCALE = 'en_GB'
ENABLE_ROOMBOOKING = False
CACHE_DIR = '/opt/indico/cache'
TEMP_DIR = '/opt/indico/tmp'
LOG_DIR = '/opt/indico/log'
ASSETS_DIR = '/opt/indico/assets'
STORAGE_BACKENDS = {'default': 'fs:/opt/indico/archive', 'legacy': 'fs-readonly:/opt/indico/legacy-archive'}
ATTACHMENT_STORAGE = 'default'
ROUTE_OLD_URLS = True

# Email settings
SMTP_SERVER = ('127.0.0.1', 25)
SMTP_USE_TLS = False
SMTP_LOGIN = ''
SMTP_PASSWORD = ''
SUPPORT_EMAIL = 'my_email'
PUBLIC_SUPPORT_EMAIL = 'my_email'
NO_REPLY_EMAIL = 'my_email'

STATIC_FILE_METHOD = 'xsendfile'

Do you have any idea that how to solve these problems?

We have just returned to version 1.2 but we have opportunity to test any way on test machine.

Thank you.

Regards,
Janos BODOR

pferreir · September 5, 2018, 9:46am

Hello,

That seems strange. Is there anything in common about those events? Are all of them conferences? Do their IDs show at all on the migration log?

yaky · September 6, 2018, 9:14am

Hy,

To tell you truth, I do not know. Our indico instance has more than 500 events, and I could not look through them all. My opinion is all event is conferences.
We suspect it was a category dependent problem, but it is not sure. As you can see the migration log contains some event id, where the regform is problematic, but we do not know that is all.
Do you have any idea?

Thank you.

pferreir · September 10, 2018, 2:40pm

It’s hard to hint at anything without having an idea of how many events are affected and whether there is a pattern. Perhaps if you look up the events that show up on the log and try to see whether they have all failed to migrate?

dattila · September 21, 2018, 10:14am

Dear pferreir,

Unfortunately, we did not find any pattern. I tried to clone and create new test events in several categories, but the migration script didn’t transfer the registration forms.

Example, I cloned the 187th event to 779. In the old indico the registration form is presented. After the migration, the form of the 779 is missed.

$ grep '187  regform' migration.log | head
✓    187  regform     2014-01-05 - C-C++ @ Registration Form
i    187  regform     Section/Personal - Personal Data
i    187  regform     Field/single_choice - Title
i    187  regform     Field/text - First Name
i    187  regform     Field/text - Surname
i    187  regform     Field/text - Position
i    187  regform     Field/text - Institution
i    187  regform     Field/text - Address
i    187  regform     Field/text - City
i    187  regform     Field/text - ZIP Code

$ grep '779  regform' migration.log
✓    779  regform     2018-07-16 - C-C++ @ Registration Form
i    779  regform     Section/Personal - Personal Data
i    779  regform     Field/single_choice - Title
i    779  regform     Field/text - First Name
i    779  regform     Field/text - Surname
i    779  regform     Field/text - Position
i    779  regform     Field/text - Institution
i    779  regform     Field/text - Address
i    779  regform     Field/text - City
i    779  regform     Field/text - ZIP Code
i    779  regform     Field/country - Country
i    779  regform     Field/text - Phone
i    779  regform     Field/text - Fax
i    779  regform     Field/email - Email
i    779  regform     Field/text - Personal homepage
i    779  regform     Field/single_choice - Use e-mail to receive PRACE announcements
i    779  regform     Field/country - Nationality
i    779  regform     Section - Information about your laboratory
i    779  regform     Field/text - Name of laboratory
i    779  regform     Field/text - Name of supervisor
i    779  regform     Field/text - E-mail of supervisor
i    779  regform     Section/Reason - Reason for participation

Do you have any idea?

Thank you,
Attila

dattila · October 3, 2018, 7:41am

Dear pferreir,

we’ve found that the registration form is missing when migrating events that had been created by cloning another event.

What do you think, how can we resolve this?

If you want, we can serve more debug message, but please help for us.

Best regards,
Attila

pferreir · October 9, 2018, 8:08am

Oh, that is a little bit more interesting. OK, we’ll have a look ASAP.

ThiefMaster · October 9, 2018, 4:24pm

I just published indico-migrate v1.0.6 on PyPI. Can you please update to that version and try the migration again? I fixed the problem with the registration import.

Did those events have the survey configured? We are skipping surveys if they have no questions configured.

I was not able to reproduce this. If this still happens (in a consistent way), please provide me with as screenshot of the old (1.2) and new (2.x) version of the question where the problem can be seen.

dattila · January 10, 2019, 10:23am

sorry, for my late.