Search plugin development

Development Plugin Development
81

New question: recommended way to add an extra field to the URLs, needed for pagination.

In order to implement pagination, the code (class JSONSearchEngine @ indico_search_json/engine.py) needs to know which one is the current page, in order to pass the right value to the next call to the CERN Search API cern search api

I assume that can be done adding a new field to the URLs: &search-current_page=number

I have been investigating how, in that case, to change the code to recognize that new field and use it. I see two options here, at least:

  • straightforward one would be to simply import request:

    class JSONSearchEngine(SearchEngine):
   def process(self):
      from flask import request
      current_page = request.args['search-current_page']

    that is simple and easy. But breaks the consistency with the rest of the code. The rest of fields have been recorded in self.values, so it would be weird to have only some of them in that attribute.

  • second option is adding a new field in indico_search/forms.py

    class SearchForm(IndicoForm):
   phrase = StringField(_('Phrase'))
   field = SelectField(_('Search in'), choices=FIELD_CHOICES, default='')
   start_date = IndicoDateField('Start Date', [Optional()])
   end_date = IndicoDateField('End Date', [Optional()])
   
   # new field
   current_page = StringField()

    But this requires touching code outside the JSON plugin, and I am not sure we are allowed to do that. Could that have unexpected collateral effects?

If it is safe, I would go for option 2. Comments?

82

I’d go for the easiest option, i.e. option 1. We’ll eventually add a new react-based frontend anyway, so chances are good the whole SearchForm will go away at that point.

But if you prefer adding a field to the form, please use an IntegerField and not a StringField :wink:

83

I guess you meant option 1, right?

84

yes, that’s what i meant :wink:

85

Hello,

I am trying to test the livesync_json plugin (indico-plugins/livesync_json at Elasticsearch · penelopec/indico-plugins · GitHub). I am using the latest indico from master (I updated it yesterday).
I run into problems when I try to create a revision for the plugin executing (Adding models to your plugin — Indico 3.3-dev documentation):

$ indico db --plugin livesync_json migrate -m ‘Create table - searchapp_id_map’

Traceback (most recent call last):
File “/opt/indico/.venv/bin/indico”, line 10, in
sys.exit(cli())
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/cli/util.py”, line 110, in invoke
return self._impl.invoke(ctx)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 1134, in invoke
Command.invoke(self, ctx)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 425, in decorator
with __ctx.ensure_object(ScriptInfo).load_app().app_context():
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 381, in load_app
app = call_factory(self, self.create_app)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 117, in call_factory
return app_factory(script_info)
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/cli/util.py”, line 28, in _create_app
return make_app(set_path=True)
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/web/flask/app.py”, line 345, in make_app
raise Exception(‘Could not load some plugins: {}’.format(', '.join(plugin_engine.get_failed_plugins(app))))
Exception: Could not load some plugins: livesync_json

I would appreciate some help…

Thank you
Penelope

86

check indico.log for the real reason why the plugin couldn’t be loaded

87

The log file has the following:

 2019-10-09 16:35:48,058  ERROR    0000000000000000  indico.plugins            Could not load plugin livesync_json
 Traceback (most recent call last):
   File "/opt/indico/.venv/lib/python2.7/site-packages/flask_pluginengine/engine.py", line 76, in _import_plugins
     plugin_class = entry_point.load()
   File "/opt/indico/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2434, in load
     return self.resolve()
   File "/opt/indico/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2440, in resolve
     module = __import__(self.module_name, fromlist=['__name__'], level=0)
   File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/plugin.py", line 11, in <module>
     from indico_livesync_json.backend import livesyncjson_backend
   File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/backend.py", line 26, in <module>
     from indico_livesync_json.plugin import LiveSyncJsonPlugin
 ImportError: cannot import name LiveSyncJsonPlugin

I did check that the names are correct, rebuild the plugin but this issue was not resolved.
Thank you

88

You have a circular dependency between plugin.py and backend.py. Remove the top-level from indico_livesync_json.plugin import LiveSyncJsonPlugin from backend.py and move it inside the function where you use it.

89

Perfect it worked. Now I heed to fix the rest of the errors that created my moving things around…

90

Now I get the following with no errors reports in indico.log. The line that it complains about is the following, which is similar to other definitions for plugin settings form:

search_app_url = URLField(_('Search app URL'), [DataRequired(), URL(require_tld=False)],
                      description=_("URL <url:port> of search app import endpoint"))

$ indico db --plugin livesync_json migrate -m ‘Create table - searchapp_id_map’
Traceback (most recent call last):
File “/opt/indico/.venv/bin/indico”, line 10, in
sys.exit(cli())
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/cli/util.py”, line 110, in invoke
return self._impl.invoke(ctx)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 1134, in invoke
Command.invoke(self, ctx)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 425, in decorator
with __ctx.ensure_object(ScriptInfo).load_app().app_context():
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 381, in load_app
app = call_factory(self, self.create_app)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py”, line 117, in call_factory
return app_factory(script_info)
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/cli/util.py”, line 28, in _create_app
return make_app(set_path=True)
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/web/flask/app.py”, line 344, in make_app
if not plugin_engine.load_plugins(app):
File “/opt/indico/.venv/lib/python2.7/site-packages/flask_pluginengine/engine.py”, line 46, in load_plugins
plugins = self._import_plugins(state.app)
File “/opt/indico/.venv/lib/python2.7/site-packages/flask_pluginengine/engine.py”, line 76, in import_plugins
plugin_class = entry_point.load()
File “/opt/indico/.venv/lib/python2.7/site-packages/pkg_resources/init.py”, line 2434, in load
return self.resolve()
File “/opt/indico/.venv/lib/python2.7/site-packages/pkg_resources/init.py”, line 2440, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/plugin.py”, line 20, in
class livesyncjson_settingsform(IndicoForm):
File “/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/plugin.py”, line 21, in livesyncjson_settingsform
searchapp_url = URLField((‘Search app URL’), [DataRequired()],
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/util/i18n.py”, line 185, in ugettext
self._check_stack()
File “/opt/indico/.venv/lib/python2.7/site-packages/indico/util/i18n.py”, line 180, in check_stack
raise RuntimeError(msg)
RuntimeError: Using the gettext function (_) patched into the builtins is disallowed.
Please import it from indico.util.i18n instead.
The offending code was found in this location:
File “/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/plugin.py”, line 21, in livesyncjson_settingsform
searchapp_url = URLField((‘Search app URL’), [DataRequired()],

91

you are missing an import for the _ function

92

I am reviving this topic…

When I try to build the cern-search-rest-api docker containers as root, nginx does not start as it cannot read the /etc/nginx/nginx.conf (permission denied).
These are the commands I used to create the docker containers for the cern-search-rest-api

git clone https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api.git -b dev
cd cern-search-rest-api
make env

The following is what I see in the make logs for this error:

nginx_1            | 2020/06/23 16:52:08 [emerg] 1#1: open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
nginx_1            | nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
cern-search-rest-api_nginx_1 exited with code 1
93

This is probably once again something really specific to your setup. docker-compose mounts the file from nging/nginx.conf to /etc/nginx/nginx.conf in the container. It is hard to guess right now what the problem might be since you provided very little information. My first guess would be that SELinux is the problem here. But of course I am assuming you are using Linux etc.

94

Thank you for the SELinux hint. I had forgotten about it. Disabling SELinux solved the issue

95

I have a working vanilla docker environment for the cern-search-rest-api where the indico ES mappings have been create correctly and I can access it on port 8080 (nginx) without any issues.

I am trying to populate ES from my indico server (different that the one running the docker containers for the cern-search-rest-api) under the user indico environment.
The post request is working perfectly using curl but it fails when I use python (I tested using a simple test python code sending the same request as the curl one and using the indico livesync command) .
CURL command that works:

(.venv) [indico@web4606 indico_livesync_json]$ curl -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Authorization:Bearer UoPXZTKPh0VtmWgbm6ivP9zZdBbyCvsZaNrHfPrRMcXkv0bhPPtcWnFTbC75' https://indico-sup-dev.fnal.gov:8080/api/records/ --data '{"_data":{"speakers_chairs": [], "event_type": "meeting",  "title": "Run II PMG", "url": "https://indicoint.fnal.gov/event/100/", "location": "Comitium","$schema": "https://indico-sup-dev.fnal.gov:8080/schemas/indico/indico-events_v1.0.0.json", "category_path": ["Committees/Meetings", "FNAL Committees/Meetings", "Run II PMG"], "id": 100, "description": ""},"end_date": "2007-09-20T19:30:00+00:00","start_date": "2007-09-20T18:00:00+00:00", "creation_date": "2006-05-15T21:40:22.761000+00:00",  "_access": {"owner": [""], "read": [""], "update": [""], "delete": [""]}}'

{"created":"2020-06-24T19:40:49.231297+00:00","id":"2","links":{"self":"https://indico-sup-dev.fnal.gov:8080/api/record/2"},"metadata":{"_access":{"delete":[""],"owner":[""],"read":[""],"update":[""]},"_data":{"$schema":"https://indico-sup-dev.fnal.gov:8080/schemas/indico/indico-events_v1.0.0.json","category_path":["Committees/Meetings","FNAL Committees/Meetings","Run II PMG"],"description":"","event_type":"meeting","id":100,"location":"Comitium","speakers_chairs":[],"title":"Run II PMG","url":"https://indicoint.fnal.gov/event/100/"},"control_number":"2","creation_date":"2006-05-15T21:40:22.761000+00:00","end_date":"2007-09-20T19:30:00+00:00","start_date":"2007-09-20T18:00:00+00:00"},"updated":"2020-06-24T19:40:49.231304+00:00"}

indico livesync command to populate ES and test python code, give the following error:

(.venv) [indico@web4606 indico_livesync_json]$ indico livesync initial-export 1
Selected agent: LiveSync-JSON (LiveSync-JSON)
jsondata:        {"end_date": "2007-09-20T19:30:00+00:00", "start_date": "2007-09-20T18:00:00+00:00", "_access": {"owner": [""], "read": [""], "update": [""], "delete": [""]}, "$schema": "https://indico-sup-dev.fnal.gov:8080/schemas/indico/events_v1.0.0.json", "creation_date": "2006-05-15T21:40:22.761000+00:00", "_data": {"speakers_chairs": [], "event_type": "Meeting", "title": "Run II PMG", "url": "https://indicoint.fnal.gov/event/100/", "location": "Comitium", "category_path": ["Committees/Meetings", "FNAL Committees/Meetings", "Run II PMG"], "id": 100, "description": ""}}
change_type:     2
obj_id:          100
entry_type:      EntryType.event
self.search_url: https://indico-sup-dev.fnal.gov:8080/api/records/
self.headers:    {u'Content-Type': u'application/json', u'Authorization': u'Bearer UoPXZTKPh0VtmWgbm6ivP9zZdBbyCvsZaNrHfPrRMcXkv0bhPPtcWnFTbC75', u'Accept': u'application/json'}
/opt/indico/.venv/lib/python2.7/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host 'indico-sup-dev.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
Traceback (most recent call last):
  File "/opt/indico/.venv/bin/indico", line 8, in <module>
    sys.exit(cli())
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask_pluginengine/util.py", line 194, in wrapped
    return func(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync/cli.py", line 87, in initial_export
    agent.create_backend().run_initial_export(Event.find(is_deleted=False))
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync/base.py", line 113, in run_initial_export
    uploader.run_initial(events)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync/uploader.py", line 72, in run_initial
    self.upload_records(processed_batch, from_queue=False)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/backend.py", line 64, in upload_records
    self.upload_jsondata(jsondata, SimpleChange.created, entry.id, entry_type)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_livesync_json/backend.py", line 110, in upload_jsondata
    raise livesyncjson_uploaderError('{} - {}'.format(response.status_code, response.text))
indico_livesync_json.backend.livesyncjson_uploaderError: 502 - <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.17.4</center>
</body>
</html>

The docker log for nginx for the above requests is:

172.25.0.1 - - [24/Jun/2020:19:39:57 +0000] "POST /api/records/ HTTP/1.1" 502 157 "-" "python-requests/2.23.0"
2020/06/24 19:39:57 [error] 6#6: *68 upstream prematurely closed connection while reading response header from upstream, client: 172.25.0.1, server: , request: "POST /api/records/ HTTP/1.1", upstream: "uwsgi://172.25.0.8:5000", host: "indico-sup-dev.fnal.gov:8080"

172.25.0.1 - - [24/Jun/2020:19:40:49 +0000] "POST /api/records/ HTTP/1.1" 201 738 "-" "curl/7.29.0"

Information on the web about the 502 Bad Gateway error, points to the configuration of nginx inside docker. I do not know how to configure nginx under docker any help will be welcomed.

96

Hey, nginx here is just a proxy for another service (container) that runs cern-search-rest-api. Can you paste the logs of the cern-search-rest-api service?

docker-compose -f ./docker-compose.full.yml logs cern-search-api
97

These are the last entries of the cern-search-api logs. The first one is the successful entry using CURL and the next are the python ones that return the 502 error:


cern-search-rest-api | [pid: 25|app: 0|req: 34/41] 172.25.0.1 () {40 vars in 571 bytes} [Thu Jun 25 13:35:03 2020] POST /api/records/ => generated 738 bytes in 110 msecs (HTTP/1.1 201) 17 headers in 776 bytes (1 switches on core 0)
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 774, in resolve_from_url
cern-search-rest-api |     document = self.store[url]
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/_utils.py", line 22, in __getitem__
cern-search-rest-api |     return self.store[self.normalize(uri)]
cern-search-rest-api | KeyError: 'https://indico-sup-dev.fnal.gov:8080/schemas/indico/events_v1.0.0.json'
cern-search-rest-api | During handling of the above exception, another exception occurred:
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonresolver/contrib/jsonschema.py", line 46, in resolve_remote
cern-search-rest-api |     result = resolver.resolve(uri)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonresolver/core.py", line 48, in resolve
cern-search-rest-api |     loader, args = self.url_map.bind(parts.netloc).match(parts.path)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/werkzeug/routing.py", line 1799, in match
cern-search-rest-api |     raise NotFound()
cern-search-rest-api | werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
cern-search-rest-api | During handling of the above exception, another exception occurred:
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
cern-search-rest-api |     (self._dns_host, self.port), self.timeout, **extra_kw
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
cern-search-rest-api |     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
cern-search-rest-api |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
cern-search-rest-api | socket.gaierror: [Errno -2] Name or service not known
cern-search-rest-api | During handling of the above exception, another exception occurred:
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
cern-search-rest-api |     chunked=chunked,
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
cern-search-rest-api |     self._validate_conn(conn)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
cern-search-rest-api |     conn.connect()
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 308, in connect
cern-search-rest-api |     conn = self._new_conn()
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn
cern-search-rest-api |     self, "Failed to establish a new connection: %s" % e
cern-search-rest-api | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff301199588>: Failed to establish a new connection: [Errno -2] Name or service not known
cern-search-rest-api | During handling of the above exception, another exception occurred:
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
cern-search-rest-api |     timeout=timeout
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
cern-search-rest-api |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
cern-search-rest-api |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
cern-search-rest-api | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='indico-sup-dev.fnal.gov', port=8080): Max retries exceeded with url: /schemas/indico/events_v1.0.0.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff301199588>: Failed to establish a new connection: [Errno -2] Name or service not known',))
cern-search-rest-api | During handling of the above exception, another exception occurred:
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 777, in resolve_from_url
cern-search-rest-api |     document = self.resolve_remote(url)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonresolver/contrib/jsonschema.py", line 51, in resolve_remote
cern-search-rest-api |     return super(RefResolver, self).resolve_remote(uri)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 860, in resolve_remote
cern-search-rest-api |     result = requests.get(uri).json()
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/api.py", line 76, in get
cern-search-rest-api |     return request('get', url, params=params, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/api.py", line 61, in request
cern-search-rest-api |     return session.request(method=method, url=url, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
cern-search-rest-api |     resp = self.send(prep, **send_kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
cern-search-rest-api |     r = adapter.send(request, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
cern-search-rest-api |     raise ConnectionError(e, request=request)
cern-search-rest-api | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='indico-sup-dev.fnal.gov', port=8080): Max retries exceeded with url: /schemas/indico/events_v1.0.0.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff301199588>: Failed to establish a new connection: [Errno -2] Name or service not known',))
cern-search-rest-api | During handling of the above exception, another exception occurred:
cern-search-rest-api | Traceback (most recent call last):
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
cern-search-rest-api |     return self.wsgi_app(environ, start_response)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
cern-search-rest-api |     return app(environ, start_response)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
cern-search-rest-api |     return self.wsgi_app(environ, start_response)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
cern-search-rest-api |     response = self.handle_exception(e)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
cern-search-rest-api |     return cors_after_request(app.make_response(f(*args, **kwargs)))
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
cern-search-rest-api |     reraise(exc_type, exc_value, tb)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
cern-search-rest-api |     raise value
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
cern-search-rest-api |     response = self.full_dispatch_request()
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
cern-search-rest-api |     rv = self.handle_user_exception(e)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
cern-search-rest-api |     return cors_after_request(app.make_response(f(*args, **kwargs)))
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
cern-search-rest-api |     reraise(exc_type, exc_value, tb)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
cern-search-rest-api |     raise value
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
cern-search-rest-api |     rv = self.dispatch_request()
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
cern-search-rest-api |     return self.view_functions[rule.endpoint](**req.view_args)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/views.py", line 89, in view
cern-search-rest-api |     return self.dispatch_request(*args, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_rest/views.py", line 240, in dispatch_request
cern-search-rest-api |     *args, **kwargs
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/views.py", line 163, in dispatch_request
cern-search-rest-api |     return meth(*args, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_records_rest/views.py", line 429, in need_record_permission_decorator
cern-search-rest-api |     return f(self, record=record, *args, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_records_rest/views.py", line 696, in post
cern-search-rest-api |     record = self.record_class.create(data, id_=record_uuid)
cern-search-rest-api |   File "./cern_search_rest_api/modules/cernsearch/api.py", line 119, in create
cern-search-rest-api |     record = super(CernSearchRecord, cls).create(data, id_=id_, with_bucket=bucket_allowed, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_records_files/api.py", line 322, in create
cern-search-rest-api |     record = super(Record, cls).create(data, id_=id_, **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_records/api.py", line 180, in create
cern-search-rest-api |     record.validate(**kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_records/api.py", line 126, in validate
cern-search-rest-api |     _records_state.validate(self, self['$schema'], **kwargs)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_records/ext.py", line 41, in validate
cern-search-rest-api |     **kwargs
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 932, in validate
cern-search-rest-api |     error = exceptions.best_match(validator.iter_errors(instance))
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/exceptions.py", line 367, in best_match
cern-search-rest-api |     best = next(errors, None)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 328, in iter_errors
cern-search-rest-api |     for error in errors:
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/_validators.py", line 259, in ref
cern-search-rest-api |     scope, resolved = validator.resolver.resolve(ref)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 766, in resolve
cern-search-rest-api |     return url, self._remote_cache(url)
cern-search-rest-api |   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/jsonschema/validators.py", line 779, in resolve_from_url
cern-search-rest-api |     raise exceptions.RefResolutionError(exc)
cern-search-rest-api | jsonschema.exceptions.RefResolutionError: HTTPSConnectionPool(host='indico-sup-dev.fnal.gov', port=8080): Max retries exceeded with url: /schemas/indico/events_v1.0.0.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff301199588>: Failed to establish a new connection: [Errno -2] Name or service not known',))
cern-search-rest-api | [pid: 25|app: 0|req: 35/42] 172.25.0.1 () {44 vars in 648 bytes} [Thu Jun 25 13:36:02 2020] POST /api/records/ => generated 0 bytes in 20128 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 1)
[root@indico-sup-dev cern-search-rest-api]#
98

This is the same problem you had before. indico-sup-dev.fnal.gov can’t be resolved properly to an IP address. Do you run Indico on the same machine as the containers? If yes, then just replace indico-sup-dev.fnal.gov with 127.0.0.1 in the plugin config.

99

Unfortunately, the docker cern-search-api and indico are on different machines. I tried to put the ip address but I get an error that the ip is not trusted

100

Ok, this is a bit better then… I had the same problem. Open your .env file and locate INVENIO_APP_ALLOWED_HOSTS. Add your IP address to the list and restart all docker services.