Hello everyone!
I wanted to bring to the community a project I’ve been plugging way at (pun intended) for some time to see if folks are interested in contributing and really to get some expert insight and advice. GitHub - lucasflores/indico-assistant: AI-powered conversational assistant for Indico event management system
There is still a lot to be done before anything like this could be safely adopted by CERN or some other org so definitely lots of interesting problems to solve! Here are some things that are top of mind for me right now:
-
Verifying authentication - I have my own permissions layer but it can be intensive since it is a simple filtering of eventObject.can_access(user_id) of returned SQL query results. Have not had the chance to understand how Indico handles permissions (which events one has access to) to see if there is standard native way.
-
LLM Security - right now I have some naive guardrails that make (to the best of my knowledge) the llm be able to do ‘read-only’ from the db, this needs to be interrogated. Other threat vectors brainstormed in general.
-
Evaluation framework for retrieval - this becomes especially more nuanced when we start looking into pdfs/docs. Would want to utilize simulated event data here.
-
Scalability
-
How well does the retrieval work with tens of thousands (millions?) of events and documents.
-
For an existing db like CERN’s indico, just the initial campaign to parse/chunk/embed all existing uploaded materials sounds like a huge computational undertaking. And it very much matters what chunksize and embedding model (for example) you use for this so determining the best parameters for that is very important.
-
-
Chat stuff
-
persistent conversations/past conversation storage
-
context overflow → conversation history summarizer
-
Observability ( token usage, llm as a judge for quality of results. This is will be a part of the broader eval framework. )
-
Lots of little things that people expect to have with chatbots at this point that still need implementing
-
Anyway, would love to hear people’s thoughts!
Best,
Lucas