The quest for an AI agent answering any questions about your product or team seemed closed to be solved. Thanks to LLMs and RAG, the dream of having an AI agent answering Slack on your behalf seemed within reach. Tools like Dust and others have made great strides in connecting LLMs to internal company data—Slack, Notion, GitHub—to answer internal questions.
However, they still fail to reliably replace a human product manager or engineer in everyday operations.
- On one end we have agents working well for specific use cases, when their input material is curated and updated regularly (typically FAQ or procedures) - but these are too specific and are limited by the human work that is needed
- On the other end we have agents ingesting a wide source of uncurated data (like Slack, customer tickets, interview recordings) that make impressive POCs but fail at being useful for real-life work
How can we build agents that can reliably answer questions on behalf of a product manager or an engineer, without investing time in curating sources ?
In this article, I try to structure this question - maybe raising more questions than possible answers - through the lens of agentic architecture: multi-agent systems that are able to handle wide range of uncurated knowledge before answering questions.
Why this is a difficult problem
Having an agent answer on uncurated knowledge sources like Slack messages, documents and meeting transcripts is hard. The knowledge is available and continuously being generated, but has 2 limitations to be directly exploitable by the usual RAG-only method:
- Knowledge decays, with high variability of lifespan
- Knowledge sources have variable level of reliability
Knowledge in the input sources decays, and the lifespan is very variable
- For example a status update is only valid for one day
- A technical documentation about architecture can be true for several months
- A PRD may convey a vision that is consistent with the team”s view for a few weeks, and then become obsolete if the scope has changed during implementation
Input sources have variable levels of reliability
- Production code is the ultimate source of truth: it shows what is actually built, including all the bugs
- A debated Slack message is a low-confidence signal.
As a consequence, answers from agents directly pulling from these sources tend to less relevant or accurate as a human having the same source of information.
To be able to build the ultimate agent managing internal knowledge, agents need to factor in the reliability and the decay of the information before sharing an answer.
Should we take a "on-the-fly" approach or a "source of truth" approach ?
in my explorations for an architecture to solve this question, I found one key sub-question to be on the architecture of the agents to be able to create reliable knowledge. The architecture can follow a “source of truth” or “on-the-fly” approach.
Source of truth approach
One solution is to separate the task in two:
- One system of agents continuously ingest the generated knowledge in Slack, meetings, Notion pages, and create a source of truth
- The agent or system responsible of answering questions would only rely on the curated source of truth
On the fly approach
With this approach, the system of agents receives the question, retrieve and evaluate the answer on the fly
Which one to chose ?
At first sight, the right approach will depend on:
- The amount of data being generated (by Slack, documents etc)
- The quality of the data being generated (how dense is the data with actual reusable knowledge)
- The frequency at which knowledge needs to be accessed, and in particular how often the same knowledge is accessed (are different teams always asking the same questions to my team ?)
The “answers on the fly” approach seems to be preferable if the data generated is not dense in knowledge, decays fast and does not need to be accessed often. Differently said, no need to heat the whole ocean if you just need a glass of water.
The “source of truth” approach on the other hand seems to be preferable if the knowledge needs to be accessed often and has a reasonable lifespan.
The right solution for my problem is probably a mix:
- Slack messages could be processed to build a source of truth, namely because it contains the actual questions that are being asked
- Meeting notes or transcripts could be accessed on the fly
Source of truth approach raises a lot more interesting questions
What’s the best way to store knowledge ?
- An FAQ ?
- A graph of unitary knowledge elements linked together ?
- Should they be scored for reliability ?
Conclusion
In my quest to have an agent answering Slack on my behalf, I am starting by default with a “answers on the fly” approach and test others based on actual behavior of my agents.