Purpose
The problem stems from a legal requirement, which means it needs to be addressed from the company's point of view rather than the end user's perspective.
Scope
Only applies to the Upload & Ask assistant or files that are directly uploaded in the chat. It does not include files ingested to the knowledge base.
Key requirements
As a company I want to define the duration after which files uploaded in a chat should expire or if they should expire at all (i.e file retention in days).
If a file expires, both the file itself and all associated chunks generated for that file should be deleted. Therefore it will no longer be possible to chat regarding the expired file or open it.
User Interface
The UI for files uploaded in the chat differs based on whether the file retention in days is set or not.
File retention period defined
If a file retention period is defined (e.g. 30 days) and the file has not expired yet, the UI looks like this:
When the file has expired, the UI looks like this:
File retention period not defined
If no file retention period is defined, no information about the file's deletion will be displayed, as it will not be deleted at all.
Technical details
A scheduled job runs every day at 1 am to check if the file retention period is defined and if there are any files created before the retention period. If such files exist, they will be deleted, along with their associated chunks, and an expired date will be set for the content.
Note: The scheduled job has limitations on the number of files that can be expired. It runs in 10 rounds, checking 100 files per round, allowing a maximum of 1000 files to expire during each run of the job.
Future improvements
The following future improvements are in ideation phase, we cannot guarantee that the following improvements will be implemented.
We have planned to introduce two additional features that will significantly enhance the user experience, particularly in terms of visibility when interacting with expired files.
Automated chat message for expired files: Whenever a file expires within a chat, an automated chat message will be generated to notify the user. This feature ensures that users can easily identify if a file has expired, especially when there are a lot of chat messages, as the files are displayed at the beginning of the page, and scrolling up may be required to view them.
Handling edge case with document summarizer module: When using the document summarizer module, there is still an edge case to handle when a user tries to summarize documents of a chat but all of the documents are already expired.