Purpose

The problem stems from a legal requirement, which means it needs to be addressed from the company's point of view rather than the end user's perspective.

Scope

Only applies to the Upload & Ask assistant or files that are directly uploaded in the chat. It does not include files ingested to the knowledge base.

Key requirements

As a company I want to define the duration after which files uploaded in a chat should expire or if they should expire at all (i.e file retention in days).
When a file expires, both the file and all related chunks will be permanently deleted. As a result, you will no longer be able to open the file or chat about it.

This feature can be enabled/disabled by Unique, so for any changes the customer needs to reach out to us. Also, if enabling the feature, any custom retention period in days can be set (can be 30 or even 90 days).

User Interface

The UI for files uploaded in the chat differs based on whether the file retention in days is set or not.

File retention period defined

If a file retention period is defined (e.g. 30 days) and the file has not expired yet, the UI looks like this:

When the file has expired, the UI looks like this:

File retention period not defined

If no file retention period is defined, no information about the file's deletion will be displayed, as it will not be deleted at all.

Technical details

A scheduled job runs daily at 1 a.m. to verify if a file retention period is defined and to identify any files created before this period. If such files are found, they will be deleted along with their associated chunks, and an expired date will be set for the content.

The scheduled job has limitations on the number of files that can be expired. It runs in 10 rounds, checking 100 files per round, allowing a maximum of 1000 files to expire during each run of the job.

Future improvements

The following future improvements are in ideation phase, we cannot guarantee that the following improvements will be implemented.

We have planned to introduce two additional features that will significantly enhance the user experience, particularly in terms of visibility when interacting with expired files.

Automated chat message for expired files: Whenever a file expires within a chat, an automated chat message will be generated to notify the user. This feature ensures that users can easily identify if a file has expired, even in the midst of many chat messages. Since the files are displayed at the top of the page, it may require scrolling up to view them, ensuring they remain easily accessible.
Handling edge case with document summarizer module: When using the document summarizer module, there is still an edge case to handle when a user tries to summarize documents of a chat but all of the documents are already expired.

Setting the Retention Period:

Author	@Thea Kitzinger

Public Documentation

Expiration of Files Uploaded in a Chat

Analytics