Overview

The Feedback Interface enables administrators to access and analyze evaluations provided by chat users regarding answers to their prompts. This data can be downloaded and accessed in a .csv format, including key details such as message IDs, timestamps, content, and user feedback ratings. The Feedback solution enables effective monitoring of the AI's performance and user satisfaction by showing both positive and negative feedback along with contextual information from previous messages.

Who it is for

This feature is designed for Administrators or Product Managers who are responsible for evaluating the quality and improvement potential.

Role Requirements: To download user feedback, you must have the eligible roles (e.g., chat.data.admin role). For more information about roles, click here: https://unique-ch.atlassian.net/wiki/spaces/PUBDOC/pages/1411023168

Benefits

Feedback is used to evaluate and improve the quality of responses generated from user prompts, while also serving as a foundation for systematically analyzing areas of improvement.
Feedback can act as a ground truth for benchmarking purposes.

Step-by-Step Guide

Step 1: Download the Feedback

Navigate to the Feedback section by clicking the Feedback Icon on the Navigation Sidebar.
Select a specific date range to retrieve feedback data from that period and download feedback.
Click Download Feedback

Step 2: Format the .csv

You may find that the downloaded content does not split properly when using the ‘Text to Columns’ function in Excel. This is due to the content of some cells being too large to split.

To get around this:

From the ‘Data’ tab in Excel, select ‘Get Data’ and select your .csv file.

In the Preview select Delimiter = Semicolon (usually set per default)
Note Windows vs Mac difference:
Windows: Click on load and save

Mac:
On Mac to ignore quoted line breaks an additional step is required:
--> Click on ‘Transform data’ and in ‘Advanced Editor’ add (or replace existing term after QuoteStyle= “ “ ) the section QuoteStyle=QuoteStyle.Csv :

Step 3: Understand the Feedback Contents

The .csv file displays a structured format of feedback data collected from user interactions. Each record captures detailed information such as:

Message ID and Timestamp: Unique identifiers and precise time data for each interaction.
Text and Role: The content of the message and whether it was generated by the user or the assistant.
Feedback Ratings: Indicates if the feedback was positive or negative, along with any detailed comments that the user might have left.
Previous Message Details: Information about the prior exchange in the conversation facilitates an understanding of the feedback context.

API

You can also access the feedback through the API. Check out the details here linkhttps://unique-ch.atlassian.net/wiki/spaces/PUBDOC/pages/1385038869

Service Now Feedback Integration

To work with the service now, we will need to provide this environment variable in the next-chat environment

FEATURE_FLAG_SERVICE_NOW_FEEDBACK_UN_9322=true

In the values.yaml for the deployment, add the URL used in the SERVICE_NOW_SUBMIT_FEEDBACK_ENDPOINT in the connect-src values

CONTENT_SECURITY_POLICY_VALUE:

In the node-chat environment variables, set the following environment variables:

SERVICE_NOW_ACTIVE=
SERVICE_NOW_USERNAME=
SERVICE_NOW_PASSWORD=
SERVICE_NOW_SUBMIT_FEEDBACK_ENDPOINT=
SERVICE_NOW_FEEDBACK_REDIRECT_ENDPOINT=

Sample node-chat environment variable

SERVICE_NOW_ACTIVE=true
SERVICE_NOW_USERNAME=gandalf
SERVICE_NOW_PASSWORD=gandalf
SERVICE_NOW_SUBMIT_FEEDBACK_ENDPOINT=https://servicenow.com/api
SERVICE_NOW_FEEDBACK_REDIRECT_ENDPOINT=https://servicenow.com/api/id

Sequence Diagram

Author	@Enerel Khuyag