1 Functionality
2 Input
3 Output
4 Configuration settings (technical)
- 4.1 Tool definition
- 4.2 Global structure of config
- 4.3 chatHistory
- 4.4 Example Json Config
- 4.5 Template Json Config (for developers)

Functionality

This module makes us of the capabilities of some LLM to answers questions out of an Uploaded Image. This module uses the entirety of the conversation history (up to the model’s token limit). It also support using a summary history instead if correct tool definition is provided and the functionality is enabled from config.

Input

An Image with valid extension + user query. It’s helpful to trigger this module when specifying in the user query that you need to get information from uploaded image.
Example: ”Please describe the content of the uploaded image.”

Output

Answer to the question based on the image and conversation history.

Configuration settings (technical)

Tool definition

{
    "type": "function",
    "function": {
      "name": "ImageUpload",
      "parameters": {
        "type": "object",
        "required": [
          "query",
          "conversation_history"
        ],
        "properties": {
          "query": {
            "type": "string",
            "description": "Smart query based on the user request but augmented with the context of conversation if needed."
          },
          "image_name": {
            "type": "string",
            "description": "Name of the image file to be analyzed [Optional]."
          },
          "conversation_history": {
            "type": "string",
            "description": "Summary of the history of the conversation to provide context for the query."
          }
        }
      },
      "description": "This tool is designed to analyze, extract, and summarize information from images. It should be invoked whenever the user inquires about images."
    }
}

Global structure of config

Parameter	Description

Parameter

Description

languageModel: string

used GPT model

Default: AZURE_GPT_4_32K_0613

chatHistory

Instance of service

supportedExtensions: list[string]

List of supported image extensions.

Default:

[".jpg",".jpe",".jpeg",".png",".gif",".bmp"]

chatHistory

Parameter	Description

Parameter	Description
`enabled`: bool	Whether to use full conversation history. If False, attempt to use conversation summary from tool call. Default: `true`
`maxMessages`: integer	Max history messages to consider Default: `10`
`useModelTokenLimit`: bool	Whether to use model token limit or not Default: `true`
`defaultTokenLimit`: integer	Default token limit if model token limit is not available. Default: `4096`
`percentageOfMaxTokens`: float	Upper limit of tokens to keep from history. Default: `0.5`

Example Json Config

{
  "ImageUploadConfig": {
    "chatHistory": {
      "enabled": true,
      "maxMessages": 10
    },
    "languageModel": "AZURE_GPT_4o_2024_0513"
  }
}

Template Json Config (for developers)

{
  "ImageUploadConfig": {
  "supportedExtensions" : [
        ".jpg",
        ".jpe",
        ".jpeg",
        ".png",
        ".gif",
        ".bmp"
    ],
    "chatHistory": {
      "enabled": true,
      "maxMessages": 10,
      "useModelTokenLimit": true,
      "defaultTokenLimit": 4096,
      "percentageOfMaxTokens": 0.5 
    },
    "languageModel": "AZURE_GPT_4o_2024_0513"
  }
}

Author	@Rami Azouz

Public Documentation

Image Upload