1 Functionality
- 1.1 Flow Diagramm
2 Input
3 Output
4 Configuration settings (technical)
- 4.1 Reference in Code
- 4.2 Web Search Config
  - 4.2.1 Bing Search API
    - 4.2.1.1 Bing Custom Search Config
  - 4.2.2 Google Search API
    - 4.2.2.1 Google Custom Search Config
  - 4.2.3 Data Source Config
  - 4.2.4 Answer Generation Config
  - 4.2.5 Query Generation Config
  - 4.2.6 Configuration Example
- 4.3 Definition
- 4.4

Functionality

This module is designed to answer a user query based on web pages retrieved using a web search engine. The module behaves as a standard GPT but has access to a web search tool. When the user asks a question that the model doesn’t know the answer, it executes the following workflow:

Formulates a search query optimized for search engines (based on the entire discussion history)
Executes a web search using the configured search engine.
Scraps the HTML content of top web pages returned by the search engine
Divide the scrapped content into small chunks of text
1. [Optional] Embed the chunks locally and perform a similarity search based on the generated query.
Passes the relevant information to the context and answers the user’s question/query

Flow Diagramm

Input

A user question

Example input:

“What are the recent trends and developments in Zurich’s financial and insurance markets?“

Output

An answer based on retrieved web pages, either referencing the appropriate links or stating that no information was found in the retrieved web pages.

Configuration settings (technical)

Reference in Code

WebSearch

Web Search Config

Provided below is the general structure of the configuration for the Web Search module.

Field	Type	Description	Default Value

Field	Type	Description	Default Value
`languageModel`	`LanguageModelName`	Model name	`DEFAULT_MODEL_NAME`
`searchEngineEame`	Literal["google", "bing"]	Search Engine ['google', 'bing']	"bing"
`questionAnsweringSystemPromptTemplate`	`str`	Refer to DS Team
`questionAnsweringTriggerPromptTemplate`	`str`	Refer to DS Team
`toolSearchSystemPromptTemplate`	`str`	Refer to DS Team
`toolSearchTriggerPromptTemplate`	`str`	Refer to DS Team
`useHeadlessBrowser`	`bool`	`false`
`bingConfig`	`BingConfig`	Configuration for Bing search	`BingConfig()`
`googleConfig`	`GoogleConfig`	Configuration for Google search	`GoogleConfig()`
`dataSourcesConfig`	`DataSourcesConfig`	Configuration for data sources	`DataSourcesConfig()`
`answerGenerationConfig`	`AnswerGenerationConfig`	Configuration for answer generation	`AnswerGenerationConfig()`
`queryGenerationConfig`	`QueryGenerationConfig`	Configuration for query generation	`QueryGenerationConfig()`
`chunkRelevancySorter`	`ChunkRelevancySortConfig`	configuration of chunk relevancy sort	`ChunkRelevancySortConfig()`
`debugging`	`bool`	Enable debugging mode	False

Bing Search API

Field	Type	Description	Default Value

Field	Type	Description	Default Value
searchCountryCode	str	Search Country Code. Default is Switzerland	"CH"
searchMarket	str	Search Market. Default is German Switzerland	"de-CH"
customSearchConfig	`BingCustomSearchConfig`	Configuration for Bing Custom Search	`BingCustomSearchConfig()`

Bing Custom Search Config

Bing offers the possibility to customize the search results. For example:

Limit the search to a predefined set of domains
Promote a domain
Pin results from a specific website
Filter links
…

This Web Search module developed by Unique supports custom search through the custom_config_id parameter that can be obtained from Bing Custom Search. More details on setup can be found in the following link: Quickstart: Create your first Bing Custom Search instance - Bing Search Services

Once the id is created, it can be passed to through config to your assistant.

There is also the possibility to boost specific search domains without the need to create a custom search id. This can simply be achieved by passing the list of domains in the boosted_search_domains parameter. NOTE: contrary to custom search, this method doesn’t necessarily restrict the search domains to the list provided in boosted_search_domains.

Field	Type	Description	Default Value

Field	Type	Description	Default Value
id	str \| None	Custom Config ID. Can be setup from Bing Custom Search	None
boostedSearchDomains	list[str]	List of domains to boost in search results	[]

Google Search API

Field	Type	Description	Default Value

Field	Type	Description	Default Value
searchCountryCode	str	Search Country Code. Default is Switzerland. Must be lowercase	"ch"
customSearchConfig	`GoogleCustomSearchConfig`	Configuration for Google Custom Search	`GoogleCustomSearchConfig()`

Google Custom Search Config

Google Custom search is not supported yet, but can be implemented if requested by users.

Field	Type	Description	Default Value

Field	Type	Description	Default Value
dummyConfig	str	Dummy Config	"dummy"

Data Source Config

This configuration defines how the module deals with the data retrieved from the internet (e.g. number of sources, chunk_size, …)

Field	Type	Description	Default Value

Field	Type	Description	Default Value
maxRetries	int	Number of retries when a fetch request fails	1
fetchTimeout	int	Number of seconds to wait for a webpage to respond	2
fetchSize	int	Number of search results to fetch	4
maxWorkers	int	Number of workers to embed search results	10
chunkSize	int	Number of chunks to split the search results	600
chunkOverlap	int	Number of chunks to overlap	50
embeddingBatchSize	int	Embedding Batch Size	128
bannedDomains	list[str]	List of domains that are banned	[]
urlBlacklist	list[str]	List of urls suffixes to be blacklisted	[“.pdf“]

Answer Generation Config

This configuration defines how the module make use of the retrieved pages from the internet to generate its answer.

Field	Type	Description	Default Value

Field	Type	Description	Default Value
limitTokenSources	int	Token Source Limit	5000
maxChunksToConsider	int	Maximum Chunks to Consider	20
numberHistoryInteractionsIncluded	int	Number of history interactions included	2
embeddingReranking	bool	Apply reranking of chunks based on embeddings	True

Query Generation Config

The module automatically optimize the question of the user to better work with search engines. The user for example may want to always search the web in English or as in the default value, use the

Field	Type	Description	Default Value

Field	Type	Description	Default Value
queryInstruction	str	Instruction of the query parameter	"The user's search query, optimized for search engines, incorporates relevant details from the conversation, especially if it is a follow-up question. Always use the user's language message."

Configuration Example

{
  "WebSearchConfig": {
    "debugging": false,
    "languageModel": "AZURE_GPT_4o_2024_0513",
    "searchEngineName": "bing",
    "useHeadlessBrowser": false,
    "bingConfig": {
      "searchMarket": "de-CH",
      "searchCountryCode": "CH",
      "customSearchConfig": {
        "id": null,
        "boostedSearchDomains": []
      }
    },
    "dataSourcesConfig": {
      "maxRetries": 1,
      "fetchTimeout": 2,
      "chunkSize": 600,
      "fetchSize": 10,
      "maxWorkers": 10,
      "chunkOverlap": 50,
      "embeddingBatchSize": 128
    },
    "answerGenerationConfig": {
      "limitTokenSources": 10000,
      "maxChunksToConsider": 500,
      "numberHistoryInteractionsIncluded": 4,
      "embeddingReranking": false
    },
    "queryGenerationConfig": {
      "queryInstruction": "The user's search query, optimized for search engines, incorporates relevant details from the conversation, especially if it is a follow-up question. Always use the user's language message."
    },
  }
}

Definition

{
  "type": "function",
  "function": {
    "name": "WebSearch",
    "parameters": {
      "type": "object",
      "required": [
        "query"
      ],
      "properties": {
        "query": {
          "type": "string",
          "description": "User's query that needs to be used to retrieve information from the web using a search engine."
        }
      }
    },
    "description": "Useful to answer user's query based on information from the web."
  }
}

Author	@Rami Azouz

Public Documentation

Web Search Module

Analytics