The default pipeline currently in place may not adequately process certain PDF documents, particularly when encountering improperly formatted data (e.g., tables in financial documents).

Microsoft Document Intelligence (MDI ff.) can enhance Unique's capability to accurately ingest documents that contain complex tables and graphics.

Ingestion Modes

PDFs on Unique are ingested page by page.

There are 2 modes of ingesting documents as it is implemented:

• PDFTODOCX_ONLY: Use our default library: PDFs are converted using pdf2docx (default)

• DOC_INTELLIGENCE_DEFAULT: Use MDI on all pages of the document

Enable MDI

MS document intelligence can be switched on per scope/folder. (Even hierarchical scopes with inheritance)

On the ingestion service per scopeId. Replace the placeholders:

• <scopeId>

• <baseUrl> (e.g. *.unique.app)

• <yourToken>

curl --location --request POST 'http://https://gateway.<baseUrl>/ingestion/v1/folder/<scopeId>/properties' \
--header 'Authorization: Bearer <yourToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "properties": {
        "ingestionConfig": {
            "pdfReadMode": "DOC_INTELLIGENCE_DEFAULT"
        }
    },
    "applyToSubScopes": true
}'

Limitations and Considerations

• The MS Document Intelligence Service costs approx. 1.6 cent per page and has limited throughput.. These costs might be charged additionally by Unique as it is not covered by the Ada Tokens.

• Service is still in Preview and only available in West Europe. There are the following points to consider services which are only available in Preview:

  • In the DPA of Microsoft (Nov 2023 version) it is stated that in Preview mode they may employ lesser or different privacy and security measures than those typically present in the Products and Services.

  • Even though we have activated the opt-out of processing data for all versions and subscriptions on your tenant (incl. Preview). Microsoft still reserves the right for preview services to store and access output and prompts for harmful content despite this opt-out.

  • Some other limitations are that preview services are not covered by the SLA and do not offer European Data Boundary Service but are not as critical.

Technical

Activation

Before being able to use MDI, the service must be deployed within a tenant. Depending on your Deployment models one of the following processes must be chosen.

Authentication Methods

MS Document Intelligence can run in two modes:

• Key based authentication (taking it from the env variables (see code), used in dev)

• via Workload Identity in production

  • Unique uses only Workload Identity on all its Deployment models