Purpose

The OWASP Top 10 is a widely recognized document outlining the most critical security risks facing web applications. Specifically focusing on LLM (Large Language Model) applications, these risks encompass vulnerabilities that could compromise the integrity, availability, and confidentiality of data processed by such models. This page outlines Unique’s understanding and approach on how to mitigate these risks to ensure the security and trustworthiness of our services.

Unique’s approach to OWASP Top 10 for LLM Applications

LLM01: Prompt injections

Prompt Injection Vulnerabilities in LLMs involve crafty inputs leading to undetected manipulations. The impact ranges from data exposure to unauthorized actions, serving attackers goals.

Prevention and Mitigation Strategies

• The platform is only available to registered internal users, which reduces the attack surface.

• Unique has an AI Governance framework in place, which helps in managing risks related to Privacy and Security, Accountability, Transparency, Explainability, Reliability and Safety. This includes user feedback on responses and a human-in-the-loop concept.

• Prompts are logged in an unalterable audit log. This allows for tracing back any prompt to the user who executed it.

• User prompts are sanitized and output is escaped according to best practices.

• We run regular Benchmarking questions to ensure the output of the LLM is in line with expectations. This involves establishing appropriate risk thresholds and providing the flexibility to disable specific use cases or functions if benchmarking results fall below predefined thresholds.

• Unique offers an API that can be integrated with existing Data Leakage Prevention (DLP) systems to monitor the data being processed. The DLP system scans the prompts during post-chat analysis, ensuring that sensitive information is not inadvertently exposed during interactions

• Unique runs a managed with external researchers to continuously harden our solution against crafted inputs.

LLM02: Insecure Output Handling

These occur when plugins or apps accept LLM output without scrutiny, potentially leading to XSS, CSRF, SSRF, privilege escalation, remote code execution, and can enable agent hijacking attacks.

Prevention and Mitigation Strategies

• The platform is only available to registered internal users, which reduces the attack surface.

• User queries are sanitized in the prompts and output is escaped according to best practices.

• Unique runs a managed with external researchers to continuously harden our solution against crafted inputs.

LLM03: Training Data Poisoning

LLMs learn from diverse text but risk training data poisoning, leading to user misinformation. Over-reliance on AI is a concern. Key data sources include Common Crawl, WebText, OpenWebText, and books.

Prevention and Mitigation Strategies

• Unique has technical and contractual safeguards in place to make sure that no data is used for any model training (if not explicitly agreed with the client).

• Unique is working closely with Microsoft and only uses official Microsoft Azure OpenAI LLMs which are pre-trained and never use any data for model training.

• Unique has an framework in place, which helps in managing risks related to Privacy and Security, Accountability, Transparency, Explainability, Reliability and Safety. This includes user feedback on responses and a human-in-the-loop concept.

• We run regular questions to ensure the output of the LLM is in line with expectations. This involves establishing appropriate risk thresholds and providing the flexibility to disable specific use cases or functions if benchmarking results fall below predefined thresholds.

LLM04: Model Denial of Service

An attacker interacts with an LLM in a way that is particularly resource-consuming, causing quality of service to degrade for them and other users, or for high resource costs to be incurred.

Prevention and Mitigation Strategies

• The platform is only available to registered internal users, which reduces the attack surface. It can additionally be locked down using IP blocking so that only internal IPs of the customer can access the system.

• Azure DDoS protection can be configured with additional costs for the customer.

• User prompts are sanitized and output is escaped according to best practices.

• Unique rate-limits authentication APIs to block repeated or automated calls and significantly slow down attempts to brute-force logins or token creation.

• Unique runs a managed bug bounty program with external researchers to continuously harden our solution against crafted inputs.

LLM05: Supply Chain Vulnerabilities

LLM supply chains risk integrity due to vulnerabilities leading to biases, security breaches, or system failures. Issues arise from pre-trained models, crowdsourced data, and plugin extensions.

Prevention and Mitigation Strategies

• Unique is working closely with Microsoft and only uses official Microsoft Azure Open.AI LLMs which are pre-trained and never use any data for model training.

• Unique has a in place that includes regular scanning for outdated or vulnerable dependencies, vulnerabilities in the software or configuration mistakes.

• We run regular questions to ensure the output of the LLM is in line with expectations. This involves establishing appropriate risk thresholds and providing the flexibility to disable specific use cases or functions if benchmarking results fall below predefined thresholds.

LLM06: Sensitive Information Disclosure

LLM applications have the potential to reveal sensitive information, proprietary algorithms, or other confidential details through their output. This can result in unauthorized access to sensitive data, intellectual property, privacy violations, and other security breaches.

Prevention and Mitigation Strategies

• Unique has technical and contractual safeguards in place to make sure that no data is used for any model training (if not explicitly agreed with the client).

• Unique is working closely with Microsoft and only uses official Microsoft Azure Open.AI LLMs which are pre-trained and never use any data for model training.

• Prompt context is only extended by the data the user that is prompting already has access to.

• User prompts are sanitized and output is escaped according to best practices.

• Unique runs a managed bug bounty program with external researchers to continuously harden our solution against crafted inputs.

LLM07: Insecure Plugin Design

LLM plugins auto-engage during user interactions and can process unvalidated free-text inputs. This can potentially lead to security risks, including harmful requests causing undesired actions, even remote code execution.

Prevention and Mitigation Strategies

• Unique does not make use of LLM plugins.

LLM08: Excessive Agency

When LLMs interface with other systems, unrestricted agency may lead to undesirable operations and actions. Like web apps, LLMs should not self-police; controls must be embedded in APIs.

Prevention and Mitigation Strategies

• Unique is a knowledge management system that gives its users access to knowledge. It does neither perform actions on behalf of users nor does it make use of LLM plugins to do so.

• Unique has an framework in place, which helps in managing risks related to Privacy and Security, Accountability, Transparency, Explainability, Reliability and Safety. This includes user feedback on responses and a human-in-the-loop concept.

LLM09: Overreliance

Overreliance on LLMs can lead to misinformation or inappropriate content due to "hallucinations." Without proper oversight, this can result in legal issues and reputational damage.

Prevention and Mitigation Strategies

• Unique offers fast and easy fact-checking via the “source function” to verify the source of information provided by LLMs before utilizing it for decision-making, information dissemination, or other critical functions. When an output is generated, the respective sources including page numbers are highlighted making it easy for the user to check for factual correctness.

• Unique has an framework in place, which helps in managing risks related to Privacy and Security, Accountability, Transparency, Explainability, Reliability and Safety. This includes user feedback on responses and a human-in-the-loop concept.

• We run regular questions to ensure the output of the LLM is in line with expectations. This involves establishing appropriate risk thresholds and providing the flexibility to disable specific use cases or functions if benchmarking results fall below predefined thresholds.

• Unique has a in place that includes regular scanning for outdated or vulnerable dependencies, vulnerabilities in the software or configuration mistakes.

LLM10: Model Theft

Model theft discusses the risk of malicious actors stealing LLM models, leading to significant losses and unauthorized use. It emphasizes the need for robust security measures to protect these valuable assets and mitigate potential risks.

Prevention and Mitigation Strategies

• Unique implements role-based access control (RBAC) for employees who need access to privileged systems or services. All access automatically expires and needs to be renewed at given intervals. This adherence to the least-privilege principle and minimizes the risk of data exposure.

• Unique enforces two-factor authentication (2FA) for access to privileged systems or services and for data centre operations.

• Unique is working closely with Microsoft and only uses official Microsoft Azure Open.AI LLMs which are pre-trained and never use any data for model training. As Unique is not training models with additional data there is nothing to steal that is not already part of the official Azure OpenAI model.