Testing AI systems requires constant awareness of developments within the AI domain.
Generative AI systems are complex are mostly black box based tested, which increases difficulty to understand and predict responses. ML-based systems rely heavily on the quality and quantity of data, both for training and inference. Systems that continuously learn from input data create high-value targets for attackers, as this data can be exploited to craft further attacks. Security vulnerabilities often arise in data collection, storage, and processing.
Generative AI security risks can be grouped into four components:
- Model: Vulnerabilities within the model itself, e.g., prompt injections or unsafe outputs.
- Data: Training and inference data, including its handling and storage.
- Application: Integration points, such as web apps using AI features, where traditional application vulnerabilities may exist.
- System: Underlying hardware, OS, and deployment configurations; includes resource exhaustion or denial-of-service risks.
Red Teaming Considerations: Traditional red teams use tactics, techniques, and procedures (TTPs) like phishing, malware deployment, and lateral movement. When targeting generative AI systems, TTPs must be adapted to the unique risks of each component, addressing model-specific, data-related, application, and system vulnerabilities.
Attacking the Model
Attacking the model consists of weights and biases and the training process.
| Risk | Description | Consequences / Notes |
|---|---|---|
| Model Poisoning | Occurs when adversaries manipulate model parameters during training. | Lower performance, biased or erratic behavior, harmful content generation. Difficult to detect, critical in sensitive applications (healthcare, finance, autonomous systems). |
| Evasion Attacks | Attacks during inference using crafted inputs to trick the model. | Incorrect outputs, harmful or illegal content generation. Jailbreaks in LLMs can bypass restrictions for malicious purposes. |
| Model Theft (Extraction) | Stealing models, which are valuable intellectual property, by inferring structure, parameters, or behavior through strategic querying. | Financial loss, IP theft, enable further attacks like model poisoning or |
Tactics, Techniques, and Procedures (TTPs):
- Adversaries typically start by analyzing the model via numerous inputs to understand behavior and detect vulnerabilities.
- They may craft malicious inputs (e.g., prompt injections) to manipulate outputs or extract sensitive data.
- For model extraction, techniques include adaptive querying and extensive input sampling to reconstruct the model, enabling replication, IP theft, or subsequent attacks.
Attacking Data Components
The data component includes all data used by the model, such as training and inference data. Since ML models are highly data-dependent, attackers often target data to influence model behavior, degrade quality, or exploit sensitive information.
| Risk | Description | Consequences / Notes |
|---|---|---|
| Improper or Biased Training Data | Training data that is unrepresentative or biased. | Low-quality, discriminatory, or harmful model outputs. Emphasizes the importance of clean, representative data. |
| Data Poisoning | Adversaries inject malicious or biased data into training sets. | Misleading, biased, or harmful outputs. Backdoor attacks can trigger malicious behavior under specific inputs. |
| Data Leakage & Theft | Unauthorized access to sensitive or valuable datasets. | Financial, legal, or competitive harm. Stolen data can be used to reverse-engineer models or craft adversarial inputs. |
Tactics, Techniques, and Procedures (TTPs):
- Training Data Manipulation: Attackers may inject biased or false data to compromise performance, with serious ethical, legal, or safety implications in applications like healthcare, legal document generation, or content creation.
- Exploiting Security Weaknesses: Poor cloud configurations, weak encryption, insecure data pipelines, and vulnerable APIs can allow attackers to steal data.
- Supply Chain Attacks: Compromising third-party data providers enables attackers to manipulate or steal data before it reaches the organization.
- Insider Threats: Employees or contractors with legitimate access may exfiltrate data for personal gain or malicious purposes, often bypassing traditional security defenses.
Attacking Application Components
The application component includes the software interfaces and systems integrating the AI model. Weaknesses here can lead to unauthorized access, data breaches, or full system compromise.
| Risk | Description | Consequences / Notes |
|---|---|---|
| Unauthorized Application Access | Attackers gain entry to sensitive areas without proper credentials. | Can lead to privilege escalation, full system compromise, and data loss. |
| Injection Attacks | Exploit improper input handling (e.g., SQL injection, command injection). | Can manipulate databases or system processes, resulting in data breaches or full system compromise. |
| Insecure Authentication | Weak passwords, lack of multi-factor authentication, and poor session management. | Enables brute-force attacks, credential misuse, and user impersonation. |
| Information Disclosure | Accidental exposure of sensitive data due to insecure coding, misconfigurations, or inadequate access controls. | Leads to privacy violations, financial losses, reputational damage, and opportunities for identity theft or fraud. |
Tactics, Techniques, and Procedures (TTPs):
- Exploit weak or missing input validation to inject malicious payloads.
- Use Cross-Site Scripting (XSS) to execute malicious scripts in victim browsers.
- Employ social engineering (phishing, pretexting, baiting) to trick users into revealing credentials or executing malware.
Attacking System Components
The system component includes the underlying hardware, operating system, system configuration, and ML model deployment. While traditional IT security risks apply, ML-specific deployment issues introduce additional vulnerabilities.
| Risk | Description | Consequences / Notes |
|---|---|---|
| System Misconfigurations | Default or weak settings, exposed admin interfaces, open ports, and default credentials. | Can lead to unauthorized access to the underlying infrastructure. |
| Insecure ML Model Deployment | Lack of authentication, encryption, or input validation exposes models to attacks. | Vulnerable to attacks and resource exhaustion (DoS/DDoS), which can disrupt operations or mask other attacks. |
| Resource Exhaustion Attacks | High-volume requests or computationally intensive inputs overwhelm CPU, RAM, network, or disk resources. | Causes operational disruption, service unavailability, and potentially increased costs in auto-scaling environments. |
Tactics, Techniques, and Procedures (TTPs):
- Use vulnerability scanners to detect outdated software or exposed services.
- Password spraying and brute-force attacks against weak credentials or default accounts.
- Exploit misconfigured server software, firewalls, or access controls to gain unauthorized access.