AI Security Testing & Validation Course Notes
Foundations
Introduction to AI Security Testing
Introduction to AI Security Testing
AI security testing: systematically evaluating AI systems (especially LLMs) and the applications built on top of them
Key categories:
- Prompt injection: Malicious instructions embedded in user input/retrieved content, that can hijack model behavior
- Jailbreaking: bypass safety guardrails and get model to produce restricted output
- Data exfiltration: Tricking model to reveal sensitive data from context, memory, or restricted systems
- Model inversion: Inferring training data from model outputs
- Indirect injection: Malicious instructions hidden in external content the model retrieves
- Tool/agent abuse: Exploiting AI systems to take unintended real-world actions
What does an AI security tester do? -> AI security assessment lifecycle
- Understanding the system
- How application works, what AI capabilities used, what data is processed, what external services are connected
- Mapping data flows
- Identify input sources, AI components, retrieval systems, databases, tool integrations, output destinations
- Threat modeling
- What can attacker influence, what assets need protection, where untrusted input enter the system, which components present the highest risk
- Developing test cases
- What will be tested, how testing performed, expected outcomes
- Executing security tests
- Reporting findings
AI Security Assessment Lifecycle & Methodology
Difference between assessment lifecycle and methodology:
- Assessment lifecycle: What are the stages of an AI security assessment?
- Structured & repeatable process
- Tell us where we are in assessment process
- Assessment methodology: How do we perform each phase of the assessment?
- Techniques, procedure, best practices used to execute assessment effectively
- Tell us how to perform activities required at each stage
Why use methodology? Follow repeatable process that ensures risks are systematically identified, tested, documented, and validated. Methodology provides:
- Consistency
- Repeatability
- Better coverage
- etc.
AI Security Assessment Lifecycle:
- Understanding the system
- Mapping data flows
- Threat modeling
- Developing test cases
- Executing security tests
- Reporting findings
- Validation & Regression Testing
- Assess Residual Risk
The OWASP AI Testing Guide
Provides:
- A standardized methodology for trustworthiness testing of AI and LLM-based systems
- Repeatable test cases that evaluate risks across:
- Application layer
- AI Model layer
- AI Infrastructure layer
- AI Data layer
How the guide fits into an assessment:
- Understand the application -> Help to identify components, assets, trust boundaries
- Threat modeling -> Identify likely attack vectors
- Test planning -> Help create structured test cases
- Security testing -> Use the guide’s recommendation to perform testing activities
- Reporting -> Map findings to test IDs
OWASP AI Testing Guide != threat framework. It complements other frameworks.
Relationship to other frameworks:
- OWASP AI Testing Guide: Security testing methodology
- OWASP Top 10 for LLM Applications: Common AI vulnerabilities
- MITRE ATLAS: Adversary tactics and techniques
- STRIDE: Threat modeling
- NIST AI RMF: Risk management and governance
Defining Assessment Objectives & Scope
Assessment objectives: Why the assessment is being performed. Common objectives:
- Security validation - determine whether security controls are functioning as intended
- Vulnerability discovery - identify unknown weaknesses
- Compliance requirements - demonstrate compliance with internal policies and external standards
- Remediation validation - verify previously identified vulnerabilities has been fixed
Scoping: defining boundaries, objectives, and focus areas of an AI security assessment before testing begins.
- What will be tested
- What will not be tested
- Assessment objectives
- Assessment constraints
- Success criteria
Scoping methodology:
- Define assessment objective
- Identify critical assets (e.g. PII)
- Inventory AI components
- Define scope boundaries (what’s in & out of scope)
- Establish Rule of Engagement
- Document assessment scope
Threat Modeling
Threat Modeling for AI Systems
Threat modeling: Identifying, analyzing, prioritizing potential threats to a system before they can be exploited
Simplified:
Using STRIDE for AI Threat Modeling
STRIDE: Help systematically evaluate a system from six different threat perspectives.
STRIDE:
- Spoofing
- Tampering
- Repudiation
- Information disclosure
- Denial of Service (DoS)
- Elevation of privilege
You walk through each component, data store, process, and trust boundary, and ask which STRIDE categories would apply
Test Planning
Developing an AI Security Test Plan
Developing AI Threat model with STRIDE:
- Identify critical assets
- Identify trust boundaries
- Apply STRIDE
- Perform risk assessment (risk = likelihood x impact)
- Calculate risk score
- Prioritize threats
Developing an AI Security Test Plan
- Prioritize threats for testing
- Convert threats into security objectives
- Define test scenarios
- Define test cases
- Identify required evidence
- Define success criteria
Example on how this leads to test cases:
- Threat model: Prompt Injection
- TC-001 Direct Prompt Injection
- TC-002 Role Impersonation
- TC-003 System Prompt Extraction
- TC-004: Context Manipulation
Writing Effective AI Security Test Cases
Test case: Documented procedure used to validate whether a specific threat/security control can be exploited, bypassed, or abused
A good security test case should clearly define:
- ID
- Test objective
- Threat
- Component
- Procedure
- Expected result
- Evidence
Writing good test cases, should be:
- Specific, clearly define what is being tested
- Repeatable, another assessor should be able to execute the same test and achieve similar results
- Threat-driven, every test case should map back to a threat identified during threat modeling
- Measurable, clearly define what success/failure