AI SECURITY / 2026
AI security is the discipline of protecting models, agents, data, tools and the systems around them from manipulation, unauthorized action, information leakage and unsafe execution. The highest-risk failures increasingly happen at the boundaries between an AI model and the tools, permissions, memory and external content it can reach.
\nLast reviewed: October 2026. Based on OWASP, NIST, MCP and primary provider security guidance.
01 / FOUNDATION
AI security protects AI systems from adversarial manipulation and limits the damage when a model behaves incorrectly. It extends conventional application and cloud security because an AI system can interpret natural language, retrieve untrusted information, decide which tool to call, generate code or commands, and sometimes act with delegated authority.
That creates a different trust problem. A web page, document, email, tool description or retrieved record may be both data for the user and instructions to the model. An agent may also hold credentials that let it cross boundaries a normal chatbot cannot: read files, send messages, modify repositories, query databases, browse authenticated sites or execute code.
For that reason, the strongest security posture does not depend on the model refusing every malicious instruction. It assumes some manipulation attempts will succeed and uses hard controls around the model: least privilege, isolation, scoped credentials, validation, approval gates, logging and the ability to stop or reverse actions.
Treat the model as a powerful but fallible decision component inside a security architecture—not as the security boundary itself.
02 / THREAT MODEL
The 2026 AI security problem spans the full path from untrusted content to privileged execution. OWASP’s GenAI guidance highlights risks such as prompt injection, sensitive information disclosure, supply-chain weaknesses, improper output handling and excessive agency. Agentic systems make those categories interact: one poisoned input can influence a model, steer a tool call and create a real-world side effect.
Malicious instructions can arrive directly from a user or indirectly through web pages, documents, emails, RAG results and tool outputs. The goal is to alter the model’s behavior or redirect the agent away from the user’s intent.
Primary concern: instruction/data confusion and downstream action.An agent with broad filesystem, SaaS, cloud or admin access has a larger blast radius. A model mistake becomes a security incident when the agent is allowed to perform destructive or sensitive actions without narrow authorization.
Primary concern: privilege scope and irreversible action.Tool descriptions, schemas, server responses and third-party MCP packages can become injection or supply-chain surfaces. Cross-server data flows can also allow one connected system to influence actions in another.
Primary concern: tool poisoning, token scope and cross-origin trust.Secrets can leak through prompts, context windows, retrieval systems, tool calls, logs or generated output. Security must cover what enters context, what leaves it and which external destinations the agent can contact.
Primary concern: credentials, PII, proprietary data and exfiltration.Persistent memory and retrieved knowledge can preserve attacker-controlled instructions beyond a single interaction. A poisoned record may quietly affect later decisions unless provenance, write access and retrieval paths are constrained.
Primary concern: durable manipulation and corrupted context.Generated SQL, shell commands, code, URLs and structured tool parameters must be treated as untrusted output. Passing model output directly into privileged systems creates classic injection and SSRF-style risks with an AI decision layer in front.
Primary concern: validation before execution.Models, adapters, datasets, packages, plugins and agent servers add dependencies with their own provenance and update risk. A trusted workflow can become unsafe when a dependency changes behavior after approval.
Primary concern: provenance, integrity and change detection.If teams cannot reconstruct which prompt, source, identity, tool, parameters and policy decision produced an action, incident response becomes guesswork. Security requires evidence at the action boundary, not just chat transcripts.
Primary concern: auditability, attribution and response.03 / DEFENSE
Prompt-level defenses are useful, but they are not enough for high-impact systems. Security should remain effective even when the model misunderstands context or follows a malicious instruction. The control plane around the model should decide what is possible.
Give each agent, tool and MCP server only the data, scopes and actions it needs. Separate read access from write access, and avoid shared credentials across unrelated tools.
Run code and local tool servers in constrained environments. Limit filesystem paths, outbound network access, process capabilities and access to host credentials.
Require explicit confirmation for destructive, financial, externally visible or sensitive operations. Show the actual target and parameters, not only a friendly summary generated by the model.
Validate model-produced URLs, file paths, SQL, code, API parameters and output schemas before execution. Use allowlists and policy engines where a hard guarantee is required.
Use short-lived, scoped tokens tied to the intended resource. Do not let the model handle master credentials or reuse one high-privilege identity across every connected service.
Record tool invocations, identity, parameters and outcomes. Detect unusual action sequences, enforce quotas, and provide a fast way to revoke tokens, disable tools or stop an agent.
04 / MCP & TOOLS
The Model Context Protocol makes it easier to connect AI applications to tools and data, but every connection adds a new trust boundary. Tool definitions and returned content can influence model behavior, while OAuth scopes and server-side permissions determine what the resulting action can reach.
Authenticate the user and authorize the resource. A connected server should not imply unlimited trust. Tokens should be scoped to the intended server and capability.
Treat tool metadata and tool results as untrusted content. A tool description can contain manipulative instructions just as a web page can.
Isolate servers from one another. Sensitive data retrieved from one tool should not automatically become input to another tool with an external side effect.
Make destructive actions explicit. When a tool can send, delete, purchase, deploy or modify permissions, the host should enforce an approval boundary rather than trusting a model-generated decision alone.
Read the full AI Agent Security guide →05 / PROMPT INJECTION
Prompt injection is not limited to a user typing “ignore previous instructions.” In agentic systems, an attacker can place instructions in content the model later reads: a website, message, document, code comment, retrieved record or tool response. OpenAI describes modern prompt-injection attacks as increasingly similar to social engineering because the model must reason about potentially manipulative content in context.
The defensive objective is therefore broader than detecting bad strings. Separate trusted instructions from untrusted data, minimize what an exposed agent can do, require confirmation where consequences are high, and validate every privileged action outside the model.
No practical production architecture should depend on that assumption. Build containment so a successful injection does not automatically become credential theft, data exfiltration, code execution or an irreversible external action.
06 / SCOPE
| Area | Main question | Typical concerns |
|---|---|---|
| AI security | Can an attacker manipulate the AI system or abuse what it can access? | Prompt injection, tool abuse, data leakage, model theft, poisoned context, excessive agency, insecure execution. |
| AI safety | Can the AI system cause harmful outcomes even without a conventional attacker? | Misalignment, dangerous capability, unintended behavior, reliability, oversight, misuse potential. |
| Cybersecurity | Can systems, identities, networks and data be compromised? | Vulnerabilities, malware, credential theft, access control, network security, incident response. |
Real deployments overlap all three. An AI agent may be manipulated through prompt injection, abuse a cloud credential, then produce an unsafe real-world outcome. Treating these as separate organizational silos creates gaps at exactly the boundaries attackers can exploit.
07 / DEPLOYMENT CHECKLIST
For an AI system that can retrieve private data or take actions, these checks provide a useful minimum baseline before production exposure.
Identify users, models, data stores, tools, MCP servers, external content and downstream systems.
Separate read-only, reversible, external, destructive and privileged actions.
Use per-tool or per-server identities and short-lived tokens wherever possible.
Constrain file access, processes, network destinations and environment secrets.
Do not execute generated commands, URLs, SQL or tool parameters without deterministic checks.
Require approval that exposes the exact target, scope and effect of the operation.
Control who can write persistent context and retain provenance for retrieved records.
Red-team web pages, emails, documents, RAG sources and tool returns—not just the chat input.
Capture identity, model, source context, tool, parameters, policy decisions and result.
Make it possible to revoke tokens, disable a tool, stop an agent and preserve evidence quickly.
08 / PRIMARY REFERENCES
SXF uses primary and standards-oriented references to define the security model on this page. These sources evolve; implementation details should always be verified against the latest source documentation.
Current risk categories for LLM and generative AI applications, including prompt injection, disclosure and agency risks.
OWASP / MCPMCP Security Cheat SheetOperational guidance for tool poisoning, OAuth scope, sandboxing, validation, isolation and monitoring.
OPENAI / SECURITYDesigning agents to resist prompt injectionWhy modern prompt injection resembles social engineering and why containment matters alongside model resistance.
NIST / AI RMFGenerative AI ProfileCross-sector guidance for governing, mapping, measuring and managing generative-AI risk.
MCP / 2026 SPECModel Context Protocol 2026-07-28Current protocol revision with authorization hardening, stateless requests and updated deployment guidance.
NIST / CYBER AICyber AI Profile workNIST work on managing cybersecurity risk related to AI systems and AI-enabled cyber capabilities.
09 / LIVE INTELLIGENCE
The reference layer above changes slowly. This feed tracks concrete security controls, cyber-defense releases and safety-relevant system changes from primary sources so the page stays connected to what is changing now.
10 / QUICK ANSWERS
There is no single universal risk, but the most dangerous combination is untrusted input plus excessive permissions. Prompt injection becomes materially worse when the affected agent can access secrets, execute code or take external actions without independent controls.
No. A jailbreak generally tries to bypass model behavior restrictions through direct interaction. Prompt injection targets an application or agent by placing instructions in inputs or external content so the model follows attacker-controlled directions.
MCP can be deployed safely, but a connection should not be treated as inherently trusted. Security depends on server provenance, narrow authorization, sandboxing, tool isolation, input/output validation, approval policies and monitoring.
Only when necessary, and then with the narrowest practical scope. Prefer task-specific or per-tool identities, short-lived tokens, resource restrictions and approval for high-impact operations rather than broad standing credentials.
No. System prompts are one layer of defense, not a hard security boundary. High-impact systems need deterministic controls outside the model so a bad decision cannot directly become an unauthorized action.
Test the entire workflow: direct and indirect prompt injection, poisoned retrieval, malicious tool output, permission abuse, credential exposure, unsafe generated parameters, cross-tool data flow, approval bypass and incident containment.