GhostWriter: Poisoning Long-Term Memory in LLM Agents

Link to the original paper

Introduction

The paper examines a threat to LLM agents with long-term memory. Such agents read email and calendars, remember contacts, agreements, and prior actions, then use that information in new tasks.

The authors call the attack GhostWriter. Its fundamental difference from conventional prompt injection is that malicious content does not need to take effect immediately. It arrives through an untrusted channel, such as an external email or calendar invitation, is saved in the agent’s memory, and activates later when the user submits an apparently harmless but topically related request.

The issue does not rely on an attacker gaining direct access to the memory store or the user’s account. Instead, the paper assumes a more realistic scenario in which the attacker can only send a message that the agent will process. If the system indiscriminately saves incoming data and later treats retrieved memories as trusted context, a single message can influence the agent’s actions in future sessions.

Threat Model

The authors model an office personal assistant that handles email and meetings, answers user questions, and can act through tools—for example, drafting and sending emails. Long-term memory stores facts, event summaries, or prior action plans; retrieval is typically based on semantic similarity to the current request.

The attacker:

• has no access to the agent’s API, user account, or memory store;
• cannot alter data outside the application;
• can send external emails and invitations disguised as ordinary work communication;
• aims to make the agent save a false fact or hidden directive and retrieve it during a later user task.

The authors identify four outcomes: integrity corruption, sensitive-information leakage, covert exfiltration, and scope escalation beyond the agent’s intended actions. For example, a false contact change can be stored as a fact and affect the recipient of a future email.

How GhostWriter Works

The attack consists of two phases separated in time.

1. Memory injection. The attacker sends a plausible message containing seemingly useful context. The agent receives it through its usual tool, summarizes or saves it, and places the result in long-term memory.

2. Activation. The user later submits an ordinary task that is semantically related to the message’s topic. The retrieval mechanism finds the poisoned record, adds it to the context, and the agent may treat the memory’s content as a reliable instruction or fact.

To improve the chance of retrieval, the attack message is optimized not for the exact future request, but for its topical similarity. In the black-box model examined here, an attacker only needs a public email corpus and a separate embedding model: they select plausible context resembling the anticipated user task. They do not know the agent’s internal embedding model and do not control the retrieval process itself.

This makes the threat particularly important: malicious content can outlive the original email, take effect in a separate session, and appear to the model as internal system knowledge. Conventional safeguards that analyze only the current request see only the final phase of the attack.

Experiments

The authors adapted five memory-agent architectures to a common setting:

• A-Mem;
• Mem0;
• ExpeL;
• Letta (formerly MemGPT);
• MemoryOS.

Each architecture was tested with GPT-5.4-mini, DeepSeek-V4-Flash, Gemini-2.5-Flash, and Llama 3.1 8B. To create a reproducible environment, the agents first processed a synthetic five-day work week, including emails, calendar events, meetings, deadline changes, and contact-information changes. Before each test, the memory state was reset to this clean baseline.

The researchers created 16 scenarios delivered through email or calendar invitations. They measured three distinct outcomes:

The average injection success rate was about 98%, and the average activation success rate was about 60% across all tested agents and models. Injection succeeded in 100% of cases for most combinations. Mem0 and ExpeL sometimes transformed the source text so that malicious content was not fully preserved, producing lower injection rates.

ExpeL also had the lowest activation rate—between 28% and 63%, depending on the model. The authors attribute this to ExpeL storing trajectories of past actions rather than treating every record as an authoritative fact. However, this does not make the architecture universally safe: in a comparative experiment, it was especially susceptible to conventional prompt injection.

GhostWriter Versus Conventional Prompt Injection

The authors separately compared GhostWriter with prompt injection from AgentDojo. Direct injection in an incoming message rarely activated after being stored by most tested agents: P2 ranged from 0% to 16.7% for many combinations. GhostWriter differs because the message is prepared for future retrieval, so poisoned memory is more likely to return at the relevant moment.

This matters for detection as well. DataFilter did not recognize the GhostWriter variants used by the authors. A separate LLM check detected strongly worded directives more effectively, but almost never flagged descriptive, polite variants. Therefore, defenses focused only on characteristic prompt-injection phrasing do not cover memory poisoning.

AM-Sentry: A Two-Stage Defense

The proposed mitigation is Agentic Memory Sentry (AM-Sentry). It is a layer between data sources, memory, and the agent’s context. It is not tied to a particular memory-store implementation and consists of two independent barriers.

Memory-Saving Policy

Before a candidate is written to memory, the system assesses whether it should be saved and how risky it is. In stricter variants, an LLM judge labels each entry using several fields:

• origin and source-trust level;
• type: fact, contact information, preference, workflow, or instruction;
• future utility;
• signs of potentially unsafe behavior, sensitive data, contradictions, and unverifiable claims.

The final decision is made not only by an LLM but also by deterministic threshold-based logic. The strictest mode uses a seven-item checklist, hard-rejects entries that show signs of changing the agent’s behavior, and sends the assessment to a second LLM judge to identify potential compromise of the first one.

Retrieval Screen

Even a sound saving policy cannot guarantee that a dangerous record will never enter memory. The second barrier therefore runs at retrieval time: every retrieved record is examined before it enters the model context.

The screen excludes a memory if it:

• is not directly relevant to the current task;
• contains directives or behavioral rules rather than data;
• comes from an untrusted source and makes unverifiable claims;
• contradicts already trusted information.

The paper expresses the idea well: memory should be data, not commands. Origin, trust, and risk should be stored as metadata rather than inferred again from the text alone each time.

Defense Results and Limitations

Security and utility were measured separately. The strictest memory-saving policy reduced GhostWriter’s average success rate to 15% for ChatGPT, DeepSeek, and Gemini, but performed worse with Llama: the model admitted 77% of malicious records in the strict format. Combined with the retrieval screen, the strict configuration reduced the end-to-end attack success rate to below 12% for all models except Llama, where the average was 20%.

The authors do not claim the defense is final. Their utility test is a synthetic work week rather than real company data and processes. Only email and calendar inputs were evaluated, whereas a real agent may also receive untrusted content from documents, web pages, and code repositories. The AM-Sentry rule weights were selected heuristically, and the attacker in the evaluation did not adapt to the defense itself.

There is also a practical trade-off: strong filtering can block useful memories. Utility metrics generally fell only slightly in the experiments, but Llama more often over-blocked legitimate data in strict modes. Deploying a defense like this therefore requires measuring not only the rate of blocked attacks, but also failures in real operational scenarios.

Conclusion

GhostWriter shows that an LLM agent’s long-term memory is not merely a context-window optimization, but state with its own security requirements. If any external message can become trusted memory, the agent gains a delayed control channel: the attack begins when an email arrives but manifests during an ordinary user task.

The paper’s central practical conclusion is that both transitions must be filtered: external input → memory and memory → model context. This requires data provenance, structured metadata, a rule against treating memory as an instruction, and a separate check of records at retrieval time. With these measures, security becomes part of the memory architecture rather than another prompt layered on top of the agent.