SXF Original Research · November 4, 2026
AI Marketing Agent Benchmark (2026): Make, Zapier, n8n & Dify
How accurately do AI marketing agents detect campaign anomalies? SXF compared Make, Zapier, n8n and Dify across 16 operator-observed runs (12 detection and four recommendation tests) using one open, synthetic six-channel dataset. Explore precision, recall, false alarms and the exact prompts—not claims of general platform superiority.
Direct answer: On this one fictional campaign benchmark, Dify returned the exact six expected anomaly flags in 3/3 runs; Make and Zapier each did so in 2/3, while n8n returned 0/3 exact sets because it added false alerts. The four environments used different model configurations. Compare the full precision and recall results.
AI Marketing Agent Accuracy: Precision, Recall and Exact Matches
The anomaly challenge had six correct flags distributed across three of six channels. Each platform received the same fixture and thresholds in three repeated runs. The n8n experiment used an OpenAI gpt-4.1-mini chat model connected to an AI Agent in n8n Cloud; Dify used a model labeled Gemini 3.5 Flash in its Preview agent. Other platform model configurations were different or incompletely recorded. We scored alert codes rather than fluent explanations.
| Metric across 3 repeated runs | Make AI Agent | Zapier Agents | n8n AI Agent | Dify Agent |
|---|---|---|---|---|
| True-positive alert codes | 17 | 17 | 18 | 18 |
| False-positive alert codes | 0 | 2 | 6 | 0 |
| Missed correct alert codes | 1 | 1 | 0 | 0 |
| Precision = TP / (TP + FP) | 100.0% | 89.5% | 75.0% | 100.0% |
| Recall = TP / (TP + FN) | 94.4% | 94.4% | 100.0% | 100.0% |
| Runs with exactly correct alert set | 2 of 3 | 2 of 3 | 0 of 3 | 3 of 3 |
| JSON-only format in visible responses | 3 of 3 | Not consistently; calculation prose was shown | Uncertain; operator-pasted output table showed fenced JSON | 3 of 3 in operator-pasted responses |
Each platform had 18 required flags across three repeats of one dataset. They are not model-accuracy confidence intervals or benchmarks for marketing automation in general.
How We Tested AI Marketing Agents: Synthetic Campaign Anomaly Detection
The test operator ran Zapier's Agent Management Hub, Make AI Agent using Make's AI Provider (Medium setting), n8n Cloud with Manual Trigger → AI Agent → OpenAI Chat Model (gpt-4.1-mini), and Dify Agent Preview with a model labeled Gemini 3.5 Flash. No live Google Ads, Meta Ads, LinkedIn Ads, CRM or analytics accounts were connected for this challenge. This matters: the study evaluates text-based analysis and suggested diagnoses, not actual marketing execution.
The 12 CSV rows describe two seven-day periods for six channels: Brand Search, Nonbrand Search, Meta Prospecting, Meta Retargeting, LinkedIn B2B and Organic Search. Four metrics were derived from the raw counts: click-through rate (CTR), cost per click (CPC), conversion rate (CVR) and return on ad spend (ROAS). For zero-spend Organic Search, CPC and ROAS were excluded.
| Alert code | Rule, relative change from baseline |
|---|---|
CTR_DROP_10PCT | CTR falls at least 10% |
CPC_RISE_30PCT | CPC rises at least 30% |
CVR_DROP_25PCT | CVR falls at least 25% |
ROAS_DROP_40PCT | ROAS falls at least 40% |
Relative change was computed on unrounded metric values: (current / baseline - 1) × 100%. The reference answer contains three flags for Nonbrand Search, two for Meta Prospecting and one for Organic Search. See the exact Test 03 prompt, the CSV and the answer key.
Reference calculations readers can verify
| Channel | Metric | Baseline → current | Correct relative change |
|---|---|---|---|
| Nonbrand Search | CPC | $1.20 → $1.90 | +58.33% |
| Nonbrand Search | CVR | 3.00% → 2.00% | −33.33% |
| Nonbrand Search | ROAS | 2.50 → 1.0526 | −57.89% |
| Meta Prospecting | CVR | 3.00% → 1.3333% | −55.56% |
| Meta Prospecting | ROAS | 2.40 → 1.00 | −58.33% |
| Organic Search | CTR | 8.00% → 7.00% | −12.50% |
Nonbrand Search CTR fell only about 8.54%, so it should not be flagged by the 10% rule. Organic Search CVR fell 10%, so it should not be flagged by the 25% rule. These near-miss negatives help expose over-flagging.
AI Agent Benchmark: Run-by-Run Detection Results
| Platform | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
| Make | 6 correct / 0 false / 0 missing | 6 / 0 / 0 | 5 / 0 / 1 |
| Zapier | 5 correct / 2 false / 1 missing | 6 / 0 / 0 | 6 / 0 / 0 |
| n8n | 6 correct / 2 false / 0 missing | 6 / 2 / 0 | 6 / 2 / 0 |
| Dify | 6 correct / 0 false / 0 missing | 6 / 0 / 0 | 6 / 0 / 0 |
Zapier, run 1: it marked Nonbrand CTR and Organic CVR incorrectly, while missing the Nonbrand CPC increase. Its subsequent two runs returned the exact set. Make, run 3: it omitted the Organic Search CTR decline despite finding it in earlier runs. n8n, all three runs: it found all six true flags but also marked Nonbrand CTR and Organic CVR incorrectly every time. Dify, all three runs: its responses exactly matched the six expected alert codes with no extras or omissions. These outcomes illustrate different operational risks: unnecessary escalation and missed escalation.
Make visibility: the operator copied displayed reasoning-time values of 17.7, 15.1 and 16.0 seconds, and displayed credits of 2.04 for run 2 and 2.11 for run 3. We do not have comparable end-to-end Zapier timing, nor credit usage for every Make run. We therefore make no speed or price-performance winner claim.
Make returned JSON-only outputs for these three runs. Zapier's visible responses sometimes wrapped JSON in explanatory arithmetic and narrative despite an instruction to return only JSON. The operator-pasted n8n Test 03 tables displayed Markdown code fences; we cannot determine whether these came from the agent itself or the n8n output display, so raw JSON compliance for n8n is not established. Dify's three operator-pasted Test 03 responses were JSON-only. That is material when an automation expects machine-readable output without manual cleanup.
Marketing AI Agent Recommendation Accuracy: Test 04
One additional run per platform (four runs total) required: report the qualifying metric changes, rank HIGH if ROAS fell at least 40% and otherwise MEDIUM, give exactly two diagnostic checks, and label any suspected cause as an unverified hypothesis. Make and Zapier identified Nonbrand Search and Meta Prospecting as HIGH and Organic Search as MEDIUM, with two checks per channel. n8n also identified the three genuinely affected channels, but added Meta Retargeting incorrectly and assigned Organic Search HIGH rather than MEDIUM. Dify returned exactly the three qualifying channels, all six qualifying metric changes to two-decimal accuracy, correct HIGH/HIGH/MEDIUM priorities, two diagnostic checks per channel, and explicitly unverified hypotheses.
| Review area | Make | Zapier | n8n | Dify |
|---|---|---|---|---|
| Correct channels / priorities | 3 / 3 | 3 / 3 | 3 true channels + 1 false channel; 2 of 3 true-channel priorities correct | 3 / 3 correct channels and priorities |
| Triggered metrics represented | 6 / 6 | 6 / 6 | 6 / 6 true alerts, plus 3 nonqualifying metric claims | 6 / 6, no false triggers |
| Hypotheses clearly unverified | Yes | Yes | Yes | Yes |
| Exactly two checks per included channel | Yes | Yes | Yes, including extra channel | Yes, for 3 true channels |
| Structured JSON-only response | Yes | No; prefatory analysis included | Yes in operator-pasted Test 04 response | Yes in operator-pasted response |
| Numeric note | Stated Nonbrand ROAS drop −58.89%; correct is −57.89% | Mostly rounded correctly; one small CVR rounding inconsistency | Incorrect Meta Retargeting CVR −26.63% (actual −0.41%); Organic CVR −25% (actual −10%); Nonbrand CTR incorrectly treated as a trigger | All 6 qualifying relative changes correct to 2 decimals; no invented trigger |
| Organic search diagnosis | General engagement/tracking checks | More directly connected to search queries, rankings and title snippets | Relevant ranking and landing-page checks, but incorrect HIGH priority | Relevant Search Console query and SERP snippet checks; correct MEDIUM priority |
Illustrative diagnostic workflow: For the organic CTR drop, inspect Google Search Console query and page breakdowns, changes in average positions and SERP presentation before rewriting titles. For rising CPC and declining ROAS, inspect keyword-level traffic, conversion definitions, landing pages and attribution windows. No test proved that those factors caused the changes.
The n8n recommendation's extra Meta Retargeting alert relied on a fabricated 26.63% relative CVR decline: the actual change was only about −0.41%. Organic Search CVR changed −10% (not −25%); only its CTR decline qualified, so its priority should be MEDIUM. The Test 04 prompt and operator-transcribed JSON responses allow readers to inspect the individual checks and hypotheses.
Separate, qualitative Zapier SEO audit: plausible-looking mistakes
In earlier exploratory tests on the live AI agents guide and AI agent use cases guide, the agent claimed some H1 and canonical tags were absent. Inspection of the repository HTML found them present. A duplicate-description claim also did not match the distinct descriptions. This highlights a serious failure mode: an agent auditing extracted page text may misread the site's actual HTML.
A suggestion for reciprocal contextual internal links was useful, but a claimed keyword-cannibalization issue lacked supporting search performance data. This work was not included in the Test 03 precision/recall numbers; read the verification note.
Download the Marketing AI Agent Benchmark Dataset and Results
Download the synthetic campaign dataset (CSV) and the 12-run anomaly scorecard (CSV), along with the complete prompts, gold answer and operator-transcribed run records. These plain-text files are in the public SXF GitHub repository. They can be downloaded, diffed or copied into another agent. The responses are operator-transcribed records, not provider-signed session exports; the full surrounding narratives and UI captures are not archived.
Benchmark Limitations and Reproducibility
- One small fictional dataset was repeated three times per platform. Repeating the same fixture does not establish general reliability across campaigns.
- We did not connect ad platforms, change budgets, deploy autonomous actions, validate live account permissions or demonstrate marketing lift.
- Model versions and generation settings were not fully captured. n8n used gpt-4.1-mini, Dify used a model labeled Gemini 3.5 Flash, Make used its AI Provider (Medium), and Zapier's underlying model was not captured. Results cannot be attributed solely to the automation platforms.
- No blinded evaluation or independent reproduction was performed. The outputs are records supplied by the test operator to SXF's research process, not provider-signed exports. Dify's model name and setup were reported in the user-guided workflow, not independently retrieved from its execution API.
- Test 04 includes one qualitative recommendation review per platform, not a scaled ground-truth benchmark for recommendation helpfulness.
- Reported Make reasoning seconds are not a stopwatch measurement of overall workflow time; usage credits were available only for specific runs.
- The separate SEO audit is qualitative. It did not test search traffic, indexing, rankings or SERP growth.
We do not publish a fake composite score, platform-wide ranking, proprietary model benchmark or price comparison derived from this evidence. This is a dated original observation set with a replicable task.
How to Evaluate Marketing AI Agents Before Deployment
- Calculate key metrics deterministically. Compute CTR, CPC, CVR and ROAS in code or a trusted analytics system, not only inside natural-language reasoning.
- Validate the JSON schema. Require a known list of issue codes and reject unparseable responses, extra claims and missing records.
- Use evaluation cases including near misses. Include borderline changes that should not raise alerts, plus zero-spend channels where CPC and ROAS are undefined.
- Record evidence. Keep input snapshots, tool access, model and version where available, output, errors, timestamps and approval decisions.
- Require human approval. Agents may draft recommendations; budget changes, campaign pauses, publication and outbound messages need explicit authorization.
For implementation details, see the AI agent evaluation checklist, agent observability, authorization controls, cost planning and reliability research.