What “full study” means here
Research attribution. Research conducted by SXF Research. SXF Research owns the research design and question selection, and documented and published this original study. Google AI Mode and ChatGPT Search are the evaluated platforms, not the study authors or publishers.
This appendix publishes the complete wording of all 14 prompts, each first-run preliminary score and the editorial review notes, with primary-source starting links. A user submitted 28 original answers during the research conversation. The spreadsheet archives some of them in normalized or abridged form, and the available excerpt file is not a verified verbatim transcript. The full raw 28-answer archive has not been independently reconstructed.
ChatGPT originally graded both tools, including itself, and awarded itself full marks on every item. There has been no independent blind review, no additional repeat runs and no statistical validation. Do not use these numbers as a general accuracy benchmark.
Study methodology
Research responsibilities. SXF Research selected the 14 questions for this pilot and documented and published the results; the 14 prompts were submitted once to each platform, and 28 first-run answers were collected. ChatGPT assisted in preliminary scoring of both tools, including itself.
Collection. The same 14 English prompts were manually run once in Google AI Mode and once in ChatGPT Search. The user pasted the Google answer first and ChatGPT answer second for each task. Questions covered recent developments, official sources, peer-reviewed science, regulations, arithmetic, Python, SEO, and crawler controls.
Sampling. SXF Research deliberately chose questions; no randomly sampled real-search query set was used. The responses were reviewed on November 5, 2026. Exact timestamps and device/search settings for each original run were not recorded; the date is a study-session context, not independently logged per-response execution time.
Score rule. Factual accuracy 40%; sources 30%; freshness and relevance 20%; clarity/usefulness 10%. Each dimension received a 0–5 score, and the total out of 100 was 8×accuracy + 6×sources + 4×freshness + 2×usefulness. Fractional dimension ratings were permitted.
Independence. ChatGPT assigned the preliminary scores for Google AI Mode and ChatGPT Search, including itself. This was unblinded scoring by a competing product, not independent human review or independent validation. In particular, the provisional 100/100 ChatGPT mean is an artifact of that scoring process and not validated evidence of universal perfection. Scores and examples may change after neutral claim-level re-evaluation.
Original descriptive figures. Google AI Mode: 1296 total points, averaging 92.57/100. ChatGPT Search: 1400 points, averaging 100/100. Six first-pass ties, eight provisional ChatGPT leads, zero independent inter-rater reliability measurements.
All original prompts, scores and source records
All 14 questions below reproduce the source prompt ledger exactly. Expand each entry to read the old editorial comments. Notes are preserved as historical evaluator observations, not freshly independently verified outcomes. Data downloads: prompts.csv and initial-editorial-scores.csv.
Anthropic vault particles
Original question:
On November 4, 2026, what did Anthropic report about vault particles, how many protein copies form the phage-derived structure, and which experimental technique was used? Cite the original research post.
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: All three requested facts supported by Anthropic's Nov 4 research. Primary research post and technical PDF linked; additional third-party article. PDF link present but PDF not separately reviewed. No factual error identified.
ChatGPT: All three requested facts and additional details align with Anthropic's Nov 4 research. One direct primary article; no factual error identified.
Original source starting point:
Anthropic FlashProve
Original question:
In Anthropic's November 2, 2026 FlashProve announcement, which model was tested, what proof throughput was reported, and roughly how much extra GPU time did proving take? Cite Anthropic's publication.
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: All required claims corroborated by Anthropic Nov 2 publication. Primary and secondary sources cited. Extra optional follow-up mentions STARKs without asserting FlashProve uses STARKs; no scored factual error. 110 tok/s is batched throughput, not single-request latency; question did not ask for that distinction.
ChatGPT: All required claims corroborated by Anthropic Nov 2 publication. 4-bit quantization, ~19MB proof, and ~72x/94x details supported by primary source. Model-reported benchmarks have not been independently replicated in this comparison.
Original source starting point:
Google AI source links
Original question:
What changes to links and source discovery did Google announce for AI Mode and AI Overviews on October 15, 2026? List at least two, with the official Google source.
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: All four claims corroborated by Google Oct 15 primary post. Omits the related-articles change, but prompt asked for at least two changes; therefore no deduction. One official primary source plus one secondary site.
ChatGPT: All five updates match Google Oct 15 post; additional related-article links with thumbnails supported. Primary official Google source cited directly; no error identified.
Original source starting point:
EU AI Act obligations
Original question:
As of November 5, 2026, when did the EU AI Act's obligations for general-purpose AI models start applying, and when did enforcement of those rules begin? Cite European Commission pages and distinguish the dates.
Accuracy 4.5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Core dates verified by Commission. Minor legal overstatement: under Article 88 the Commission/AI Office has exclusive enforcement powers for GPAI provider obligations, rather than co-equal national authority enforcement of those model rules. -0.5/5 accuracy.
ChatGPT: Core dates and legacy GPAI 2027 transition verified by official EU Commission guidance and Article 111. Three official source pages. No error identified.
Original source starting point:
Original Transformer paper
Original question:
In which year and conference proceedings did 'Attention Is All You Need' appear? Name the first two authors and link to the original proceedings page.
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Confirmed in official NeurIPS proceedings. Google also cited a Wikipedia page and supplied Google Search links for people/conference, but the original proceedings link independently satisfies the request; equal-contributor note corroborated.
ChatGPT: All four requested bibliographic facts and primary conference citation confirmed in official NeurIPS proceedings; no substantive error identified.
Original source starting point:
Webb and K2-18 b life claim
Original question:
As of November 5, 2026, a viral post says NASA's James Webb Space Telescope has confirmed life on the exoplanet K2-18 b. Fact-check the claim. Distinguish what Webb observed from proposed biosignatures, cite an official NASA source and at least one peer-reviewed reanalysis, and explain the uncertainty.
Accuracy 4/5; sources 3.3333333333333335/5; currency 4.25/5; usefulness 4.5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Scientific uncertainty: strongest reliable evidence is methane; 2025 Schmidt reanalysis did not find reliable CO2 or DMS, contrary to Google's 'robustly detected CO2'. Google also described DMS as exclusively biological without Earth-only caveat; abiotic cometary DMS has been reported (Hänni et al., 2024). Main requirement to cite at least one peer-reviewed reanalysis directly was missed (NPR article cited instead). Weighted rubric = 78/100.
ChatGPT: All principal conclusions and direct peer-reviewed reanalyses checked against journal or institutional sources. Correctly acknowledges scientific disagreement and 2026 Pica-Ciamarra alternative analysis rather than equating contested tentative signals with life. Copied T05 response prefix in user's pasted text excluded from T06 scoring. Weighted rubric = 100/100.
Original source starting point:
Hubble and Webb mission comparison
Original question:
Compare NASA's Hubble and James Webb Space Telescopes using official NASA sources. Where does each operate relative to Earth (with approximate distance), which wavelengths are their instruments mainly optimized for, and what is one trade-off of those different designs? Cite NASA pages.
Accuracy 4.25/5; sources 5/5; currency 4.5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: No extended narrative note was recorded for this item.
ChatGPT: No extended narrative note was recorded for this item.
Original source starting point:
Nobel Prize in Chemistry 2026
Original question:
What was the 2026 Nobel Prize in Chemistry awarded for and who received it? Link directly to the Nobel Prize organization's official press release. Explain the scientific contribution in one paragraph without overstating possible applications.
Accuracy 4.75/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Only minor issue: quote is an official descriptive sentence, not the formal award motivation. Scientific account is broadly sound; official motivation is 'for the discovery of non-linear effects and autocatalysis in asymmetric organic synthesis'. Editorial 2-point accuracy deduction.
ChatGPT: All requested elements supported by Royal Swedish Academy official Oct 7 press release. Correctly distinguishes mechanistic insight from final explanation of life's homochirality.
Original source starting point:
95% confidence interval
Original question:
A research article claims that 'a 95% confidence interval means there is a 95% probability that the fixed true population parameter lies in this exact computed interval.' Is the claim correct in frequentist statistics? Explain the distinction between coverage and probability with one intuitive example, citing NIST or a university statistics reference.
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Fully aligned with NIST: after a sample is fixed, it covers or misses the fixed parameter; 95% belongs to the interval procedure. Phrase '100% or 0% probability' is best understood as a coverage indicator, not a subjective posterior probability.
ChatGPT: Fully aligned with NIST: repeated-sample coverage interpretation, one computed interval not 95% probability, and clear example. Conditions on model assumptions appropriately stated.
Original source starting point:
Agent pass^k vs pass@k
Original question:
For an AI agent with an 80% success probability on each independent attempt at the same task, what is the probability that all five attempts succeed? Contrast that with the chance that at least one of five succeeds. Explain the difference between pass^k and pass@k, give your assumptions, and cite an original agent-evaluation paper or benchmark documentation.
Accuracy 5/5; sources 4.5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Google's arithmetic and core definitions are correct. Slight citation gap: Codex paper supports pass@k; tau-bench is the primary source introducing agent pass^k. 'Perfect oracle' is relevant to selecting a successful result, not to event probability itself.
ChatGPT: Direct original tau-bench citation and important limitation: averaging per-task reliability in empirical evaluation cannot be replaced without assumptions by raising a pooled average to the fifth power.
Original source starting point:
Standard API token pricing
Original question:
As of November 5, 2026, use the official OpenAI and Anthropic API pricing pages to compare standard, uncached input and output costs per 1 million tokens for GPT-6 Astra and Claude Sonnet 4.6. Calculate the total model-token cost for 2 million input tokens and 500,000 output tokens on each model. Do not mix in Batch, Fast or cached-input rates; cite both official pricing pages, state any caveats, and do not claim this compares model quality.
Accuracy 2/5; sources 3/5; currency 4/5; usefulness 4/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Major pricing-tier mistake: $60 input/$300 output is GPT-6 Astra Ultrafast, not requested Standard $10/$50. $270 is mathematically right for that wrong tier; correct Standard total is $45. Claude total $13.50 is right. Link to OpenAI primary source did not prevent selecting incorrect tab/tier.
ChatGPT: Matched both official Standard pricing tables and calculations. Qualifies GPT-6 Astra >272K-input-token request surcharge and distinguishes aggregate usage from a single prompt. No quality comparison implied.
Original source starting point:
Python 3.14 and GIL
Original question:
A viral developer post claims: 'Python 3.14 removed the Global Interpreter Lock (GIL) by default, so every Python program now runs CPU-bound threads in parallel without changes.' Fact-check this claim using official Python 3.14 documentation. Explain what free-threaded CPython actually requires, whether it is the default build, and one limitation or compatibility caveat. Cite the relevant Python documentation sections.
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: All requested claims supported by Python 3.14 official docs. Free-threading is officially supported but optional; pyperformance overhead and C-extension GIL fallback also documented. Broad phrase 'pure Python execution won't crash interpreter' should not be interpreted as a guarantee; the answer later cites a potential crash involving frame.f_locals.
ChatGPT: The requested claims are correct and sourced to official Python 3.14 docs. The pasted code snippet lost line breaks in the transcript; excluded from score because example code was not requested. Docs version 3.14.10 was released Oct 25, 2026.
Original source starting point:
llms.txt and Google AI Search
Original question:
An SEO consultant claims that adding an llms.txt file is mandatory for a website to appear in Google AI Mode and AI Overviews, and that it directly improves Google rankings. As of November 5, 2026, fact-check both claims using official Google Search Central documentation. Explain what website owners actually need to do, distinguish requirements from optional practices, and cite the exact official sources.
Accuracy 4.25/5; sources 5/5; currency 5/5; usefulness 5/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Original claims correctly fact-checked; however, says standard Google search eligibility is sufficient/automatic, without mentioning the separate Search Console inclusion control rolled out Aug 31 2026. Google documentation expressly requires inclusion for AI features. No deduction for Lighthouse audit: separate tool, correctly distinguished from Search ranking.
ChatGPT: Both core claims and extra Search Console control verified from official Google source. Explicitly states inclusion does not guarantee being surfaced; correct separation of requirements vs SEO recommendations.
Original source starting point:
Search and AI training crawlers
Original question:
A publisher wants their website to remain eligible for discovery in both Google AI Mode and ChatGPT Search, while opting out of model-training uses wherever official crawler controls allow it. As of November 5, 2026, explain the differences between Googlebot, Google-Extended, OAI-SearchBot, and GPTBot. Propose a minimal robots.txt example, distinguish search visibility from AI training permissions, identify important limitations, and cite official Google and OpenAI documentation.
Accuracy 3.75/5; sources 4.5/5; currency 4.5/5; usefulness 4/5
Accuracy 5/5; sources 5/5; currency 5/5; usefulness 5/5
Read first-pass editorial notes
Google: Correct proposed rules, but overclaims robots-based removal, mischaracterizes Google-Extended as a separate crawler and frames the proposed shared control as current/legally recognized. Google Search generative AI inclusion option omitted. Scored provisionally by ChatGPT; blind review required.
ChatGPT: Matches official Google and OpenAI docs on scope of crawler controls and 2026 Search Console inclusion. Sample snippet is accurate for a public site without conflicting rules. No claim of guaranteed indexing. Still a self-score; blind review required.
Original source starting point:
Evidence grading criteria
| Component | Weight | What a reviewer should check |
|---|---|---|
| Accuracy | 40% | Precise facts, calculations, cautious scientific interpretation and no misleading omissions |
| Source quality | 30% | Primary citations that actually substantiate the exact claimed fact |
| Freshness | 20% | Relevant dates, current product tiers, law applicability and source vintage |
| Usefulness | 10% | Whether the response clearly and directly answers the literal question |
These weights describe the original grading process; they were not independently pre-registered or calibrated. Linking to a government, provider or scientific page by itself does not mean that the page supports every nearby claim.
Evidence review: the highest-impact observed examples
T11: a cited but mis-selected API price tier
The first Google answer used $60 / $300 per million GPT-6 Astra tokens instead of the Standard $10 / $50 per million requested. For the specified 2 million input and 0.5 million output tokens, that created a $270 estimate where the Standard-token calculation was $45. Its arithmetic was internally consistent for the wrong tier. This example supports checking pricing context, not any claim about typical error rate. Original reference: OpenAI API pricing.
T06: biosignatures are not confirmed life
Both tools rejected the K2-18 b viral claim. The editorial review noted differences in distinguishing methane and other atmospheric inference from tentative potential biosignatures, independent reanalyses, and non-biological alternatives. Starting references: NASA Science and peer-reviewed reanalysis.
T13–T14: search visibility is not AI model training
Googlebot, Google-Extended, OAI-SearchBot and GPTBot have different control meanings; allowing search discovery is not identical to granting training permission. Neither crawler permission nor a citation-rich answer guarantees search placement. Starting sources: Google Search Central, Google Crawlers and OpenAI Bots.
Downloadable research records
Independent-review status (5 November 2026): Protocol prepared; 28 blinded case IDs and two blank reviewer score forms prepared. The original conversation transcript has not been exported into a verified 28-answer raw archive; zero external judges have completed evaluations. Read the public independent review protocol and pending requirements.
- Exact 14-question English prompt ledger (CSV) — research tasks, categories and starting reference URLs.
- Initial scores (CSV) — 28 paired grades, four dimensions, recorded notes and citations.
- Workbook answer-text excerpts (Markdown) — normalized/abbreviated material, explicitly not verified verbatim.
Provenance notice: This page is a complete question-by-question public research record of the pilot, but not a fully independent raw-answer dataset. The 28 original submissions exist in the user conversation, while some archived excerpts differ in format or length. Their original session times, screenshots and public share links were not all saved. No simulated data or fabricated test sessions were used to fill those gaps.
What these results can and cannot tell us
These 14 paired tasks can illustrate source selection errors, scientific uncertainty and the need to check technical guidance. They do not identify overall population error rates for either product. Without repeated testing we cannot measure stability. Without independent blind graders we cannot confidently separate tool quality from evaluator preference. Without verified verbatim logs we cannot claim full third-party reproducibility of every original answer.
The transparent next step is to freeze raw outputs, conduct blinded evaluation and add repeat sessions as a separately versioned study. This page should be updated with those records rather than silently revising the original pilot.