Who this is for
- Finance brokers
- Legal firms
- Accounting firms
- Consultants
- Trades
- Clinics
The stories
A new legal AI test checks for made-up facts, and most passing answers had one
Published 8 October
Artificial Analysis, an independent firm that tests AI models, published version 1.1 of its legal benchmark on 8 October. It runs on 120 private legal tasks built by Harvey, a legal AI company: deal reports from a folder of contracts, deposition outlines, commercial lease reviews, disclosure schedules and tax compliance reviews.
The new version checks every finished document against the task's source documents for hallucinations, facts the AI made up or got wrong. A material hallucination is one that would mislead a reader on a substantive point, and the example given is a wrong contractually required date. One of those and the task scores zero.
The result: more than 60% of otherwise passing results contained a material hallucination. A pass means the model met every point on the marking sheet for that task. Muse Spark 1.3 fell from 26.7% to 8.9%. The best score after the check is 9.4% (Grok 4.7), then 8.9% (Muse Spark 1.3) and 8.6% (OpenAI's GPT-6 Astra, which barely moved). Several models scored 0%.
Google's new Gemini agent gets its own email address, calendar and Drive
Announced 8 October
At its Gemini at Work event on 8 October, Google Cloud announced a single Gemini agent, software that carries out a task step by step on its own. Google says it plans the work, uses tools, connects to your systems and "brings back something finished", working inside Gmail, Drive, Docs, Sheets, Slides, Chat and Calendar.
The change that matters for a small firm: an agent can be set up as a coworker with its own Workspace account, meaning its own email address, calendar, Drive storage and a place in the company directory. Google says it acts under its own identity rather than yours and sees only what you share with it. Every action is written to an audit trail in the agent's name, not a person's.
Two money controls. You can set a hard spending limit on a project, and the agent pauses when it hits it. And it routes work between Google's Gemini models and Anthropic's Claude models, choosing the cheapest one that does the job well.
Google's own post gives no availability or pricing. VentureBeat reports it is in private preview for a select group of customers, with wider release promised but no date, and pricing details to come.
Older OpenAI models switch off on 23 October, and some of your tools may run on them
Reminder sent 8 October, shutdown 23 October
OpenAI emailed developers on 8 October reminding them that a set of older models will be shut down in its API, the connection suppliers use to build software on OpenAI, on 23 October. The plan was first announced on 22 April.
The list includes gpt-4-turbo, gpt-4o-2024-05-13, gpt-4.1-nano, o1, o3-mini, o4-mini, gpt-3.5-turbo and the image model gpt-image-1, plus custom-trained (fine-tuned) versions of several of them. OpenAI names a newer replacement for each, mostly from its GPT-5.6 family.
ChatGPT itself is not affected. This hits software built on OpenAI: the chatbot on a website, an email sorting tool, a quote generator, an automation a freelancer set up two years ago that nobody has touched since.
Clear takeaways
- Polished is not the same as correct. The new legal test shows a well-written AI draft can still carry a wrong date, figure or clause.
- The safer agents have their own identity: their own account, only the files you share, every action logged.
- AI tools have a shelf life. The model under a tool you bought can be switched off, and your supplier has to move it in time.
Do this now
- Pick the last AI-assisted document that went to a client. Check every date, figure, name and clause reference against the source. Count the errors. That number sets your review standard.
- Write one rule into your process: whoever signs a document checks the facts against the source, not just the wording.
- Email every supplier whose tool uses AI: "Which OpenAI model does this run on, and will it be moved before 23 October?" Ask for the date they will test it.
- On 23 or 24 October, run one real enquiry through each AI tool you rely on (web chat, email sorting, quote builder) and confirm it still answers.
- If you run Google Workspace, note for later: when Gemini agents reach you, give each one its own account, share only the folders it needs, and set a spending cap before it starts.
