Moonlit benchmark finds 81% of AI research answers were verifiable

Vulture Labs says internal tests of Moonlit, its paid research service, found 4,280 of 5,283 AI-generated answers could be verified across 187 synthetic-email runs. The company says the results show promise, but they are controlled development data and not proof of live participant performance or accuracy at scale.

Why it matters: - Vulture Labs is testing whether AI assistants can reliably support paid research tasks by grounding answers in evidence. - The benchmark points to a possible path for faster research collection, but the company says real-world validation is still needed before the results can be treated as market performance. - The findings matter for buyers because Moonlit is designed to deliver purchase evidence from participants more quickly than traditional research workflows.

What happened: - Vulture Labs reported an internal benchmark for Moonlit, its service for paid research prepared by people's own AI assistants. - The company ran 187 tests using synthetic email data. - Across those runs, 4,280 of 5,283 answers were classified as verifiable, or about 81%. - The tests covered four models and up to 40 questions per run. - Reported performance ranged from 70% to 94% by model.

The details: - The benchmark used controlled development data, not real participants' inboxes. - The results do not establish an 81% payment rate, a population coverage figure or a guarantee of accuracy. - The dataset was synthetic and in English. - Counting tasks performed worse than the broader set, with reported results of 62% to 75%. - The company reported privacy-rule compliance on 916 of 935 opportunities. - Three runs disclosed personal information. - Thirteen instruction variations were explored, usually with two to five runs per condition. - Researchers can request the benchmark methodology and discuss a pilot at More information. - Participants can review the instructions at Participant instructions.

Between the lines: - The benchmark suggests Moonlit can extract useful answers from AI-assisted research workflows, but the test design also shows clear limits around privacy, counting tasks and model variation. - The company built safeguards into the product flow by asking for the basis of each answer and requiring participants to review the draft and reward before submission. - Moonlit also directs the assistant to distinguish records from estimates and to say when it cannot tell. - The emphasis on controlled testing signals an early-stage product that still needs outside validation before buyers can judge reliability at scale.

What's next: - Vulture Labs says the next step is to validate Moonlit in real-world research settings. - The company says its one-day research cycle remains a target. - Published panel and validation results in live use are still needed to assess performance at scale. - Companies can request a study at Request a study. - People can connect an existing AI assistant or set up a supported one through Setup guide.

The bottom line: - Moonlit's internal benchmark suggests AI-assisted research can produce a high share of verifiable answers, but the result is still a development signal, not proof of durable real-world performance.