PaperOrchestra

Open original

This article introduces PaperOrchestra, a multi-agent system that transforms raw idea summaries and experimental logs into submission-ready AI research papers using agents for outlining, plotting, literature review, writing, and refinement. It outperforms single-agent and state-of-the-art baselines on PaperWritingBench, a new benchmark of 200 CVPR and ICLR papers, in both literature review and overall manuscript quality.

  • PaperOrchestra uses a multi-agent pipeline (outline, plots, citation graph via Semantic Scholar, LaTeX writing, peer-review-style refinement) to turn raw idea summaries and experiment logs into full draft papers.
  • Tested on a new 200-paper benchmark (100 CVPR 2025, 100 ICLR 2025), it beat a monolithic LLM pipeline and AI Scientist-v2 by 50–68 points on literature-review quality and 14–38 points on overall manuscript quality in blind human evaluations by 11 AI researchers.
  • Grounding citations in real API data and using iterative self-reflection agents reduced hallucinated references and shallow reviews compared to baselines.
  • The tool is positioned as a drafting accelerator, not a replacement for authors—humans must still verify facts, originality, and ethics despite built-in flags for dubious content.

PaperOrchestra tackles the gap between raw research materials and a polished conference submission. You feed it an idea summary, an experimental log with extracted numbers, and a LaTeX template. The system then splits the work across specialized agents: one drafts a structured outline, another generates plots and diagrams, a third builds a citation graph via Semantic Scholar API, a fourth writes each section in LaTeX, and a final agent refines the draft using simulated peer-review. To test its flexibility, the authors built PaperWritingBench from 200 top-tier AI papers (100 CVPR 2025, 100 ICLR 2025). That mix forces the system to handle both double-column CVPR layouts and single-column ICLR formats. Each benchmark item includes the core methodology summary, data tables turned into logs, and venue rules—no polished prose. This setup isolates writing from experimentation, reflecting real workflows where experiments are done but drafting isn’t. In human evaluations against two baselines—a monolithic LLM pipeline and AI Scientist-v2—PaperOrchestra won literature-review quality by 50–68 percentage points and overall manuscript quality by 14–38 points. Eleven AI researchers compared blind pairs: PaperOrchestra versus each baseline and versus ground truth. The system’s API-grounded citation checks and iterative self-reflection agents cut down on hallucinations and shallow reviews. The team emphasizes that PaperOrchestra speeds up drafting rather than replacing authors. Researchers must verify facts, origins, and ethics. Built-in safeguards flag dubious citations and content, but final responsibility for accuracy and originality stays with the human user.

Ask about this article