Peer Review in the Agentic Era

As science enters the agentic era, our ability to generate new knowledge is accelerating across disciplines. For centuries we have transmitted such knowledge in the form of scholarly papers, articles, theses, and conference proceedings. And while the medium for knowledge transmission will continue to evolve, we still think that the right unit of abstraction in the AI era continues to be something that looks like those traditional works: there is a list of authors who take responsibility for its contents, a title and abstract that capture the essence of the result, an introduction and a conclusion, a compact representation of the unique contributions in that work, citations to other works, and enough evidence to help others test and reproduce the findings.

The academy receives and absorbs such work as it passes it through the process of peer review. Indeed an academic work that has been read, digested, and vetted by experts in our respective subfield is considered to have forded a critical hurdle in becoming a bona fide addition to the academy. In the abstract, passing successfully through peer review signals that a work is a unique contribution, understands the field in which it lands, and is likely to stand the test of time¹.

But what does peer review look like for works generated (or partially constructed) by AI? Today, humans are still asked (by journals and conferences) to review papers written almost entirely by AI. But the time required to review carefully, the sheer number of papers to review, and the expertise needed are rapidly becoming impedance-mismatched with the rate at which work is produced². A logical development, then, is the advent of AI reviewing systems, which can act as the high-velocity, high-volume counterpart to AI-accelerated knowledge generation.

While tiny nuggets of computer-assisted review have been with us for years (cf. spell and grammar checking), it’s only in the past six months or so that AI reviewer capabilities have reached a level of sophistication to be taken seriously as a legitimate alternative to human review. Products like Google’s Deep Review are the latest entrants to the market, but many others are in the market, both in open source (like OpenAIReview) and as private offerings (like Reviewer3). There’s growing evidence, too, that these kinds of review systems are more sophisticated and better at catching real issues in scientific papers than human reviewers are³.

One of the great things about human peer review is that it tends to be freeform, written in natural language, and digestible by the people it’s intended to help (authors and publishers). The downside is that even if the reviews conform to the guidelines set out by a particular journal or conference, it’s challenging to capture the quantitative outcomes of the review. AI reviewing systems are at risk of emulating the human readability of reviews at the cost of standardization.

A common language for reviews

We’re proposing the Open Canonical Review Schema (OCRS) as an expressive data model to capture both the subjective and quantitative components of peer review. A review in OCRS is a single JSON object, and its fields fall into six high-level groups (see the schema overview for the details):

1. The record. The review’s ID and version, when it was written, and its provenance: the system it came from, the original IDs, the review’s own DOI and license, and how the record was mapped.
2. Who reviewed what. One or more referees (a single reviewer or a whole panel), each human, AI, or hybrid, with an identity mode (from open to triple blind), expertise, and competing interests; the exact version of the work under review; the context of the review (its stage, round, and who requested it); and the other reviews it responds to, replaces, or summarizes, as a meta-review does.
3. Overall judgment. The reviewer’s opinion and stance, an acceptance recommendation, and how much revision is required. Metrics allow this to also come with a credence or confidence for how firmly the reviewer holds this opinion, as well as any scores (each on its own declared scale).
4. Work summary. The reviewer’s summary of the work including any paraphrased contributions or claims that the paper makes.
5. Findings. These are specific atomised strengths, weaknesses, questions, comments and requested changes. Findings can carry evidence links to parts of the paper itself, topic tags, and uncertainty expressed as probability and associated credence. Weaknesses and changes can carry a severity label, and changes can address other findings and include suggested edits. Findings can also directly evaluate claims, contributions, or check results, including ethical concerns and the availability or adequacy of code and data.
6. Checks and evidence. Reported human, AI, or tool examinations, with their procedures, execution status, outcomes, and individual results. This might include things like checking that associated code compiles and runs, ensuring all references resolve to an actual paper, or even reproducing a statistical analysis. The results of these checks can point to source materials through a common evidence-link format (e.g. to a specific page, line of code, or text span).
7. Text and documents. The review’s text as written, in parts and in order, including any metadata about what each part is or how to read its markup and math.
8. Integrity and extension. Cryptographic signatures, so a reader can check who wrote a review and that it hasn’t changed, as well as a home for any source data that has no OCRS field yet.

Two ideas do most of the work in this layout. The first is the hierarchy itself, in which each record in a review has exactly one home. For example the overall opinion does not need to restate the scores that justify it, and a weakness does not re-quote the passage of the paper it is about. A review in OCRS is a set of small records, each stated once, arranged from the most summarised (the verdict) down to the least (the paper itself).

The second idea is the links. What makes a review persuasive is not necessarily its verdict, but the chain of reasoning that supports it. One example for how this might work is an opinion pointing at the metrics that justify it, a metric pointing at the findings that support it, a requested change saying which weakness it addresses, and at the bottom, any finding, claim, or check carrying an evidence link into the source paper. Together these two things buy a great deal of expressiveness from a small schema.

For AI referees, OCRS also records the model, the execution trace, token usage, and cost, so a reader can see how an AI reviewer reached its conclusions.

Like what the Model Context Protocol (MCP) has done for the interoperability of LLMs with the rest of the world, we hope OCRS can unleash the ability for people and machines to disseminate and understand the results of review at scale.

The schema repository includes mappers that take reviews in existing formats (OpenReview, CMT, HotCRP, Editorial Manager, ScholarOne, review reports published in JATS as eLife, F1000Research and PeerJ do, plain journal narratives, and the AI reviewers above) and map them to OCRS, normalizing their particular numerical ranges to a common scale wherever the source states its scale. We’ve tried to create an expressive schema (at some level it’s a union of all the types of reviews we saw in the wild) that allows existing reviews to be mapped into OCRS and back again: written back to its own format, a review OCRS has read comes back unchanged. We test that round trip both ways for every system, including on published reviews from ICLR, NeurIPS, COLM, TMLR, eLife and F1000Research, and we list the few source fields OCRS has no place for rather than dropping them silently. We also think it’s important for people and machines to declare their level of confidence in the statements they make, so OCRS makes affordances for a credence to be stated (how firmly the reviewer holds a judgment) on a recommendation, a finding, a claim, or a score alike. We’ll write about how we’re viewing the use of credence in another post (we have already made the broader case for keeping the probabilities around in Why be uncertain?), as well as what agentic reviewing might mean for instrumenting reproducibility in science.

One way in which we are excited to use the OCRS data model is to push for a more dynamic model of the scientific paper. As it stands today, most papers are published and never change, or else the changes they underwent over the course of peer review are behind closed doors. OCRS gives us some of the pieces that can help improve this system, especially for preprints. Because weaknesses and change requests are linked upwards to overall opinions and metrics, and downwards to checks and evidence anchors, an OCRS formatted review is a small acyclic graph rather than a list of comments.

Consider, then, a new version of a paper landing. We can diff this new version against the previous reviewed version’s evidence anchors. If the previous review called for a specific table or figure to be changed, and the new version updates exactly those parts of the paper, we can walk up the graph to flag the associated weaknesses and associated metrics as potentially stale, while leaving other aspects of the review as likely unchanged. The nice side effect of this is that minimal rereview becomes cheaper. Instead of asking a reviewer to reread the whole paper, we can point them to exactly the findings whose evidence moved. We think this is roughly what peer review should look like—a “living” record attached to a “living” document.

A new version arrives

00/00

§3.2 Growth of noveltyv1 · reviewed Simulate v2 upload

ev-3

ev-1

ev-2

ev-3 ✓

opinionInteresting model; the derivation needs its symbols defined. stance: mixedmetric_refs: metric-1, metric-2downstream

metric-1 · Clarity2 of 5 supported_by: weakness-1score may be stale

metric-2 · Originality4 of 5untouched

change-1Define s̄ₙ where it first appears and state the assumption that gives κ. addresses: weakness-1possibly done

weakness-1s̄ₙ in Eq. (2) and κ in Eq. (3) are used without definition; the mean-field step is asserted. evidence: ev-1, ev-2re-read

weakness-2That each discovery adds exactly ν + 1 possibilities is asserted, not motivated. evidence: ev-3untouched

ev-1work v1 · Eq. (2)
The total average salience “weight” of the undiscovered balls isanchor edited

ev-2work v1 · Eq. (3)
Under a mean-field assumption, for sufficiently many discoveries,anchor edited

ev-3work v1 · §3.2
introduces ν + 1 new onesanchor intact

v1 reviewedThree evidence links point into the sheet. Simulate uploading version 2 to see what they do.

We’ve been developing OCRS internally at Valency for the past ten months, and we’re excited for the community to begin to kick the tires, send feedback, and contribute to the maturation of the schema. Any new data model only makes sense to curate in the long run if it’s actually used—we hope you do. The schema is released under the Apache 2.0 license and is hosted on GitHub at valency-oss/open-canonical-review-schema. We particularly welcome those working on AI reviewing systems to start working with OCRS and hopefully emitting the reviews from their systems in the OCRS format. We have an accepted presentation⁴ on OCRS at AGU26 on December 10, and we’d welcome the community’s feedback ahead of that.

If you’d like to see what a review you wrote or received would look like in OCRS, head over to the playground at ocrs.valency.io. Reviews pasted in a format OCRS has a mapper for are converted entirely in your browser, using the same mappers as the repo. Freeform reviews are mapped with the help of TypeSafe’s Jev, which classifies each passage of the review into its place in the schema, so every word in the result is the reviewer’s own.