ReproCrowd

Crowdsourcing AI reproductions

With our infrastructure, you can run a check for numerical reproducibility within minutes. Check it, sign it with your name, and upload it for the entire community to find it.

ReproCrowd hex sticker

What is ReproCrowd?

Many researchers already attempt to reproduce the results of papers in their field, but in an unsystematic way. We provide a standardized prompt, a way for you to sign and submit results, and the means to disseminate your reports in a human- and machine-friendly way.

  1. 1

    Choose an article

    Pick a paper with openly available data and code from your field. Copy its DOI.

  2. 2

    Run the reproducibility check

    Go to the ReproCrowd app and complete the process.

  3. 3

    Submit & get signed

    Your agent submits a standardized, machine- and human-readable report to a GitHub repo, where a human reviews and signs it.

  4. 4

    Share & compare

    The report lands in FLoRA with an AI tag, and anyone can re-run your reprocheck or comment. Reports are comparable because the engine and model are recorded.

How a reproduction is made

The end-to-end flow: from a paper someone wants to check, through acquiring and parsing the full text, to tagging targets, generating code, and publishing the reproduction. Annotations and open questions are omitted here for clarity. 

flowchart TD
    A[Someone uses AI to run a paper]
    B[User enters DOI]
    C{Already in our DB?}
    D[Inform user]
    E{Wants a new repro anyway?}
    F[Initialize local project store]
    G[Acquire metadata - OpenAlex]
    H{Open access?}
    I{Can user get free text?}
    J[Acquire full text - pdf, xml, ...]
    K[Parse text - IDs for section / paragraph / sentence, figures, tables]
    L[User adds files - data, code]
    M[Local store has metadata + full parsed text]
    N{User satisfied with state?}
    O[User updates metadata / text]
    P[User tags content as procedures and results]
    Q[Parse tagged content into structured targets - LLM assisted]
    R[Local store has structured targets]
    S[User adds / edits / removes targets]
    T[User sets code generation options]
    U[Code generation - LLM assisted]
    V[User edits code directly]
    W[Run code, generate report]
    X{User approves?}
    Y[Publish to DB]
    Z[END]

    A --> B
    B --> C
    C -->|No| F
    C -->|Yes| D
    D --> E
    E -->|No| Z
    E -->|Yes| F
    F --> G
    G --> H
    H -->|Yes| J
    H -->|No| I
    I -->|Yes| J
    I -->|No| Z
    J --> K
    K --> M
    L --> M
    M --> N
    N -->|No| O
    O --> M
    N -->|Yes| P
    P --> Q
    Q --> R
    S --> R
    R --> T
    T --> U
    U --> W
    V --> U
    W --> X
    X -->|No| V
    X -->|Yes| Y