Building Trust Without Peer Review

Establishing Reproducibility Standards in Industrial Statistical Consulting

Dean Marchiori

deanmarchiori.com

2025-11-25

Play along




📊 Slides: deanmarchiori.github.io/biometrics-2025-talk

💻 GitHub Repo: github.com/deanmarchiori/biometrics-2025-talk

QR Code to slides

The Trust Problem

The Stakes

Statistical consulting in industry drives high-stakes decisions

  • Regulatory submissions
  • Product development
  • Business strategy
  • Public health interventions

The Gap

  • Academia: Peer review (imperfect but present)
  • Industry: Often no formal review mechanism
  • Result: Trust based on credentials, not verifiable process

The Question: “How do we make our work trustworthy by design, not by reputation?”

Why Peer Review Isn’t the Answer

Limitations of Traditional Peer Review

  • Time constraints (weeks to months)
  • Limited scope (reviewers rarely re-run analyses)
  • Imperfect even in academia
    • Replication crisis
    • Pressure to publish
    • Reviewer variability

In Industry

  • Client confidentiality
  • Proprietary data
  • Speed requirements
  • Cost considerations

Key message: We need something different, not just “peer review lite”

A Framework for Self-Validating Work

Three Pillars

1. Computational Reproducibility

“Can someone else get the same results from the same data?”

2. Methodological Transparency

“Can someone else understand and evaluate your approach?”

3. Defensible Documentation

“Can you defend this work 2 years from now?”

Pillar 1: Computational Reproducibility in R

Core Principles

Project Structure

  • Standardised way of organising projects
  • Keeps raw and processed data separate
  • Keeps source code organised
project/
├── data/
│   ├── raw/           # Never modified
│   └── processed/     # Scripted transformations only
├── scripts/
│   ├── 01_import.R
│   ├── 02_clean.R
│   └── 03_analyze.R
├── reports/
├── renv.lock          # Package versions
└── README.md

Question: How do you run this workflow?

make System

From the manual1:

  • The targets package is a Make-like pipeline tool for statistics and data science in R.
  • The package skips costly runtime for tasks that are already up to date, orchestrates the necessary computation with implicit parallel computing, and abstracts files as R objects.
  • If all the current output matches the current upstream code and data, then the whole pipeline is up to date, and the results are more trustworthy than otherwise.

Version Control Everything

  • Git for code (even internal projects)
  • Push everything to a remote repository
  • Pick a git flow1 and stick to it

Source: https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow

Environment Management with {renv}

  • Automatically track all dependencies and their versions using the {renv} package (Ushey and Wickham 2025)
  • Easily setup collaboration with others
# At project start
renv::init()

# Before delivering work
renv::snapshot()

# Client/reviewer recreates environment
renv::restore()

Key point: “Your analysis should be a recipe, not a magic trick”

Validation Artifacts

  • Unit tests for custom functions -> {testthat}1
  • Data validation checks -> {pointblank}2
  • Quality control outputs -> {assertthat}3

Example validation check:

The old fashioned way..

# Validate that merging didn't lose patients
stopifnot(
  nrow(merged_data) == nrow(patient_data),
  all(patient_data$id %in% merged_data$id)
)

Pillar 2: Methodological Transparency

Make Decisions Visible

Document the “Why” Not Just the “What”

  • Include insightful comments in the code
  • Better yet, include this in the long form report
# BAD
model <- lm(y ~ x1 + x2, data = df)

# GOOD
# Linear model chosen over GLM because:
# 1. Residual diagnostics show normality (see Section 2.1)
# 2. Client requires interpretable coefficients
# 3. Model comparison (AIC) showed no improvement with GLM
model <- lm(y ~ x1 + x2, data = df)

Cite your tools

  • For any package or method used- create a citation.
  • Easy to do in R1 with:
citation("pkgname")
  • Easy to include citations in Quarto with a .bib file

Assumption Documentation

Create an “Assumptions Log”

  • Statistical assumptions
  • Biological/domain assumptions
  • Data quality assumptions
  • When assumptions were checked

Pillar 3: Defensible Documentation

Documentation as Future-Proofing

The README Test

Can someone (including future you) understand this project from the README alone?

Essential README components:

## Project Overview
Brief description and business question

## Data Description
- Source and date accessed
- Key variables and units
- Known limitations

## Analysis Approach
- Methods used and why
- Software versions (see renv.lock)
- Key assumptions

## How to Reproduce
1. Clone repository
2. Run `renv::restore()`
3. Execute scripts in order
4. Compare results/output.html

## Contact
Your email for questions

Literate Programming: Quarto

  • Integrate your prose with your code using Quarto (Allaire and Dervieux 2025)
  • The final report should be generated programmatically
# Not just code comments
# Full narrative of decisions, assumptions, limitations

---
title: "Analysis Report"
author: "Your Name"
date: "`r Sys.Date()`"
output:
  html_document:
    code_folding: show
    toc: true
---

Reproducibility Report

  • Automatically generate a receipt1 when your report renders that outputs:
    • Timestamp
    • Which github repo is this from
    • Which branch and commit this version run on
    • Is the project environment controlled through {renv}
    • Full sessionInfo() output
      • R Version and Platform
      • All loaded packages and versions

AI Transparency statement

Until we have a mechanism to test for artificial intelligence, writers need a tool to maintain trust in their work.

  • Has this work been * using AI?
    • generated?
    • improved?
    • suggested?
    • corrected?

Source: Brewin (2024)

Source: https://www.linkedin.com/posts/speidel_llm-genai-science-activity-7384967753985781760-4oD6

Overcoming Barriers

Common Objections and Responses

“This takes too much time”

  • Saves time when revisions are requested
  • Prevents “analysis archaeology” later
  • Upfront cost, long-term savings

“Clients don’t value this”

  • Reframe as risk mitigation
  • Prevents costly errors
  • Many clients will value it once explained

“Our data is confidential”

  • Reproducibility ≠ making everything public
  • Focus on internal reproducibility

The Role of Community

You’re Not Alone

Open Source Tooling

  • R community’s emphasis on reproducibility
  • Growing ecosystem: renv, targets, pointblank
  • GitHub for collaboration and transparency
  • Community Peer Review: ROpenSci

Professional Standards

  • Professional association (SSA, ASA, RSS) could play an important role in establishing codes of conduct

Building Local Practice Communities

  • Internal working groups
  • Conference sessions like this one!

The Broader Impact

  • Increased trust in statistical consulting
  • Fewer errors reaching decision-makers
  • Stronger statistical practice across industry
  • Bridge between academia and industry

Key message: “We’re not just writing better code—we’re building better trust”

Summary & Call to Action

Key Takeaways

Three Pillars of Trust Without Peer Review:

  1. Computational Reproducibility - Make it replicable
    • Formal project structure and make system
    • VCS
    • Env management
  2. Methodological Transparency - Make it understandable
    • Document your decisions
    • Cite your tools
  3. Defensible Documentation - Make it future-proof
    • Literate programming
    • README
    • Reproducibility receipt

Practical R Tools

Stage Tool
Project Structure {targets}
Environment Mgmt {renv}
Project Documentation README.md
Data Documenation {pointblank}
Data Tests {pointblank}
Unit Tests {testthat}
Report writing Quarto
Version Control git

Starting Small: MVP

For your next project, commit to:

  1. Use renv for package management
  2. Write the analysis in R Markdown / Quarto
  3. Create a project README that tells the user how to run the project
  4. Use version control (even if not shared)

Lasting Wins

  • Quarto templates for reports and projects
  • Checklists for project handoff
  • “Reproducibility review” before delivery

Thank You + Contact

Contact & Connect

📧 Email: deanmarchiori@gmail.com

🌐 Website: deanmarchiori.com

💼 LinkedIn: linkedin.com/in/deanmarchiori

Resources

📊 This Talk: deanmarchiori.github.io/biometrics-2025-talk

💻 GitHub Repo: github.com/deanmarchiori/biometrics-2025-talk

QR Code to slides

Questions? Let’s connect!

References

Allaire, JJ, and Christophe Dervieux. 2025. Quarto: R Interface to ’Quarto’ Markdown Publishing System. https://github.com/quarto-dev/quarto-r.
Brewin. 2024. “Why i Wrote an AI Transparency Statement for My Book, and Think Other Authors Should Too.” The Guardian. https://www.theguardian.com/books/2024/apr/04/why-i-wrote-an-ai-transparency-statement-for-my-book-and-think-other-authors-should-too.
Iannone, Richard, Mauricio Vargas, and June Choe. 2024. Pointblank: Data Validation and Organization of Metadata for Local and Remote Tables. https://rstudio.github.io/pointblank/.
Landau, William Michael. 2021. “The Targets r Package: A Dynamic Make-Like Function-Oriented Pipeline Toolkit for Reproducibility and High-Performance Computing.” Journal of Open Source Software 6 (57): 2959. https://doi.org/10.21105/joss.02959.
Marwick, Ben, Carl Boettiger, and Lincoln Mullen. 2018. “Packaging Data Analytical Work Reproducibly Using r (and Friends).” The American Statistician 72 (1): 80–88.
Ushey, Kevin, and Hadley Wickham. 2025. Renv: Project Environments. https://rstudio.github.io/renv/.
Wickham, Hadley. 2011. “Testthat: Get Started with Testing.” The R Journal 3: 5–10. https://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf.
———. 2019. Assertthat: Easy Pre and Post Assertions.