Define
State the learning outcome and the scientific decision before asking learners to produce code, prose, or a result.
Study portfolio
SL-006Stage 05: Analysis and reportingCompleted methods synthesis
Teaching reproducible research as a learning outcome
Research questionWhich teaching and reporting practices make reproducibility teachable, assessable, and useful to scientific learners?
SL-006 public status
- Record status
- Completed Completed 28 August 2026 as a cross-standard methods synthesis; it evaluates teaching and reporting requirements, not learner impact.
- Completed
- 28 August 2026 Public synthesis record
- Participants
- None recruited No Santaros participant data used
- Findings
- Bounded Interpretation and limits are stated below
COMPLETED EVIDENCE RECORD
A Learning Question With a Bounded Answer
This completed record asks what a research-methods lesson must make visible if reproducibility is meant to be learned rather than merely requested. We compared public guidance across education research, data stewardship, evidence synthesis, and research reporting. The result is a practical teaching frame, not a claim that any curriculum has already improved learner outcomes.
- Learning context
- Graduate methods, statistics, data science, laboratory, and research-software teaching where learners move from source material to an analysis, interpretation, and claim.
- Decision this study should support
- Give instructors a defensible baseline for teaching provenance and reproducibility, and define the outcomes a future learner study should measure directly.
Synthesis objectives
- 01
Identify requirements that recur across authoritative guidance on evidence, data, code, and reporting.
- 02
Translate those requirements into observable learner practices and assessable artifacts.
- 03
Separate a well-documented workflow from evidence that learners can reproduce or transfer it independently.
- 04
Specify the next empirical study without implying that this methods synthesis measured impact.
SYNTHESIS DESIGN
What We Reviewed
This record makes its source frame, extraction choices, interpretation rule, and completion criteria visible. It does not estimate a new learner effect.
- Record type
- Completed cross-standard methods synthesis; no learner-effect estimate
- Source frame
- IES SEER, FAIR, PRISMA 2020, Cochrane methods, and education data-sharing guidance
- Unit of comparison
- Requirement or reporting practice, mapped across source documents
- Teaching translation
- Observable learner action, artifact, or independent reproduction check
- Causal claim
- None. The synthesis defines a future evaluation target
- Completion rule
- Source matrix, teaching checklist, limits, and follow-on protocol specification reviewed
Learning sequence
Trace
Link source data, transformations, code, environment, model assistance, and human decisions in a record another learner can inspect.
Reproduce
Give an independent learner a governed package and score whether the reported result and reasoning can be reconstructed.
Transfer
Test the same practice on a new dataset or question without the original worked example or hidden support.
SYNTHESIS OUTCOMES
What the Record Separates
The table distinguishes what the reviewed sources report from the limits and future outcomes they cannot resolve. No new participant outcome was collected here.
| Outcome | Role | Operational definition | Timing |
|---|---|---|---|
| Outcome definition | Synthesis requirement | A learning or research outcome stated precisely enough to be assessed without relying on fluency or completion alone. | Before instruction or analysis |
| Provenance completeness | Synthesis requirement | Whether sources, transformations, tools, environments, and material decisions are findable and interpretable. | At artifact review |
| Independent reproduction | Future empirical outcome | Whether another learner can reconstruct the analysis and obtain an agreed result within prespecified tolerances. | In a planned learner study |
| Transfer | Future empirical outcome | Whether a learner applies the reproducibility practice to a new scientific task without the original scaffold. | In a planned learner study |
COMPLETED RECORD
Findings With Their Boundaries
These are the conclusions supported by the completed source review. They are not claims about a Santaros intervention or a universal teaching effect.
- 01
Across the reviewed guidance, a reproducible learning outcome depends on more than a final answer: sources, transformations, code, environment, and material decisions must remain inspectable.
- 02
The standards consistently separate transparent reporting from evidence of effectiveness. A complete record is a condition for evaluation, not an outcome estimate.
- 03
FAIR principles apply to data, code, tools, and workflows, while education research standards add relevance, implementation, generalizability, and uncertainty requirements.
- 04
The most defensible next step is an empirical teaching study that measures independent reproduction and transfer rather than rewarding documentation volume alone.
Limits of this record
- This record synthesizes standards and guidance; it does not analyze learner work or estimate curriculum effects.
- Source documents differ in purpose, vocabulary, and disciplinary scope, so the crosswalk is an interpretive tool rather than a new standard.
- Institutional, privacy, security, and licensing requirements may change which artifacts can be released for a specific course or project.
- Future empirical work is needed to test which teaching practices learners retain and use independently.
INTERPRETATION DISCIPLINE
An Interpretation That Keeps Uncertainty Visible
The synthesis keeps source context, design limits, and unresolved questions attached to every conclusion. It does not convert standards or one study into a general recommendation.
- 01
Map requirements across sources without treating a reporting checklist as proof of research quality or learner learning.
- 02
Distinguish documentation completeness, technical rerun success, scientific interpretation, and independent transfer.
- 03
Use the cross-standard map to define candidate outcomes and failure categories for a future controlled teaching study.
- 04
Record where privacy, licensing, security, or institutional rules constrain open release of learner or research data.
- 05
Report disagreements between guidance documents and avoid presenting one standard as a universal substitute for local review.
RESEARCH INTEGRITY
Implementation, Access, and AI Disclosure
The record must be detailed enough to audit what learners experienced, what the AI system could do, and where qualified humans remained responsible.
Implementation record
- Source documents, versions, publication dates, and access date recorded in the evidence register
- Requirement matrix linking each source principle to a candidate teaching practice
- Checklist reviewed for assessability, learner burden, and dependence on tool-specific features
- Future study outcomes separated from the completed methods synthesis findings
- Licensing, privacy, security, and governance constraints recorded for any planned artifact
Equity and access
- Reproducibility instruction should not depend on expensive software, high-end hardware, or a private AI subscription.
- Teaching materials should offer accessible text, keyboard-friendly workflows, lower-compute alternatives, and clear accommodation routes.
- A learner's documentation style or language fluency should not substitute for the target ability to trace and reproduce a scientific workflow.
AI-system disclosure
- The completed synthesis did not use an AI system to make source-inclusion or interpretation decisions.
- The future empirical study specification requires provider, model, version, prompts, tools, retrieval sources, and human approval points to be recorded.
- AI-generated code or prose is treated as a material workflow event when it changes data, analysis, or a scientific claim.
- No finding here establishes that AI assistance improves reproducibility or learning; that question remains planned work.
VALIDITY REGISTER
Main Risks and Planned Responses
These responses reduce specific risks. They do not eliminate uncertainty or guarantee that the final design will support a causal claim.
Checklist compliance mistaken for learning
Separate artifact completeness from independent reproduction and transfer outcomes.
Normative guidance mistaken for causal evidence
Label this record as a methods synthesis and define, rather than estimate, future learner outcomes.
Tool-specific provenance
Require configuration and service details while keeping the teaching principles tool-agnostic.
Open sharing conflicts
Use governed access, licensing review, and explicit explanations when data or environments cannot be public.
Hidden human judgment
Teach learners to record decisions, alternatives considered, and why a change was accepted or rejected.
STUDY GATES
Status Changes Require an Exit Record
A stage label is a public claim. The record moves forward only when its stated exit condition is documented.
Question
CompleteTeaching problem, intended decision, and non-causal scope recorded.
Source frame
CompleteAuthoritative standards and guidance selected and versioned.
Crosswalk
CompleteRequirements mapped to observable teaching practices and artifact checks.
Teaching translation
CompleteChecklist and future learner outcomes separated from normative guidance.
Reporting
CompleteMethods synthesis, limits, and next empirical study specification made public in this case file.
EVIDENCE CONTEXT
Real Sources That Inform the Protocol
These sources are the public evidence and methods guidance reviewed for this completed record. They are not Santaros Labs outputs or endorsements.
Institute of Education Sciences · Living standard
Standards for Excellence in Education Research
Connects education research to relevant student outcomes, implementation, generalizability, open science, and uncertainty.
Open sourceScientific Data · 2016
The FAIR Guiding Principles for scientific data management and stewardship
Defines findability, accessibility, interoperability, and reusability across data and the workflows that create them.
Open sourceThe BMJ · 2021
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Provides reporting guidance and explicitly distinguishes reporting completeness from assessing the conduct or quality of a review.
Open sourceCochrane · Version 6.5
Cochrane Handbook for Systematic Reviews of Interventions
Provides methods for planning reviews, collecting data, assessing bias, analyzing evidence, and interpreting certainty.
Open sourcePUBLIC RECORD
Artifacts That Keep the Record Auditable
- Completed cross-standard comparison
- Teaching checklist
- Instructor implementation prompts
- Future learner-study specification
- Correction and update log
This record can be updated when sources, corrections, or a future replication change the evidence boundary.
Next portfolio record
SL-001Scientific reasoning instruction with AI assistanceEVIDENCE COLLABORATION
Help Keep the Evidence Current
We welcome educators, learning scientists, and domain researchers who can identify a missing source, challenge an interpretation, or propose a careful replication.
Corrections and updates are welcome. A cited source does not imply endorsement by Santaros Labs.