Academy pathway 06 · self-guided beta

Testing the Reading

Separate hypothesis, point-in-time data, validation, costs, calibration, and prospective resolution so a backtest cannot smuggle in the answer.

the question

How do you try to disprove an indicator before trusting its historical chart?

The reading material and fixed self-check are available now. The project is a guided brief only. There is no upload, grading, instructor review, cohort, paid enrollment, certification, or accreditation.

Level: advanced
Lessons: 4
Delivery: Self-guided beta
Advice: None

learning outcomes

What you should be able to demonstrate

01
Identify look-ahead, survivorship, selection, multiple-testing, and execution-cost risks.
02
Design a train, validation, and untouched test process with walk-forward evaluation.
03
Report calibration, false alarms, drawdown, costs, sample size, and negative results alongside favourable outcomes.

lesson soundings

Follow each question into the instrument

Every lesson ends with an evidence practice and a source-backed Bathymark destination.

01
lesson 1
Precommit the hypothesis
What must be written before inspecting the answer?
- Define the input, transformation, event, horizon, benchmark, outcome, and invalidation before selecting the best-looking result.
- Record every tested variation so unsuccessful trials remain in the denominator.
- A reversal label that needs future candles is a retrospective label, not a live input at the focal time.
Evidence practice
Write a one-page test specification that can be timestamped before data inspection.
Open the Public LogStudy precommitted calls and immutable outcomes.
Sources and limits for this lesson
- Bathymark Public LogA small or selective sample cannot establish a persistent predictive edge.
- The Probability of Backtest OverfittingThe paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.
02
lesson 2
Point-in-time data
Could the historical test know this fact at the decision time?
- Use the source value and metadata available at the simulated time, not today's corrected or backfilled state.
- Publication, observation, effective, and ingestion times can differ.
- Asset universes, contract versions, listings, and labels can change, creating survivorship or revision bias.
Evidence practice
Audit ten candidate fields and mark the earliest time each one could have been known.
Inspect material-event timingCompare observed and effective times in stored change receipts.
Sources and limits for this lesson
- Bathymark Notice to MarinersNo recorded event does not prove that no historical or unrecognized change occurred.
- QuantConnect Time Modeling GuidanceThe implementation details are specific to QuantConnect. Bathymark adapts the point-in-time principle, not its platform.
03
lesson 3
Walk forward with real costs
Does the result survive outside the period that shaped it?
- Separate rule design, training, validation, and untouched testing.
- Repeat the process through time without tuning on the future segment.
- Include spread, slippage, fees, funding, liquidity limits, turnover, and realistic entry timing where a strategy simulation is claimed.
Evidence practice
Draw a walk-forward schedule and list every cost or market constraint that enters the simulated result.
Inspect execution constraintsConnect strategy assumptions to observed route and size limits.
Sources and limits for this lesson
- QuantConnect Learning CenterBathymark is not affiliated with QuantConnect and does not provide its execution or backtesting environment.
- QuantConnect Time Modeling GuidanceThe implementation details are specific to QuantConnect. Bathymark adapts the point-in-time principle, not its platform.
- Bathymark Exit SoundingA quote is a point-in-time simulation. It does not guarantee later execution or holder eligibility.
- The Probability of Backtest OverfittingThe paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.
04
lesson 4
Calibration and negative results
Does a stated probability occur at the promised frequency?
- A hit rate alone can hide coverage, false alarms, class imbalance, and the size of misses.
- Use Brier score and reliability bands only when a real probability was precommitted.
- Publish sample size, uncertainty, regime splits, methodology versions, and negative findings.
Evidence practice
Design a result table that makes a weak sample and a high false-alarm rate impossible to hide.
Inspect calibration recordsRead resolved and unresolved calls with their sample context.
Sources and limits for this lesson
- Bathymark Public LogA small or selective sample cannot establish a persistent predictive edge.
- The Probability of Backtest OverfittingThe paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.

private progress

Mark what you have worked through

Loading locally

This checklist stays in this browser. There is no account, streak, ranking, grading, or credential.

Precommit the hypothesisPoint-in-time dataWalk forward with real costsCalibration and negative resultsGuided project: Attempt to disprove a Bathymark indicatorDeterministic self-check

guided project · no submission

Attempt to disprove a Bathymark indicator

Specify and run a point-in-time test that reports negative evidence as prominently as positive evidence.

Brief available · review planned

Deliverables

Precommitted hypothesis, input universe, horizon, benchmark, outcome, and invalidation.
Train, validation, untouched test, and walk-forward boundaries.
Bias audit covering look-ahead, survivorship, selection, revisions, and multiple testing.
Results with sample size, uncertainty, false alarms, costs, drawdown where relevant, and negative findings.

Evidence gate

Every factual claim names its source and links to it.
Every market observation includes the observation time or source cadence.
Every number includes a unit, denominator, and calculation path where derived.
Missing inputs remain missing and do not become zero, calm, flat, or safe.
The conclusion includes a limit and a measurable condition that would change it.

Bathymark does not currently accept, grade, review, or certify this project. Keep your work locally or publish it through a tool whose terms and source rights you have reviewed.

deterministic self-check

Test the evidence boundary

Choose an answer, then check it. These fixed questions do not call a model and do not produce a grade or credential.

01A reversal label needs two later candles. When is that label first available to a live strategy?

At the central candleOnly after both later candles are knownBefore the central candle

02Why is the best result from hundreds of tested variants risky?

More variants guarantee better scienceSelection can find chance patterns unless the testing process and untouched evaluation are controlledThe best chart is always the most tradable

evidence shelf

Sources for this pathway

Each source has a narrow job. Its limit remains attached so a lesson cannot claim more than the evidence supports.

bathymark

Bathymark Exit Sounding

Reviewed 2026-07-11

Supports

Stored standardized direct-market quote observations and separately sourced primary-redemption terms.

Does not prove

A quote is a point-in-time simulation. It does not guarantee later execution or holder eligibility.

bathymark

Bathymark Notice to Mariners

Reviewed 2026-07-11

Supports

Stored material control and source-document changes after a valid comparable observation.

Does not prove

No recorded event does not prove that no historical or unrecognized change occurred.

bathymark

Bathymark Public Log

Reviewed 2026-07-11

Supports

Timestamped calls, declared invalidations, immutable resolutions, and visible unresolved history.

Does not prove

A small or selective sample cannot establish a persistent predictive edge.

official guidance

QuantConnect Learning Center

Reviewed 2026-07-11

Supports

A public model for teaching research through the same tools used to test an analytical idea.

Does not prove

Bathymark is not affiliated with QuantConnect and does not provide its execution or backtesting environment.

official guidance

QuantConnect Time Modeling Guidance

Reviewed 2026-07-11

Supports

Point-in-time handling and the need to prevent later information from entering an earlier decision.

Does not prove

The implementation details are specific to QuantConnect. Bathymark adapts the point-in-time principle, not its platform.

research

The Probability of Backtest Overfitting

Reviewed 2026-07-11

Supports

Research context for why repeated strategy selection can produce an apparently strong historical result by chance.

Does not prove

The paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.

sounding open dataReading the waterlineLive sources are loading. Nothing is filled with placeholder market data.

Academy pathway 06 · self-guided beta

Testing the Reading

Separate hypothesis, point-in-time data, validation, costs, calibration, and prospective resolution so a backtest cannot smuggle in the answer.

the question

How do you try to disprove an indicator before trusting its historical chart?

Level: advanced
Lessons: 4
Delivery: Self-guided beta
Advice: None

learning outcomes

What you should be able to demonstrate

01
Identify look-ahead, survivorship, selection, multiple-testing, and execution-cost risks.
02
Design a train, validation, and untouched test process with walk-forward evaluation.
03
Report calibration, false alarms, drawdown, costs, sample size, and negative results alongside favourable outcomes.

lesson soundings

Follow each question into the instrument

Every lesson ends with an evidence practice and a source-backed Bathymark destination.

01
lesson 1
Precommit the hypothesis
What must be written before inspecting the answer?
- Define the input, transformation, event, horizon, benchmark, outcome, and invalidation before selecting the best-looking result.
- Record every tested variation so unsuccessful trials remain in the denominator.
- A reversal label that needs future candles is a retrospective label, not a live input at the focal time.
Evidence practice
Write a one-page test specification that can be timestamped before data inspection.
Open the Public LogStudy precommitted calls and immutable outcomes.
Sources and limits for this lesson
- Bathymark Public LogA small or selective sample cannot establish a persistent predictive edge.
- The Probability of Backtest OverfittingThe paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.
02
lesson 2
Point-in-time data
Could the historical test know this fact at the decision time?
- Use the source value and metadata available at the simulated time, not today's corrected or backfilled state.
- Publication, observation, effective, and ingestion times can differ.
- Asset universes, contract versions, listings, and labels can change, creating survivorship or revision bias.
Evidence practice
Audit ten candidate fields and mark the earliest time each one could have been known.
Inspect material-event timingCompare observed and effective times in stored change receipts.
Sources and limits for this lesson
- Bathymark Notice to MarinersNo recorded event does not prove that no historical or unrecognized change occurred.
- QuantConnect Time Modeling GuidanceThe implementation details are specific to QuantConnect. Bathymark adapts the point-in-time principle, not its platform.
03
lesson 3
Walk forward with real costs
Does the result survive outside the period that shaped it?
- Separate rule design, training, validation, and untouched testing.
- Repeat the process through time without tuning on the future segment.
- Include spread, slippage, fees, funding, liquidity limits, turnover, and realistic entry timing where a strategy simulation is claimed.
Evidence practice
Draw a walk-forward schedule and list every cost or market constraint that enters the simulated result.
Inspect execution constraintsConnect strategy assumptions to observed route and size limits.
Sources and limits for this lesson
- QuantConnect Learning CenterBathymark is not affiliated with QuantConnect and does not provide its execution or backtesting environment.
- QuantConnect Time Modeling GuidanceThe implementation details are specific to QuantConnect. Bathymark adapts the point-in-time principle, not its platform.
- Bathymark Exit SoundingA quote is a point-in-time simulation. It does not guarantee later execution or holder eligibility.
- The Probability of Backtest OverfittingThe paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.
04
lesson 4
Calibration and negative results
Does a stated probability occur at the promised frequency?
- A hit rate alone can hide coverage, false alarms, class imbalance, and the size of misses.
- Use Brier score and reliability bands only when a real probability was precommitted.
- Publish sample size, uncertainty, regime splits, methodology versions, and negative findings.
Evidence practice
Design a result table that makes a weak sample and a high false-alarm rate impossible to hide.
Inspect calibration recordsRead resolved and unresolved calls with their sample context.
Sources and limits for this lesson
- Bathymark Public LogA small or selective sample cannot establish a persistent predictive edge.
- The Probability of Backtest OverfittingThe paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.

private progress

Mark what you have worked through

Loading locally

This checklist stays in this browser. There is no account, streak, ranking, grading, or credential.

Precommit the hypothesisPoint-in-time dataWalk forward with real costsCalibration and negative resultsGuided project: Attempt to disprove a Bathymark indicatorDeterministic self-check

guided project · no submission

Attempt to disprove a Bathymark indicator

Specify and run a point-in-time test that reports negative evidence as prominently as positive evidence.

Brief available · review planned

Deliverables

Precommitted hypothesis, input universe, horizon, benchmark, outcome, and invalidation.
Train, validation, untouched test, and walk-forward boundaries.
Bias audit covering look-ahead, survivorship, selection, revisions, and multiple testing.
Results with sample size, uncertainty, false alarms, costs, drawdown where relevant, and negative findings.

Evidence gate

Every factual claim names its source and links to it.
Every market observation includes the observation time or source cadence.
Every number includes a unit, denominator, and calculation path where derived.
Missing inputs remain missing and do not become zero, calm, flat, or safe.
The conclusion includes a limit and a measurable condition that would change it.

Bathymark does not currently accept, grade, review, or certify this project. Keep your work locally or publish it through a tool whose terms and source rights you have reviewed.

deterministic self-check

Test the evidence boundary

Choose an answer, then check it. These fixed questions do not call a model and do not produce a grade or credential.

01A reversal label needs two later candles. When is that label first available to a live strategy?

At the central candleOnly after both later candles are knownBefore the central candle

02Why is the best result from hundreds of tested variants risky?

More variants guarantee better scienceSelection can find chance patterns unless the testing process and untouched evaluation are controlledThe best chart is always the most tradable

evidence shelf

Sources for this pathway

Each source has a narrow job. Its limit remains attached so a lesson cannot claim more than the evidence supports.

bathymark

Bathymark Exit Sounding

Reviewed 2026-07-11

Supports

Stored standardized direct-market quote observations and separately sourced primary-redemption terms.

Does not prove

A quote is a point-in-time simulation. It does not guarantee later execution or holder eligibility.

bathymark

Bathymark Notice to Mariners

Reviewed 2026-07-11

Supports

Stored material control and source-document changes after a valid comparable observation.

Does not prove

No recorded event does not prove that no historical or unrecognized change occurred.

bathymark

Bathymark Public Log

Reviewed 2026-07-11

Supports

Timestamped calls, declared invalidations, immutable resolutions, and visible unresolved history.

Does not prove

A small or selective sample cannot establish a persistent predictive edge.

official guidance

QuantConnect Learning Center

Reviewed 2026-07-11

Supports

A public model for teaching research through the same tools used to test an analytical idea.

Does not prove

Bathymark is not affiliated with QuantConnect and does not provide its execution or backtesting environment.

official guidance

QuantConnect Time Modeling Guidance

Reviewed 2026-07-11

Supports

Point-in-time handling and the need to prevent later information from entering an earlier decision.

Does not prove

The implementation details are specific to QuantConnect. Bathymark adapts the point-in-time principle, not its platform.

research

The Probability of Backtest Overfitting

Reviewed 2026-07-11

Supports

Research context for why repeated strategy selection can produce an apparently strong historical result by chance.

Does not prove

The paper describes a statistical risk and method. It does not score a Bathymark reading unless that method is actually applied.