AutoResearch and evaluation

Extension API. Configure optimization loops and persist evaluation evidence with explicit identities.

AutoResearchConfig dataclass

AutoResearchConfig(
    experiment_name,
    experiment_id,
    evaluator_id,
    rollout_contract_id,
    episode_config=EpisodeConfig(),
    num_episodes=10,
    parallel=False,
    max_iterations=100,
    improvement_threshold=0.0,
    destroy_forks_on_complete=False,
    record_to_ledger=True,
)

Configuration for one autoresearch loop.

experiment_id, evaluator_id, and rollout_contract_id are stable caller-provided identities used for resumption and score comparability. Higher scores are better; a candidate becomes the incumbent when it exceeds the current score by at least improvement_threshold.

Field	Type	Default
experiment_name	str	required
experiment_id	str	required
evaluator_id	str	required
rollout_contract_id	str	required
episode_config	EpisodeConfig	generated by EpisodeConfig
num_episodes	int	10
parallel	bool	False
max_iterations	int	100
improvement_threshold	float	0.0
destroy_forks_on_complete	bool	False
record_to_ledger	bool	True

AutoResearchResult dataclass

AutoResearchResult(
    experiment_name,
    iterations_completed,
    final_score,
    initial_score,
    iterations=list(),
    lab_world_id="",
)

Summarize a completed or stopped autoresearch loop.

improved property

improved

Field	Type	Default
experiment_name	str	required
iterations_completed	int	required
final_score	float	required
initial_score	float	required
iterations	list[IterationResult]	generated by list
lab_world_id	str	''

CandidateContext dataclass

CandidateContext(
    experiment_id,
    experiment_name,
    iteration,
    run_id,
    base_world_id,
)

Context passed to a candidate-preparation callback.

Field	Type	Default
experiment_id	str	required
experiment_name	str	required
iteration	int	required
run_id	str	required
base_world_id	str	required

EvaluationResult dataclass

EvaluationResult(
    score, evaluator, evidence=dict(), metadata=dict()
)

Return a score with evaluator identity and supporting evidence.

Field	Type	Default
score	float	required
evaluator	str	required
evidence	dict[str, Any]	generated by dict
metadata	dict[str, Any]	generated by dict

IterationResult dataclass

IterationResult(
    iteration,
    rollout,
    score,
    evaluation,
    improved,
    incumbent_score,
)

Result of one autoresearch iteration.

Field	Type	Default
iteration	int	required
rollout	RolloutResult	required
score	float	required
evaluation	EvaluationResult	required
improved	bool	required
incumbent_score	float	required

Outcome dataclass

Outcome(status, score=None, evidence=dict())

Represent a validated grading conclusion.

status must be pass, fail, invalid, or inconclusive. A supplied score must be finite.

Field	Type	Default
status	str	required
score	float \| None	None
evidence	dict	generated by dict

GraderContract dataclass

GraderContract(
    grader_id,
    implementation_version,
    config=dict(),
    thresholds=dict(),
    seed=None,
)

Identify the grader configuration used for a durable receipt.

Two receipts are directly comparable only when their contract digests match. Change implementation_version, configuration, thresholds, or seed whenever that comparison should no longer be valid.

digest

digest()

Field	Type	Default
grader_id	str	required
implementation_version	str	required
config	dict	generated by dict
thresholds	dict	generated by dict
seed	int \| None	None

EvalReceipt

Persist the evidence produced by one evaluation.

Receipts are historical facts rather than active simulation entities. They record what a grader concluded under a specific contract; callers decide what that conclusion means for policy or promotion.

Field	Type	Default
evaluation_id	str	''
subject_digest	str	''
contract_digest	str	''
grader_id	str	''
outcome	str	''
score	float \| None	None
graded_at_ms	int	0
evidence_json	str	'{}'