You are working on a research idea from the Modal Research Exchange: dimuon-lite vs AdamW, LoRA SFT on Qwen3.5-2B, no_robots held-out NLL dimuon-lite is Spindle's rebuild of Tinker's DiMuon LoRA optimizer; we compare its held-out no_robots NLL against AdamW at equal steps, with IFEval as a secondary check. Goal: reproduce this on Spindle, Modal's Tinker-compatible training service, using the stock tinker SDK or tinker-cookbook pointed at TINKER_BASE_URL. Report real metrics only. If it cannot run on Spindle, say why. Selected experiment version: exp_71911f0fb9 The saved experiment spec below defines the hypothesis, methods and success criterion. Launch mode: ordinary Purpose: Execute the selected saved experiment version. Source: https://modal.enterprise.slack.com/archives/C0BMATL1S4F/p1792028936612819 Triage: The shared Spindle exposes dimuon-lite (spindle#85) through DimuonOptimizerConfig/DimuonParams. A short tinker SDK SFT script can run DiMuon and AdamW arms on the same data and step count. Experiment spec: { "title": "dimuon-lite vs AdamW, LoRA SFT on Qwen3.5-2B, no_robots held-out NLL", "hypothesis": "With per-optimizer learning rates picked on a validation split, LoRA SFT with dimuon-lite cuts held-out no_robots token_mean NLL by at least 0.02 nats versus AdamW. Both arms use the same 100 steps, batches and seeds.", "method": "Train a rank-32 LoRA on Qwen/Qwen3.5-2B over HuggingFaceH4/no_robots, using one tinker SDK script for both arms so the only difference is the optimizer. The cookbook chat_sl recipe is AdamW-only, so running AdamW through the same script removes any recipe difference. DiMuon is selected at creation with optimizer=tinker.types.DimuonOptimizerConfig() and stepped with DimuonParams(learning_rate, beta1=0.9, grad_clip_norm=0.0). AdamW uses tinker.AdamParams(learning_rate, beta1=0.9, beta2=0.95, eps=1e-8). dimuon-lite is a recently landed reverse-engineered update rule: Polar Express orthogonalization with about 5 iterations, rank_lr_correction_exponent 0.175, Muon on LoRA B, and no beta2. Its update is checked against the trusted DiMuon reconstruction before the comparison runs. Phase 1, LR selection: each optimizer runs 40 steps at seed 0 on a 3-point LR grid fixed now (AdamW {5e-5, 1e-4, 2e-4}; DiMuon {3e-4, 1e-3, 3e-3}). Each run is scored by token_mean NLL on 500 validation items held out from the train split, and the best LR per optimizer is frozen. Phase 2, confirmation: each optimizer trains 100 steps at its frozen LR for seeds {0,1,2}. Batches are identical across arms within a seed: the same shuffled order of the remaining 9,000 train rows, batch_size 32, max_seq_len 2048. Data order and LoRA init seed are shared per seed. Primary eval is token_mean NLL on the full 500-example no_robots test split, computed from per-example summed logprobs times loss weights via forward(). Secondary, descriptive only: IFEval prompt-level strict accuracy (541 prompts, greedy, max_tokens 1024) on each final checkpoint. IFEval is dropped first if feasibility does not fit.", "base_model": "Qwen/Qwen3.5-2B", "lora_rank": 32, "training_type": "sft", "dataset": { "name": "no_robots", "hf_path": "HuggingFaceH4/no_robots", "split": "train (9000 for training, 500 held out for LR selection); test (500) for confirmation", "n_examples": 9500 }, "optimizer": { "name": "dimuon-lite", "lr": 0.001 }, "steps": 100, "batch_size": 32, "max_seq_len": 2048, "eval": { "benchmark": "no_robots test split (HuggingFaceH4/no_robots, split=test)", "metric": "held-out token_mean NLL (nats per assistant loss token)", "n_examples": 500, "aggregation": "token_mean" }, "primary_endpoint": "no_robots_test_token_mean_nll_at_step_100", "arms": [ { "name": "lrsearch_adamw_5e-5_s0", "phase": "selection", "optimizer": "adam", "lr": 5e-05, "steps": 40, "seed": 0 }, { "name": "lrsearch_adamw_1e-4_s0", "phase": "selection", "optimizer": "adam", "lr": 0.0001, "steps": 40, "seed": 0 }, { "name": "lrsearch_adamw_2e-4_s0", "phase": "selection", "optimizer": "adam", "lr": 0.0002, "steps": 40, "seed": 0 }, { "name": "lrsearch_dimuon_3e-4_s0", "phase": "selection", "optimizer": "dimuon-lite", "lr": 0.0003, "steps": 40, "seed": 0 }, { "name": "lrsearch_dimuon_1e-3_s0", "phase": "selection", "optimizer": "dimuon-lite", "lr": 0.001, "steps": 40, "seed": 0 }, { "name": "lrsearch_dimuon_3e-3_s0", "phase": "selection", "optimizer": "dimuon-lite", "lr": 0.003, "steps": 40, "seed": 0 }, { "name": "baseline_adamw_s0", "phase": "confirmation", "optimizer": "adam", "lr": "frozen from selection", "steps": 100, "seed": 0 }, { "name": "baseline_adamw_s1", "phase": "confirmation", "optimizer": "adam", "lr": "frozen from selection", "steps": 100, "seed": 1 }, { "name": "baseline_adamw_s2", "phase": "confirmation", "optimizer": "adam", "lr": "frozen from selection", "steps": 100, "seed": 2 }, { "name": "treatment_dimuon_s0", "phase": "confirmation", "optimizer": "dimuon-lite", "lr": "frozen from selection", "steps": 100, "seed": 0 }, { "name": "treatment_dimuon_s1", "phase": "confirmation", "optimizer": "dimuon-lite", "lr": "frozen from selection", "steps": 100, "seed": 1 }, { "name": "treatment_dimuon_s2", "phase": "confirmation", "optimizer": "dimuon-lite", "lr": "frozen from selection", "steps": 100, "seed": 2 } ], "baseline": "AdamW (tinker.AdamParams, beta1 0.9, beta2 0.95, eps 1e-8) through the same SDK script, with the same model, LoRA rank, init seed, batches and order, 100 steps, and eval. Its LR is chosen by the same 3-point validation search.", "experiment_features": { "custom_loss": false, "custom_optimizer": true, "paired_comparison": true, "paired_training_arms": true, "checkpoint_restore": false, "warmup_adaptation": false, "generated_training_data": false, "learned_generator": false, "heldout_loss": true, "adaptive_selection": true }, "correctness_preflight": { "required": true, "status": "unresolved: spindle and miles commits for #85/#17 and the commit of Aaron's DiMuon reconstruction must be pinned before any full sweep", "paths": [ { "id": "spindle_dimuon_lite_optimizer", "kind": "optimizer", "baseline_revision": "UNRESOLVED: pin the modal-projects/spindle commit deployed at the shared Spindle URL (post-#85). AdamW updates on this deployment must match the pre-#85 AdamW update on identical grads/state.", "reference_revision": "UNRESOLVED: pin the commit of Aaron's DiMuon reconstruction (0.09% relative update error vs Tinker) and the modal-projects/miles commit of #17", "tolerances": { "baseline_update": { "atol": 1e-06, "rtol": 0.001, "min_cosine": 0.99999 }, "reference_update": { "atol": 1e-05, "rtol": 0.01, "min_cosine": 0.999 } }, "lr_scaling": { "learning_rates": [ 0.0001, 0.001, 0.01 ], "rtol": 0.01 } } ], "notes": "Tolerances are fixed for bf16 LoRA weights with fp32 optimizer math before measuring. reference_update rtol 1e-2 leaves headroom over the reported 0.09% error. Feed the same LoRA state and gradient to the Spindle DiMuon step and to the reference for one step. With grad_clip_norm=0 and beta1 momentum seeded from the first gradient, the update should scale linearly in LR across a 100x span." }, "activation_gate": { "applicable": true, "reason": "The paired comparison only tests DiMuon if the executed optimizer really orthogonalizes LoRA B updates and both arms ran the declared configuration. Polar Express makes the first-step B update near-orthogonal (flat singular spectrum); AdamW's is not. This is measured from saved weight diffs, not SDK introspection.", "checks": [ { "metric": "executed_base_model", "measures": "model name passed to create_lora_training_client in every arm", "one_of": [ "Qwen/Qwen3.5-2B" ] }, { "metric": "executed_optimizer_step_type", "measures": "type of params object passed to optim_step in each arm (logged client-side)", "one_of": [ "DimuonParams", "AdamParams" ] }, { "metric": "dimuon_B_update_normalized_stable_rank_step1", "measures": "||dB||_F^2 / (||dB||_2^2 * r) for the first DiMuon update of LoRA B, averaged over layers, from downloaded checkpoints at step 0 and 1", "min": 0.6, "max": 1.05 }, { "metric": "batch_hash_match_across_arms", "measures": "fraction of the 100 training batches whose token-id hash is identical between paired arms within a seed", "min": 1.0, "max": 1.0 }, { "metric": "pilot_train_nll_drop_dimuon_10steps", "measures": "train token_mean NLL at step 0 minus step 10 in the DiMuon pilot at the middle LR; must be finite and positive", "min": 0.0 } ] }, "restoration_preflight": { "applicable": true, "reason": "AdamW and DiMuon arms are trained from a shared fresh-LoRA start within each seed. Before the first update, each arm calls save_weights and downloads the adapter. The SHA-256 of the LoRA tensors must match across the paired arms; B=0 at init, and A comes from the seeded init. Base-model logprobs on a fixed tokenized batch must also match. A hash mismatch makes that seed inconclusive. Optimizer type is verified by the activation-gate step-1 spectrum check, not by invented introspection.", "fixed_batch_required": true, "groups": [ { "arms": [ { "name": "baseline_adamw_s0", "optimizer": "adam", "optimizer_config": { "beta1": 0.9, "beta2": 0.95, "eps": 1e-08, "lr": "frozen" } }, { "name": "treatment_dimuon_s0", "optimizer": "dimuon-lite", "optimizer_config": { "beta1": 0.9, "grad_clip_norm": 0.0, "lr": "frozen" } } ] }, { "arms": [ { "name": "baseline_adamw_s1", "optimizer": "adam", "optimizer_config": { "beta1": 0.9, "beta2": 0.95, "eps": 1e-08, "lr": "frozen" } }, { "name": "treatment_dimuon_s1", "optimizer": "dimuon-lite", "optimizer_config": { "beta1": 0.9, "grad_clip_norm": 0.0, "lr": "frozen" } } ] }, { "arms": [ { "name": "baseline_adamw_s2", "optimizer": "adam", "optimizer_config": { "beta1": 0.9, "beta2": 0.95, "eps": 1e-08, "lr": "frozen" } }, { "name": "treatment_dimuon_s2", "optimizer": "dimuon-lite", "optimizer_config": { "beta1": 0.9, "grad_clip_norm": 0.0, "lr": "frozen" } } ] } ] }, "success_criterion": "Validated if the simultaneous 95% paired_t interval, across 3 training seeds, for (AdamW NLL \u2212 DiMuon NLL) on no_robots test token_mean NLL at step 100 lies entirely above 0.02 nats. Failed if it lies entirely below 0.02. Otherwise inconclusive. IFEval strict accuracy is reported descriptively and does not decide.", "decision_rule": { "version": 1, "estimand": "Reduction in held-out no_robots test token_mean NLL at step 100 from replacing AdamW with dimuon-lite (each at its validation-selected LR) for rank-32 LoRA SFT on Qwen3.5-2B; larger is better for dimuon-lite.", "unit": "training_seed", "seeds": [ 0, 1, 2 ], "alpha": 0.05, "method": "paired_t", "assumptions": { "independent_seed_effects": true, "identically_distributed_seed_effects": true, "normal_seed_effects": true }, "criteria": [ { "id": "primary", "metric": "no_robots_test_token_mean_nll_at_step_100", "weights": { "baseline": 1, "treatment": -1 }, "margin": 0.02 } ], "adaptive_selection": { "kind": "hyperparameter", "rationale": "Muon-family and AdamW LoRA optimizers have very different good LRs. Each gets a prespecified 3-point grid, scored only on validation items held out from train, and the choice is frozen before the single confirmation eval on the test split.", "search_item_ids": "no_robots train rows held out by seeded shuffle (seed 1234), first 500 after shuffle; never used for training", "confirmation_item_ids": "all 500 rows of no_robots split=test" } }, "budget": { "gpu_hours": 2.0, "max_wall_minutes": 150 }, "generation_audit": { "required": false, "answer_comparison": null, "min_sample_size": 0, "max_label_errors": 0, "max_eval_overlaps": 0, "reason": "fixed public dataset; no generated training data" }, "feasibility_required": true, "feasibility": { "config": null, "effective_config": { "base_model": "Qwen/Qwen3.5-2B", "arms": 12, "selection_steps": 40, "confirmation_steps": 100, "batch_size": 32, "max_seq_len": 2048, "n_eval_validation": 500, "n_eval_test": 500, "n_ifeval_secondary": 541, "planned_arm_steps": 840 }, "primary_endpoint": "no_robots_test_token_mean_nll_at_step_100", "source": null, "measurements": null, "generation": null, "recovery": null, "downsizing_order": "drop IFEval secondary, then shrink LR grid to 2 points per optimizer, then shorten selection runs to 25 steps; keep 3 seeds and the 100-step test NLL. If that still does not fit, report inconclusive." }, "command": "# Custom part: one SDK SFT loop shared by both optimizers (cookbook chat_sl is AdamW-only).\nimport os, sys, tinker, datasets, random\nfrom tinker_cookbook import renderers, tokenizer_utils\nfrom tinker_cookbook.supervised.data import conversation_to_datum\nOPT, LR, SEED, STEPS = sys.argv[1], float(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4])\nMODEL='Qwen/Qwen3.5-2B'; BS=32; MAXLEN=2048\ntok=tokenizer_utils.get_tokenizer(MODEL); rend=renderers.get_renderer(renderers.get_recommended_renderer_name(MODEL) if hasattr(renderers,'get_recommended_renderer_name') else 'qwen3', tok)\nds=datasets.load_dataset('HuggingFaceH4/no_robots'); tr=list(ds['train']); random.Random(1234).shuffle(tr)\nval, train = tr[:500], tr[500:]; random.Random(SEED).shuffle(train)\nmk=lambda r: conversation_to_datum(r['messages'], rend, MAXLEN)\nsc=tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\nkw={'optimizer': tinker.types.DimuonOptimizerConfig()} if OPT=='dimuon' else {}\ntc=sc.create_lora_training_client(base_model=MODEL, rank=32, seed=SEED, **kw)\ntc.save_weights_for_sampler(name=f'init-{OPT}-s{SEED}').result() # hash for restoration preflight\nstep_params = (lambda: tinker.types.DimuonParams(learning_rate=LR, beta1=0.9, grad_clip_norm=0.0)) if OPT=='dimuon' else (lambda: tinker.AdamParams(learning_rate=LR, beta1=0.9, beta2=0.95, eps=1e-8))\nfor s in range(STEPS):\n batch=[mk(r) for r in train[s*BS:(s+1)*BS]]\n tc.forward_backward(batch, loss_fn='cross_entropy').result(); tc.optim_step(step_params()).result()\ndef nll(rows): # token_mean from per-example totals, never batch loss:mean\n tot=0.0; n=0.0\n for i in range(0,len(rows),32):\n d=[mk(r) for r in rows[i:i+32]]; out=tc.forward(d, loss_fn='cross_entropy').result()\n for di,o in zip(d,out.loss_fn_outputs):\n w=di.loss_fn_inputs['weights'].to_numpy(); lp=o['logprobs'].to_numpy(); tot+=-(lp*w).sum(); n+=w.sum()\n return tot/n\nsplit = val if STEPS==40 else list(ds['test'])\nprint({'opt':OPT,'lr':LR,'seed':SEED,'steps':STEPS,'token_mean_nll':nll(split)})\ntc.save_weights_for_sampler(name=f'final-{OPT}-lr{LR}-s{SEED}').result()\n# run: python sft_opt.py {adam|dimuon} {40|100}", "feasibility_check": { "status": "unknown", "problems": [ "Effective configuration not recorded", "Pilot evidence source not recorded", "Representative training timing not recorded", "Representative setup timing not recorded", "Representative evaluation timing not recorded", "Positive recovery reserve not recorded" ], "wall_seconds": null, "gpu_hours": null, "known_gpu_hours": 0, "phases": {}, "generation_counts": null, "generation_applicability": { "check": "generation_audit", "status": "not_applicable", "enforce": false, "reasons": [ "spec.generation_audit opts out and no feature it covers is true" ], "features": { "generated_training_data": false } } } } How to work: - Run the experiment on Spindle with the stock tinker SDK (or tinker-cookbook) pointed at $TINKER_BASE_URL. Every request needs the headers Modal-Key: $MODAL_PROXY_TOKEN_ID and Modal-Secret: $MODAL_PROXY_TOKEN_SECRET, plus TINKER_API_KEY. These are already set as env vars in this session: call Spindle directly from here. Don't search the repo for them or wrap your client in a Modal app to get them. Never print their values. - Any Modal app you do need (a Spindle fork, a GPU eval) goes in the environment research-exchange-runs, named with run_739abc9a8b. Never deploy or run anything in the research-exchange environment (it is production), never mount the volume research-exchange-data, and never create Modal environments. - Keep the run small and inside the spec's budget. Real numbers only; never estimate or invent a metric. - Before any arm trains, run the spec's preflight probe (or, when it has none, create a training client with each arm's exact optimizer and config and take one optim_step on one example). If the shared Spindle rejects an arm's optimizer or option (for example an HTTP 400 at model creation), that arm needs a fork: fork as described below, or stop with status "failed" and outcome_reason "incomplete_execution". Never swap in a different optimizer silently. - Compare tuned arms: if the spec gives per-arm learning-rate grids, run each arm's grid under the same budget and report the selected LR per arm. A selected LR at the edge of its grid makes the verdict "inconclusive". - Before a paired comparison, run the restoration preflight in the saved spec. Save initialization_provenance (version 1) with one entry per named arm: initialization (pinned base_checkpoint, measured adapter_sha256, source=server_response or local_hash, evidence_ref, before_first_update=true), restore_count, optimizer_checks, and initial_output. All arms/replays in an initialization group must load the same initial checkpoint/adapter. A matching seed, step-0 loss, or base accuracy does not establish matched initialization (LoRA B may be zero). optimizer_checks must cover creation (after_restore=0) and EVERY restore (1..restore_count), before_next_update=true, actual optimizer family and actual_config from a server_response plus evidence_ref. Compare against each arm's requested optimizer/config. Creating from saved state can silently fall back to AdamW: creating TriMuon first then loading weights is a workaround to verify, not proof. Stop on mismatch. Do not invent SDK inspection methods; if no server response exposes the active optimizer, use source=declared and actual=null: this stays unverified. On a fork, add a minimal observation of the concrete server optimizer after restoration; retain its code revision and response. Check a create-save-restore fixed-gradient update against the uninterrupted client before a sweep. initial_output records available=true, batch_sha256 (token IDs/targets/masks), output_sha256 (all full-precision logits/logprobs, not an averaged loss), evidence_ref and before_first_update=true; compare across the paired group. If capture is unavailable, record available=false with a reason. For exact replay, require it in the spec. Retain the evidence before cleanup. Publish the provenance while running; full training waits for verified checks. - Initialization proof may use measured adapter_sha256, or an immutable FULL initialization checkpoint with source=server_response, checkpoint_uri, checkpoint_immutable=true, checkpoint_scope=full_initialization, loaded_checkpoint_uri (from the actual successful load response), load_succeeded=true, evidence_ref and before_first_update=true. Keep base_checkpoint in both proof types. A declared URI or base-model-only load does not prove matching adapters. Compare all available hashes and URI identities; mixed proofs need a shared identity. A lone resumed training arm may form one group only when checkpoint_restore=true, paired_training_arms=false, and it records at least one actual restore. Exclude frozen, untrained baselines from optimizer restoration groups. - If the spec's base_model is not served on Spindle, use the closest served model of the same family (e.g. Qwen/Qwen3.5-2B for a smaller Qwen3.5) and say so in the summary; a missing model is not a reason to stop. - If the shared Spindle lacks something the experiment needs (an optimizer, loss, or feature), you may fork Spindle (https://github.com/modal-projects/spindle), add it, and deploy your fork following its README as a throwaway Modal app named spindle-fork-run_739abc9a8b in the research-exchange-runs environment. Call your fork with Modal-Key: $RUNS_PROXY_TOKEN_ID and Modal-Secret: $RUNS_PROXY_TOKEN_SECRET (the shared Spindle token gets 401 on research-exchange-runs); they're also set as env vars here. Run every arm on the fork so the comparison is fair. Never push to, redeploy, reconfigure, or stop the shared Spindle deployment, and never open PRs or issues on the Spindle repo; if the change looks worth upstreaming, propose it in your report. Include the fork's diff and app name in your report, and when you are done stop spindle-fork-run_739abc9a8b and every app your fork deployed (Spindle deploys separate trainer, lora and inference apps; give them names containing run_739abc9a8b), with `modal app stop -e research-exchange-runs `, then check `modal app list -e research-exchange-runs` shows none of yours still deployed. Stop nothing else. - Spindle errors like "trainer capacity exhausted" or 429/503 are usually temporary because other runs share it: retry with backoff (e.g. 1, 2, 4 minutes) and update your structured output while you wait. If no comparison training step has run yet and it is still refused after about 10 minutes, don't keep waiting: release the Spindle clients this session created where the SDK lets you (yours only), then finish with "status": "failed", "verdict": "inconclusive", "outcome_reason": "incomplete_execution" and "capacity_block". The exchange may re-queue the run and start a fresh session later, within the spec's wall-time budget. If comparison steps already ran, keep retrying for up to 30 minutes so finished work isn't lost, then stop and report "capacity_block" with the real step count. - Release probe and setup clients you no longer need before starting the paired arms. - Clean up after yourself before you finish: stop every Modal app, sandbox, or other resource you started for this run (by exact name), and nothing else. Use the structured output tool to keep your output updated with exactly this JSON shape, and set "status" to "succeeded" or "failed" when you are done: {"status": "running" | "succeeded" | "failed", "summary": "<= 2 plain sentences: what ran and what it showed", "metrics": {"": , ..., "arms": {"": {"": }}}, "completed_arms": [""], "verdict": "supported" | "refuted" | "inconclusive", "outcome_reason": "supported" | "refuted" | "tie" | "mixed_criteria" | "insufficient_evidence" | "incomplete_execution", "capacity_block": {"error": "", "comparison_steps_completed": , "released_clients": true | false}} Include "capacity_block" only when Spindle capacity stopped the run. A finished result must include a valid "verdict". Finishing execution alone does not support the hypothesis. Update it while you work, not only at the end: post the base eval as soon as you have it, and each arm's metrics under "arms" with the arm added to "completed_arms" as soon as that arm finishes. Keep every number you posted earlier in later updates. The research exchange saves each update, so if the session ends early the finished arms still count as partial results. Leave "verdict" out until you are done. Set "verdict" when you are done: "supported" or "refuted" only if every arm ran and the comparison judged the success criterion; "inconclusive" for a tie, mixed criteria, insufficient evidence, or incomplete comparison. Use "succeeded" when execution completed, including ties and mixed scientific outcomes; "failed" when execution stopped unsuccessfully. Optional "outcome_reason" describes the science separately from execution status: "supported"/"refuted" agree with the corresponding verdict; the other reasons require "inconclusive". "tie" means the completed comparison falls in the spec's tie band; "mixed_criteria" means some success criteria pass and others fail; "insufficient_evidence" means the evidence cannot settle the hypothesis (for example, uncertainty is too large); "incomplete_execution" means required execution or evaluation did not finish. Report the reason only with a final result. Never infer a tie or support just because execution succeeded. Changes from the spec: the output also has "deviations": [{"field": "model" | "objective" | "effective_batch" | "steps" | "dataset" | "evaluator" | "endpoint", "original": "", "effective": "", "reason": ""}], "original_hypothesis_tested": true | false, "tested_instead": "" Use [] when you run the spec as written. effective_batch is sequences per optimizer step after accumulation. Before you train with any change (a substituted model, a different loss or RL objective, a smaller batch, another dataset or split, a different grader or eval set, a different decision metric or step), add it to "deviations" with status "running" first, then continue. A change you only report at the end is flagged as unrecorded. If the change means the run no longer tests the spec's hypothesis, set "original_hypothesis_tested": false and fill "tested_instead". Changing the objective (e.g. GRPO to REINFORCE) or the endpoint always counts as not testing it: the exchange then records the hypothesis as inconclusive and keeps your verdict for the changed experiment. Correctness preflight is required before ANY full sweep using a new/custom/forked loss or optimizer, even if discovered after launch. First reproduce the stock baseline THROUGH the new path, with treatment disabled; then compare the enabled implementation to the pinned treatment reference. Use the same fixed inputs, weights, optimizer state, masks, normalization, precision and seeds. Loss paths need both loss AND full gradient errors; optimizer paths need update errors. Similar gradient norms alone do not prove parity. Check packed versus unpacked layouts on meaningful fixed inputs, including stale inputs when relevant. Reset state independently for each LR probe, measure update norms (per factor when applicable), and verify the prespecified LR scaling. STOP before full training/sweeps if any check is missing or fails. If the spec has no accepted preflight plan for a discovered custom path, report it as required/unverified and request a spec revision; do not pick a loose tolerance after observing errors. Activation and checkpoint provenance are separate gates. Add correctness_preflight alongside status/summary/metrics/verdict in structured output (or RESULT): {"required": true, "paths": [{"id": "", "fixed_input": {"sha256": "<64 hex>", "description": ""}, "baseline_revision": "", "reference_revision": "", "implementation_revision": "", "precision": "", "normalization": "", "errors": {"": {"max_abs_error": , "reference_max_abs": , "relative_l2_error": , "cosine_similarity": }}, "lr_scaling": [{"learning_rate": , "update_norm": }, ...]}]}. For stock-only execution use {"required": false, "paths": []}. Keep probe scripts and inputs as report artifacts. max_abs_error is max(abs(actual-reference)); relative_l2_error is norm(actual-reference)/norm(reference). Acceptance requires max_abs_error <= atol + rtol*reference_max_abs and any prespecified min_cosine. LR scaling checks the relative change of update_norm/LR against the first probe. These are measured errors, not self-declared pass flags. Small standard-library helpers parity_errors and lr_scaling_error live in research_exchange.correctness; `python -m research_exchange.correctness spec.json preflight.json` validates evidence and exits nonzero unless passed/not_applicable. Gate full-sweep launch on that exit code when available. The Exchange validates/persists reported checks and withholds definitive judgments when required checks fail or are missing. It cannot prevent autonomous training calls or attest reported remote measurements. Treatment activation gate: when the spec's "activation_gate" is applicable, first run a short pilot of the treatment arm in the exact regime of the full run (same model, loss path, batch, updates per batch, lag, dtype). Measure every check's metric and record it in "activation_calibration": {"values": {"": }, "note": ""}, before any full training. Keep the spec's accepted ranges fixed. If a check is missing or outside its range, do not start full training; report the evidence and an inconclusive verdict. The exchange validates reported calibration; it cannot prevent autonomous training calls. Freeze the evaluator before scoring any arm. Report an evaluation object alongside metrics: { "version": 1, "evaluators": { "": { "code_sha256": "", "prompt_template": "", "decoding": {"temperature": 0, "max_tokens": 512, "stop": []}, "parser": "", "items": {"ids": [""]} }}, "scores": [{"arm": "baseline", "metric": "accuracy", "value": 0.8, "evaluator_id": "", "seed": 0, "predictions_ref": "predictions.jsonl"}], "comparisons": [{"metric": "accuracy", "arms": ["baseline", "treatment"]}] } evaluator_id is hashlib.sha256(json.dumps(provenance, sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False).encode()).hexdigest(). Include seed per score when applicable. For large item lists use items {"manifest_ref": "items.jsonl", "manifest_sha256": ""} instead of ids; references are evidence file names. Include evaluator_id, metric, arm and item_id in each prediction record. Keep the scored model/checkpoint separate from evaluator identity. Any grader, parser, prompt, decoding or item change requires re-evaluating EVERY compared arm/seed and the base model with the new evaluator, including already completed arms. Until then report inconclusive. Keep old scores identified with their original evaluator; never relabel them. When spec.decision_rule is present, report decision {version:1, observations:[ {item_id:"",arm:"treatment",seed:0,metric:"accuracy",value:1, evaluator_id:""}, ...]} for every arm, item and scheduled seed. For large evaluations, avoid putting all rows in structured output. Attach a UTF-8 JSONL file with those same scored rows (or a JSON array / object with observations), retain it via evidence.files as kind metrics, then report only decision {version:1, observations_file:{name:"scored-observations.jsonl",sha256:""}}. Use exactly one of observations or observations_file. The reference is a retained file name and hash, never a path or URL. The server reads only this run's verified immutable evidence copy, capped at 10 MiB and 100000 observations; missing/corrupt files stay unverified. Upload a new immutable file name/hash for each update. Keep selection metadata in the decision object. No scores, item IDs, seeds or evaluator IDs are inferred. Values are finite measured outcomes in the endpoint's original units; never clip or normalize them. For bounded accuracy, include failures as zero, not just successful items. Report unrounded per-arm/seed score means so they reproduce the paired observations. Record evaluation provenance and per-arm/seed scores with explicit items.ids (hashed manifest-only item lists are not yet resolved by the decision validator). Persist the raw predictions through evidence too. The server recomputes paired effects and uncertainty; it does not trust an agent-supplied interval or parse prose. Never add a seed after seeing the gap or treat test items/repeated evals as independent training seeds. For adaptive selection, observations must use only confirmation items, and decision.selection must contain frozen_before_confirmation:true. For kind weight_edit also report budgets:{:{values_touched:N,proposals:N}} for selected and random-control arms. Report supported/refuted only when every criterion's confidence interval settles the prespecified margin; otherwise inconclusive. Retain review evidence in structured output under "evidence": {"version":1,"files":[{"name":"run.py","kind":"script","content":""}, {"name":"effective-config.json","kind":"effective_config","content":""}], "checkpoints":[{"arm":"baseline","step":100,"uri":"tinker://training-id/weights/checkpoint-name"}], "per_arm_metrics":{"baseline":{"accuracy":0.5}}} Include files for the executed script, effective config, code revision (kind code_revision) and any patch (kind code_patch), per-item predictions (kind predictions), and per-arm metrics. Include generated_tasks when training on generated data. Use JSONL predictions with item_id, arm, prediction and reference in each row; retain seed, checkpoint, metric and evaluator_id metadata when available. Invalid outputs must remain rows. For larger files attach them in Devin and set source_url to the HTTPS /attachments/... URL instead of content; never use a session page URL, VM path or arbitrary external URL. Optional sha256 must match the original bytes. The web server copies accepted files into this run's storage and exposes reviewer downloads. Post evidence as stages finish, before cleanup. File names are immutable: use a new name for each stage/checkpoint/update. Each file is capped at 10 MiB, with 64 files / 40 MiB total per run. Never include .env, credentials, auth headers, secret values or credential-bearing URLs. Read credentials from environment variables in scripts. Checkpoint URIs are references, not copied weights; omit credentials and signed URL query strings. Missing evidence is shown as unknown or partial, never as verified. Do not invent an artifact or checkpoint that does not exist. Held-out loss: this spec's metric is a loss and asks for "token_mean". Save the module below as loss_eval.py and score every arm's held-out loss with it, on the same examples: run forward(..., "cross_entropy") and pass each output plus each example's loss weights to LossEval("token_mean").add_output. Do not average per-batch loss:mean values, and do not normalise weights per example to get a token mean. Report loss_eval.report("token_mean", {arm name: evaluator}) as "loss_eval" in the structured output, with one entry per arm. The server checks this record against the spec and treats a missing or mismatched record as inconclusive. ```python """Shared held-out loss evaluator: one definition of "mean NLL" for every arm of a comparison. Standard library only, so experiment scripts can copy this file next to themselves and `import loss_eval`. Masking: a token counts when its loss weight is > 0. Weights are only a mask; their size is ignored, so a recipe that normalises weights to sum to 1 per example gets the same numbers as one that uses 0/1 weights. Aggregations (pick one before training and report it): - token_mean: sum of NLL over every counted token / number of counted tokens. Long responses weigh more. - example_mean: mean over examples of (that example's NLL sum / its counted tokens). Every example weighs the same. Examples with no counted tokens are left out of both and reported as n_empty. Totals are kept per example, so results do not depend on how examples were split into batches. Never average per-batch `loss:mean` values: that is neither of the two. """ import math AGGREGATIONS = ("token_mean", "example_mean") DEFAULT_AGGREGATION = "token_mean" REL_TOLERANCE = 1e-6 def _floats(values): if hasattr(values, "tolist"): values = values.tolist() elif hasattr(values, "data"): values = values.data return [float(v) for v in values] def example_nll(logprobs, weights): """(NLL summed over tokens with weight > 0, number of those tokens) for one example.""" logprobs, weights = _floats(logprobs), _floats(weights) if len(logprobs) != len(weights): raise ValueError(f"{len(logprobs)} logprobs but {len(weights)} weights") counted = [lp for lp, w in zip(logprobs, weights) if w > 0] return -sum(counted), len(counted) class LossEval: """Accumulates per-example NLL totals for one arm and reports both aggregations.""" def __init__(self, aggregation=DEFAULT_AGGREGATION): if aggregation not in AGGREGATIONS: raise ValueError(f"aggregation must be one of {AGGREGATIONS}, got {aggregation!r}") self.aggregation = aggregation self.examples = [] def add(self, logprobs, weights): self.examples.append(example_nll(logprobs, weights)) def add_output(self, output, weights_per_example): """Add a Tinker forward/forward_backward output ("cross_entropy" loss) for the batch it scored.""" for item, weights in zip(output.loss_fn_outputs, weights_per_example, strict=True): self.add(item["logprobs"], weights) def record(self): counted = [(s, n) for s, n in self.examples if n > 0] nll_sum = sum(s for s, _ in counted) n_tokens = sum(n for _, n in counted) token_mean = nll_sum / n_tokens if n_tokens else None example_mean = sum(s / n for s, n in counted) / len(counted) if counted else None return { "aggregation": self.aggregation, "value": token_mean if self.aggregation == "token_mean" else example_mean, "token_mean": token_mean, "example_mean": example_mean, "nll_sum": nll_sum, "n_tokens": n_tokens, "n_examples": len(counted), "n_empty": len(self.examples) - len(counted), } def report(aggregation, arms): """The `loss_eval` record a run reports: {"aggregation", "arms": {arm name: LossEval.record()}}.""" return {"aggregation": aggregation, "arms": {name: arm.record() for name, arm in arms.items()}} def requested_aggregation(spec): """The aggregation a spec asks for: eval.aggregation when set, else read from the eval metric text, else None.""" eval_spec = (spec or {}).get("eval") or {} explicit = str(eval_spec.get("aggregation") or "").strip().lower().replace("-", "_").replace(" ", "_") if explicit in AGGREGATIONS: return explicit text = " ".join(str(eval_spec.get(k) or "") for k in ("metric", "benchmark")).lower().replace("-", " ") if any(p in text for p in ("per example", "example mean", "example weighted", "mean of example", "per sequence")): return "example_mean" if any(p in text for p in ("per token", "token mean", "token weighted", "per assistant token", "nats/token")): return "token_mean" return None def is_loss_metric(spec): eval_spec = (spec or {}).get("eval") or {} text = " ".join(str(eval_spec.get(k) or "") for k in ("metric", "benchmark")).lower() return any(word in text for word in ("nll", "loss", "perplexity", "cross entropy", "cross-entropy", "log likelihood")) def check(spec, loss_eval): """Problems that make a run's loss comparison unusable for the spec's verdict; [] when it is consistent. Checks that the reported aggregation is the one the spec asked for, that every arm used it on the same tokens and examples, and that each arm's value really is that aggregation of its own totals.""" requested = requested_aggregation(spec) if not isinstance(loss_eval, dict) or not isinstance(loss_eval.get("arms"), dict) or not loss_eval["arms"]: if requested or is_loss_metric(spec): return ["no loss_eval record: the loss aggregation behind the reported numbers is unknown"] return [] problems = [] aggregation = loss_eval.get("aggregation") if aggregation not in AGGREGATIONS: problems.append(f"loss_eval aggregation {aggregation!r} is not one of {', '.join(AGGREGATIONS)}") elif requested and aggregation != requested: problems.append(f"spec asks for {requested} but the run reported {aggregation}") counts = {} for name, arm in loss_eval["arms"].items(): if not isinstance(arm, dict): problems.append(f"arm {name}: no loss record") continue if arm.get("aggregation", aggregation) != aggregation: problems.append(f"arm {name} used {arm.get('aggregation')}, not the run's {aggregation}") problems += _arm_problems(name, arm, aggregation) counts[name] = (arm.get("n_tokens"), arm.get("n_examples")) if len(set(counts.values())) > 1: problems.append("arms were scored on different tokens or examples: " + ", ".join(f"{n} {t} tokens / {e} examples" for n, (t, e) in counts.items())) return problems def _arm_problems(name, arm, aggregation): nll_sum, n_tokens, value = arm.get("nll_sum"), arm.get("n_tokens"), arm.get("value") if not all(_number(v) for v in (nll_sum, n_tokens, value)) or n_tokens <= 0: return [f"arm {name}: needs numeric value, nll_sum and n_tokens > 0"] token_mean = nll_sum / n_tokens reported_token_mean = arm.get("token_mean", token_mean) if not _number(reported_token_mean) or not _close(reported_token_mean, token_mean): return [f"arm {name}: token_mean {reported_token_mean} is not nll_sum / n_tokens = {token_mean:.6g}"] if aggregation == "token_mean" and not _close(value, token_mean): return [f"arm {name}: value {value} is not the token mean {token_mean:.6g} (nll_sum / n_tokens)"] if aggregation == "example_mean" and _number(arm.get("example_mean")) and not _close(value, arm["example_mean"]): return [f"arm {name}: value {value} is not its example_mean {arm['example_mean']}"] return [] def _number(value): return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) def _close(a, b): return math.isclose(a, b, rel_tol=REL_TOLERANCE, abs_tol=1e-12) ``` Feasibility is a measured projection, separate from scientific metrics and actual GPU usage. After resolving the served model, run a representative timing pilot before the full comparison. Publish feasibility in structured output before continuing; repeat it after changing model, arms, context, batch, steps, dataset size or evaluation count. Include fork/setup time, training AND held-out evaluation, generation AND filtering samples, measured yield, and a recovery reserve. Never scale timing by parameter count or assume concurrency is free. Measure the actual schedule. Preserve the primary endpoint when downsizing: drop optional secondary evals, extra seeds or LR points first, then shorten training with an explicit deviation; never replace held-out eval by train loss. If the primary comparison cannot fit, stop and report inconclusive. Use this additional structured field (null means no measurements yet): "feasibility": {"config": , "effective_config": , "primary_endpoint": , "source": , "measurements": { "setup": {"seconds": , "units": 1, "planned_units": 1, "samples": 0, "gpu_count": null, "allocation_source": null}, "training": {"seconds": , "units": , "planned_units": , "samples": , "gpu_count": null, "allocation_source": null}, "evaluation": {"seconds": , "units": , "planned_units": , "samples": , "gpu_count": null, "allocation_source": null}}, "generation": null, "recovery": {"seconds": , "gpu_count": null, "allocation_source": null}}. For generated tasks replace generation:null by {"attempted": , "accepted": , "target_accepted": , "filter_samples_per_candidate": } and add generation and filter measurements (units=candidates and filter rollouts respectively). Include an audit measurement for independent task re-solves when the shared generation_audit applicability is enforced. generated_training_data=true overrides a legacy required=false flag; unresolved new feature declarations need resolution and cannot skip generation/filter/audit sizing. Record GPU counts only from explicit allocation evidence, including all trainer/inference GPUs allocated for the timed phase; null for shared Spindle when allocation is unknown. CPU-only phases may explicitly use 0. Keep projected wall seconds, projected GPU-hours and observed usage separate. Do not invent timings or GPU-hours to fill this contract. Resolve applicability with experiment_features.check_applicability(spec,"generation_audit"). An explicit generated_training_data=true requires the audit. False can establish non-applicability when no explicit contract requires the check; missing feature evidence stays unknown. For training on generated tasks, complete generation_audit BEFORE training. Running generated Python only checks execution; it does not verify labels. Save all candidates as JSONL in evidence.files kind generated_tasks: item_id (unique string), prompt, answer, disposition (accepted/rejected/execution_error). Retain generated code and execution errors when available. Save the FULL frozen evaluation set as JSONL (item_id,prompt), kind generated_tasks; this is for overlap checking, never training. Freeze both files before sampling. Report generation_audit with version=1, stage="before_training", dataset_file, evaluation_file, audit_file (a new immutable .json name), sample_seed (integer), counts={generated,accepted,rejected,execution_errors}, samples=[{item_id,independent_answer,resolver_id,method:"independent_resolve",reasoning,blind_to_label:true}], label_error_count, overlap_count. Select min(accepted_count, spec.generation_audit.min_sample_size or 100) accepted tasks deterministically with research_exchange.generation_audit.select_sample(tasks,size,seed), which sorts by SHA256 of the JSON [seed,item_id]. Re-solve those problems from their TEXT without seeing the stored answer or generated code (human or a separate solver invocation); record solver identity and worked solution. Never reuse the generator's Python as independent evidence. Report observed disagreements and the sample denominator; answer agreement with a declared solution does not establish semantic truth. New specs must explicitly set generation_audit.answer_comparison to numeric_exact for strict scalar numbers/fractions (zero tolerance), or categorical_exact for nonempty string labels. Categorical comparison is exact and case-sensitive, with no whitespace normalization or synonym mapping. Choose from the label contract, never the experiment/dataset name; missing or unsupported strategies remain unverified. Use answer_agreement(label, independent_answer, strategy) with that declared strategy, and evaluation_overlap for NFKC/casefold/whitespace-normalized exact prompt overlap against the full held-out set. This does not detect paraphrases or semantic leakage. Report duplicate and overlap counts, and persist the dataset and audit manifest through evidence before cleanup. Failed/missing audits block a definitive scientific verdict. Call generation_audit.validate(report, spec.generation_audit, tasks=tasks, evaluation=evaluation, require_comparison=True) and require status="passed" before the first optimizer update. Do not invent passing re-solve evidence to get past this check. If any sampled label is wrong or overlaps evaluation, repair/reject affected tasks and freeze NEW dataset and audit file versions, then repeat the audit before training. Do not train on known bad labels as-is.