🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-2B",
  "baseline": "The identical chat_sl command on unmodified shared Spindle (stock cross-entropy), with the same seed, data order, LoRA rank, learning rate, batch size and step count.",
  "batch_size": 64,
  "budget": {
    "gpu_hours": 1.6,
    "max_wall_minutes": 90
  },
  "command": "# treatment: TINKER_BASE_URL points at the forked Spindle deployment; baseline: same command against shared Spindle\npython -m tinker_cookbook.recipes.chat_sl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL dataset=no_robots learning_rate=1e-4 lora_rank=32 batch_size=64 max_steps=200 log_path=/tmp/rx-ce-fix",
  "dataset": {
    "hf_path": "HuggingFaceH4/no_robots",
    "n_examples": 9500,
    "name": "no_robots",
    "split": "train (eval on test)"
  },
  "eval": {
    "benchmark": "no_robots test split",
    "metric": "mean per-token NLL computed with a float64 reference loss, plus max gradient error vs float64 from the unit check",
    "n_examples": 500
  },
  "hypothesis": "Training with a numerically fixed log-softmax/cross-entropy op lowers held-out no_robots NLL by at least 0.02 nats/token versus Spindle's stock op, under otherwise identical LoRA SFT.",
  "lora_rank": 32,
  "max_seq_len": 2048,
  "method": "Needs a Spindle fork: swap the server-side log-softmax/cross-entropy forward and backward for the corrected implementation from Barron, Gnanapragasam & Tschannen (arXiv 2610.31642). Read the paper first, because the linked issue does not describe the algorithm. If the paper's code is unavailable, reimplement it.\n\nBefore training, add a unit check in the fork. Run real Qwen3.5-2B bf16 logits on 8 no_robots batches through both ops and compare each against a float64 reference. Log the max per-token gradient error and the deviation of the vocab gradient sum from zero. This shows whether Softmax Collapse happens at realistic margins.\n\nThen run two LoRA SFT arms that differ only in the loss op. The baseline uses stock shared Spindle. The treatment uses the fork deployed for this run. Both arms share the same seed, data order, renderer and hyperparameters. Every 25 steps, log the train loss curve and the gradient-norm curve.\n\nHeld-out NLL for both final checkpoints is scored with the same float64-reference loss, so the eval itself is not biased by either op. The paper's KL/teacher-logit (Softmax Cancellation) case is left for a follow-up DPO or distillation check.",
  "optimizer": {
    "lr": 0.0001,
    "name": "adam"
  },
  "steps": 200,
  "success_criterion": "Validated if the fixed op's held-out NLL is at least 0.02 nats/token below the baseline and the unit check shows a gradient-sum deviation in the stock op of at least 10x float64 error. A difference under 0.01 nats/token counts as no practical effect. Anything in between is inconclusive and gets a second seed per arm.",
  "title": "Stock vs fixed server-side cross-entropy for LoRA SFT on no_robots",
  "training_type": "sft"
}

[
  {
    "arm": "baseline",
    "step": 148,
    "uri": "tinker://5eba5b64-616b-4f91-a8ea-007b531b8ee4:train:0/weights/rx-baseline-final-run_192d61aae0"
  },
  {
    "arm": "treatment",
    "step": 148,
    "uri": "tinker://6e086d34-6bf7-4ed8-b2a5-95dc720313d1:train:0/weights/rx-treatment-final-run_192d61aae0"
  },
  {
    "arm": "baseline_rerun",
    "step": 148,
    "uri": "tinker://47ca5f0b-5f67-496a-a682-b84bfbeb5c8d:train:0/weights/rx-baseline_rerun-final-run_192d61aae0"
  },
  {
    "arm": "init",
    "step": 0,
    "uri": "tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0"
  }
]

{
  "baseline": {
    "heldout_nll_token_mean": 2.0361143335948904
  },
  "baseline_rerun": {
    "heldout_nll_token_mean": 2.0361872034823914
  },
  "treatment": {
    "heldout_nll_token_mean": 2.036132229860809
  }
}

preflight_fork_fixed_vs_stock_forward_max_abs_logprob_diff

9.537e-07

preflight_fork_stock_exact_zero_logprobs

0

preflight_base_nll_fork_stock_8ex

2.351

preflight_base_nll_shared_stock_8ex

2.351

preflight_step_seconds_treatment

2.369

unit_check_n_tokens

5,940

unit_stock_max_grad_abs_err_vs_fp64

4.434e-07

unit_fixed_max_grad_abs_err_vs_fp64

1.087e-06

unit_stock_max_abs_gradsum

5.393e-07

unit_fixed_max_abs_gradsum

4.384e-07

unit_fp64_max_abs_gradsum

4.002e-15

unit_stock_target_grad_exact_zero_tokens

0

unit_margin_max_nats

15.25

unit_margin_p99_nats

11.12

unit_tokens_margin_gt_16p6

0

unit_stock_mean_abs_gradsum

1.281e-07

unit_fixed_mean_abs_gradsum

7.903e-08

unit_stock_max_loss_abs_err_vs_fp64

1.184e-06

unit_fixed_max_loss_abs_err_vs_fp64

1.184e-06

delta_nll_baseline_minus_treatment

-1.79e-05

delta_nll_bootstrap95_lo

-0.0002544

delta_nll_bootstrap95_hi

0.0001985

noise_floor_nll_baseline_minus_rerun

-7.287e-05

• checkpoint · baseline · step 148 · tinker://5eba5b64-616b-4f91-a8ea-007b531b8ee4:train:0/weights/rx-baseline-final-run_192d61aae0
• checkpoint · treatment · step 148 · tinker://6e086d34-6bf7-4ed8-b2a5-95dc720313d1:train:0/weights/rx-treatment-final-run_192d61aae0
• checkpoint · baseline_rerun · step 148 · tinker://47ca5f0b-5f67-496a-a682-b84bfbeb5c8d:train:0/weights/rx-baseline_rerun-final-run_192d61aae0
• checkpoint · init · step 0 · tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0
• code_revision · code-revision.txt
• code_patch · fork-diff-summary.txt

1. Nov 4, 00:01 UTC · succeeded · done: baseline, treatment, baseline_rerun · Ran 148-step LoRA SFT (rank 32, Adam 1e-4, batch 64) of Qwen3.5-2B on no_robots on a Spindle fork: stock vs Barron-et-al fixed cross-entropy plus a second stock run, all starting from one init checkp…
2. Nov 3, 23:48 UTC · running · Unit check on 5940 real Qwen3.5-2B no_robots tokens: max target margin 15.25 nats (below the 16.6-nat fp32 cliff), so the stock op showed no Softmax Collapse (0 zeroed target grads; gradient-sum devi…
3. Nov 3, 23:41 UTC · running · Spindle fork spindle-fork-run_192d61aae0 deployed with the Barron et al. fixed hard-target CE behind loss_fn_config fixed_ce=1; fork-side parity preflight done, float64 unit check on real Qwen3.5-2B …

adaptation_manifests

[{'arm': 'baseline', 'pair': 'seed0', 'warmup_batches': 0, 'adaptation_batch_ids_rule': '148 ids r{64k}-{64k+63} for k=0..147 (rows of the seed-0 shuffled no_robots train split, consumed in order; full ids with batch token sha256 in train_baseline.jsonl)', 'adaptation_batch_hash': '294588d42df16f1c84329570f9f8bc845c6df05a08e99e029a6148d3f560aad1'}, {'arm': 'treatment', 'pair': 'seed0', 'warmup_batches': 0, 'adaptation_batch_ids_rule': 'same 148 ids as baseline', 'adaptation_batch_hash': '294588d42df16f1c84329570f9f8bc845c6df05a08e99e029a6148d3f560aad1'}, {'arm': 'baseline_rerun', 'pair': 'seed0', 'warmup_batches': 0, 'adaptation_batch_ids_rule': 'same 148 ids as baseline', 'adaptation_batch_hash': '294588d42df16f1c84329570f9f8bc845c6df05a08e99e029a6148d3f560aad1'}]

correctness_preflight

{'required': True, 'paths': [{'id': 'fixed_cross_entropy_fork', 'fixed_input': {'sha256': '14b34a6429df8e56c9e171c8fbca902ae2842665400742c9456edf14406664e5', 'description': 'unit_batches.json: 8 no_robots train batches x 4 examples (5940 assistant tokens), real Qwen3.5-2B bf16 logits upcast to fp32; API fixture fixed batch sha 40ae6a33d7bf92b24618629be6188f5df6106dd0bc94cc0f19e26432eb8b6e50'}, 'baseline_revision': 'modal-projects/spindle afd2f22 (stock Megatron fused_vocab_parallel_cross_entropy via Miles compute_log_probs)', 'reference_revision': 'float64 torch log_softmax; pinned treatment reference = arXiv 2610.31642v3 JAX cross_entropy listing transcribed to torch', 'implementation_revision': 'spindle fork e51e9c17448030d90915272fcdcd08e047cc591d', 'precision': 'bf16 logits upcast to fp32 for CE; reference float64', 'normalization': 'per-token NLL over weight>0 tokens; per-token logit gradients; token_mean eval', 'errors': {'baseline_fork_stock_vs_shared_stock_logprob': {'max_abs_error': 0.23758649826049805, 'reference_max_abs': 30.25650405883789, 'relative_l2_error': 0.006280671552338041, 'cosine_similarity': 0.9999802793422653}, 'baseline_stock_op_grad_vs_float64': {'max_abs_error': 4.434179126100446e-07, 'reference_max_abs': 0.9999999999999061, 'relative_l2_error': 1.1704608897813963e-07, 'cosine_similarity': None}, 'reference_fixed_vs_paper_grad': {'max_abs_error': 1.1324882507324219e-06, 'reference_max_abs': 1.0, 'relative_l2_error': 2.078577654426149e-07, 'cosine_similarity': 0.9999999999999784}, 'reference_fixed_vs_paper_loss': {'max_abs_error': 0.0, 'reference_max_abs': None, 'relative_l2_error': None, 'cosine_similarity': None}, 'reference_fixed_grad_vs_float64': {'max_abs_error': 1.0872522725469125e-06, 'reference_max_abs': 0.9999999999999061, 'relative_l2_error': 1.985197974289911e-07, 'cosine_similarity': None}, 'fork_fixed_vs_fork_stock_logprob_api': {'max_abs_error': 9.5367431640625e-07, 'reference_max_abs': None, 'relative_l2_error': None, 'cosine_similarity': None}}, 'lr_scaling': [], 'notes': 'Only the loss op changed, so LR scaling does not apply. Fork-stock vs shared-stock differences come from bf16 GPU-layout nondeterminism (1 vs 4 trainer GPUs), and the tolerance for that check was not prespecified. Reference tolerances (atol 1e-6, rtol 1e-5, min cosine 0.99999) were fixed before the unit check ran.'}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [], 'reason': 'Correctness preflight was not recorded.', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'unknown', 'reported_verdict': 'inconclusive'}

evaluation

{'version': 1, 'evaluators': {'5695f13669f67e58346a46567fb4249bdf1478867126f5edfef1b35c89d3006e': {'code_sha256': 'e84df3249ef5849e771580f32cd29ec98f0888f25ee1ec2fa14c4c04253f3597', 'prompt_template': 'tinker_cookbook NoRobotsBuilder conversation_to_datum, renderer qwen3_5, TrainOnWhat.ALL_ASSISTANT_MESSAGES, max_length 2048', 'decoding': {'temperature': 0, 'max_tokens': 0, 'stop': []}, 'parser': 'teacher-forced: Spindle fork forward(cross_entropy, loss_fn_config fixed_ce=1) logprobs; loss_eval.LossEval token_mean over weight>0 tokens', 'items': {'manifest_ref': 'items.jsonl', 'manifest_sha256': 'a8f62e48eef840aac72265117cc4e374d8ee382728eed3e9fa1bff9ec24b1083', 'note': 'ids no_robots_test_0..no_robots_test_499 = first 500 rows of the test split'}}}, 'scores': [{'arm': 'baseline', 'metric': 'heldout_nll_token_mean', 'value': 2.0361143335948904, 'evaluator_id': '5695f13669f67e58346a46567fb4249bdf1478867126f5edfef1b35c89d3006e', 'seed': 0, 'predictions_ref': None}, {'arm': 'treatment', 'metric': 'heldout_nll_token_mean', 'value': 2.036132229860809, 'evaluator_id': '5695f13669f67e58346a46567fb4249bdf1478867126f5edfef1b35c89d3006e', 'seed': 0, 'predictions_ref': None}, {'arm': 'baseline_rerun', 'metric': 'heldout_nll_token_mean', 'value': 2.0361872034823914, 'evaluator_id': '5695f13669f67e58346a46567fb4249bdf1478867126f5edfef1b35c89d3006e', 'seed': 0, 'predictions_ref': None}], 'comparisons': [{'metric': 'heldout_nll_token_mean', 'arms': ['baseline', 'treatment']}]}

evaluator_validation

{'status': 'comparable', 'issues': [], 'comparisons': [{'metric': 'heldout_nll_token_mean', 'status': 'comparable', 'arms': ['baseline', 'baseline_rerun', 'treatment'], 'evaluator_ids': ['5695f13669f67e58346a46567fb4249bdf1478867126f5edfef1b35c89d3006e'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Primary endpoint changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0.5887777777777777, 'phases': {'setup': {'wall_seconds': 83.2, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': 0.023111111111111114}, 'training': {'wall_seconds': 1974.6000000000001, 'planned_units': 444, 'planned_samples': 28416.0, 'gpu_hours': 0.5485}, 'evaluation': {'wall_seconds': 61.800000000000004, 'planned_units': 3000, 'planned_samples': 1500.0, 'gpu_hours': 0.017166666666666667}, 'recovery': {'wall_seconds': 900, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'unknown', 'enforce': False, 'reasons': ['spec has no experiment_features: generated_training_data'], 'features': {'generated_training_data': None}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['baseline', 'treatment'], 'batch_size': 64, 'max_seq_len': 2048, 'n_eval': 500, 'endpoint': 'fork, 1xH100 trainer'}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['baseline', 'treatment', 'baseline_rerun'], 'steps': 148, 'batch_size': 64, 'max_seq_len': 2048, 'n_eval': 500}, 'primary_endpoint': 'mean per-token held-out NLL on no_robots test (500)', 'source': 'preflight.json, result_*.json', 'measurements': {'setup': {'seconds': 83.2, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': 1, 'allocation_source': 'fork config trainer_gpus_per_node=1'}, 'training': {'seconds': 658.2, 'units': 148, 'planned_units': 444, 'samples': 9472, 'gpu_count': 1, 'allocation_source': 'fork config; 3 arms ran concurrently on one trainer GPU; treatment arm wall time'}, 'evaluation': {'seconds': 20.6, 'units': 1000, 'planned_units': 3000, 'samples': 500, 'gpu_count': 1, 'allocation_source': 'fork config; treatment arm, 500 items x 2 loss paths'}}, 'generation': None, 'recovery': {'seconds': 900, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793750490.612935}

feasibility_required

True

initialization_provenance

{'version': 1, 'arms': {'baseline': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'source': 'server_response', 'checkpoint_uri': 'tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0', 'load_succeeded': True, 'evidence_ref': 'preflight.json, run_baseline.log', 'before_first_update': True}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'requested': {'family': 'adam', 'lr': 0.0001, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': None, 'actual_config': None, 'source': 'declared', 'evidence_ref': 'no server response exposes the active optimizer'}], 'initial_output': {'available': True, 'batch_sha256': '40ae6a33d7bf92b24618629be6188f5df6106dd0bc94cc0f19e26432eb8b6e50', 'output_sha256': '05ae9a78bc32cbf08938312e5eefe58e83dd763911043fcc993f79340edcf9a4', 'evidence_ref': 'preflight.json', 'before_first_update': True}}, 'treatment': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'source': 'server_response', 'checkpoint_uri': 'tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0', 'load_succeeded': True, 'evidence_ref': 'preflight.json, run_treatment.log', 'before_first_update': True}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'requested': {'family': 'adam', 'lr': 0.0001, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': None, 'actual_config': None, 'source': 'declared', 'evidence_ref': 'no server response exposes the active optimizer'}], 'initial_output': {'available': True, 'batch_sha256': '40ae6a33d7bf92b24618629be6188f5df6106dd0bc94cc0f19e26432eb8b6e50', 'output_sha256': '05ae9a78bc32cbf08938312e5eefe58e83dd763911043fcc993f79340edcf9a4', 'evidence_ref': 'preflight.json', 'before_first_update': True}}, 'baseline_rerun': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'source': 'server_response', 'checkpoint_uri': 'tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://25529c1f92b86b52ef9215e36949960e/weights/rx-init-run_192d61aae0', 'load_succeeded': True, 'evidence_ref': 'preflight.json, run_baseline_rerun.log', 'before_first_update': True}, 'restore_count': 0, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'requested': {'family': 'adam', 'lr': 0.0001, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': None, 'actual_config': None, 'source': 'declared', 'evidence_ref': 'no server response exposes the active optimizer'}], 'initial_output': {'available': True, 'batch_sha256': '40ae6a33d7bf92b24618629be6188f5df6106dd0bc94cc0f19e26432eb8b6e50', 'output_sha256': '05ae9a78bc32cbf08938312e5eefe58e83dd763911043fcc993f79340edcf9a4', 'evidence_ref': 'preflight.json', 'before_first_update': True}}}}

launch

{'experiment_id': 'exp_1914726325', 'context_status': 'legacy_unknown', 'superseded': False, 'mode': 'redo', 'purpose': "Repeat the selected saved experiment version to check reproducibility. This repetition tests that version's hypothesis and methods."}

progress_updated_at

1793750467.4792688

reported_outcome_reason

tie

reported_verdict

inconclusive

restoration_validation

{'status': 'unverified', 'required': False, 'checks': [], 'reason': 'Initialization and optimizer evidence was not required by this saved spec.', 'applicability': 'unknown'}

started_at

1793747570.1732779