🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-2B",
  "baseline": "The identical chat_sl command on unmodified shared Spindle (stock cross-entropy), with the same seed, data order, LoRA rank, learning rate, batch size and step count.",
  "batch_size": 64,
  "budget": {
    "gpu_hours": 1.6,
    "max_wall_minutes": 90
  },
  "command": "# treatment: TINKER_BASE_URL points at the forked Spindle deployment; baseline: same command against shared Spindle\npython -m tinker_cookbook.recipes.chat_sl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL dataset=no_robots learning_rate=1e-4 lora_rank=32 batch_size=64 max_steps=200 log_path=/tmp/rx-ce-fix",
  "dataset": {
    "hf_path": "HuggingFaceH4/no_robots",
    "n_examples": 9500,
    "name": "no_robots",
    "split": "train (eval on test)"
  },
  "eval": {
    "benchmark": "no_robots test split",
    "metric": "mean per-token NLL computed with a float64 reference loss, plus max gradient error vs float64 from the unit check",
    "n_examples": 500
  },
  "hypothesis": "Training with a numerically fixed log-softmax/cross-entropy op lowers held-out no_robots NLL by at least 0.02 nats/token versus Spindle's stock op, under otherwise identical LoRA SFT.",
  "lora_rank": 32,
  "max_seq_len": 2048,
  "method": "Needs a Spindle fork: swap the server-side log-softmax/cross-entropy forward and backward for the corrected implementation from Barron, Gnanapragasam & Tschannen (arXiv 2610.31642). Read the paper first, because the linked issue does not describe the algorithm. If the paper's code is unavailable, reimplement it.\n\nBefore training, add a unit check in the fork. Run real Qwen3.5-2B bf16 logits on 8 no_robots batches through both ops and compare each against a float64 reference. Log the max per-token gradient error and the deviation of the vocab gradient sum from zero. This shows whether Softmax Collapse happens at realistic margins.\n\nThen run two LoRA SFT arms that differ only in the loss op. The baseline uses stock shared Spindle. The treatment uses the fork deployed for this run. Both arms share the same seed, data order, renderer and hyperparameters. Every 25 steps, log the train loss curve and the gradient-norm curve.\n\nHeld-out NLL for both final checkpoints is scored with the same float64-reference loss, so the eval itself is not biased by either op. The paper's KL/teacher-logit (Softmax Cancellation) case is left for a follow-up DPO or distillation check.",
  "optimizer": {
    "lr": 0.0001,
    "name": "adam"
  },
  "steps": 200,
  "success_criterion": "Validated if the fixed op's held-out NLL is at least 0.02 nats/token below the baseline and the unit check shows a gradient-sum deviation in the stock op of at least 10x float64 error. A difference under 0.01 nats/token counts as no practical effect. Anything in between is inconclusive and gets a second seed per arm.",
  "title": "Stock vs fixed server-side cross-entropy for LoRA SFT on no_robots",
  "training_type": "sft"
}

eval_token_nll_stock

2.095

eval_token_nll_fixed

2.095

eval_token_nll_base

2.325

eval_token_nll_stock_minus_fixed

5.919e-05

eval_weighted_nll_stock

2.029

eval_weighted_nll_fixed

2.029

eval_n_examples

500

eval_n_tokens

95,351

train_steps

40

batch_size

64

lora_rank

32

final_train_nll_stock

1.912

final_train_nll_fixed

1.913

final_grad_norm_stock

22.29

final_grad_norm_fixed

22.33

unit_n_tokens

13,067

unit_stock_bf16_max_gsum_dev

0.006864

unit_stock_fp32_max_gsum_dev

4.969e-07

unit_fixed_bf16_max_gsum_dev

0.003898

unit_ref_f64_max_gsum_dev

8.327e-16

unit_stock_bf16_max_grad_err

0.005262

unit_stock_fp32_max_grad_err

4.742e-07

unit_fixed_bf16_max_grad_err

0.01579

unit_stock_bf16_target_grad_zeroed

1,027

unit_fixed_bf16_target_grad_zeroed

0