Does a fixed cross-entropy op change LoRA SFT loss on Qwen3.5-2B?
Standard softmax and cross-entropy can zero out or invent gradients in bf16. We compare held-out SFT loss with Spindle's stock op against a fork with the fixed op.
Budget
2 GPU-h (default)
· each run plans 1.6 GPU-h, 90 min wall cap
· 2 runs
·
change
Reproducible on Spindle: yes
— Needs a Spindle fork: replace the server-side log-softmax/cross-entropy op with the paper's fixed version, then run matched chat_sl LoRA SFT on the fork and on stock Spindle.
Spec
Stock vs fixed server-side cross-entropy for LoRA SFT on no_robots
Hypothesis. Training with a numerically fixed log-softmax/cross-entropy op lowers held-out no_robots NLL by at least 0.02 nats/token versus Spindle's stock op, under otherwise identical LoRA SFT.
Method. Needs a Spindle fork: swap the server-side log-softmax/cross-entropy forward and backward for the corrected implementation from Barron, Gnanapragasam & Tschannen (arXiv 2610.31642). Read the paper first, because the linked issue does not describe the algorithm. If the paper's code is unavailable, reimplement it.
Qwen/Qwen3.5-2B
SFT
200 steps
batch 64
rank 32
no_robots
eval no_robots test split · mean per-token NLL computed with a float64 reference loss, plus max gradient error vs float64 from the unit check
1.6 GPU-h
90 min cap
Success criterion. Validated if the fixed op's held-out NLL is at least 0.02 nats/token below the baseline and the unit check shows a gradient-sum deviation in the stock op of at least 10x float64 error. A difference under 0.01 nats/token counts as no practical effect. Anything in between is inconclusive and gets a second seed per arm.
full spec
{
"base_model": "Qwen/Qwen3.5-2B",
"baseline": "The identical chat_sl command on unmodified shared Spindle (stock cross-entropy), with the same seed, data order, LoRA rank, learning rate, batch size and step count.",
"batch_size": 64,
"budget": {
"gpu_hours": 1.6,
"max_wall_minutes": 90
},
"command": "# treatment: TINKER_BASE_URL points at the forked Spindle deployment; baseline: same command against shared Spindle\npython -m tinker_cookbook.recipes.chat_sl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL dataset=no_robots learning_rate=1e-4 lora_rank=32 batch_size=64 max_steps=200 log_path=/tmp/rx-ce-fix",
"dataset": {
"hf_path": "HuggingFaceH4/no_robots",
"n_examples": 9500,
"name": "no_robots",
"split": "train (eval on test)"
},
"eval": {
"benchmark": "no_robots test split",
"metric": "mean per-token NLL computed with a float64 reference loss, plus max gradient error vs float64 from the unit check",
"n_examples": 500
},
"hypothesis": "Training with a numerically fixed log-softmax/cross-entropy op lowers held-out no_robots NLL by at least 0.02 nats/token versus Spindle's stock op, under otherwise identical LoRA SFT.",
"lora_rank": 32,
"max_seq_len": 2048,
"method": "Needs a Spindle fork: swap the server-side log-softmax/cross-entropy forward and backward for the corrected implementation from Barron, Gnanapragasam & Tschannen (arXiv 2610.31642). Read the paper first, because the linked issue does not describe the algorithm. If the paper's code is unavailable, reimplement it.\n\nBefore training, add a unit check in the fork. Run real Qwen3.5-2B bf16 logits on 8 no_robots batches through both ops and compare each against a float64 reference. Log the max per-token gradient error and the deviation of the vocab gradient sum from zero. This shows whether Softmax Collapse happens at realistic margins.\n\nThen run two LoRA SFT arms that differ only in the loss op. The baseline uses stock shared Spindle. The treatment uses the fork deployed for this run. Both arms share the same seed, data order, renderer and hyperparameters. Every 25 steps, log the train loss curve and the gradient-norm curve.\n\nHeld-out NLL for both final checkpoints is scored with the same float64-reference loss, so the eval itself is not biased by either op. The paper's KL/teacher-logit (Softmax Cancellation) case is left for a follow-up DPO or distillation check.",
"optimizer": {
"lr": 0.0001,
"name": "adam"
},
"steps": 200,
"success_criterion": "Validated if the fixed op's held-out NLL is at least 0.02 nats/token below the baseline and the unit check shows a gradient-sum deviation in the stock op of at least 10x float64 error. A difference under 0.01 nats/token counts as no practical effect. Anything in between is inconclusive and gets a second seed per arm.",
"title": "Stock vs fixed server-side cross-entropy for LoRA SFT on no_robots",
"training_type": "sft"
}Runs
| status | summary | took | backend | started by | started |
|---|---|---|---|---|---|
| Succeeded weak evidence against | Ran 148-step LoRA SFT (rank 32, Adam 1e-4, batch 64) of Qwen3.5-2B on no_robots on a Spindle fork: stock vs Barron-et-al fixed cross-entropy plus a second stock run, all starting from one init checkpoint and seeing identical batches. | 48m 42s | agentcore | — | 1d ago |
| Succeeded weak evidence against | On a throwaway Spindle fork with a per-request fixed CE (arXiv 2610.31642), matched 40-step Qwen3.5-2B rank-32 LoRA SFT arms on no_robots (cut from 200 steps to fit the budget; same init, data and hyperparameters) reached float64-scored held-out token NLL 2.094693 (stock) vs 2.094634 (fixed), a 0.00006 nats/token gap, well under the 0.01 no-effect threshold. | 1h 11m | Devin ↗ | — | 1d ago |
Source
raw text
softmax, log-softmax, and cross-entropy numerical bugs across all pytorch, jax, etc. https://arxiv.org/abs/2610.31642 [jason.mancuso] also for any KL objective [rasool] ofc as kl and cross entropy are related [aaron] Working on upstreaming some of this into PyTorch actually. Tracking integration here: https://github.com/pytorch/pytorch/issues/201986
links triage read (2)
Standard softmax, log-softmax, and cross-entropy implementations have two numerical bugs. Softmax Collapse rounds the target logit's gradient to zero when that logit is much larger than the others, so the gradients no longer sum to zero. Softmax Cancellation produces non-zero gradients even when student and teacher logits are equal. The cited paper says it has simple implementations that fix both, and the issue asks PyTorch to adopt them.
brief
method: The GitHub issue does not describe the algorithm. It only quotes the paper's abstract, which says the authors 'present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs.' The actual formulation, any custom backward pass, and the dtype handling are not stated, so the paper itself (arXiv 2610.31642) must be read before reimplementing the fix. What the issue does state: the bugs are numerical (rounding to zero, cancellation), and the issue author saw collapse in bfloat16 and worked around it by computing the loss in float32. The issue also links two possibly related PyTorch issues, #113708 and #201659, without describing them.
evidence: The issue offers no quantitative evidence: no numbers, models, datasets, hyperparameters, or baselines. The only evidence is the abstract's qualitative description of the two failure modes and an anecdote from the issue author ('I personally have encountered softmax collapse in bfloat16') who fixed it by computing the loss in float32. Whether the paper shows an effect on training outcomes is not stated here. The source is a short feature request plus a quoted abstract, so the paper's results are unseen.
reproduction: Smallest test of the core claim, with no training needed. (1) Build synthetic logits where the target logit exceeds the others by a large margin, in bf16, fp16, and fp32. Compare the autograd gradient of standard cross-entropy against a float64 reference, and check whether the gradient sums to zero across the vocabulary. (2) For distillation or KL, set student logits equal to teacher logits, or very close. Check whether the KL or cross-entropy-with-soft-targets gradient is exactly zero or only approximately so, in each dtype. (3) Use real logits from a Qwen3.5 or gpt-oss checkpoint on a few batches, where large margins are common late in SFT, to see how often and how badly the bugs occur at vocabulary scale (large vocab, bf16 logits). Only if the bugs show up in realistic conditions, run a small SFT or DPO LoRA comparison of standard vs fixed loss, with matched seeds, tracking loss curves and eval metrics. This would need the paper's reference implementation or a reimplementation of it. Hyperparameters and compute are not stated in the source; the unit test needs one GPU for minutes. What may not carry over: if Spindle computes the loss server-side with its own kernel, as in Tinker's built-in losses, users can't swap in a fixed loss unless the service supports custom loss functions. If logits are already upcast to fp32 before the loss, which the issue author's workaround implies is common, the collapse effect may largely vanish in practice. KL terms in RL (e.g. a reference-policy penalty) and DPO log-ratios would need a separate check, since the abstract speaks only of cross-entropy and student/teacher logits.
context: The idea text asks whether these numerical bugs skew training losses across PyTorch, JAX, and similar frameworks. Commenters note the issue also applies to any KL objective, since KL and cross-entropy are related. Aaron says he is working on upstreaming parts of the fix into PyTorch, and this issue tracks that work. The issue links #113708 and #201659 as possibly related but does not describe them. Modal's training team might care because SFT, DPO, and RL with KL penalties all use log-softmax over large vocabularies, often in bf16. A silent gradient error, or a spurious gradient when the policy equals the reference, could affect stability or final quality. If the fix lands upstream, it could be a free correctness improvement. It also affects whether Spindle's loss path needs a custom implementation.
caveats: The source is only a feature request quoting the abstract, so the paper's method, evidence, and any ablations are unseen. The issue gives no data on whether the bugs change downstream training quality, only that they exist numerically. The author's float32 workaround suggests the practical impact is avoidable with standard mixed-precision practice, which weakens the case for large effects. Open questions: how often the margins that trigger collapse occur in real LoRA fine-tuning, whether the fix costs extra memory or compute at large vocabularies, and whether it is compatible with fused or chunked cross-entropy kernels. The arXiv ID and the paper's contents were not verified from this source.
triage output
{
"adjacent_experiment": "",
"reproducibility_reason": "Needs a Spindle fork: replace the server-side log-softmax/cross-entropy op with the paper's fixed version, then run matched chat_sl LoRA SFT on the fork and on stock Spindle.",
"reproducible_on_spindle": true,
"sources": [
{
"brief": null,
"error": "HTTPStatusError: Client error '429 Unknown Error' for url 'https://export.arxiv.org/api/query?id_list=2610.31642'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429",
"full_text": false,
"kind": "link",
"title": "https://arxiv.org/abs/2610.31642",
"url": "https://arxiv.org/abs/2610.31642"
},
{
"brief": {
"caveats": "The source is only a feature request quoting the abstract, so the paper's method, evidence, and any ablations are unseen. The issue gives no data on whether the bugs change downstream training quality, only that they exist numerically. The author's float32 workaround suggests the practical impact is avoidable with standard mixed-precision practice, which weakens the case for large effects. Open questions: how often the margins that trigger collapse occur in real LoRA fine-tuning, whether the fix costs extra memory or compute at large vocabularies, and whether it is compatible with fused or chunked cross-entropy kernels. The arXiv ID and the paper's contents were not verified from this source.",
"claim": "Standard softmax, log-softmax, and cross-entropy implementations have two numerical bugs. Softmax Collapse rounds the target logit's gradient to zero when that logit is much larger than the others, so the gradients no longer sum to zero. Softmax Cancellation produces non-zero gradients even when student and teacher logits are equal. The cited paper says it has simple implementations that fix both, and the issue asks PyTorch to adopt them.",
"context": "The idea text asks whether these numerical bugs skew training losses across PyTorch, JAX, and similar frameworks. Commenters note the issue also applies to any KL objective, since KL and cross-entropy are related. Aaron says he is working on upstreaming parts of the fix into PyTorch, and this issue tracks that work. The issue links #113708 and #201659 as possibly related but does not describe them. Modal's training team might care because SFT, DPO, and RL with KL penalties all use log-softmax over large vocabularies, often in bf16. A silent gradient error, or a spurious gradient when the policy equals the reference, could affect stability or final quality. If the fix lands upstream, it could be a free correctness improvement. It also affects whether Spindle's loss path needs a custom implementation.",
"evidence": "The issue offers no quantitative evidence: no numbers, models, datasets, hyperparameters, or baselines. The only evidence is the abstract's qualitative description of the two failure modes and an anecdote from the issue author ('I personally have encountered softmax collapse in bfloat16') who fixed it by computing the loss in float32. Whether the paper shows an effect on training outcomes is not stated here. The source is a short feature request plus a quoted abstract, so the paper's results are unseen.",
"method": "The GitHub issue does not describe the algorithm. It only quotes the paper's abstract, which says the authors 'present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs.' The actual formulation, any custom backward pass, and the dtype handling are not stated, so the paper itself (arXiv 2610.31642) must be read before reimplementing the fix. What the issue does state: the bugs are numerical (rounding to zero, cancellation), and the issue author saw collapse in bfloat16 and worked around it by computing the loss in float32. The issue also links two possibly related PyTorch issues, #113708 and #201659, without describing them.",
"reproduction": "Smallest test of the core claim, with no training needed. (1) Build synthetic logits where the target logit exceeds the others by a large margin, in bf16, fp16, and fp32. Compare the autograd gradient of standard cross-entropy against a float64 reference, and check whether the gradient sums to zero across the vocabulary. (2) For distillation or KL, set student logits equal to teacher logits, or very close. Check whether the KL or cross-entropy-with-soft-targets gradient is exactly zero or only approximately so, in each dtype. (3) Use real logits from a Qwen3.5 or gpt-oss checkpoint on a few batches, where large margins are common late in SFT, to see how often and how badly the bugs occur at vocabulary scale (large vocab, bf16 logits). Only if the bugs show up in realistic conditions, run a small SFT or DPO LoRA comparison of standard vs fixed loss, with matched seeds, tracking loss curves and eval metrics. This would need the paper's reference implementation or a reimplementation of it. Hyperparameters and compute are not stated in the source; the unit test needs one GPU for minutes. What may not carry over: if Spindle computes the loss server-side with its own kernel, as in Tinker's built-in losses, users can't swap in a fixed loss unless the service supports custom loss functions. If logits are already upcast to fp32 before the loss, which the issue author's workaround implies is common, the collapse effect may largely vanish in practice. KL terms in RL (e.g. a reference-policy penalty) and DPO log-ratios would need a separate check, since the abstract speaks only of cross-entropy and student/teacher logits.",
"title": "A Fix for Softmax and Cross-Entropy (PyTorch feature request #201986, citing Barron, Gnanapragasam & Tschannen, arXiv 2610.31642)"
},
"error": null,
"full_text": true,
"kind": "github",
"title": "A Fix for Softmax and Cross-Entropy (PyTorch feature request #201986, citing Barron, Gnanapragasam & Tschannen, arXiv 2610.31642)",
"url": "https://github.com/pytorch/pytorch/issues/201986"
}
],
"spec": {
"base_model": "Qwen/Qwen3.5-2B",
"baseline": "The identical chat_sl command on unmodified shared Spindle (stock cross-entropy), with the same seed, data order, LoRA rank, learning rate, batch size and step count.",
"batch_size": 64,
"budget": {
"gpu_hours": 1.6,
"max_wall_minutes": 90
},
"command": "# treatment: TINKER_BASE_URL points at the forked Spindle deployment; baseline: same command against shared Spindle\npython -m tinker_cookbook.recipes.chat_sl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL dataset=no_robots learning_rate=1e-4 lora_rank=32 batch_size=64 max_steps=200 log_path=/tmp/rx-ce-fix",
"dataset": {
"hf_path": "HuggingFaceH4/no_robots",
"n_examples": 9500,
"name": "no_robots",
"split": "train (eval on test)"
},
"eval": {
"benchmark": "no_robots test split",
"metric": "mean per-token NLL computed with a float64 reference loss, plus max gradient error vs float64 from the unit check",
"n_examples": 500
},
"hypothesis": "Training with a numerically fixed log-softmax/cross-entropy op lowers held-out no_robots NLL by at least 0.02 nats/token versus Spindle's stock op, under otherwise identical LoRA SFT.",
"lora_rank": 32,
"max_seq_len": 2048,
"method": "Needs a Spindle fork: swap the server-side log-softmax/cross-entropy forward and backward for the corrected implementation from Barron, Gnanapragasam & Tschannen (arXiv 2610.31642). Read the paper first, because the linked issue does not describe the algorithm. If the paper's code is unavailable, reimplement it.\n\nBefore training, add a unit check in the fork. Run real Qwen3.5-2B bf16 logits on 8 no_robots batches through both ops and compare each against a float64 reference. Log the max per-token gradient error and the deviation of the vocab gradient sum from zero. This shows whether Softmax Collapse happens at realistic margins.\n\nThen run two LoRA SFT arms that differ only in the loss op. The baseline uses stock shared Spindle. The treatment uses the fork deployed for this run. Both arms share the same seed, data order, renderer and hyperparameters. Every 25 steps, log the train loss curve and the gradient-norm curve.\n\nHeld-out NLL for both final checkpoints is scored with the same float64-reference loss, so the eval itself is not biased by either op. The paper's KL/teacher-logit (Softmax Cancellation) case is left for a follow-up DPO or distillation check.",
"optimizer": {
"lr": 0.0001,
"name": "adam"
},
"steps": 200,
"success_criterion": "Validated if the fixed op's held-out NLL is at least 0.02 nats/token below the baseline and the unit check shows a gradient-sum deviation in the stock op of at least 10x float64 error. A difference under 0.01 nats/token counts as no practical effect. Anything in between is inconclusive and gets a second seed per arm.",
"title": "Stock vs fixed server-side cross-entropy for LoRA SFT on no_robots",
"training_type": "sft"
},
"summary": "Standard softmax and cross-entropy can zero out or invent gradients in bf16. We compare held-out SFT loss with Spindle's stock op against a fork with the fixed op.",
"tags": [
"numerics",
"sft",
"infra",
"paper",
"eval"
],
"title": "Does a fixed cross-entropy op change LoRA SFT loss on Qwen3.5-2B?",
"value_estimate": 3,
"value_rationale": "A silent loss-op bug would affect every Spindle SFT, RL and DPO user, so a clear yes or no is worth having. The bf16-to-fp32 workaround suggests the effect may be small. Triage saw only the GitHub issue, because the arXiv paper returned 429, so the fix's details are unverified."
}Comments
@prior-art · 2d ago · 22/100 · Talk to Aaron before anything else. He's already upstreaming these fixes, so he likely knows which o…
22/100 · Talk to Aaron before anything else. He's already upstreaming these fixes, so he likely knows which ops are affected and in which conditions. Then run the compute_logprobs trainer-versus-sampler mismatch check on GSM8K, naming the specific failure regimes from the paper you expect to see.
2/5 novelty: Applies an external paper's numerical-bug findings to Spindle; the audit itself is a known-method check in our stack, and the paper's claim isn't tested by us.
3/5 positioning: Cites the arXiv paper and the PyTorch issue appears in Slack, but the idea doesn't state which ops or versions Spindle uses, or what the audit adds.
1/5 duplication-risk: Aaron is already upstreaming fixes into PyTorch (issue 201986); the idea doesn't link or coordinate with that effort, so overlap is near-certain.
2/5 insight: Rationale is only that silent loss errors could affect runs. It gives no mechanism for which inputs trigger the bugs, such as extreme logits, bf16, or large vocabularies.
Prior Art rubric v1
@pre-mortem · 2d ago · 20/100 · Before spending compute, feed the paper's failing inputs through Spindle's compute_logprobs and comp…
20/100 · Before spending compute, feed the paper's failing inputs through Spindle's compute_logprobs and compare with an fp64 reference. If errors are below a pre-set tolerance, stop. Otherwise GSM8K mismatch can't tell you whether the bug exists or your setup missed it.
Dealbreaker, untestable: Swapping fixed and buggy loss ops is server-side kernel work the Tinker API can't do; the adjacent logprob-mismatch experiment can't isolate the paper's bugs.
2/5 riskiest-assumption: Unstated. The key assumption is that the paper's bugs trigger on Spindle's dtypes, vocab sizes and logit ranges; nothing tests that first. The adjacent experiment is implied only.
2/5 negative-result-value: Flat logprob mismatch on GSM8K can't separate 'no bug' from 'bug masked by trainer/sampler noise' or inputs that never hit the bad regime.
2/5 setup-sanity-checks: No known reference reproduced. Nothing like fp64 reference logprobs or the paper's failing inputs run through Spindle to prove the check can detect the bug.
1/5 kill-criteria: No stopping rule or mismatch threshold. An audit of every loss op could absorb effort indefinitely, and the original op swap isn't runnable anyway.
Pre-mortem rubric v1
@reproducibility · 2d ago · 19/100 · Replacing loss kernels isn't doable from the client side, so the core audit can't run here. Rescope …
19/100 · Replacing loss kernels isn't doable from the client side, so the core audit can't run here. Rescope to the compute_logprobs trainer-versus-sampler mismatch on a named GSM8K split with pinned seed, model, and command, and read the paper's numbers first.
Dealbreaker, closed-dependency: The core claim needs replacing server-side softmax/cross-entropy/KL kernels inside Spindle's trainer, a capability the Tinker API does not expose.
2/5 data-available: No training or eval data specified in the idea; the only hint is GSM8K rollouts in the adjacent experiment, with no hf_path, split, or rollout set named.
1/5 stock-tooling: Swapping softmax, log-softmax, cross-entropy or KL kernels is server-side work. The Tinker API cannot replace loss ops; only compute_logprobs mismatch checks fit.
1/5 rerunnable: No command, config, seed, or model version defined. The audit is infra engineering rather than a pinned experiment, so nobody could rerun it as written.
3/5 claim-evidence: An arXiv paper plus a PyTorch tracking issue (#201986) suggest numbers and upstream work, but the source doesn't confirm released code or comparable figures.
Reproducibility Auditor rubric v1
@scope-cutter · 2d ago · 34/100 · Drop the kernel-swap audit and run only the adjacent probe: one small model, about 100 fixed GSM8K r…
34/100 · Drop the kernel-swap audit and run only the adjacent probe: one small model, about 100 fixed GSM8K rollouts, compute_logprobs against sampler logprobs, with a pre-set tolerance. If mismatch tracks low-probability tokens, then escalate to infra; otherwise stop.
2/5 one-variable: No spec exists. The original swap of fixed versus current ops across softmax, log-softmax, CE and KL bundles several ops and can't be done via the API. Adjacent probe is cleaner but unspecified.
3/5 smallest-version: The adjacent compute_logprobs check on fixed GSM8K rollouts is small, but no model, rollout count, or tolerance is named. The audit framing is broad (SFT, RL, KL).
2/5 incremental-path: No ladder is spelled out. If a mismatch appears, next steps (isolate op, upstream fix, rerun) are unstated, and the PyTorch upstream issue is only a pointer.
Scope Cutter rubric v1
@reviewer-2 · 2d ago · 29/100 · As posed, this is an infra audit, not an experiment. Compute a float64 reference logprob offline on …
29/100 · As posed, this is an infra audit, not an experiment. Compute a float64 reference logprob offline on fixed rollouts, set a tolerance in advance (say 1e-3 nats/token), and compare Spindle against it, including extreme-logit cases. Trainer-vs-sampler mismatch alone cannot attribute error to softmax.
Dealbreaker, no-baseline: No fixed-op reference implementation is available through the API, so the buggy vs. corrected comparison the claim needs cannot be run.
2/5 falsifiable-hypothesis: Question 'do bugs skew losses?' has no stated threshold, such as a logprob discrepancy magnitude or loss delta, that would count as yes or no.
2/5 fair-baseline: No fixed-op reference is specified. Adjacent experiment compares trainer vs sampler logprobs, which differ in kernels anyway, so it is not a clean fixed-vs-buggy baseline.
2/5 confounds: Trainer-sampler logprob mismatch conflates numerics bugs with bf16, kernel fusion, batching, and temperature differences. Cannot isolate softmax or cross-entropy errors without swapping ops, which the API forbids.
3/5 noise: Logprob comparison on fixed rollouts is deterministic-ish, so seed noise is small, but no sample size or run-to-run nondeterminism estimate is given.
2/5 eval-validity: Mismatch on GSM8K rollouts is a weak proxy: it shows disagreement, not which side is wrong, and the bugs likely appear in edge cases (extreme logits) GSM8K won't hit.
Reviewer 2 rubric v1
@spindle-customer · 2d ago · 34/100 · Drop the paper audit; it's kernel work customers can't act on. Reframe as a compute_logprobs trainer…
34/100 · Drop the paper audit; it's kernel work customers can't act on. Reframe as a compute_logprobs trainer-vs-sampler mismatch diagnostic on GSM8K rollouts, with a number and a threshold. Customers debugging RL instability would copy that, and it tests Spindle's logprob path.
2/5 customer-action: No customer decision changes unless a bug is found; Spindle can't swap server-side ops, so the main outcome is an internal audit, not a user choice.
2/5 recipe-potential: The adjacent compute_logprobs trainer-vs-sampler mismatch on fixed GSM8K rollouts could become a small diagnostic recipe, but the original audit is a one-off.
3/5 product-feedback: compute_logprobs mismatch checks touch a less-used path and could expose logprob discrepancies, but kernel-level loss bugs sit outside the API.
3/5 story: 'Does trainer logprob match sampler logprob on GSM8K?' is clear, but tying that to the paper's softmax bugs needs a caveat.
Spindle Customer Lens rubric v1
@instrument-check · 1d ago · 31/100 · Add a one-step probe of both endpoints first, and make the unit check a hard gate: if stock bf16 sho…
31/100 · Add a one-step probe of both endpoints first, and make the unit check a hard gate: if stock bf16 shows no collapse versus float64, stop. Also sweep LR per arm, with an edge rule, since 1e-4 on one seed can't resolve 0.02 nats.
Dealbreaker, untuned-baseline: Both arms share a single untuned learning rate (1e-4), with no per-arm sweep. A 0.02-nat gap could just reflect LR sensitivity, not the op.
3/5 capability-probe: A unit check compares both ops against float64 on real logits, but no one-step probe of each arm's exact client, fork deployment and chat_sl config runs first.
1/5 tuning-parity: Both arms share one untuned lr=1e-4 with no grid, no per-arm tuning and no edge rule; 200 steps on a single seed.
3/5 riskiest-assumption: The unit check tests whether Softmax Collapse happens at realistic margins, but nothing says failure stops the run. Reimplementing a paper never read is unflagged.
Instrument Check rubric v1
@compute-hawk · 2d ago · 66/100 · Skip training runs. Generate a few hundred fixed GSM8K rollouts once, score them with compute_logpro…