Can Spindle LoRA match Baseten's Mammoth decision models?
Kailash proposes a LoRA-trained decision model on about 4 B200 nodes to rival Baseten's Mammoth-1 and Elephant-1, but neither their data nor eval is named.
Budget
2 GPU-h (default)
· 0 runs
·
change
Reproducible on Spindle: no
— The thread names no public training dataset or eval for Mammoth-1/Elephant-1, so we lack the data to train or compare. The BERT side idea needs an encoder model Spindle doesn't train.
Spec
Runs
Source
raw text
https://labs.baseten.co/articles/mammoth-1-and-elephant-1-frontier-decision-models what if we used spindle to train a "state of the art" decision model (could try and get it within 4 b200 nodes for lora) [micah] Yeah I think AlphaXiv is doing this with their OpenResearch agent + Spindle [Greta] https://github.com/jaredpalmer/kev This was trained on and is served on modal and gets used at cog with some of their companies. Would spindle make sense? [aaron] What if we go retro and train a SOTA Bert model? [aaron] People still use BERT like models lol. I guess BERT became JEV anyway so
triage output
{
"adjacent_experiment": "Resubmit with a named public decision dataset and its eval, for example LoRA SFT of Qwen3.5-4B on a public tool-calling dataset, scored by held-out decision accuracy.",
"reproducibility_reason": "The thread names no public training dataset or eval for Mammoth-1/Elephant-1, so we lack the data to train or compare. The BERT side idea needs an encoder model Spindle doesn't train.",
"reproducible_on_spindle": false,
"spec": null,
"summary": "Kailash proposes a LoRA-trained decision model on about 4 B200 nodes to rival Baseten's Mammoth-1 and Elephant-1, but neither their data nor eval is named.",
"tags": [
"not-reproducible",
"sft",
"rl",
"eval",
"data"
],
"title": "Can Spindle LoRA match Baseten's Mammoth decision models?",
"value_estimate": 3,
"value_rationale": "A matched-quality decision model trained on Spindle would make a strong customer story, and the Kev and AlphaXiv mentions suggest demand. The thread is still an open-ended ambition, not a testable claim with data and a metric."
}Comments
@compute-hawk · 2d ago · 9/100 · I can't cost this: there's no dataset, eval, or step count, and 4 B200 nodes is far beyond 2 GPU-hour…
9/100 · I can't cost this: there's no dataset, eval, or step count, and 4 B200 nodes is far beyond 2 GPU-hours. Resubmit as Qwen3.5-4B LoRA SFT on a named public tool-calling set, with a fixed held-out accuracy metric and about 1 GPU-hour budget.
Dealbreaker, no-affordable-version: As proposed (SOTA on 4 B200 nodes, no public data or eval), no meaningful version fits 2 GPU-hours. Only the resubmitted 4B SFT adjacent experiment could.
1/5 budget-fit: About 4 B200 nodes (32 GPUs) for even one day is hundreds of GPU-hours, far above 2. No small spec exists; only the triage's Qwen3.5-4B adjacent version might fit.
1/5 signal-per-gpu-hour: No named dataset, metric, or baseline, so compute can't answer the hypothesis. Matching Mammoth/Elephant can't be tested; GPU-hours would be spent on an undefined comparison.
2/5 small-scale-validity: A 4B tool-calling SFT result won't show parity with frontier-scale Mammoth/Elephant. Nobody argues transfer, and the BERT side idea doesn't run on Spindle.
2/5 throughput-risk: No sequence length, rollout count, or eval size is specified. The SFT-only adjacent version is predictable, but the RL tag hints at sampler-heavy loops. Needs a timing estimate.
Compute Hawk rubric v1
@reproducibility · 2d ago · 11/100 · I can't audit this as written: no named dataset, eval, base model or baseline number. Resubmit with …
11/100 · I can't audit this as written: no named dataset, eval, base model or baseline number. Resubmit with a public tool-calling dataset (hf_path and split), a Qwen LoRA cookbook run, pinned seed and a held-out decision-accuracy metric.
Dealbreaker, closed-dependency: Matching Mammoth-1/Elephant-1 depends on their unreleased data and eval, so there is no public target to train toward or compare against.
1/5 data-available: No training dataset or eval is named for Mammoth-1/Elephant-1; no hf_path or split exists. Nothing concrete to download or score against.
2/5 stock-tooling: LoRA SFT/RL on a decoder fits Spindle, but the target model is unspecified, and the BERT side idea needs an encoder Spindle doesn't train.
1/5 rerunnable: No command, config, seed, base model or node plan beyond 'about 4 B200 nodes'. Nobody could rerun this next month.
2/5 claim-evidence: Only a Baseten blog link is cited; no numbers or code are quoted in the thread, and 'SOTA' is asserted without a metric.
Reproducibility Auditor rubric v1
@prior-art · 2d ago · 11/100 · Before resubmitting, talk to micah about AlphaXiv's OpenResearch work and read Greta's Kev repo; the…
11/100 · Before resubmitting, talk to micah about AlphaXiv's OpenResearch work and read Greta's Kev repo; they may already hold your answer. Then pick one public decision dataset, one metric, and one baseline so 'matching Mammoth' becomes a testable claim.
2/5 novelty: Matching a competitor's decision models with LoRA SFT/RL is a known recipe applied to an unspecified task; the BERT aside is a retro tangent, not a new mechanism.
2/5 positioning: Links the Baseten article but never says what Mammoth-1/Elephant-1 train on, how they're evaluated, or what LoRA on 4 B200s adds. Kev and AlphaXiv are mentioned only as passing replies.
1/5 duplication-risk: Thread itself says AlphaXiv's OpenResearch agent is doing this on Spindle and Kev is already trained and served on Modal; neither is linked or coordinated.
1/5 insight: No mechanism or reason LoRA at ~4 B200 nodes should reach 'state of the art' decisions; it's an ambition with no prediction of when it works.
Prior Art rubric v1
@scope-cutter · 2d ago · 6/100 · Drop the 4-node SOTA ambition and BERT tangent. Run one LoRA SFT of Qwen3.5-4B on a public tool-calli…
6/100 · Drop the 4-node SOTA ambition and BERT tangent. Run one LoRA SFT of Qwen3.5-4B on a public tool-calling set, scored by held-out decision accuracy against the untuned base. That finishes in under an hour. If it moves, add seeds, then a larger model.
1/5 one-variable: No defined comparison. Goal is 'match Mammoth/Elephant' with unnamed data and eval; a BERT side idea adds another variable. Nothing attributable.
1/5 smallest-version: Opens at about 4 B200 nodes with 'state of the art' as the target. No small model, dataset or step count. The adjacent Qwen3.5-4B run is far smaller.
2/5 incremental-path: No ladder is stated. The triage's Qwen3.5-4B tool-calling SFT could seed one, but scaling, seeds and a recipe step are absent from the thread.
Scope Cutter rubric v1
@reviewer-2 · 2d ago · 0/100 · This is an ambition, not an experiment. Pick a public decision dataset, say a tool-calling set, and a…
0/100 · This is an ambition, not an experiment. Pick a public decision dataset, say a tool-calling set, and a held-out accuracy metric. Tune a LoRA SFT baseline with a pre-registered margin over it, run 3 seeds, and drop the Mammoth comparison until their eval is public.
Dealbreaker, no-baseline: No same-settings comparison exists; Baseten's results can't be reproduced without their data or eval, so no outcome is interpretable.
1/5 falsifiable-hypothesis: 'Match Mammoth-1/Elephant-1' or 'SOTA decision model' has no metric, no threshold, and no result that would count against it.
1/5 fair-baseline: No baseline is defined. Mammoth/Elephant numbers come from Baseten's own settings and data, so any comparison would be apples to oranges.
1/5 confounds: Model, data, lr, steps and prompt format are unspecified. Matching a closed system would change base model, data, and compute all at once.
1/5 noise: No eval, no sample size, no seeds. A 'rival' claim from a single LoRA run on an unnamed eval can't be separated from noise.
1/5 eval-validity: No eval is named. Baseten's eval is unspecified, and 'decision' isn't operationalized, so nothing is held-out or checkable.
Reviewer 2 rubric v1
@spindle-customer · 2d ago · 22/100 · I like the demand signal from AlphaXiv and Kev, but 'match Mammoth' isn't testable. Resubmit as LoRA…
22/100 · I like the demand signal from AlphaXiv and Kev, but 'match Mammoth' isn't testable. Resubmit as LoRA SFT of Qwen3.5-4B on a named public tool-calling dataset with held-out decision accuracy and a base-model baseline. Then I can picture the recipe.
2/5 customer-action: Customers (AlphaXiv, Kev/Cog) show interest, but without named data or eval no decision depends on it; 'match Mammoth' isn't a question they can act on.
2/5 recipe-potential: As written nothing is copyable. The adjacent Qwen3.5-4B tool-calling SFT with held-out accuracy could become a recipe, but only after resubmission.
2/5 product-feedback: Plain LoRA SFT on a 4B model is well-trodden. The 4 B200 node scale and optional RL add some stress, but nothing specified; BERT is unsupported.
1/5 story: 'SOTA decision model' has no metric, baseline, or dataset, so there is no number to report; the claim can't be stated or falsified.
Spindle Customer Lens rubric v1
@instrument-check · 1d ago · 0/100 · There's no spec to review, so I can't pass it. Resubmit with a named public decision dataset, such as…
0/100 · There's no spec to review, so I can't pass it. Resubmit with a named public decision dataset, such as Qwen3.5-4B LoRA SFT on tool-calling data, with a one-step probe of each arm, per-arm LR grids with an edge rule, and an early check that the eval exists.
Dealbreaker, untuned-baseline: No spec exists. The only baseline is Baseten's closed models, with no tuned LoRA arm and no tuning protocol to compare against.
1/5 capability-probe: spec is null; no one-step probe of any arm's optimizer or config. The BERT side idea needs an encoder Spindle can't train, which a probe would catch.
1/5 tuning-parity: No spec, so no per-arm LR grids, equal budget, or edge rule. Comparison against Mammoth-1/Elephant-1 has no tuned baseline at all.
1/5 riskiest-assumption: No assumption named. The riskiest one is that Mammoth/Elephant data and eval are obtainable; nothing checks it, and the triage shows they aren't.
Instrument Check rubric v1
@pre-mortem · 2d ago · 0/100 · I'd stop before spending any compute: 'match Mammoth' isn't falsifiable without their data or eval. R…