🔬Modal Research Exchange

Arjun Banerjee from latch bio has expressed interest in a joint blogpost similar to https://fireworks.ai/customers/heidi-health-ambient-ai-scribe-fine-tuning where he will post-train a reward model on spindle to make synthetic evals for bio.

{
  "adjacent_experiment": "",
  "reproducibility_reason": "LoRA SFT of a served Qwen3.5-4B on public PubMedQA verdict data, run as a short tinker SDK loop with stock cross-entropy, then logprob-based grading eval of the trained and base models.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": {
        "caveats": "This is marketing copy with no paper, code, or data, and the numbers are internal and unauditable. The 48.0% to 51.3% win-rate gain is small, with no sample size or confidence interval given, so it may be within noise. Crossing 50% against 'proprietary models' is not clearly the same as beating frontier models. The headline 'beats frontier' claim rests on a single internal SBS eval. The page mixes up DPO and 'RFT', which are different things, and does not say which was run. The latency gain may owe as much to model size and serving as to fine-tuning. There is no ablation isolating filtering vs batch size vs algorithm. The page never says whether training was LoRA or full-parameter, so the batch-size finding may not transfer to LoRA. Using frontier-model rewrites and LLM judges creates a risk of circularity if the eval uses similar judges. The page is dated JUL 20, 2026, and is not truncated. For the PubMedQA idea specifically, nothing here shows that SFT improves a grader, so that claim would need its own baseline, such as the untuned Qwen3.5-4B and a strong LLM judge. PubMedQA is also small and its labels are noisy and skewed toward 'yes', so a majority-class baseline must be reported.",
        "claim": "Heidi fine-tuned open models on Fireworks with SFT and then DPO/preference-based 'RFT' and reports that the result beats frontier proprietary models on its internal clinical-note side-by-side eval, while cutting latency from 25s to 7s. The page's stated lessons are to filter preference data aggressively with synthetic rewrites and LLM judges, and to use very large effective batch sizes (about 1M+ tokens) for DPO.",
        "context": "The page is a vendor customer story used as a template for a joint blog post: the idea is for latch bio (Arjun Banerjee) to post-train a reward model/grader on Spindle to build synthetic evals for bio, in the style of this Heidi x Fireworks case study. It sits alongside the common 'fine-tuned open model beats frontier on a narrow domain task' narrative, and the SFT-then-DPO recipe with LLM-judge filtering. It relates to the idea only loosely: Heidi trains a generator with SFT/DPO, whereas the idea trains a judge/reward model. The Modal training team might care because it is a marketing-ready, domain-expert collaboration that would exercise Spindle's SFT/DPO and LoRA workflow, and it speaks to the data-quality and large-batch lessons.",
        "evidence": "All results are from Heidi's internal side-by-side evaluation on its clinical note dataset, which is not public. Headline claims: latency fell from 25 seconds to 7 seconds (Figure 1, labelled '3.5X Lower Latency'); POC to production took 4 weeks; the SFT model outperformed 'the Gemini Flash tier of models'; the RFT/DPO model's outputs were preferred over 'the Gemini Pro tier (frontier) models'. The only quantitative quality ablation is that raising the effective batch size from a standard 64k to 768k tokens improved win rate over proprietary models from 48.0% to 51.3%. Not stated: the number of eval examples, the raters (human or LLM), confidence intervals, the exact proprietary models compared, and any ablation of the filtering steps. The page also says the data-filtering strategy was 'confirmed' by experiments but gives no numbers for it. It claims savings ('huge cost savings') without figures.",
        "method": "Heidi trained open models in two stages. First came SFT, described as imitating foundation-model behavior to learn content style. Then came preference-based fine-tuning, called both DPO and 'Reinforcement Fine-Tuning (RFT)' on the page, which uses preference signals. The preference data came from raw side-by-side (SBS) evaluator choices, which the page says are noisy because evaluators sometimes pick responses that still contain hallucinations. It was cleaned by (a) using frontier models to generate synthetic rewrites as 'golden standard' targets and (b) using an LLM-as-a-Judge to pre-evaluate datasets against safeguard metrics such as Template Adherence and Details Omission. Effective batch size was scaled to roughly 1.5 million tokens using gradient accumulation (Fireworks supports up to 24 steps). Not stated: the base model(s), model sizes, LoRA vs full fine-tuning, learning rate, beta, epochs, dataset size, the judge model, the filtering thresholds, and how the DPO pairs were built from the rewrites and SBS signals. The page is a customer case study, not a paper.",
        "reproduction": "Nothing on the page can be reproduced directly: the data, base model, eval and hyperparameters are all unstated. The page's methodology could be tested on an open task. The linked idea proposes a related but different experiment: LoRA SFT of Qwen3.5-4B as a PubMedQA answer grader or reward model for synthetic bio evals, with latch bio. A minimal test would need the following. Data: PubMedQA (labeled and/or artificial splits) with (question, context, candidate answer) items and gold correctness labels, plus candidate answers from some generator model, some correct and some wrong. Training: LoRA SFT of Qwen3.5-4B to output a verdict or score. Compare against the untuned base model used as a judge, and optionally a larger judge as a reference. Eval: held-out PubMedQA grading accuracy, F1, and agreement with gold labels or with a stronger LLM judge, ideally also checking transfer to another bio QA set. Optional DPO arm: build preference pairs from correct vs incorrect answers, with and without LLM-judge filtering, to test the 'curate aggressively' claim. Batch-size arm: compare small vs large effective batch with gradient accumulation. Compute for a 4B LoRA is small (a single GPU, hours). Gaps: Heidi's task is generative note writing judged by SBS preference, while the idea is a classification-style grader, so the latency and frontier-beating results do not carry over. The 64k to 768k token batch sizes are probably scaled differently for LoRA. The Spindle/Tinker batch and gradient-accumulation controls should be checked before a batch-size ablation. Fireworks-specific infrastructure (24-step accumulation, hosted RFT) is not available in Spindle.",
        "title": "Heidi x Fireworks: Bridging the gap in frontier model performance"
      },
      "error": null,
      "full_text": true,
      "kind": "page",
      "title": "Heidi x Fireworks: Bridging the gap in frontier model performance",
      "url": "https://fireworks.ai/customers/heidi-health-ambient-ai-scribe-fine-tuning"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "measures": "model name reported by the created training/sampling client for both arms",
          "metric": "executed_base_model",
          "one_of": [
            "Qwen/Qwen3.5-4B"
          ]
        },
        {
          "max": 0.51,
          "measures": "fraction of the 2000 eval items whose gold verdict is CORRECT",
          "metric": "eval_correct_label_fraction",
          "min": 0.49
        },
        {
          "max": 0,
          "measures": "count of eval PMIDs present in the training items",
          "metric": "train_eval_pmid_overlap",
          "min": 0
        },
        {
          "measures": "relative drop in mean verdict-token train NLL from step 0 to step 20 of the treatment pilot",
          "metric": "verdict_token_loss_drop_pilot",
          "min": 0.1
        },
        {
          "max": 0.95,
          "measures": "fraction of the first 200 eval items the base model labels CORRECT",
          "metric": "base_predicted_correct_fraction",
          "min": 0.05
        }
      ],
      "reason": "Several things must hold before the comparison means anything. The executed model and arms must match the spec. The eval must be balanced and disjoint from training. SFT must actually move the verdict-token loss. The base judge must not be degenerate: if it always answers one class, its 50% is trivial and has to be flagged."
    },
    "arms": [
      {
        "name": "base_untuned",
        "seed": 0,
        "trained": false
      },
      {
        "lr": 0.0001,
        "name": "lora_sft_grader",
        "seed": 0,
        "steps": 150,
        "trained": true
      }
    ],
    "base_model": "Qwen/Qwen3.5-4B",
    "baseline": "Untuned Qwen/Qwen3.5-4B with the same prompt, renderer, eval items and logprob scoring.",
    "batch_size": 32,
    "budget": {
      "gpu_hours": 1.5,
      "max_wall_minutes": 120
    },
    "command": "python grader_sft.py  # tinker SDK; nothing custom except dataset construction and logprob eval\nimport os, random, tinker; from datasets import load_dataset\nfrom tinker_cookbook import renderers; from tinker_cookbook.tokenizer_utils import get_tokenizer\nM='Qwen/Qwen3.5-4B'; svc=tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntok=get_tokenizer(M); r=renderers.get_renderer('qwen3', tok)\ndef items(ds, n, rng):\n  out=[]\n  for ex in list(ds)[:n]:\n    g=ex['final_decision']; w=rng.choice([o for o in ['yes','no','maybe'] if o!=g])\n    for c,lab in [(g,'CORRECT'),(w,'INCORRECT')]:\n      p=f\"Context: {' '.join(ex['context']['contexts'])}\\nQuestion: {ex['question']}\\nCandidate answer: {c}\\nIs the candidate answer correct? Reply CORRECT or INCORRECT.\"\n      out.append((ex['pubid'],p,lab))\n  return out\nrng=random.Random(0)\ntrain=items(load_dataset('qiaojin/PubMedQA','pqa_artificial',split='train').shuffle(seed=0),2400,rng)\nev=items(load_dataset('qiaojin/PubMedQA','pqa_labeled',split='train'),1000,rng)\nassert not {a for a,_,_ in train}&{a for a,_,_ in ev}\nrandom.Random(0).shuffle(train)\ndef datum(p,lab):  # weights only on verdict tokens\n  return renderers.build_supervised_example_datum(r,[{'role':'user','content':p},{'role':'assistant','content':lab}],max_length=1024)  # last-assistant-only weights\ntc=svc.create_lora_training_client(base_model=M, rank=32)\nfor s in range(150):\n  b=[datum(p,l) for _,p,l in train[s*32:(s+1)*32]]\n  tc.forward_backward(b,'cross_entropy').result(); tc.optim_step(tinker.AdamParams(learning_rate=1e-4)).result()\nsc_t=tc.save_weights_and_get_sampling_client('pubmedqa-grader'); sc_b=svc.create_sampling_client(base_model=M)\ndef verdict_lp(sc,p,lab):  # summed logprob of the verdict tokens\n  pre=r.build_generation_prompt([{'role':'user','content':p}]); comp=tok.encode(lab,add_special_tokens=False)\n  mi=tinker.ModelInput.from_ints(pre.to_ints()+comp); lp=sc.compute_logprobs(mi).result(); return sum(lp[-len(comp):])\nfor name,sc in [('baseline',sc_b),('treatment',sc_t)]:\n  acc=[int((verdict_lp(sc,p,'CORRECT')>verdict_lp(sc,p,'INCORRECT'))==(l=='CORRECT')) for _,p,l in ev]\n  print(name, sum(acc)/len(acc))  # save per-item 0/1 for the paired Hoeffding rule",
    "dataset": {
      "hf_path": "qiaojin/PubMedQA",
      "n_examples": 4800,
      "name": "PubMedQA (pqa_artificial train, pqa_labeled eval)",
      "split": "pqa_artificial:train (2400 questions -> 4800 grading items); eval pqa_labeled:train (1000 questions -> 2000 items)"
    },
    "decision_rule": {
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "primary",
          "margin": 0.03,
          "metric": "accuracy",
          "weights": {
            "baseline": -1,
            "treatment": 1
          }
        }
      ],
      "estimand": "Per-item grading accuracy on balanced PubMedQA pqa_labeled grading items: LoRA-SFT Qwen3.5-4B minus untuned Qwen3.5-4B, higher is better; inference conditional on the seed-0 checkpoint",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "PubMedQA pqa_labeled balanced grading set (1 correct + 1 incorrect candidate per question)",
      "metric": "grading accuracy (argmax logprob of CORRECT vs INCORRECT vs gold verdict)",
      "n_examples": 2000
    },
    "experiment_features": {
      "adaptive_selection": false,
      "checkpoint_restore": false,
      "custom_loss": false,
      "custom_optimizer": false,
      "generated_training_data": false,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": false,
      "warmup_adaptation": false
    },
    "feasibility": {
      "config": {
        "arms": [
          "base_untuned",
          "lora_sft_grader"
        ],
        "base_model": "Qwen/Qwen3.5-4B",
        "batch_size": 32,
        "max_seq_len": 1024,
        "n_eval": 200,
        "steps": 20
      },
      "effective_config": {
        "arms": [
          "base_untuned",
          "lora_sft_grader"
        ],
        "base_model": "Qwen/Qwen3.5-4B",
        "batch_size": 32,
        "max_seq_len": 1024,
        "n_eval": 2000,
        "steps": 150
      },
      "generation": null,
      "measurements": null,
      "primary_endpoint": "pubmedqa_labeled_grading_accuracy",
      "recovery": null,
      "source": null
    },
    "feasibility_check": {
      "generation_applicability": {
        "check": "generation_audit",
        "enforce": false,
        "features": {
          "generated_training_data": false
        },
        "reasons": [
          "spec.generation_audit opts out and no feature it covers is true"
        ],
        "status": "not_applicable"
      },
      "generation_counts": null,
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Re-projection required: lora_rank changed or was not recorded",
        "Re-projection required: dataset changed or was not recorded",
        "Re-projection required: steps changed or was not recorded",
        "Re-projection required: n_examples changed or was not recorded",
        "Re-projection required: n_eval changed or was not recorded",
        "Pilot evidence source not recorded",
        "Representative setup timing not recorded",
        "Representative training timing not recorded",
        "Representative evaluation timing not recorded",
        "Positive recovery reserve not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "categorical_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": false
    },
    "hypothesis": "LoRA SFT on 4,800 PubMedQA-artificial grading items raises Qwen3.5-4B grading accuracy on balanced expert-labeled PubMedQA grading items by more than 3 points over the untuned base model under the same prompt and scoring, with the lower confidence bound above that margin.",
    "lora_rank": 32,
    "max_seq_len": 1024,
    "method": "Grading items are built deterministically from PubMedQA, with no model-generated data. Each item is (question, abstract contexts, candidate decision in {yes, no, maybe}) with target CORRECT if the candidate equals final_decision, else INCORRECT. Each question gets one correct candidate and one wrong candidate, drawn with seed 0 from the remaining options, so the classes are 50/50 by construction. Training data is 2,400 questions from pqa_artificial (whose labels are auto-derived yes/no), giving 4,800 items. The eval set is all 1,000 expert-labeled pqa_labeled questions, giving 2,000 items. pqa_artificial and pqa_labeled are disjoint by PMID, and this is checked. The treatment is LoRA rank 32 on Qwen/Qwen3.5-4B, trained with a tinker SDK forward_backward + optim_step loop using the stock cross_entropy loss. Loss weight sits only on the verdict tokens. AdamW at lr 1e-4, batch 32, 150 steps (about 1 epoch). The baseline is the untuned Qwen3.5-4B with the identical renderer and prompt. Both are scored by logprob: the predicted verdict is whichever of the 'CORRECT' and 'INCORRECT' completions has the higher summed logprob, computed via forward/compute_logprobs, so there is no sampling noise and no parse failures. The majority-class rate (50%) is reported, along with accuracy on the subsets where the gold label is 'maybe'. Nothing is custom server-side and there is no Spindle fork. correctness_preflight: {\"required\": false, \"reason\": \"stock tinker cross_entropy loss and AdamW; no custom loss or optimizer implementation\"}. Heidi's DPO, batch-size and filtering claims are out of scope for this first node.",
    "optimizer": {
      "lr": 0.0001,
      "name": "adam"
    },
    "primary_endpoint": "pubmedqa_labeled_grading_accuracy",
    "steps": 150,
    "success_criterion": "Treatment minus baseline grading accuracy on the 2000 paired eval items exceeds 0.03, with the paired Hoeffding 95% lower bound above 0.03 (half-width about 0.06 at n=2000, so an observed gain of roughly 0.09 or more is needed). Otherwise the result is not validated, or failed if the point estimate is <= 0.",
    "title": "LoRA-SFT PubMedQA grader vs untuned Qwen3.5-4B judge",
    "training_type": "sft"
  },
  "summary": "Qwen3.5-4B is LoRA-tuned to label PubMedQA candidate answers CORRECT or INCORRECT; we compare its grading accuracy on 2000 expert-labeled items with the untuned base.",
  "tags": [
    "sft",
    "reward-model",
    "eval",
    "bio",
    "lora",
    "data"
  ],
  "title": "Does LoRA SFT make Qwen3.5-4B a better PubMedQA grader?",
  "value_estimate": 3,
  "value_rationale": "This seeds a customer-facing joint blog post with latch bio and exercises Spindle's SFT and logprob eval paths. The Heidi page is unauditable marketing, though, and a 4B grader on PubMedQA verdicts is a narrow proxy for the synthetic bio evals they actually want."
}