🔬Modal Research Exchange

Can you verify if this paper actually works? <https://thinkingmachines.ai/blog/lora/>

{
  "adjacent_experiment": "",
  "reproducibility_reason": "Spindle trains LoRA only, so the FullFT baseline cannot run. The post's RL claim that rank-1 LoRA matches high-rank LoRA is testable with stock math_rl GSM8K runs that vary only lora_rank.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": {
        "caveats": "- Most evidence is in plots without numeric tables, error bars or seed counts. The authors acknowledge RL noise limits precision.\n- Models are older or smaller than Modal's targets. The post does not test Qwen3.5, gpt-oss, DeepSeek or Nemotron. The one MoE (Qwen3-30B-A3B) uses a custom per-expert LoRA.\n- SFT is judged by log loss, not downstream task accuracy, so 'same performance' is about loss, not sampled quality. SFT is one epoch, and the post itself says capacity needs for generalizable learning are an open question.\n- RL tests are math-only with binary correctness rewards. The post does not test long-horizon, agentic or dense-reward settings, and the 1-bit-per-episode argument applies only to policy gradient.\n- The 8192-token DeepMath cap limits CoT length.\n- LoRA underperforms at large batch size, which is a practical concern for RL with large batches. The post argues best loss occurs at small batch sizes, which is only shown for one dataset.\n- The LoRA LR 10x rule has no theory, and the derived hidden_size/(2\u00b7alpha) ratio contradicts it. The 15x short-run multiplier is anecdotal.\n- The finding that LoRA is better than FullFT on one 1B dataset is attributed to random variation without investigation.\n- The capacity-estimate framework is rough, and forecasting when LoRA falls short is listed as an open problem.\n- The authors did not improve on the peft defaults, but sweeps were done per condition. Reproductions with untuned LRs may show spurious gaps.\n- The post is a blog, not peer reviewed, and no code or configs are mentioned.",
        "claim": "With a few key details right (LoRA on all layers, especially MLP/MoE; enough capacity for the dataset; a learning rate about 10x the FullFT optimum), LoRA matches full fine-tuning in sample efficiency and final performance for post-training-scale SFT and for RL. For RL, even rank-1 LoRA matches FullFT, because policy gradient absorbs only O(1) bits per episode.",
        "context": "The post builds on the LoRA paper (Hu et al., 2021) and on Biderman et al. 2024 ('LoRA Learns Less and Forgets Less'), which it agrees with on LoRA's weakness in pre-training-like regimes and on the roughly 10x LR ratio. It cites QLoRA, Allen-Zhu & Li's 2 bits/parameter capacity result, and Malladi et al.'s eNTK view. It reinterprets LoRA+ and Unsloth's high-alpha advice in terms of its two-parameter basis, and lists PiSSA and MoE LoRA as open questions. The linked idea asks only whether the result holds up on Spindle. Modal's team trains LoRA adapters on Qwen3.5, gpt-oss, DeepSeek and Nemotron via a Tinker-compatible service, and Thinking Machines is the company behind Tinker. The post thus effectively gives the default recipe for such a service: all-layer LoRA, an LR about 10x the FullFT value, and low ranks being sufficient for RL. If it holds, it supports cheap rank choices for RL and a simple LR heuristic.",
        "evidence": "All results are reported as plots, and the post gives few raw numbers.\n- SFT rank sweeps: FullFT and high-rank LoRA have similar curves, with loss falling linearly in log(steps). Lower-rank LoRAs fall off the curve at a step count that correlates with rank. On the 1B model, high-rank LoRA beats FullFT on one dataset and loses on the other.\n- LR: the optimal LoRA LR is about 10x FullFT's. The fitted multiplier is 9.8, with LoRA pow = 0 (the same hidden-size dependence as FullFT). The optimal LR changes by less than 2x between rank 4 and rank 512. Preliminary, anecdotal evidence suggests about 15x for runs under about 100 steps.\n- Batch size: LoRA shows a persistent loss gap versus FullFT at large batch sizes, independent of rank. The gap is smaller and shrinks over time at batch size 32.\n- Layers: attention-only LoRA underperforms MLP-only, and MLP-only is about equal to MLP+attention. This holds on Llama-3.1-8B and Qwen3-30B-A3B-Base. Attention-only at rank 256 (0.25B params) underperforms MLP-only at rank 128 (0.24B params). The same pattern appears in small-data SFT with rank 256 and in RL on MATH.\n- RL: on GSM and MATH with Llama-3.1-8B, LoRA reaches the same peak reward as FullFT and has a wider range of good LRs, even at rank 1. On DeepMath with Qwen3-8b-base, learning curves are nearly identical across ranks and FullFT. AIME 2024/2025 held-out scores are similar, and CoT length grows similarly in both.\n- Capacity arithmetic: MATH RL is about 10,000 problems x 32 samples, roughly 320,000 bits, versus about 3M parameters for rank-1 LoRA on Llama-3.1-8B. DeepSeek-R1-Zero is about 5.3M episodes (10,400 steps x 32 questions x 16 samples).\n- Compute: LoRA uses slightly more than 2/3 of the FullFT FLOPs per pass, ignoring attention FLOPs.\nNo seeds or error bars are stated.",
        "method": "The authors compare LoRA to full fine-tuning (FullFT) on Llama 3 series and Qwen3 models, including the MoE Qwen3-30B-A3B-Base. LoRA rank is swept from 1 to 512, and the learning rate is swept for every condition. They use a constant LR schedule with no warmup or cooldown. LoRA is parametrized as W' = W + (alpha/r)BA with alpha=32, the HF peft default: A uniform with scale 1/sqrt(d_in), B zero-initialized, the same LR for A and B. SFT runs use Tulu3 (instruction following) and OpenThoughts3 (reasoning) for a single epoch, and the metric is log loss rather than sampled evals. Learning curves are drawn as the pointwise minimum over LRs at each step. Other SFT ablations vary batch size (on a 10,000-example OpenThoughts3 subset) and which layers get LoRA (attention-only, MLP-only, all). For MoE they train a separate LoRA per expert, each with rank equal to total rank divided by the number of active experts (8 for Qwen3 MoE). RL uses a basic policy gradient with an importance sampling correction, objective sum_t (p_sampler/p_learner)\u00b7Adv. Advantages come from a GRPO-like scheme that samples multiple completions per problem and subtracts the per-group mean reward. Tasks are MATH and GSM8K on Llama-3.1-8B base, plus DeepMath on Qwen3-8b-base with samples capped at 8192 tokens, with answer correctness as the reward. Separately, they sweep LR across 14 Llama and Qwen models on Tulu3 and fit an optimal-LR formula in hidden size.",
        "reproduction": "A minimal test fits Spindle well.\n\nRL (the cheapest and most relevant check):\n- Run GRPO-style RL with verifiable rewards on MATH or GSM8K with a base model (the paper used Llama-3.1-8B base to avoid math-contaminated Qwen). Compare LoRA at rank 1 and at a high rank, such as 32 or 256, with a LoRA LR sweep. Include a FullFT baseline only if Spindle or another stack can run one.\n- The paper's exact RL hyperparameters (group size beyond the stated 32 samples per problem on MATH, batch size, LR values, KL, steps) are not stated.\n- Success is a matching reward-versus-LR curve, with LoRA showing a wider good-LR range.\n\nSFT:\n- Use a Tulu3 or OpenThoughts3 subset, one epoch, constant LR, and a sweep over rank (for example 4, 32, 256) and LR. Measure held-out log loss.\n- Check the LoRA-LR-about-10x-FullFT rule and attention-only versus MLP-only versus all-layers at matched parameter counts.\n- Use a 10k-example subset and a small model, such as a Qwen3 or other small open model, for the quick version.\n\nLR rule:\n- Start from an LR for LoRA of about 10x the FullFT LR, with 15x for runs under about 100 steps. A sweep of 3-5 LRs around that value is enough.\n\nCompute:\n- A small model gives a single-GPU-scale experiment. The RL runs on 8B models need more sampling time.\n\nWhat may not carry over:\n- FullFT baselines probably can't run through a Tinker-compatible LoRA-only service, so the comparison may have to be against the paper's reported trends or rank-vs-rank. This is the main limit on verifying the 'matches FullFT' claim directly.\n- The batch-size finding needs control of batch size.\n- Whether Spindle exposes rank, per-layer targeting (attention vs MLP vs MoE experts), and alpha is unknown. The paper fixes alpha=32 and uses the peft init.\n- The per-expert MoE LoRA scheme depends on Spindle's MoE adapter implementation (relevant for gpt-oss and Qwen3 MoE models).\n- The 8192-token DeepMath setup and the AIME evals are expensive.\n- The FLOPs claim is not testable through a service.",
        "title": "LoRA Without Regret"
      },
      "error": null,
      "full_text": true,
      "kind": "page",
      "title": "LoRA Without Regret",
      "url": "https://thinkingmachines.ai/blog/lora/"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "measures": "lora_rank reported in the training client config / logged run config for rank-1 runs",
          "metric": "executed_lora_rank_rank1_arm",
          "one_of": [
            "1"
          ]
        },
        {
          "measures": "lora_rank reported in the training client config / logged run config for rank-32 runs",
          "metric": "executed_lora_rank_rank32_arm",
          "one_of": [
            "32"
          ]
        },
        {
          "measures": "fraction of the first step's 8 groups whose 8 rewards are not all equal",
          "metric": "frac_groups_nonzero_advantage_step0",
          "min": 0.25
        },
        {
          "max": 0.85,
          "measures": "mean correctness reward of base-model rollouts at step 0",
          "metric": "mean_train_reward_step0",
          "min": 0.05
        }
      ],
      "reason": "The treatment is a rank change, so the executed adapter rank must be verified. Policy gradient also only learns when groups have mixed rewards, so the pilot must show a nonzero advantage signal and some reward headroom for both arms."
    },
    "arms": [
      {
        "lora_rank": 1,
        "lr": 0.0001,
        "name": "rank1_lr1e-4",
        "seed": 0
      },
      {
        "lora_rank": 1,
        "lr": 0.0003,
        "name": "rank1_lr3e-4",
        "seed": 0
      },
      {
        "lora_rank": 1,
        "lr": 0.001,
        "name": "rank1_lr1e-3",
        "seed": 0
      },
      {
        "lora_rank": 32,
        "lr": 0.0001,
        "name": "rank32_lr1e-4",
        "seed": 0
      },
      {
        "lora_rank": 32,
        "lr": 0.0003,
        "name": "rank32_lr3e-4",
        "seed": 0
      },
      {
        "lora_rank": 32,
        "lr": 0.001,
        "name": "rank32_lr1e-3",
        "seed": 0
      }
    ],
    "base_model": "Qwen/Qwen2.5-3B",
    "baseline": "Rank-32 LoRA on the same model, recipe, data order, group size, batch, max_tokens, steps and seed, with its own LR grid {1e-4, 3e-4, 1e-3}. The best LR is chosen on test items 0-199.",
    "batch_size": 8,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 120
    },
    "command": "# 1) Pilot, before the sweep: run rank1_lr3e-4 and rank32_lr3e-4 for 3 steps, check activation_gate (executed rank, nonzero-advantage fraction, step-0 reward) and record feasibility timing; do not change settings without a recorded deviation.\n# 2) Train six runs (stock recipe, no custom parts):\nfor R in 1 32; do for LR in 1e-4 3e-4 1e-3; do python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen2.5-3B base_url=$TINKER_BASE_URL env=gsm8k lora_rank=$R group_size=8 groups_per_batch=8 learning_rate=$LR max_tokens=512 max_steps=40 seed=0 log_path=/tmp/rx-lwr/r${R}_lr${LR}; done; done\n# 3) Eval (short custom script on the tinker SDK): sampling_client = service.create_sampling_client(model_path=<final sampler path>); greedy (temperature=0) max_tokens=512 with the recipe's GSM8K prompt/renderer; parse the final number and compare by numeric exact match. Score test[0:200] for all 6 runs plus the base model, then pick the best LR per rank (a tie goes to the lower LR). If the pick is on a grid edge, add 3e-5 or 3e-3 for that rank only if the measured feasibility allows; otherwise mark the run inconclusive. Freeze the choice, then score test[200:1319] once for the two selected checkpoints and the base, and save per-item correctness for the paired Hoeffding CI.\n# Activation-failure plan: if the step-0 nonzero-advantage fraction is <0.25 or the reward is outside [0.05,0.85], keep the failed pilot log, record a revised spec (e.g. max_tokens or prompt fix from the cookbook defaults) before any rerun, and never relax the gate.",
    "correctness_preflight": {
      "reason": "Both arms use the unchanged math_rl recipe loss and the stock AdamW optimizer. Only lora_rank and LR differ, and no custom loss or optimizer is implemented.",
      "required": false
    },
    "dataset": {
      "hf_path": "openai/gsm8k",
      "n_examples": 7473,
      "name": "gsm8k",
      "split": "train (training) / test (selection items 0-199, confirmation items 200-1318)"
    },
    "decision_rule": {
      "adaptive_selection": {
        "confirmation_item_ids": "gsm8k/test[200:1319]",
        "kind": "hyperparameter",
        "rationale": "Each rank's LR is chosen from {1e-4, 3e-4, 1e-3} by accuracy on test items 0-199 and frozen before the single confirmation eval on items 200-1318.",
        "search_item_ids": "gsm8k/test[0:200]"
      },
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "primary_noninferiority",
          "margin": -0.09,
          "metric": "accuracy",
          "weights": {
            "baseline": -1,
            "treatment": 1
          }
        }
      ],
      "estimand": "Difference in GSM8K test (items 200-1318) greedy exact-match accuracy between the LR-tuned rank-1 and LR-tuned rank-32 LoRA RL checkpoints on Qwen2.5-3B, at a fixed 40-step endpoint; larger means rank-1 is better",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "GSM8K test items 200-1318 (confirmation)",
      "metric": "greedy numeric exact-match accuracy",
      "n_examples": 1119
    },
    "experiment_features": {
      "adaptive_selection": true,
      "checkpoint_restore": false,
      "custom_loss": false,
      "custom_optimizer": false,
      "generated_training_data": false,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": true,
      "warmup_adaptation": false
    },
    "feasibility_check": {
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Measured feasibility not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "numeric_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": false
    },
    "hypothesis": "With each arm at its best learning rate, rank-1 LoRA GRPO on Qwen2.5-3B base reaches GSM8K confirmation accuracy no more than 9 points below rank-32 LoRA, using the lower bound of a simultaneous 95% CI. Rank-1 at least matching rank-32 is the post's claim.",
    "lora_rank": 1,
    "max_seq_len": 1024,
    "method": "Scaled-down version of the post's RL setup, which ran GSM8K/MATH on Llama-3.1-8B base with group-mean-subtracted advantages and an importance-sampled policy gradient. Llama is not served here, so we use Qwen/Qwen2.5-3B, a served base model. Training uses the stock tinker_cookbook.recipes.math_rl.train with env=gsm8k and the recipe's default importance-sampling loss. Settings: constant LR, AdamW, group_size=8, groups_per_batch=8, max_tokens=512, 40 steps. The two arms differ only in lora_rank (1 vs 32), and the cookbook default alpha/init is used for both. Each rank runs an LR grid of {1e-4, 3e-4, 1e-3}, bracketing the post's roughly 10x-FullFT LoRA LR for a model with hidden size 2048. All six runs use seed 0 and the same environment ordering. Each final checkpoint is scored greedily on GSM8K test with numeric exact match. Items 0-199 select each arm's LR, and items 200-1318 are the single confirmation evaluation. The FullFT comparison from the post is out of scope: Spindle is LoRA-only, and no fork is planned. This tests only the post's 'rank-1 suffices for RL' claim, not 'LoRA matches FullFT'.",
    "optimizer": {
      "lr": 0.0003,
      "name": "adam"
    },
    "preflight": {
      "probe": "Create training clients for Qwen/Qwen2.5-3B with lora_rank=1 and lora_rank=32. For each, run forward on a fixed tokenized batch and check the logprobs match each other (B=0 init). Then take one forward_backward plus optim_step(tinker.AdamParams(learning_rate=1e-3, beta1=0.9, beta2=0.95, eps=1e-8)) on one GSM8K example and confirm both succeed and the logged rank is the requested one.",
      "riskiest_assumption": "math_rl on Spindle actually honors lora_rank=1 for Qwen2.5-3B (a valid rank-1 adapter on all layers, not silently clamped), and the base model gets enough mixed-reward GSM8K groups in 512 tokens for policy gradient to learn within 40 steps.",
      "tuning": "Each rank gets its own grid {1e-4, 3e-4, 1e-3} under an identical 40-step, 8x8-rollout budget. Selection is on GSM8K test items 0-199, disjoint from confirmation items 200-1318. If a best LR sits on a grid edge, the grid is extended one point for that arm if feasibility allows; otherwise the comparison is inconclusive, never a win."
    },
    "primary_endpoint": "gsm8k_test_confirmation_exact_match_accuracy",
    "restoration_preflight": {
      "applicable": true,
      "fixed_batch_required": true,
      "groups": [
        {
          "arms": [
            {
              "name": "rank1_lr1e-4",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 0.0001
              }
            },
            {
              "name": "rank1_lr3e-4",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 0.0003
              }
            },
            {
              "name": "rank1_lr1e-3",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 0.001
              }
            },
            {
              "name": "rank32_lr1e-4",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 0.0001
              }
            },
            {
              "name": "rank32_lr3e-4",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 0.0003
              }
            },
            {
              "name": "rank32_lr1e-3",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 0.001
              }
            }
          ]
        }
      ],
      "reason": "Trained arms share the Qwen2.5-3B base with freshly initialized LoRA (B=0), so both arms start from the same function. Before the first update, verify this with identical logprobs on a fixed tokenized batch, and verify each client's active AdamW config after creation. Server identity beyond this stays unverified."
    },
    "spec_preflight": {
      "findings": [],
      "status": "passed"
    },
    "steps": 40,
    "success_criterion": "Validated if the simultaneous 95% paired Hoeffding CI lower bound on (best-rank1 minus best-rank32) GSM8K confirmation accuracy over 1119 items exceeds -0.09, and both selected arms beat the untrained base by at least 0.05 point estimate. Failed if the CI upper bound is below -0.09. Otherwise the result is inconclusive, and so is a selected LR on the grid edge that cannot be extended.",
    "title": "Rank-1 vs rank-32 LoRA under GRPO-style RL on GSM8K, best LR vs best LR",
    "training_type": "rl"
  },
  "summary": "LoRA Without Regret says rank-1 LoRA is enough for policy-gradient RL. Each rank gets its own LR sweep, and we compare GSM8K test accuracy of tuned rank-1 and rank-32 adapters.",
  "tags": [
    "paper",
    "rl",
    "lora",
    "rank"
  ],
  "title": "Does rank-1 LoRA match rank-32 LoRA for GSM8K RL?",
  "value_estimate": 4,
  "value_rationale": "If rank-1 matches rank-32 for RL on Spindle with a simple LR heuristic, users can confidently choose tiny adapters for RL. The FullFT half of the claim cannot be checked on a LoRA-only service."
}