Does GS1 one-rollout REINFORCE match GRPO on GSM8K?
GS1 gives each prompt one rollout and uses a similarity-weighted leave-one-out baseline across the batch; we compare its GSM8K accuracy against GRPO at equal rollouts per step.
Budget
2 GPU-h (default)
· each run plans 2 GPU-h, 150 min wall cap
· 3 runs
·
change
Reproducible on Spindle: yes
— GS1 changes only the client-side advantage (one rollout per prompt with a similarity-weighted leave-one-out baseline). It fits the tinker-cookbook RL loop with stock importance-sampling loss on shared Spindle.
Spec
GS1 (group size 1, similarity-weighted LOO baseline) vs GRPO (group size 8) on GSM8K with Qwen3.5-2B
Hypothesis. At an equal budget of 64 rollouts per step for 40 steps, GS1 reaches GSM8K test exact-match accuracy no more than 5 points below GRPO with group size 8, from the same LoRA initialization.
Method. Triage saw only the Slack summary of trajectory.ai's 'You Only Rollout Once'. No paper brief, code or hyperparameters were fetched, so the similarity function and temperature below are our assumption and must be checked against the paper before the full run. Both arms run in one custom tinker SDK script built on the tinker-cookbook RL loop (rl/train.py with the math_rl gsm8k env, same renderer and reward).
Qwen/Qwen3.5-2B
RL
40 steps
batch 64
rank 32
gsm8k
eval GSM8K test · exact-match accuracy (greedy, final numeric answer)
2 GPU-h
150 min cap
Success criterion. GS1 minus GRPO paired accuracy on the 1319 GSM8K test items has a 95% two-sided Hoeffding lower bound above -0.05. The Hoeffding half-width at n=1319 is about 0.075, so this passes only if GS1's point estimate is at least ~2.5 points above GRPO. Otherwise the result is inconclusive, not a failure, unless the upper bound is below -0.05.
full spec
{
"activation_gate": {
"applicable": true,
"checks": [
{
"max": 1,
"measures": "rollouts per prompt actually sampled in the GS1 arm",
"metric": "gs1_executed_group_size",
"min": 1
},
{
"max": 64,
"measures": "total rollouts per optimizer step actually executed in each arm",
"metric": "rollouts_per_step_both_arms",
"min": 64
},
{
"max": 40,
"measures": "median over i of 1/sum_j w_ij^2. It must be well below the uniform value of 63 and above degenerate single-neighbor weighting",
"metric": "gs1_effective_contributors_median",
"min": 2
},
{
"measures": "mean |b_i - uniform LOO mean|. If ~0, the similarity weighting does nothing",
"metric": "gs1_mean_abs_baseline_minus_uniform_loo",
"min": 0.02
},
{
"max": 0.95,
"measures": "mean train reward in pilot steps. Mid-range reward is needed so that advantages are nonzero",
"metric": "batch_mean_reward_pilot",
"min": 0.1
},
{
"measures": "model name reported by the training client",
"metric": "executed_base_model",
"one_of": [
"Qwen/Qwen3.5-2B"
]
}
],
"reason": "These checks confirm that the GS1 baseline is actually prompt-weighted rather than a uniform batch mean, that GS1 has learning signal where GRPO groups would be all-equal, and that the executed configs match the plan. They are measured on a 3-step pilot of each arm."
},
"arms": [
{
"advantage_mode": "grpo",
"group_size": 8,
"groups_per_batch": 8,
"lr": 4e-05,
"name": "grpo_gs8_seed0",
"seed": 0,
"steps": 40
},
{
"advantage_mode": "gs1",
"group_size": 1,
"groups_per_batch": 64,
"lr": 4e-05,
"name": "gs1_seed0",
"seed": 0,
"similarity": "tfidf_prompt_cosine",
"steps": 40,
"tau": 0.1
}
],
"base_model": "Qwen/Qwen3.5-2B",
"baseline": "GRPO with group_size=8 and groups_per_batch=8 (64 rollouts/step) and a group-mean advantage. It runs through the same script with advantage_mode=grpo and shares model, LoRA init, lr, max_tokens, steps, reward, renderer and eval.",
"batch_size": 64,
"budget": {
"gpu_hours": 2.0,
"max_wall_minutes": 150
},
"command": "# Custom part: the advantage computation only. Everything else is the stock cookbook RL loop and loss.\n# Stock reference for the baseline arm:\n# python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=8 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=40 log_path=/tmp/rx-grpo\n# Both arms through one script so they differ only in advantage_mode:\npython gs1_vs_grpo.py --base_url $TINKER_BASE_URL --model Qwen/Qwen3.5-2B --lora_rank 32 --lr 4e-5 --max_tokens 512 --steps 40 --rollouts_per_step 64 --seed 0 --advantage_mode {grpo|gs1} --tau 0.1 --log_path /tmp/rx-{arm}\n# gs1_vs_grpo.py:\n# - client = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\n# - tc = client.create_lora_training_client(base_model=..., rank=32)\n# - per step: sample via tc.save_weights_and_get_sampling_client()\n# - grpo: 8 prompts x 8 samples\n# - gs1: 64 prompts x 1 sample\n# - reward from the cookbook gsm8k env\n# - advantages:\n# - grpo: r - group_mean\n# - gs1: r_i - sum_{j!=i} softmax_j(cos(tfidf_i, tfidf_j)/tau) r_j [CUSTOM]\n# - tc.forward_backward(datums, loss_fn='importance_sampling'); tc.optim_step(tinker.AdamParams(learning_rate=4e-5))\n# - eval: greedy sample on all 1319 GSM8K test items from the step-40 sampler, numeric exact match, per-item results saved for pairing",
"correctness_preflight": {
"paths": [
{
"baseline_revision": "UNRESOLVED: pin tinker-cookbook commit SHA (rl/train.py, importance_sampling loss) before running",
"id": "gs1_advantage_rl_loss",
"kind": "loss",
"reference_revision": "UNRESOLVED: pin trajectory.ai 'You Only Rollout Once' reference code commit SHA, or a reviewed independent float64 reference implementation at a pinned SHA",
"tolerances": {
"baseline_gradient": {
"atol": 1e-06,
"min_cosine": 0.99999,
"rtol": 1e-05
},
"baseline_loss": {
"atol": 1e-06,
"rtol": 1e-06
},
"reference_gradient": {
"atol": 1e-05,
"min_cosine": 0.9999,
"rtol": 0.0001
},
"reference_loss": {
"atol": 1e-05,
"rtol": 1e-05
}
}
}
],
"required": true,
"status": "unresolved: the GS1 reference implementation and the tinker-cookbook revision are not yet pinned. No full run until both commit SHAs are recorded and checks pass."
},
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 7473,
"name": "gsm8k",
"split": "train (config main)"
},
"decision_rule": {
"alpha": 0.05,
"criteria": [
{
"bounds": [
0,
1
],
"id": "primary_noninferiority",
"margin": -0.05,
"metric": "accuracy",
"weights": {
"baseline": -1,
"treatment": 1
}
}
],
"estimand": "Difference in GSM8K test exact-match accuracy (GS1 minus GRPO) of Qwen3.5-2B LoRA after 40 steps at 64 rollouts/step, larger is better",
"method": "paired_hoeffding",
"seeds": [
0
],
"unit": "fixed_checkpoint_items",
"version": 1
},
"eval": {
"aggregation": null,
"benchmark": "GSM8K test",
"metric": "exact-match accuracy (greedy, final numeric answer)",
"n_examples": 1319
},
"experiment_features": {
"adaptive_selection": false,
"checkpoint_restore": false,
"custom_loss": true,
"custom_optimizer": false,
"generated_training_data": false,
"heldout_loss": false,
"learned_generator": false,
"paired_comparison": true,
"paired_training_arms": true,
"warmup_adaptation": false
},
"feasibility": null,
"feasibility_check": {
"gpu_hours": null,
"known_gpu_hours": 0,
"phases": {},
"problems": [
"Measured feasibility not recorded"
],
"status": "unknown",
"wall_seconds": null
},
"feasibility_required": true,
"generation_audit": {
"answer_comparison": "numeric_exact",
"max_eval_overlaps": 0,
"max_label_errors": 0,
"min_sample_size": 100,
"required": false
},
"hypothesis": "At an equal budget of 64 rollouts per step for 40 steps, GS1 reaches GSM8K test exact-match accuracy no more than 5 points below GRPO with group size 8, from the same LoRA initialization.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "Triage saw only the Slack summary of trajectory.ai's 'You Only Rollout Once'. No paper brief, code or hyperparameters were fetched, so the similarity function and temperature below are our assumption and must be checked against the paper before the full run. Both arms run in one custom tinker SDK script built on the tinker-cookbook RL loop (rl/train.py with the math_rl gsm8k env, same renderer and reward). The script has an advantage_mode switch. Only the advantage computation is custom; the server loss is the stock importance_sampling loss.\n\nBaseline arm (advantage_mode=grpo): each step samples 8 prompts x 8 rollouts. Advantage = reward minus the group mean, as in stock cookbook.\n\nTreatment arm (advantage_mode=gs1): each step samples 64 distinct prompts x 1 rollout.\n- The advantage for rollout i is r_i - b_i.\n- b_i = sum over j != i of w_ij * r_j, with w_ij = softmax over j != i of (cos(e_i, e_j) / tau).\n- e is a TF-IDF vector of the prompt text, and tau = 0.1. Both are assumptions pending the paper.\n- Before the GS1 arm runs, also log the offline B_eff of the uniform-LOO baseline and the similarity baseline on the pilot rollouts. This is a diagnostic only, not the endpoint.\n\nShared settings:\n- Model: Qwen/Qwen3.5-2B, LoRA rank 32, AdamW lr 4e-5, max_tokens 512, temperature 1.0.\n- Each arm: 40 steps, 1 seed, fresh LoRA from the same init seed.\n- Prompts: GSM8K train in a fixed shuffled order. GRPO consumes 320 prompts and GS1 consumes 2560.\n- Eval: greedy decoding on the full GSM8K test split (1319 items), exact match on the final number, from the step-40 checkpoint of each arm.\n\nThe correctness preflight must show that advantage_mode=grpo reproduces the stock cookbook loss and gradient on fixed rollouts. GS1 advantages are compared against a reference implementation pinned to a revision. That revision is currently unresolved, so the GS1 run is blocked until it is pinned.",
"optimizer": {
"lr": 4e-05,
"name": "adam"
},
"primary_endpoint": "gsm8k_test_exact_match_accuracy_step40",
"restoration_preflight": {
"applicable": true,
"fixed_batch_required": true,
"groups": [
{
"arms": [
{
"name": "grpo_gs8_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"init_seed": 0,
"learning_rate": 4e-05,
"lora_rank": 32
}
},
{
"name": "gs1_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"init_seed": 0,
"learning_rate": 4e-05,
"lora_rank": 32
}
}
]
}
],
"reason": "Both arms train fresh LoRAs from the same base model and init seed. Initial identity is verified by forward logprobs on a fixed tokenized batch before the first update, along with the optimizer config observed after creation. Server identity beyond this stays unverified."
},
"steps": 40,
"success_criterion": "GS1 minus GRPO paired accuracy on the 1319 GSM8K test items has a 95% two-sided Hoeffding lower bound above -0.05. The Hoeffding half-width at n=1319 is about 0.075, so this passes only if GS1's point estimate is at least ~2.5 points above GRPO. Otherwise the result is inconclusive, not a failure, unless the upper bound is below -0.05.",
"title": "GS1 (group size 1, similarity-weighted LOO baseline) vs GRPO (group size 8) on GSM8K with Qwen3.5-2B",
"training_type": "rl"
}previous versions (2)
GS1 (group-size-1, similarity-weighted LOO batch baseline) vs GRPO on GSM8K at equal rollout budget
spec
{
"base_model": "Qwen/Qwen3.5-0.8B",
"baseline": "GRPO via the stock tinker_cookbook math_rl recipe: env=gsm8k, group_size=8, groups_per_batch=16 (128 rollouts/step), lr 4e-5, rank 32, max_tokens 512, 100 steps. The same model, renderer, reward, eval and total rollouts as GS1; only the rollout-per-prompt structure and the baseline differ.",
"batch_size": 128,
"budget": {
"gpu_hours": 1.6,
"max_wall_minutes": 120
},
"command": "# Baseline (GRPO, stock recipe):\npython -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-0.8B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=16 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=100 log_path=/tmp/rx-gs1-grpo\n\n# Treatment (GS1, custom advantage; tinker SDK sketch reusing the cookbook gsm8k renderer/grader):\nimport os, numpy as np, tinker\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nsvc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = svc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\nprompts, answers = load_gsm8k_train() # same renderer as cookbook gsm8k env\nE = TfidfVectorizer().fit_transform(prompts) # L2-normalized rows\nfor step in range(100):\n sc = tc.save_weights_and_get_sampling_client(name=f'gs1-{step}')\n idx = np.random.choice(len(prompts), 128, replace=False)\n outs = [sc.sample(prompt=render(prompts[i]), num_samples=1, sampling_params=tinker.types.SamplingParams(max_tokens=512, temperature=1.0)) for i in idx]\n r = np.array([grade(o, answers[i]) for o, i in zip(outs, idx)], dtype=float)\n S = (E[idx] @ E[idx].T).toarray(); W = np.exp(S / 0.1); np.fill_diagonal(W, 0.0) # CUSTOM: similarity-weighted LOO\n b = (W @ r) / W.sum(1); A = r - b # CUSTOM: GS1 advantage\n datums = [make_is_datum(o, adv) for o, adv in zip(outs, A)] # tokens, sampling logprobs, per-token advantage\n tc.forward_backward(datums, loss_fn='importance_sampling').result()\n tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\ntc.save_weights_for_sampler(name='gs1-final')\n# Eval both checkpoints: greedy decoding on all 1319 GSM8K test examples, exact match.",
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 7473,
"name": "GSM8K",
"split": "main/train"
},
"eval": {
"benchmark": "GSM8K test (main config)",
"metric": "exact-match accuracy of the extracted final answer, greedy decoding",
"n_examples": 1319
},
"hypothesis": "At an equal budget of 128 rollouts per step for 100 steps, GS1 reaches GSM8K test exact-match accuracy no more than 1.5 points below GRPO (group_size=8, 16 groups) on Qwen3.5-0.8B. A gap larger than 1.5 points falsifies the paper's claim that GS1 matches GRPO.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "Both arms train a rank-32 LoRA on Qwen/Qwen3.5-0.8B with RL on the GSM8K train split, using the stock cookbook gsm8k renderer, prompt format and correctness reward (1 if the extracted final answer matches, else 0). Settings are identical in both arms: AdamW at lr 4e-5, temperature 1.0, max_tokens 512, 100 optimizer steps and 128 sampled completions per step.\n\nBaseline arm (GRPO): the stock math_rl recipe with group_size=8 and groups_per_batch=16, so 16 prompts \u00d7 8 rollouts. Advantages are reward minus the group mean.\n\nTreatment arm (GS1): a short Tinker SDK loop. Each step draws 128 distinct prompts and samples 1 rollout per prompt. For rollout i, the baseline is b_i = \u03a3_{j\u2260i} w_ij r_j / \u03a3_{j\u2260i} w_ij, where w_ij = exp(cos(e_i, e_j)/\u03c4) and \u03c4 = 0.1. The embeddings e are TF-IDF vectors of the prompts, computed once over the train set; this stands in for the paper's similarity kernel. The advantage is A_i = r_i \u2212 b_i, broadcast to every completion token. The update uses forward_backward with the importance_sampling loss, then optim_step with tinker.AdamParams. Only the prompt sampling and the advantage computation are custom; the sampler, loss, optimizer and grader match the baseline.\n\nBoth final checkpoints are evaluated with greedy decoding on the full GSM8K test set. Training-reward curves are logged as a secondary signal. Seeds are fixed. If the gap lands within \u00b11.5 points, repeat with a second seed before calling the result.",
"optimizer": {
"lr": 4e-05,
"name": "adam"
},
"steps": 100,
"success_criterion": "Validated if GS1 test accuracy is at least GRPO accuracy minus 1.5 points on all 1319 GSM8K test examples; the standard error is about \u00b11.3 points at this n. If GS1 beats GRPO by more than 2 points, report it as an improvement. Failed if GS1 trails GRPO by more than 1.5 points, confirmed on a second seed when the first-seed gap is between 1.5 and 3 points. The base-model accuracy is also reported, and both arms must improve over it by at least 3 points for the comparison to count.",
"title": "GS1 (group-size-1, similarity-weighted LOO batch baseline) vs GRPO on GSM8K at equal rollout budget",
"training_type": "rl"
}GS1 (one rollout per prompt, similarity-weighted LOO baseline) vs GRPO on GSM8K at matched rollout budget
spec
{
"base_model": "Qwen/Qwen3.5-0.8B",
"baseline": "GRPO via the stock math_rl recipe: env=gsm8k, group_size=8, groups_per_batch=16 (128 rollouts/step), same model, LoRA rank, learning rate, max_tokens=512, 100 steps, same eval.",
"batch_size": 128,
"budget": {
"gpu_hours": 2.0,
"max_wall_minutes": 120
},
"command": "Baseline (GRPO): python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-0.8B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=16 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=100 log_path=/tmp/rx-gs1-grpo. Treatment (GS1, custom advantage only): python gs1_train.py, a short tinker SDK script. It runs sc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']) and tc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32), then loops 100 steps. Each step: (1) sampler = tc.save_weights_and_get_sampling_client(...); (2) draw 128 distinct GSM8K train prompts, render them with the same cookbook renderer, sample 1 completion each (max_tokens=512, temperature=1.0) and keep the sampling logprobs; (3) score each completion with the cookbook gsm8k grader to get r_i in {0,1}; (4) CUSTOM: compute TF-IDF prompt vectors, set w_ij = max(0, cos), set b_i as the weighted leave-one-out mean of r_j, and set A_i = r_i - b_i; (5) build tinker.Datum objects with per-token advantages A_i and sampling logprobs, then call tc.forward_backward(data, loss_fn='importance_sampling'); (6) call tc.optim_step(tinker.AdamParams(learning_rate=4e-5)). At the end, save with save_weights_for_sampler and run the shared GSM8K test eval on both checkpoints and on the base model.",
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 7473,
"name": "GSM8K",
"split": "train (main config); eval on test"
},
"eval": {
"benchmark": "GSM8K test (openai/gsm8k main, test split)",
"metric": "exact-match accuracy of final numeric answer, greedy decoding",
"n_examples": 1319
},
"hypothesis": "At an equal budget of 128 rollouts per step for 100 steps, GS1 reaches GSM8K test exact-match accuracy no worse than GRPO minus 2 points, and both arms beat the untrained base model.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "Both arms train a rank-32 LoRA on Qwen/Qwen3.5-0.8B with AdamW, using the same GSM8K train prompts, renderer, binary correctness reward (math_rl gsm8k env grader), max_tokens, learning rate, and REINFORCE-style importance-sampling loss. Each step samples 128 rollouts in both arms. GRPO (baseline) uses 16 prompts x 8 rollouts, and each rollout's advantage is its reward minus the mean reward of its group. GS1 (treatment) uses 128 distinct prompts x 1 rollout. For rollout i, the advantage is r_i - b_i, where b_i = sum_{j!=i} w_ij r_j / sum_{j!=i} w_ij. The weights are w_ij = max(0, cos(e_i, e_j)), where e is a TF-IDF vector of the prompt text computed client-side. If all weights are zero, the code falls back to the uniform leave-one-out batch mean. Neither arm normalizes advantages by standard deviation, so the baseline estimator is the only difference. Evaluate the final checkpoint of each arm and the untrained base model on the full GSM8K test set with greedy decoding. Also log train reward and gradient-norm variance per step, and save the rollouts so B_eff can be computed offline later.",
"optimizer": {
"lr": 4e-05,
"name": "adam"
},
"steps": 100,
"success_criterion": "Validated if GS1 test accuracy >= GRPO test accuracy - 2.0 points (about 1.5x the standard error of a binomial at n=1319, p~0.5) and GS1 improves over the base model by >= 3 points. Failed if GS1 trails GRPO by more than 2.0 points or does not beat the base model by 3 points. If GS1 exceeds GRPO by more than 2.0 points, report it as a win.",
"title": "GS1 (one rollout per prompt, similarity-weighted LOO baseline) vs GRPO on GSM8K at matched rollout budget",
"training_type": "rl"
}Runs
Source
raw text
Peyton shared trajectory.ai's 'You Only Rollout Once' in #training-internal. GS1 = group-size-1 RL: REINFORCE where each prompt gets a single rollout and the baseline is a similarity-weighted leave-one-out average over the rest of the batch, replacing GRPO's per-prompt groups. The paper introduces B_eff, a metric for scoring candidate baselines offline. The advantage computation is client-side, so it fits a tinker-cookbook RL loop pointed at Spindle.
triage output
{
"adjacent_experiment": "",
"reproducibility_reason": "GS1 changes only the client-side advantage (one rollout per prompt with a similarity-weighted leave-one-out baseline). It fits the tinker-cookbook RL loop with stock importance-sampling loss on shared Spindle.",
"reproducible_on_spindle": true,
"sources": [],
"spec": {
"activation_gate": {
"applicable": true,
"checks": [
{
"max": 1,
"measures": "rollouts per prompt actually sampled in the GS1 arm",
"metric": "gs1_executed_group_size",
"min": 1
},
{
"max": 64,
"measures": "total rollouts per optimizer step actually executed in each arm",
"metric": "rollouts_per_step_both_arms",
"min": 64
},
{
"max": 40,
"measures": "median over i of 1/sum_j w_ij^2. It must be well below the uniform value of 63 and above degenerate single-neighbor weighting",
"metric": "gs1_effective_contributors_median",
"min": 2
},
{
"measures": "mean |b_i - uniform LOO mean|. If ~0, the similarity weighting does nothing",
"metric": "gs1_mean_abs_baseline_minus_uniform_loo",
"min": 0.02
},
{
"max": 0.95,
"measures": "mean train reward in pilot steps. Mid-range reward is needed so that advantages are nonzero",
"metric": "batch_mean_reward_pilot",
"min": 0.1
},
{
"measures": "model name reported by the training client",
"metric": "executed_base_model",
"one_of": [
"Qwen/Qwen3.5-2B"
]
}
],
"reason": "These checks confirm that the GS1 baseline is actually prompt-weighted rather than a uniform batch mean, that GS1 has learning signal where GRPO groups would be all-equal, and that the executed configs match the plan. They are measured on a 3-step pilot of each arm."
},
"arms": [
{
"advantage_mode": "grpo",
"group_size": 8,
"groups_per_batch": 8,
"lr": 4e-05,
"name": "grpo_gs8_seed0",
"seed": 0,
"steps": 40
},
{
"advantage_mode": "gs1",
"group_size": 1,
"groups_per_batch": 64,
"lr": 4e-05,
"name": "gs1_seed0",
"seed": 0,
"similarity": "tfidf_prompt_cosine",
"steps": 40,
"tau": 0.1
}
],
"base_model": "Qwen/Qwen3.5-2B",
"baseline": "GRPO with group_size=8 and groups_per_batch=8 (64 rollouts/step) and a group-mean advantage. It runs through the same script with advantage_mode=grpo and shares model, LoRA init, lr, max_tokens, steps, reward, renderer and eval.",
"batch_size": 64,
"budget": {
"gpu_hours": 2.0,
"max_wall_minutes": 150
},
"command": "# Custom part: the advantage computation only. Everything else is the stock cookbook RL loop and loss.\n# Stock reference for the baseline arm:\n# python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=8 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=40 log_path=/tmp/rx-grpo\n# Both arms through one script so they differ only in advantage_mode:\npython gs1_vs_grpo.py --base_url $TINKER_BASE_URL --model Qwen/Qwen3.5-2B --lora_rank 32 --lr 4e-5 --max_tokens 512 --steps 40 --rollouts_per_step 64 --seed 0 --advantage_mode {grpo|gs1} --tau 0.1 --log_path /tmp/rx-{arm}\n# gs1_vs_grpo.py:\n# - client = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\n# - tc = client.create_lora_training_client(base_model=..., rank=32)\n# - per step: sample via tc.save_weights_and_get_sampling_client()\n# - grpo: 8 prompts x 8 samples\n# - gs1: 64 prompts x 1 sample\n# - reward from the cookbook gsm8k env\n# - advantages:\n# - grpo: r - group_mean\n# - gs1: r_i - sum_{j!=i} softmax_j(cos(tfidf_i, tfidf_j)/tau) r_j [CUSTOM]\n# - tc.forward_backward(datums, loss_fn='importance_sampling'); tc.optim_step(tinker.AdamParams(learning_rate=4e-5))\n# - eval: greedy sample on all 1319 GSM8K test items from the step-40 sampler, numeric exact match, per-item results saved for pairing",
"correctness_preflight": {
"paths": [
{
"baseline_revision": "UNRESOLVED: pin tinker-cookbook commit SHA (rl/train.py, importance_sampling loss) before running",
"id": "gs1_advantage_rl_loss",
"kind": "loss",
"reference_revision": "UNRESOLVED: pin trajectory.ai 'You Only Rollout Once' reference code commit SHA, or a reviewed independent float64 reference implementation at a pinned SHA",
"tolerances": {
"baseline_gradient": {
"atol": 1e-06,
"min_cosine": 0.99999,
"rtol": 1e-05
},
"baseline_loss": {
"atol": 1e-06,
"rtol": 1e-06
},
"reference_gradient": {
"atol": 1e-05,
"min_cosine": 0.9999,
"rtol": 0.0001
},
"reference_loss": {
"atol": 1e-05,
"rtol": 1e-05
}
}
}
],
"required": true,
"status": "unresolved: the GS1 reference implementation and the tinker-cookbook revision are not yet pinned. No full run until both commit SHAs are recorded and checks pass."
},
"dataset": {
"hf_path": "openai/gsm8k",
"n_examples": 7473,
"name": "gsm8k",
"split": "train (config main)"
},
"decision_rule": {
"alpha": 0.05,
"criteria": [
{
"bounds": [
0,
1
],
"id": "primary_noninferiority",
"margin": -0.05,
"metric": "accuracy",
"weights": {
"baseline": -1,
"treatment": 1
}
}
],
"estimand": "Difference in GSM8K test exact-match accuracy (GS1 minus GRPO) of Qwen3.5-2B LoRA after 40 steps at 64 rollouts/step, larger is better",
"method": "paired_hoeffding",
"seeds": [
0
],
"unit": "fixed_checkpoint_items",
"version": 1
},
"eval": {
"aggregation": null,
"benchmark": "GSM8K test",
"metric": "exact-match accuracy (greedy, final numeric answer)",
"n_examples": 1319
},
"experiment_features": {
"adaptive_selection": false,
"checkpoint_restore": false,
"custom_loss": true,
"custom_optimizer": false,
"generated_training_data": false,
"heldout_loss": false,
"learned_generator": false,
"paired_comparison": true,
"paired_training_arms": true,
"warmup_adaptation": false
},
"feasibility": null,
"feasibility_check": {
"gpu_hours": null,
"known_gpu_hours": 0,
"phases": {},
"problems": [
"Measured feasibility not recorded"
],
"status": "unknown",
"wall_seconds": null
},
"feasibility_required": true,
"generation_audit": {
"answer_comparison": "numeric_exact",
"max_eval_overlaps": 0,
"max_label_errors": 0,
"min_sample_size": 100,
"required": false
},
"hypothesis": "At an equal budget of 64 rollouts per step for 40 steps, GS1 reaches GSM8K test exact-match accuracy no more than 5 points below GRPO with group size 8, from the same LoRA initialization.",
"lora_rank": 32,
"max_seq_len": 1024,
"method": "Triage saw only the Slack summary of trajectory.ai's 'You Only Rollout Once'. No paper brief, code or hyperparameters were fetched, so the similarity function and temperature below are our assumption and must be checked against the paper before the full run. Both arms run in one custom tinker SDK script built on the tinker-cookbook RL loop (rl/train.py with the math_rl gsm8k env, same renderer and reward). The script has an advantage_mode switch. Only the advantage computation is custom; the server loss is the stock importance_sampling loss.\n\nBaseline arm (advantage_mode=grpo): each step samples 8 prompts x 8 rollouts. Advantage = reward minus the group mean, as in stock cookbook.\n\nTreatment arm (advantage_mode=gs1): each step samples 64 distinct prompts x 1 rollout.\n- The advantage for rollout i is r_i - b_i.\n- b_i = sum over j != i of w_ij * r_j, with w_ij = softmax over j != i of (cos(e_i, e_j) / tau).\n- e is a TF-IDF vector of the prompt text, and tau = 0.1. Both are assumptions pending the paper.\n- Before the GS1 arm runs, also log the offline B_eff of the uniform-LOO baseline and the similarity baseline on the pilot rollouts. This is a diagnostic only, not the endpoint.\n\nShared settings:\n- Model: Qwen/Qwen3.5-2B, LoRA rank 32, AdamW lr 4e-5, max_tokens 512, temperature 1.0.\n- Each arm: 40 steps, 1 seed, fresh LoRA from the same init seed.\n- Prompts: GSM8K train in a fixed shuffled order. GRPO consumes 320 prompts and GS1 consumes 2560.\n- Eval: greedy decoding on the full GSM8K test split (1319 items), exact match on the final number, from the step-40 checkpoint of each arm.\n\nThe correctness preflight must show that advantage_mode=grpo reproduces the stock cookbook loss and gradient on fixed rollouts. GS1 advantages are compared against a reference implementation pinned to a revision. That revision is currently unresolved, so the GS1 run is blocked until it is pinned.",
"optimizer": {
"lr": 4e-05,
"name": "adam"
},
"primary_endpoint": "gsm8k_test_exact_match_accuracy_step40",
"restoration_preflight": {
"applicable": true,
"fixed_batch_required": true,
"groups": [
{
"arms": [
{
"name": "grpo_gs8_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"init_seed": 0,
"learning_rate": 4e-05,
"lora_rank": 32
}
},
{
"name": "gs1_seed0",
"optimizer": "adam",
"optimizer_config": {
"beta1": 0.9,
"beta2": 0.95,
"eps": 1e-08,
"init_seed": 0,
"learning_rate": 4e-05,
"lora_rank": 32
}
}
]
}
],
"reason": "Both arms train fresh LoRAs from the same base model and init seed. Initial identity is verified by forward logprobs on a fixed tokenized batch before the first update, along with the optimizer config observed after creation. Server identity beyond this stays unverified."
},
"steps": 40,
"success_criterion": "GS1 minus GRPO paired accuracy on the 1319 GSM8K test items has a 95% two-sided Hoeffding lower bound above -0.05. The Hoeffding half-width at n=1319 is about 0.075, so this passes only if GS1's point estimate is at least ~2.5 points above GRPO. Otherwise the result is inconclusive, not a failure, unless the upper bound is below -0.05.",
"title": "GS1 (group size 1, similarity-weighted LOO baseline) vs GRPO (group size 8) on GSM8K with Qwen3.5-2B",
"training_type": "rl"
},
"summary": "GS1 gives each prompt one rollout and uses a similarity-weighted leave-one-out baseline across the batch; we compare its GSM8K accuracy against GRPO at equal rollouts per step.",
"tags": [
"rl",
"grpo",
"baseline",
"math",
"lora",
"paper"
],
"title": "Does GS1 one-rollout REINFORCE match GRPO on GSM8K?",
"value_estimate": 4,
"value_rationale": "If one rollout per prompt matches GRPO, Spindle RL users could cover far more prompts at the same rollout cost, which is a directly actionable recipe change. Triage saw only the Slack summary, not the paper itself, so the similarity function and B_eff details are unconfirmed."
}Comments
@compute-hawk · 2d ago · 53/100 · Cost is fine, but parallelize the GS1 sampling calls and time one step of each arm first. Your 1.5-p…
53/100 · Cost is fine, but parallelize the GS1 sampling calls and time one step of each arm first. Your 1.5-point margin sits near 1 SE, so a single seed can't decide it. Use two seeds up front, or lean on the training-reward curves, rather than adding a second seed afterward.
4/5 budget-fit: Two arms at 100 steps × 128 rollouts × ≤512 tokens on a 0.8B model, plus 1319-example greedy evals, fit 1.6 GPU-hours. A second seed has limited headroom.
3/5 signal-per-gpu-hour: Equal-rollout comparison is the right one, but ±1.3 SE on 1319 examples is noisy against a 1.5-point margin. A cheaper proxy is training-reward curves.
2/5 small-scale-validity: 0.8B on GSM8K for 100 steps is mostly format learning with a high base pass rate. Baseline quality depends on batch size and task diversity, so transfer is unargued.
3/5 throughput-risk: Sampling dominates wall clock. The GS1 sketch calls sample() serially per prompt (128 calls), unlike the batched recipe, and saves weights every step. Needs a timing estimate.
Compute Hawk rubric v1
@prior-art · 2d ago · 31/100 · Before building, ask in #training-internal whether Peyton or anyone has already run GS1 or RLOO on S…
31/100 · Before building, ask in #training-internal whether Peyton or anyone has already run GS1 or RLOO on Spindle. Then say what this adds beyond the paper: the TF-IDF kernel on GSM8K is the real delta, so add an unweighted-LOO ablation to test whether similarity weighting matters.
2/5 novelty: Reproduction of the trajectory.ai GS1 claim on GSM8K with Qwen3.5-0.8B. The new setting is small, and the TF-IDF kernel is a substitute for the paper's kernel.
3/5 positioning: Names the GS1 paper and the GRPO baseline, but not the delta versus the paper's setup: the kernel is swapped to TF-IDF and B_eff is dropped. Other LOO/RLOO work is not cited.
2/5 duplication-risk: Peyton shared the paper in #training-internal, and RLOO/LOO baselines are supported in Spindle. No search of Slack or the exchange for existing GS1 or RLOO runs is mentioned.
2/5 insight: The hypothesis only says GS1 matches GRPO within 1.5 points. It doesn't argue when similarity weighting should help or hurt, and TF-IDF similarity of math prompts is barely tied to reward correlation.
Prior Art rubric v1
@reproducibility · 2d ago · 61/100 · The experiment is well specified and data is exact, but the GS1 arm is a truncated sketch and the cl…
61/100 · The experiment is well specified and data is exact, but the GS1 arm is a truncated sketch and the claim has no cited numbers. Pin seed values and versions, commit the full GS1 script, and note that TF-IDF replaces the paper's kernel, so this tests a variant.
5/5 data-available: openai/gsm8k, main/train (7473) for training and main/test (1319) for eval are named exactly, with the stock cookbook renderer and grader.
3/5 stock-tooling: GRPO arm is stock math_rl; GS1 arm needs a custom SDK loop with custom advantage computation, though the loss and optimizer are stock. The sketch is incomplete and truncated.
3/5 rerunnable: GRPO command is recorded, but 'seeds are fixed' gives no values, there is no SDK or model version pin, and the GS1 script is a truncated sketch.
2/5 claim-evidence: Source is a trajectory.ai post relayed via Slack. No numbers or code are cited, and the TF-IDF kernel is a substitute for the paper's kernel.
Reproducibility Auditor rubric v1
@scope-cutter · 2d ago · 50/100 · Cut the confound first. Add a GS1 arm with uniform leave-one-out weights (no TF-IDF) to separate gro…
50/100 · Cut the confound first. Add a GS1 arm with uniform leave-one-out weights (no TF-IDF) to separate group size from the similarity kernel. Run both arms for 30 steps on a 300-example eval to get a signal, then scale to seeds and a larger model.
3/5 one-variable: Rollout structure and baseline change together (1 vs 8 per prompt, TF-IDF kernel), plus a custom loop versus the stock recipe. Equal budget helps, but gains can't be attributed.
3/5 smallest-version: Qwen3.5-0.8B and 1.6 GPU-hours is small, but 100 steps × 128 rollouts and the full 1319-example test set exceed a first signal; 30 steps on a test subset would do.
3/5 incremental-path: A second seed is specified, and B_eff is deferred. Beyond that there is no ladder: no larger model, no ablation of the kernel, and no recipe step.
Scope Cutter rubric v1
@reviewer-2 · 2d ago · 58/100 · Your 1.5-point margin is roughly one standard error of the eval, and RL seed variance on a 0.8B mode…
58/100 · Your 1.5-point margin is roughly one standard error of the eval, and RL seed variance on a 0.8B model is probably larger. Run 3 seeds per arm, estimate the spread first, and set the margin from it. Also sweep lr for both arms, and run the stock recipe through the same custom loop.
4/5 falsifiable-hypothesis: Clear non-inferiority margin (1.5 pts) with a stated failure rule. But the margin is about equal to the eval SE (~1.3), so it is a weak threshold.
3/5 fair-baseline: Same model, data, and rollouts, but both arms use lr 4e-5 with no sweep. GS1 and GRPO likely have different optimal lrs, and the arms use different code paths (stock recipe vs custom loop).
3/5 confounds: Arms are matched on rollouts, lr, and temperature. Untreated: the stock recipe versus a hand-written loop (renderer, loss, and stop-token handling), and TF-IDF is a stand-in kernel, so the result tests a GS1 variant.
2/5 noise: One seed per arm by default; the second seed runs only if the gap falls in a narrow range. SE ~1.3 on 1319 items, and RL seed variance on a 0.8B model is likely larger than the 1.5 margin.
4/5 eval-validity: The full held-out GSM8K test set with greedy exact-match is standard and directly tests the claim. A 0.8B model's low base accuracy and the 512-token truncation risk are minor gaps.
Reviewer 2 rubric v1
@spindle-customer · 2d ago · 75/100 · I like that this tests whether customers can drop group sampling, and it stresses a custom RL loop o…
75/100 · I like that this tests whether customers can drop group sampling, and it stresses a custom RL loop on Spindle. Be careful: TF-IDF on GSM8K prompts is a weak similarity proxy, and the ±1.3 SE makes a 1.5-point margin noisy. Report that caveat, and log any SDK friction as product feedback.
4/5 customer-action: Customers pay for GRPO's 8x rollouts per prompt; if GS1 matches within 1.5 points on GSM8K at equal budget, it changes the default RL recipe and diversifies prompt coverage.
4/5 recipe-potential: Clean before/after against stock math_rl; the GS1 loop is short, though the SDK sketch needs cleanup into a cookbook-style recipe.
4/5 product-feedback: Custom client-side advantages with importance_sampling loss and per-step save_weights_and_get_sampling_client on Qwen3.5-0.8B exercise a less-used RL path outside the stock recipe.
4/5 story: 'Same rollouts, one per prompt instead of eight: GS1 lands X points from GRPO on GSM8K.' Caveats: TF-IDF stand-in kernel, small model, 100 steps.
Spindle Customer Lens rubric v1
@instrument-check · 1d ago · 31/100 · Run a small LR sweep per arm (e.g. 1e-5 to 1e-4) under equal step budget, pick each arm's best on th…
31/100 · Run a small LR sweep per arm (e.g. 1e-5 to 1e-4) under equal step budget, pick each arm's best on the same eval, and extend the grid if the best sits at an edge. Also add a one-step probe of both arms' exact configs, and name LR sensitivity as the key risk.
Dealbreaker, untuned-baseline: GRPO and GS1 share a single untuned lr=4e-5 (optimizer.lr, baseline text). With the batch composition so different, the result could reflect LR mismatch rather than the method.
3/5 capability-probe: Both arms use stock adam and importance_sampling, so fork-only risk is low. Pilot and fixed-batch preflight cover config and model, but no explicit one-step probe of each arm's exact client.
1/5 tuning-parity: Both arms share one untuned lr=4e-5 (optimizer.lr); no per-arm grid, no edge rule. GS1 uses 64 prompts/step against GRPO's 8, so the optimal LR likely differs.
3/5 riskiest-assumption: Method admits the TF-IDF similarity and tau=0.1 are guesses and the reference revision is unpinned, which blocks the GS1 run. But the leading risk, that a mistuned shared LR decides the comparison, is unnamed.
Instrument Check rubric v1
@pre-mortem · 2d ago · 30/100 · Name TF-IDF similarity as the riskiest assumption and test it first. Add controls: GS1 with a unifor…