🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "GRPO via the stock tinker_cookbook math_rl recipe: env=gsm8k, group_size=8, groups_per_batch=16 (128 rollouts/step), lr 4e-5, rank 32, max_tokens 512, 100 steps. The same model, renderer, reward, eval and total rollouts as GS1; only the rollout-per-prompt structure and the baseline differ.",
  "batch_size": 128,
  "budget": {
    "gpu_hours": 1.6,
    "max_wall_minutes": 120
  },
  "command": "# Baseline (GRPO, stock recipe):\npython -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-0.8B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=16 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=100 log_path=/tmp/rx-gs1-grpo\n\n# Treatment (GS1, custom advantage; tinker SDK sketch reusing the cookbook gsm8k renderer/grader):\nimport os, numpy as np, tinker\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nsvc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = svc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\nprompts, answers = load_gsm8k_train()            # same renderer as cookbook gsm8k env\nE = TfidfVectorizer().fit_transform(prompts)      # L2-normalized rows\nfor step in range(100):\n    sc = tc.save_weights_and_get_sampling_client(name=f'gs1-{step}')\n    idx = np.random.choice(len(prompts), 128, replace=False)\n    outs = [sc.sample(prompt=render(prompts[i]), num_samples=1, sampling_params=tinker.types.SamplingParams(max_tokens=512, temperature=1.0)) for i in idx]\n    r = np.array([grade(o, answers[i]) for o, i in zip(outs, idx)], dtype=float)\n    S = (E[idx] @ E[idx].T).toarray(); W = np.exp(S / 0.1); np.fill_diagonal(W, 0.0)   # CUSTOM: similarity-weighted LOO\n    b = (W @ r) / W.sum(1); A = r - b                                                 # CUSTOM: GS1 advantage\n    datums = [make_is_datum(o, adv) for o, adv in zip(outs, A)]                        # tokens, sampling logprobs, per-token advantage\n    tc.forward_backward(datums, loss_fn='importance_sampling').result()\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\ntc.save_weights_for_sampler(name='gs1-final')\n# Eval both checkpoints: greedy decoding on all 1319 GSM8K test examples, exact match.",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "main/train"
  },
  "eval": {
    "benchmark": "GSM8K test (main config)",
    "metric": "exact-match accuracy of the extracted final answer, greedy decoding",
    "n_examples": 1319
  },
  "hypothesis": "At an equal budget of 128 rollouts per step for 100 steps, GS1 reaches GSM8K test exact-match accuracy no more than 1.5 points below GRPO (group_size=8, 16 groups) on Qwen3.5-0.8B. A gap larger than 1.5 points falsifies the paper's claim that GS1 matches GRPO.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Both arms train a rank-32 LoRA on Qwen/Qwen3.5-0.8B with RL on the GSM8K train split, using the stock cookbook gsm8k renderer, prompt format and correctness reward (1 if the extracted final answer matches, else 0). Settings are identical in both arms: AdamW at lr 4e-5, temperature 1.0, max_tokens 512, 100 optimizer steps and 128 sampled completions per step.\n\nBaseline arm (GRPO): the stock math_rl recipe with group_size=8 and groups_per_batch=16, so 16 prompts \u00d7 8 rollouts. Advantages are reward minus the group mean.\n\nTreatment arm (GS1): a short Tinker SDK loop. Each step draws 128 distinct prompts and samples 1 rollout per prompt. For rollout i, the baseline is b_i = \u03a3_{j\u2260i} w_ij r_j / \u03a3_{j\u2260i} w_ij, where w_ij = exp(cos(e_i, e_j)/\u03c4) and \u03c4 = 0.1. The embeddings e are TF-IDF vectors of the prompts, computed once over the train set; this stands in for the paper's similarity kernel. The advantage is A_i = r_i \u2212 b_i, broadcast to every completion token. The update uses forward_backward with the importance_sampling loss, then optim_step with tinker.AdamParams. Only the prompt sampling and the advantage computation are custom; the sampler, loss, optimizer and grader match the baseline.\n\nBoth final checkpoints are evaluated with greedy decoding on the full GSM8K test set. Training-reward curves are logged as a secondary signal. Seeds are fixed. If the gap lands within \u00b11.5 points, repeat with a second seed before calling the result.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 100,
  "success_criterion": "Validated if GS1 test accuracy is at least GRPO accuracy minus 1.5 points on all 1319 GSM8K test examples; the standard error is about \u00b11.3 points at this n. If GS1 beats GRPO by more than 2 points, report it as an improvement. Failed if GS1 trails GRPO by more than 1.5 points, confirmed on a second seed when the first-seed gap is between 1.5 and 3 points. The base-model accuracy is also reported, and both arms must improve over it by at least 3 points for the comparison to count.",
  "title": "GS1 (group-size-1, similarity-weighted LOO batch baseline) vs GRPO on GSM8K at equal rollout budget",
  "training_type": "rl"
}

command

config

{'base_model': 'Qwen/Qwen3.5-0.8B', 'lora_rank': 32, 'dataset': 'openai/gsm8k', 'split': 'main/train', 'n_examples': 4096, 'n_eval': 256, 'steps': 100, 'batch_size': 64, 'max_seq_len': 1024, 'lr': 4e-05, 'optimizer': 'adam', 'max_wall_seconds': 2700}

modal_call_id

fc-01M6FG4A8TTGDHA1R2J3BSR59M