🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "GRPO (std-normalized) advantages under identical lag-0 and lag-4 settings: same seeds, data order, clip range, LR, group size and eval. The comparison is the lag-4 minus lag-0 change for each objective.",
  "batch_size": 256,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "# custom SDK loop (custom parts: advantage fn, stale-sampler queue, clipped-IS loss + metrics); run for adv in {maxrl,grpo} x lag in {0,4} x seed in {0,1}\nimport os, collections, tinker, torch\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\ndef adv(r, kind):  # r: tensor[8] of 0/1\n    k, N = r.sum(), len(r)\n    if kind == 'maxrl': return torch.zeros_like(r) if k == 0 else torch.where(r > 0, N / k - 1, -1.0)\n    return (r - r.mean()) / (r.std() + 1e-6)\nsamplers = collections.deque(maxlen=LAG + 1)\nfor step in range(60):\n    samplers.append(tc.save_weights_and_get_sampling_client(name=f's{step}'))\n    behav = samplers[0]  # lag-LAG checkpoint (fresh when LAG=0)\n    groups = rollout(behav, gsm8k_batch(step, seed), group_size=8, n_groups=32, max_tokens=512, temperature=1.0)  # keep behavior logprobs; reward = exact match, format_coef=0\n    data = build_datums(groups, lambda r: adv(r, ADV))\n    def loss(data, logprobs):  # PPO-clipped IS: ratio = exp(cur - behav), clip [0.8,1.2]; logs ratio p95, KL, clip frac, |A|-weighted clip frac by k-bucket\n        return clipped_is_loss(data, logprobs, lo=0.8, hi=1.2)\n    tc.forward_backward_custom(data, loss).result()\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\nevaluate_pass_at_k(tc.save_weights_and_get_sampling_client(name='final'), split='test', n=500, k=8, temperature=1.0)",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "train (config main); eval on test"
  },
  "eval": {
    "benchmark": "GSM8K test (first 500 problems)",
    "metric": "Primary: |A|-weighted clipped fraction at lag 4, by k-bucket. Secondary: sampled pass@1 and pass@8 (temp 1, 8 samples, unbiased)",
    "n_examples": 500
  },
  "hypothesis": "At 4-step rollout lag with PPO clip [0.8,1.2], TailRL/MaxRL loses at least 1.5x the |A|-weighted clipped advantage mass that GRPO loses, concentrated in groups with k<=2 successes of 8. Its lag-induced pass@8 drop is also larger than GRPO's.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Train Qwen3.5-0.8B LoRA rank 32 on GSM8K train with binary reward: exact-match correctness, format_coef=0, so TailRL equals MaxRL exactly. Use group_size 8 and 32 groups per batch. The only objective difference is the per-group advantage. TailRL/MaxRL uses A = N/k - 1 for successes, -1 for failures, and 0 when k=0. GRPO uses A = (r - p)/(std + eps). Both are group-mean centered. The lag factor is fresh rollouts (lag 0) vs rollouts sampled from the sampler checkpoint saved 4 optimizer steps earlier (lag 4). Behavior logprobs come from the stale sampler. The loss is the PPO-clipped IS objective with clip [0.8,1.2], no KL, one update per batch. The full 2x2 (objective x lag) runs with 2 seeds per cell. Primary readouts are logged every step and bucketed by k in {1,2,3-5,6-7}: IS-ratio p95 and spread, approx KL(behavior||current), raw clip fraction, and |A|-weighted clipped fraction. Secondary readout is sampled pass@1 and pass@8 at the end (temperature 1, 8 samples, unbiased estimator). Sanity checks: run a step-0 base eval. At lag 0, confirm the IS ratio is about 1 and the clip fraction is about 0. Confirm lag 4 actually raises KL and clipping before reading accuracy. Triage could not fetch the paper, site or repo. The estimator follows the refine notes citing the official advantages.py (commit 5682c6a) and modal-projects/lilo codegolf/reward.py.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 60,
  "success_criterion": "Validated if, averaged over steps 20-60 and 2 seeds, TailRL's lag-4 |A|-weighted clipped fraction is >=1.5x GRPO's, the gap exceeds 2x the seed-to-seed range, and the excess sits in the k<=2 buckets. Accuracy is supporting evidence only: TailRL's lag-induced pass@8 drop exceeds GRPO's by >=3 pts. Refuted if the ratio is <1.2x or within seed noise. A 500-problem accuracy diff-of-diffs is underpowered, so it alone cannot validate.",
  "title": "Lag-0 vs lag-4 staleness damage for TailRL/MaxRL vs GRPO advantages",
  "training_type": "rl"
}

served_models

20

base_eval_problems

500

base_eval_samples

4,000

base_pass_at_1

0.6015

base_pass_at_8

0.878

base_truncated_fraction

0.3127

base_eval_wall_seconds

57.67

local_math_tests_passed

4

required_arms

8

arms_with_training_updates

1

completed_60_step_arms

0

completed_training_updates

17

final_evaluated_arms

0

observed_capacity_rejections

29

cleanup_model_ids_checked

3

remaining_run_checkpoints

0

tailrl_lag0_seed0_diagnostic_mean_ratio_mean

1

tailrl_lag0_seed0_diagnostic_mean_ratio_p95

1.019

tailrl_lag0_seed0_diagnostic_mean_ratio_p95_p05_spread

0.04107

tailrl_lag0_seed0_diagnostic_mean_approx_kl_behavior_current

0.0002808

tailrl_lag0_seed0_diagnostic_mean_raw_clip_fraction

0.0009203

tailrl_lag0_seed0_diagnostic_mean_abs_adv_weighted_clipped_fraction

0.0009676

tailrl_lag0_seed0_diagnostic_mean_abs_adv_weighted_active_clipped_fraction

0.0004924

tailrl_lag0_seed0_completed_updates

17