🔬Modal Research Exchange

is TailRL more susceptible to rollout lag than traditional GRPO? the original experiments were purely sync, which was a detractor for the work. TailRL theory / explanation was quite compelling but if it can't be used async this would be a major downside for using it in prod runs. <https://zanette-labs.github.io/TailRL-website/|zanette-labs.github.io/TailRL-website> <https://arxiv.org/abs/2609.02987|arxiv.org/abs/2609.02987>

[Research Exchange] Hello! *TailRL under async rollout lag* looks interesting to try, so we're running it now. <https://rx.modal.dev/runs/run_smoke|Follow the run>

[Research Exchange] :green-light-blinker: *TailRL under async rollout lag* succeeded.
Smoke test: no training happened.
steps 0
<https://rx.modal.dev/ideas/idea_smoke|See the idea on the exchange>

[micah] (Sorry ignore, just testing the Slackbot UI flow)

{
  "adjacent_experiment": "",
  "reproducibility_reason": "Fits client-side RL on the tinker SDK. The TailRL loss runs through forward_backward_custom, and lag comes from sampling a checkpoint saved k steps earlier. The TailRL update rule is still unknown because the paper could not be fetched.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": {
        "caveats": "The brief rests on a title-page-only retrieval, so the claim, method and evidence are all unknown. The idea's premise that the original experiments were purely sync is unverified. The arXiv ID (2609.02987) should be checked, since it may be mistyped or a placeholder. Any experiment design here is conditional on reading the actual paper. Staleness handling may also interact with GRPO details such as importance-ratio clipping, so the comparison needs matched correction mechanisms (e.g., clipping or truncated importance sampling applied equally to both methods). Otherwise differences may reflect the correction rather than the method.",
        "claim": "Not stated in the text available. Only the title, author list and affiliations were retrieved from the project page (CMU, UC Berkeley, Impossible Inc., Together AI, Aurora Innovation; 'Preprint, 2026'). The title suggests an RL method that targets the tail of the likelihood distribution, but this is an inference from the name alone. The linked idea text says TailRL has a 'compelling' theory/explanation and that its original experiments were purely synchronous. Neither statement can be checked against the retrieved text.",
        "context": "The idea asks whether TailRL is more fragile than GRPO under off-policy staleness from asynchronous rollouts, which matters for production runs where inference and training overlap. GRPO is the explicit baseline. The thread's bot messages ('smoke test: no training happened', 'steps 0') and Micah's note ('Sorry ignore, just testing the Slackbot UI flow') show the exchange pipeline was being tested, not that the idea was evaluated. The idea itself (the first paragraph) is still legitimate. Modal's team might care because async rollout is a practical throughput lever, and a method that only works synchronously would be hard to adopt.",
        "evidence": "None available. The source text has no results, models, datasets, baselines or numbers. The retrieved text is effectively truncated to the title page. Claims in the idea text (sync-only experiments, strong theory) are the idea author's characterization and are unverified here.",
        "method": "Not stated. The retrieved page text contains no abstract, algorithm, loss, or hyperparameters, only a title block and a placeholder for a video ('Your browser cannot play this video'). The page text gives no description of how TailRL differs from GRPO, such as the objective, the weighting, or how 'tail' is defined. A triage agent should not assume any algorithmic detail. Before designing anything, fetch the arXiv abstract and paper (the idea links arxiv.org/abs/2609.02987; that ID looks unusual and should be verified) to get the actual update rule.",
        "reproduction": "The method cannot be specified from this source, so only the experimental frame for the idea's question is given.\n- Data and eval: choose a verifiable-reward task (e.g., math with exact-match reward) on a small open model such as a Qwen3.5 small variant with LoRA. Use the paper's own tasks if they can be recovered from the full text.\n- Implementation: TailRL's loss must be taken from the paper. GRPO is the baseline, with the same data, reward, LoRA rank, learning rate, group size and step budget.\n- Core manipulation: rollout lag k, meaning rollouts are sampled from a policy k optimizer steps stale. Sweep k in {0, 1, 2, 4, 8} (suggested values, not from the source), with k=0 being the sync control. Report reward/pass@1 curves, stability (KL to the sampler policy, importance-ratio statistics, gradient norm), and final held-out accuracy for both methods at each k.\n- Smallest test: 2 methods x about 3 lag levels x 2-3 seeds, short runs.\n- Spindle/Tinker fit: a Tinker-style API normally decouples sampling from training, so lag can likely be induced by sampling from an older saved sampler checkpoint. Confirm whether Spindle supports this. If the paper's method needs full-model access, custom losses, or logprobs beyond what the API exposes, that part may not carry over. This cannot be assessed from the retrieved text.\n- Compute: not stated; likely modest for a small-model LoRA run.\n- Note: the Slack thread shows the exchange run was only a smoke test with 0 training steps, so no empirical result exists yet.",
        "title": "Tail-Likelihood Reinforcement Learning"
      },
      "error": null,
      "full_text": true,
      "kind": "page",
      "title": "Tail-Likelihood Reinforcement Learning",
      "url": "https://zanette-labs.github.io/TailRL-website/"
    },
    {
      "brief": null,
      "error": "HTTPStatusError: Client error '429 Unknown Error' for url 'https://export.arxiv.org/api/query?id_list=2609.02987'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429",
      "full_text": false,
      "kind": "link",
      "title": "https://arxiv.org/abs/2609.02987",
      "url": "https://arxiv.org/abs/2609.02987"
    },
    {
      "brief": null,
      "error": "HTTPStatusError: Client error '404 Not Found' for url 'https://rx.modal.dev/runs/run_smoke'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404",
      "full_text": false,
      "kind": "link",
      "title": "https://rx.modal.dev/runs/run_smoke",
      "url": "https://rx.modal.dev/runs/run_smoke"
    },
    {
      "brief": null,
      "error": "HTTPStatusError: Client error '404 Not Found' for url 'https://rx.modal.dev/ideas/idea_smoke'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404",
      "full_text": false,
      "kind": "link",
      "title": "https://rx.modal.dev/ideas/idea_smoke",
      "url": "https://rx.modal.dev/ideas/idea_smoke"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "max": 4,
          "measures": "optimizer step minus the step of the sampler checkpoint that generated each batch, after the buffer fills (step >= 4), for lag-4 arms",
          "metric": "executed_sampler_lag_steps",
          "min": 4
        },
        {
          "measures": "mean per-token KL(sampler || current trainer), computed from sampler vs trainer logprobs on the sampled tokens over real loss tokens, at lag 4 divided by the same quantity at lag 0, steps 4-8",
          "metric": "kl_behavior_current_ratio_lag4_over_lag0",
          "min": 3.0
        },
        {
          "max": 0.2,
          "measures": "fraction of real loss tokens with nonzero advantage whose importance ratio exceeds the 2.0 truncation, at lag 4. Signed and absolute p90/p99/max of the log-ratio are also saved",
          "metric": "is_ratio_truncation_fraction_lag4"
        },
        {
          "measures": "fraction of real loss tokens where the TailRL per-token advantage/weight differs from the GRPO advantage on the same batch by more than 1e-4",
          "metric": "tailrl_weight_divergence_fraction",
          "min": 0.1
        },
        {
          "measures": "model name reported by the created training client",
          "metric": "executed_base_model",
          "one_of": [
            "Qwen/Qwen3.5-2B"
          ]
        }
      ],
      "reason": "Lag only matters if the lag-4 behavior policy is measurably further from the trained policy than the lag-0 policy, which only carries trainer/sampler mismatch. TailRL only matters if its per-token weights actually differ from GRPO's. The pilot is 8 steps of tailrl_lag4 and grpo_lag4, with a lag-0 reference KL from 8 steps of grpo_lag0."
    },
    "arms": [
      {
        "lag": 0,
        "lr": 4e-05,
        "method": "grpo",
        "name": "grpo_lag0",
        "seed": 0
      },
      {
        "lag": 4,
        "lr": 4e-05,
        "method": "grpo",
        "name": "grpo_lag4",
        "seed": 0
      },
      {
        "lag": 0,
        "lr": 4e-05,
        "method": "tailrl",
        "name": "tailrl_lag0",
        "seed": 0
      },
      {
        "lag": 4,
        "lr": 4e-05,
        "method": "tailrl",
        "name": "tailrl_lag4",
        "seed": 0
      }
    ],
    "base_model": "Qwen/Qwen3.5-2B",
    "baseline": "GRPO (group-mean advantages, same truncated importance correction) at lag 0 and lag 4. Same model, data order, reward, renderer, LoRA rank 32, AdamW lr 4e-5, group_size 8, 16 groups per batch, max_tokens 512 and 40 steps.",
    "batch_size": 128,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 120
    },
    "command": "Custom tinker SDK script (custom parts: the lag ring buffer and the TailRL advantage). It uses svc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']) and, per arm, tc = svc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32, seed=0). Data is the GSM8K train prompts in a fixed shuffled order (seed 0), shared by all arms, with 16 prompts x group_size 8 per step and max_tokens 512. Each step t: sampler = sampler_ring[max(0, t-LAG)], where the ring holds tc.save_weights_for_sampler at every step; sample with sampler logprobs; reward is exact-match on the GSM8K '####' answer. GRPO computes reward minus group mean. TAILRL computes the paper's advantage/weight (pinned). Then tc.forward_backward_custom(batch, loss = -sum(min(exp(lp - sampler_lp), 2.0) * adv * lp_weight)) and tc.optim_step(tinker.AdamParams(learning_rate=4e-5)), for 40 steps. Eval: greedy sampling at the step-40 checkpoint on all 1319 GSM8K test items, exact-match, with per-item correctness saved for the paired test. Activation-failure plan: (1) offline, on a fixed saved batch, force TailRL weighting on and off and check gradient differences and that the off path equals GRPO; check that lag-4 sampler step indices are correct. (2) If the KL ratio is under 3.0, a recorded spec revision may raise lag to 8 (the only allowed calibration, at most 8 pilot steps, at most 10 wall-minutes) before the outcome runs. Calibration uses training-batch statistics only, never GSM8K test. The failed pilot logs are preserved and gates are never relaxed.",
    "correctness_preflight": {
      "paths": [
        {
          "baseline_revision": "UNRESOLVED: pin tinker-cookbook commit SHA of rl importance_sampling loss + GRPO advantage code",
          "id": "grpo_lagged_is_loss",
          "kind": "loss",
          "reference_revision": "UNRESOLVED: same pinned tinker-cookbook SHA (stock GRPO is its own reference)",
          "tolerances": {
            "baseline_gradient": {
              "atol": 1e-05,
              "min_cosine": 0.9999,
              "rtol": 0.001
            },
            "baseline_loss": {
              "atol": 1e-05,
              "rtol": 0.0001
            },
            "reference_gradient": {
              "atol": 1e-05,
              "min_cosine": 0.9999,
              "rtol": 0.001
            },
            "reference_loss": {
              "atol": 1e-05,
              "rtol": 0.0001
            }
          }
        },
        {
          "baseline_revision": "UNRESOLVED: same pinned tinker-cookbook SHA; TailRL path with tail weighting disabled must reproduce stock GRPO loss/gradient",
          "id": "tailrl_loss",
          "kind": "loss",
          "reference_revision": "UNRESOLVED: pinned commit of official TailRL code or a pinned arXiv version (2609.02987vN) equation transcription",
          "tolerances": {
            "baseline_gradient": {
              "atol": 1e-05,
              "min_cosine": 0.9999,
              "rtol": 0.001
            },
            "baseline_loss": {
              "atol": 1e-05,
              "rtol": 0.0001
            },
            "reference_gradient": {
              "atol": 0.0001,
              "min_cosine": 0.999,
              "rtol": 0.01
            },
            "reference_loss": {
              "atol": 0.0001,
              "rtol": 0.001
            }
          }
        }
      ],
      "reason": "The TailRL loss is unknown: the arXiv source returned 429 and the project page had only a title. No pinned reference implementation or revision exists yet. The custom lagged loop also routes GRPO through new client code. No full sweep until both paths pass.",
      "required": true,
      "status": "unresolved"
    },
    "dataset": {
      "hf_path": "openai/gsm8k",
      "n_examples": 5120,
      "name": "gsm8k",
      "split": "train"
    },
    "decision_rule": {
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "primary",
          "margin": 0.03,
          "metric": "accuracy",
          "weights": {
            "grpo_lag0": -1,
            "grpo_lag4": 1,
            "tailrl_lag0": 1,
            "tailrl_lag4": -1
          }
        }
      ],
      "estimand": "On GSM8K test items, for the fixed step-40 checkpoints of Qwen3.5-2B LoRA, the excess accuracy drop of TailRL over GRPO when rollout lag goes from 0 to 4 (larger means TailRL more lag-sensitive)",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "GSM8K test (openai/gsm8k main, test split)",
      "metric": "exact-match accuracy, greedy decoding, final-step checkpoint",
      "n_examples": 1319
    },
    "experiment_features": {
      "adaptive_selection": false,
      "checkpoint_restore": false,
      "custom_loss": true,
      "custom_optimizer": false,
      "generated_training_data": false,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": true,
      "warmup_adaptation": false
    },
    "feasibility_check": {
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Measured feasibility not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "numeric_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": false
    },
    "hypothesis": "On Qwen3.5-2B LoRA RL, TailRL's GSM8K test accuracy drop from lag 0 to lag 4 exceeds GRPO's drop by at least 0.03 absolute.",
    "lora_rank": 32,
    "max_seq_len": 1024,
    "method": "Four LoRA RL arms start from the same Qwen/Qwen3.5-2B base: {GRPO, TailRL} x {lag 0, lag 4}. All arms share the GSM8K train prompts, prompt order, exact-match reward, group size, LR, LoRA rank and step count. They also share the same token-level importance correction against the sampler's logprobs. That correction is the stock tinker importance_sampling objective, with ratio truncation at 2.0 applied identically to every arm, so differences come from the method and not the correction. Lag k means each step's rollouts are drawn from the sampler weights saved at optimizer step t-k. Every step's weights are saved with save_weights_for_sampler and kept in a ring buffer of k+1. GRPO uses group-mean-centered advantages. TailRL's advantage or loss must come from the paper (arXiv 2609.02987, not yet fetched), implemented client-side through forward_backward_custom. Its paper-default hyperparameters must be pinned before any arm runs. No Spindle fork is needed. The method's lag sensitivity is the decision under test. The primary contrast is the interaction (TailRL_sync - TailRL_lag) - (GRPO_sync - GRPO_lag). Status: blocked until correctness_preflight resolves the TailRL source and revisions. No full sweep runs before then.",
    "optimizer": {
      "lr": 4e-05,
      "name": "adam"
    },
    "preflight": {
      "probe": "Create one training client per arm (Qwen3.5-2B, rank 32, adam lr 4e-5, same seed). Run one forward_backward_custom with each arm's exact loss on one GSM8K example sampled from a saved sampler checkpoint, then one optim_step(tinker.AdamParams(learning_rate=4e-5)). Confirm save_weights_for_sampler and sampling from a 4-step-old checkpoint both work.",
      "riskiest_assumption": "That TailRL's loss can be pinned from the paper and expressed as a client-side per-token advantage/weight over sampled-token logprobs. If it needs full-vocabulary likelihoods or a server-side op, the arm cannot run as specified.",
      "tuning": "All arms share one optimizer (AdamW), so there is no per-arm LR grid. The fixed LR 4e-5 is the cookbook math_rl default scaled for the 2B model, and TailRL-specific hyperparameters use the pinned paper defaults. Nothing is selected from results."
    },
    "primary_endpoint": "gsm8k_test_exact_match_accuracy_at_step_40",
    "restoration_preflight": {
      "applicable": true,
      "fixed_batch_required": true,
      "groups": [
        {
          "arms": [
            {
              "name": "grpo_lag0",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "lr": 4e-05,
                "weight_decay": 0.0
              }
            },
            {
              "name": "grpo_lag4",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "lr": 4e-05,
                "weight_decay": 0.0
              }
            },
            {
              "name": "tailrl_lag0",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "lr": 4e-05,
                "weight_decay": 0.0
              }
            },
            {
              "name": "tailrl_lag4",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "lr": 4e-05,
                "weight_decay": 0.0
              }
            }
          ]
        }
      ],
      "reason": "Four trained arms must start from identical LoRA init and optimizer state. Lagged sampling reads saved sampler weights, which are not training restores."
    },
    "spec_preflight": {
      "findings": [],
      "status": "passed"
    },
    "steps": 40,
    "success_criterion": "Validated if the 95% simultaneous paired_hoeffding lower bound of (TailRL_lag0 - TailRL_lag4) - (GRPO_lag0 - GRPO_lag4) GSM8K test accuracy exceeds 0.03. Failed (TailRL not more lag-sensitive) if the upper bound falls below 0.03. Otherwise inconclusive. Given a single checkpoint per arm and 1319 items, a half-width of about 0.15 is expected, so this pilot may not settle and is not a training-seed claim.",
    "title": "TailRL vs GRPO lag sensitivity on GSM8K, lag 0 vs lag 4",
    "training_type": "rl"
  },
  "summary": "TailRL was only tested with synchronous rollouts. Under the same 4-step rollout lag, we compare TailRL's and GRPO's GSM8K accuracy drops.",
  "tags": [
    "rl",
    "paper",
    "async",
    "grpo",
    "lora"
  ],
  "title": "Does TailRL lose more GSM8K accuracy than GRPO under 4-step rollout lag?",
  "value_estimate": 3,
  "value_rationale": "Whether a new RL objective survives async rollouts decides if production users can adopt it. Triage only saw the TailRL project page title; the arXiv paper returned 429, so the method is unverified and the spec is blocked until it is pinned."
}