{
 "title": "DPPO divergence-masked loss vs GRPO ratio-clipped loss, Qwen3.5-2B GSM8K LoRA RL",
 "hypothesis": "On Qwen3.5-2B LoRA RL on GSM8K with 2 policy updates per rollout batch, the DPPO loss reaches higher greedy exact-match accuracy on the 1319-item GSM8K test set than the GRPO clipped-ratio loss. The lower bound of the paired 95% CI on DPPO minus GRPO must be above 0.",
 "method": "The requested Qwen3.5-0.8B is not served, so we use the closest served small model, Qwen/Qwen3.5-2B.\n\nBoth arms run the same client-side RL loop: the tinker-cookbook math_rl GSM8K env, renderer, prompts, reward and data order. Each step samples groups_per_batch=32 GSM8K train prompts with group_size=8 rollouts at temperature 1.0 and max_tokens=512. Advantages are group-mean-centered (GRPO).\n\nThe rollout batch is split into 2 minibatches. That gives 2 optim_steps per batch, so the policy drifts from the sampling policy and the trust region can act. With one update per fresh batch, neither mask nor clip fires. Both losses are computed in one forked custom loss path through forward_backward_custom, using the sampler logprobs as p_old.\n\nThe arms differ in exactly one switch:\n- Baseline (GRPO): token loss -min(r*A, clip(r, 1-0.2, 1+0.2)*A).\n- Treatment (DPPO, arXiv 2602.04879): token loss -r*A, with tokens masked out when the divergence between the current and sampling policy exceeds the threshold delta in the direction the advantage pushes. The divergence is estimated from the token probabilities, e.g. |p - p_old|, rather than the single-sample ratio.\n\nTriage only received a truncated digest of the paper. The exact divergence estimator, delta and masking direction must be pinned from the paper's released code before any training. Until then the correctness preflight is unresolved and no full sweep runs.\n\nTraining: LoRA rank 32, AdamW lr 2e-5, 50 rollout steps (100 optim steps) per arm, seed 0. Both arms start from the same base weights.\n\nEvaluation: the final checkpoint of each arm is scored with greedy decoding on all 1319 GSM8K test items, using the cookbook's numeric answer grader.",
 "base_model": "Qwen/Qwen3.5-2B",
 "lora_rank": 32,
 "training_type": "rl",
 "dataset": {
  "name": "gsm8k",
  "hf_path": "openai/gsm8k",
  "split": "main/train",
  "n_examples": 1600
 },
 "optimizer": {
  "name": "adam",
  "lr": 2e-05
 },
 "steps": 100,
 "batch_size": 256,
 "max_seq_len": 1024,
 "eval": {
  "benchmark": "GSM8K test (openai/gsm8k main/test)",
  "metric": "greedy exact-match accuracy",
  "n_examples": 1319,
  "aggregation": null
 },
 "baseline": "GRPO clipped-ratio loss (eps=0.2) through the same custom loss path. Identical rollouts per step, 2 minibatch updates per batch, lr, rank, seed, data order and eval.",
 "experiment_features": {
  "custom_loss": true,
  "custom_optimizer": false,
  "paired_comparison": true,
  "paired_training_arms": true,
  "checkpoint_restore": false,
  "warmup_adaptation": false,
  "generated_training_data": false,
  "learned_generator": false,
  "heldout_loss": false,
  "adaptive_selection": false
 },
 "correctness_preflight": {
  "required": true,
  "status": "pinned by executor from released code (Stable-RL) per method text; tolerances unchanged",
  "paths": [
   {
    "id": "grpo_dppo_custom_loss",
    "kind": "loss",
    "baseline_revision": "modal-projects/spindle@e58ebef130b4cd02de6880e62d27ec358ff7132c:src/spindle/backends/megatron_runtime/common/forward_backward.py::_loss(loss_name='ppo', clip 0.8/1.2) \u2014 Spindle's stock Tinker 'ppo' loss (tinker-cookbook 0.5.9 has no client-side ppo; loss_fn='ppo' is server stock)",
    "reference_revision": "sail-sg/Stable-RL@c6af1d119c0fe30d16dd9a837d35fdf695acd226:verl/trainer/ppo/core_algos.py::compute_policy_loss_with_mask(loss_mode='dppo_binary_tv', clip_ratio=0.15, clip_ratio_c=10)",
    "tolerances": {
     "baseline_loss": {
      "atol": 1e-05,
      "rtol": 0.0001
     },
     "baseline_gradient": {
      "atol": 1e-05,
      "rtol": 0.001,
      "min_cosine": 0.9999
     },
     "reference_loss": {
      "atol": 1e-05,
      "rtol": 0.0001
     },
     "reference_gradient": {
      "atol": 1e-05,
      "rtol": 0.001,
      "min_cosine": 0.9999
     }
    }
   }
  ]
 },
 "activation_gate": {
  "applicable": true,
  "reason": "DPPO only differs from GRPO on tokens whose divergence exceeds the trust region. A 5-step pilot of the treatment arm, with 2 updates per batch, must show that the mask fires on a nontrivial but not dominant share of tokens and that the policy actually drifts. Executed config values must match the plan.",
  "checks": [
   {
    "metric": "dppo_masked_token_fraction",
    "measures": "fraction of loss tokens masked by the DPPO divergence rule on the second minibatch update, mean over pilot steps",
    "min": 0.001,
    "max": 0.3
   },
   {
    "metric": "policy_drift_abs_p_q90",
    "measures": "90th percentile of |p_current - p_sampler| over loss tokens at the second update",
    "min": 0.005
   },
   {
    "metric": "grpo_clip_fraction",
    "measures": "fraction of tokens clipped by the eps=0.2 ratio clip in the baseline loss on the same pilot batches (confirms the regime is off-policy for the baseline too)",
    "min": 0.0005
   },
   {
    "metric": "executed_updates_per_batch",
    "measures": "optim_step calls per rollout batch as logged",
    "min": 2,
    "max": 2
   },
   {
    "metric": "executed_base_model",
    "measures": "model name reported at client creation",
    "one_of": [
     "Qwen/Qwen3.5-2B"
    ]
   }
  ]
 },
 "restoration_preflight": {
  "applicable": true,
  "reason": "Both arms are LoRA-trained from the same base. Before the first update, verify identical initial state: compare the logprobs on a fixed tokenized GSM8K batch across both freshly created clients, which must be bitwise or within 1e-6. Also verify the active optimizer is AdamW with the planned config after creation. Server identity cannot be introspected and stays unverified.",
  "fixed_batch_required": true,
  "groups": [
   {
    "arms": [
     {
      "name": "grpo_seed0",
      "optimizer": "adam",
      "optimizer_config": {
       "learning_rate": 2e-05,
       "beta1": 0.9,
       "beta2": 0.95,
       "eps": 1e-08
      }
     },
     {
      "name": "dppo_seed0",
      "optimizer": "adam",
      "optimizer_config": {
       "learning_rate": 2e-05,
       "beta1": 0.9,
       "beta2": 0.95,
       "eps": 1e-08
      }
     }
    ]
   }
  ]
 },
 "success_criterion": "On the 1319 GSM8K test items, the paired 95% Hoeffding CI for DPPO accuracy minus GRPO accuracy has a lower bound above 0.0, roughly a 7.5-point gap at this n. A smaller positive gap is reported as inconclusive, not validated.",
 "decision_rule": {
  "version": 1,
  "estimand": "Difference in greedy GSM8K test exact-match accuracy, DPPO minus GRPO, for these fixed final checkpoints of Qwen3.5-2B LoRA; larger is better",
  "unit": "fixed_checkpoint_items",
  "seeds": [
   0
  ],
  "alpha": 0.05,
  "method": "paired_hoeffding",
  "criteria": [
   {
    "id": "primary",
    "metric": "accuracy",
    "weights": {
     "treatment": 1,
     "baseline": -1
    },
    "margin": 0.0,
    "bounds": [
     0,
     1
    ]
   }
  ]
 },
 "primary_endpoint": "gsm8k_test_greedy_exact_match_accuracy@final_checkpoint (1319 items)",
 "arms": [
  {
   "name": "grpo_seed0",
   "loss": "grpo_clip_eps0.2",
   "seed": 0,
   "lr": 2e-05,
   "updates_per_batch": 2
  },
  {
   "name": "dppo_seed0",
   "loss": "dppo_divergence_mask",
   "seed": 0,
   "lr": 2e-05,
   "updates_per_batch": 2
  }
 ],
 "feasibility_required": true,
 "feasibility": null,
 "budget": {
  "gpu_hours": 2.0,
  "max_wall_minutes": 120
 },
 "generation_audit": {
  "required": false,
  "answer_comparison": "numeric_exact",
  "min_sample_size": 100,
  "max_label_errors": 0,
  "max_eval_overlaps": 0
 },
 "command": "Custom tinker SDK loop; the loss function is the custom part. Sketch:\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32)\n\n# Rollouts: reuse tinker_cookbook.recipes.math_rl GSM8K env and renderer.\n# group_size=8, groups_per_batch=32, max_tokens=512, seed=0.\nfor step in range(50):\n    sampler = tc.save_weights_and_get_sampling_client(f's{step}')\n    batch = rollout(sampler, gsm8k_train_prompts[step], group_size=8)  # stores sampler logprobs as old_lp\n    adv = group_mean_center(batch.rewards)\n    for mb in split(batch, 2):\n        tc.forward_backward_custom(mb.data, make_loss(LOSS, old_lp=mb.old_lp, adv=mb.adv)).result()\n        tc.optim_step(tinker.AdamParams(learning_rate=2e-5)).result()\n\n# LOSS='grpo': -min(r*A, clip(r, 0.8, 1.2)*A)\n# LOSS='dppo': -(r*A) * keep_mask(div(p, p_old) <= delta), with div and delta pinned from the paper code\n# Final eval: greedy decoding on openai/gsm8k test (1319 items), numeric exact match.",
 "feasibility_check": {
  "status": "unknown",
  "problems": [
   "Measured feasibility not recorded"
  ],
  "wall_seconds": null,
  "gpu_hours": null,
  "known_gpu_hours": 0,
  "phases": {}
 }
}