🔬Modal Research Exchange

<https://arxiv.org/abs/2601.20802>
how well does SDPO work on multi-turn support agent tasks (e.g. tau-bench) with a simple rubric-based LLM judge reward? is it better at improving on certain dimensions than others, e.g. invalid tool use, ungrounded claims, or clarifying user intent upfront given ambiguous circumstances?

{
  "adjacent_experiment": "",
  "reproducibility_reason": "Fits the cookbook RL loop with a client-side multi-turn tau-bench env and a custom per-token SDPO advantage passed to the importance_sampling loss. The frozen teacher's logprobs come from compute_logprobs. No fork is needed.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": {
        "caveats": "- No evidence for multi-turn agents, user simulators, tool-call trajectories, or rubric-judge feedback. The extrapolation is untested, and judge feedback may be noisier or less actionable than a compiler error (the paper's own limitation: uninformative or misleading feedback hurts).\n- Gains depend on in-context learning ability: SDPO underperforms GRPO at Qwen2.5-1.5B and barely helps at Qwen3-0.6B. A LoRA on a small model could go either way.\n- Strongest results are on Chemistry and LiveCodeBench. Several cells in Table 3 (Biology, Materials, Qwen3 Physics, Olmo tool use) show GRPO equal or better, and the aggregate gap is 70.2 vs 66.6. GRPO baselines are hyperparameter-swept but SDPO is strictly on-policy while GRPO's best configs use off-policy minibatch steps, so efficiency comparisons depend on wall-clock setup.\n- LCBv6 train/eval split: public tests (a 50% subset of private tests) are used during training and private tests for validation, with no separate held-out problems, so the 48.8 vs 41.2 comparison is on the same 131 questions the model trains on. It measures training-set learning speed more than generalization. The 4-rollout average also has high variance.\n- Mechanism ambiguity: 'successful rollout as feedback' conflates feedback with a demonstration of the answer, and the paper itself notes that it mirrors group-relative advantages. The shorter responses might reflect a loss of reasoning budget that matters on harder tasks. Why SDPO gives more concise outputs is not ablated.\n- Test-time results (Section 5) use a question set filtered to ones solved by at least one method within 512 steps, and the initial self-teacher solves almost none (0% on 78% of questions). The benefit relies on many SDPO updates, not single-shot correction.\n- SDPO needs a stable teacher: the unregularized teacher diverges, and EMA/trust-region rates are hyperparameters.\n- No evaluation on safety/behavioral dimensions, reward hacking, or judge-gaming. A rubric judge used both as training feedback and eval is vulnerable to this.\n- Hyperparameters were tuned for the paper's full-parameter settings. Nothing is reported for LoRA, RL frameworks other than verl, or models above 8B (apart from the Qwen3 scaling plot, whose sizes are shown only as a figure).",
        "claim": "RLVR's scalar outcome reward is an information bottleneck. Many environments also return tokenized feedback (runtime errors, judge evaluations). SDPO conditions the current model on that feedback to act as a self-teacher, and distills the teacher's next-token distributions back into the policy. The paper reports that this gives denser credit assignment and better sample efficiency and final accuracy than GRPO.",
        "context": "SDPO sits in the 'RL with rich feedback' space. Competitors and relatives: GRPO and variants (GSPO, CISPO, DAPO-style clipping); on-policy distillation (Agarwal et al. 2024; Thinking Machines blog) but with a stronger external teacher instead of the model itself; process reward models; Reflexion/Self-Refine-style in-context correction; off-policy self-distillation or SFT on teacher-corrected outputs (Scheurer 2023, ReSt-style), which the paper shows is worse and forgets more; and concurrent work (Chen et al. 2025c retrospective in-context learning; Shenfeld et al. 2026a self-distillation for continual learning; Zhao et al. 2026 self-distilled reasoner; Song et al. 2026 text feedback RL). It also relates to test-time training (TTT).\n\nRelation to the linked idea: the idea proposes applying SDPO to tau-bench-style multi-turn support agents where a rubric LLM judge supplies the feedback, and asking whether gains are uneven across behaviors (invalid tool use, ungrounded claims, upfront clarification). The paper's Section 3 does cover judge evaluations conceptually (the abstract lists 'judge evaluations' as an example of rich feedback), and it motivates agentic settings as future work. But it only reports experiments with code execution feedback and sibling-rollout feedback, so the idea's core setting is an extrapolation.\n\nWhy Modal's team might care: (a) a judge's textual critique is exactly the kind of feedback that scalar-reward RL discards; (b) SDPO is a small change to an RL loop (swap advantages) with small compute overhead; (c) the reported 3x shorter responses and lower forgetting on holdout benchmarks matter for agent/serving cost; (d) it is a candidate for improving sample efficiency in LoRA RL runs on Spindle, if the API exposes the needed logprobs.",
        "evidence": "Setups use Qwen3-8B and Olmo3-7B-Instruct, with an improved GRPO baseline (asymmetric clipping, no std normalization, rollout importance-sampling correction), hyperparameter-swept. Runs were on 4 GH200 GPUs, about 6h each, via verl and vLLM.\n\n(1) No rich feedback (scalar reward only; other successful rollouts serve as the feedback): SciKnowEval L3 science QA (Chemistry, Physics, Biology, Materials) plus ToolAlpaca tool use, avg@16, 3 seeds. Aggregate final accuracy is 70.2% for SDPO vs 66.6% for GRPO. Examples at 5h with Qwen3-8B (Table 3): Chemistry 80.9 vs 74.5 for GRPO and 63.4 for on-policy GRPO; tool use 68.5 vs 67.7 for GRPO and 65.7 for on-policy GRPO. With Olmo3-7B-Instruct, SDPO reaches the 5h GRPO Chemistry accuracy in 50 min (6\u00d7 faster), and its 5h accuracy is more than 10 points higher. Tool use is mixed: Olmo3 SDPO 62.1 vs GRPO 65.0 at 5h in Table 3 (62.5 vs 65.0 in Table 7), so SDPO is not uniformly better. Biology and Materials are also mixed (e.g. Qwen3-8B Biology SDPO 56.8 vs GRPO 59.9). Responses are about 3.2\u00d7 shorter on average (Qwen3-8B 255.8 vs 820.8 tokens), and 11\u00d7 shorter on Olmo3 Chemistry.\n\n(2) Rich feedback: LiveCodeBench v6 (131 questions, Feb-May 2025), LeetCode-style feedback (runtime errors, wrong-answer test cases), Qwen3-8B, avg over 4 rollouts, 3 seeds. SDPO scores 48.8% vs GRPO 41.2%, and reaches GRPO's final accuracy in 4\u00d7 fewer generations. Other RLVR baselines (Table 9): GSPO 40.1, CISPO 41.2, GRPO with high-entropy tokens only 37.8. The gain is larger on medium and hard questions. Scaling (Fig 8): the gap grows with Qwen3 size and is small at 0.6B. With Qwen2.5-Instruct, SDPO underperforms GRPO at 1.5B and outperforms it at 7B.\nAblations: logit-level > token-level > sequence-level SDPO > GRPO, so even sequence-level SDPO beats GRPO, meaning the rich feedback helps independently of dense credit. Teacher variants (Table 4, best accuracy up to step 90): unregularized teacher 36.1, frozen initial teacher 48.8, trust-region 50.6, EMA 49.3. Feedback type (Table 6): output + own solution gives 48.3 student accuracy, output alone 39.9, own solution alone 42.6, and adding the student's attempt y to the teacher context 44.5.\nForgetting (Table 5): holdout average is 42.4 for SDPO vs 41.8 for GRPO, 41.4 for SFT on the self-teacher, and 43.5 for base. So SDPO forgets less than GRPO but not zero.\nMargins are smaller than the headline suggests: GRPO with 4 off-policy minibatch steps beats SDPO in several cells.\nSDPO+GRPO is more robust at 0.6B and slightly worse at 8B.\n\n(3) Test-time self-distillation on 19 hard and 9 very hard LCBv6 questions, batch 16, 5 seeds per question. On very hard questions, discovery@2750 is 53.2% for SDPO vs 41.5% for best-of-k and 35.6% for multi-turn. On hard questions it is 78% for SDPO vs 72.3% for best-of-k and 68.4% for multi-turn. The headline is about 3\u00d7 fewer attempts to reach 22% discovery on very hard questions. The question set was filtered to questions solved by at least one method.",
        "method": "For each prompt x, sample G rollouts from the policy. Get environment feedback f_i for each. Build a self-teacher by re-prompting the same model with x plus feedback f: a successful rollout from the same group as 'correct solution' (if one exists), the environment output (e.g. runtime error) for failed attempts, and the instruction 'Correctly solve the original question.' The student's original response is NOT included in the teacher's context in the best config, since including it biases the teacher toward the student's attempt and cuts entropy. The teacher then scores the original response token by token. The loss is the per-token KL (reverse-KL, or Jensen-Shannon in Section 3) between the student's next-token distribution and the stop-gradient teacher's, approximated on the student's top-K logits plus a tail term (K=100 in Section 3, K=20 in Sections 4-5). Equivalently it is a policy gradient with logit-level advantages A = log(teacher prob / student prob), so it drops into a verl/GRPO pipeline by swapping advantages. Stability: the teacher is a regularized copy, either an EMA of student weights or a trust-region interpolation with the initial teacher (alpha 0.05 in Section 3, 0.01 in Sections 4-5). An unregularized teacher diverges. Key hyperparameters: AdamW, lr 1e-5 (Section 3) or 1e-6 (Sections 4-5), 8 rollouts (16 for test-time), question batch 32 (1 for test-time), temp 1.0, max response 8192 tokens, rollout importance-sampling clip 2, thinking off. An optional hybrid, \u03bb\u00b7A_GRPO + (1-\u03bb)\u00b7A_SDPO with \u03bb=0.9, helps weaker models.",
        "reproduction": "IMPORTANT: the paper does NOT cover tau-bench, multi-turn agents, or LLM-rubric-judge rewards. Its tasks are single-turn: science MCQ, ToolAlpaca single tool call, and code. The authors list agentic and long-horizon settings, and tasks without ground-truth verifiers, as future work. The idea's question (does SDPO work on multi-turn support agents with a rubric judge, and on which dimensions) is therefore untested by the source.\n\nWhat a reproduction would need:\n- Environment: tau-bench (retail/airline) with a user simulator, or a lighter synthetic support-agent environment with fixed policies and tools. A rubric judge returns per-dimension text such as 'invalid tool call: X', 'claimed refund policy not in the policy doc', 'did not ask which order'. This text is the feedback f. Separately, the rubric scores give a scalar reward for the GRPO baseline.\n- Teacher context design: the SDPO teacher prompt is [task/conversation] + [judge feedback] + [optional successful rollout from the same group] + 'Correctly solve\u2026', with the original response scored under it. For multi-turn trajectories, the open design choices are: score each assistant turn conditioned on the conversation prefix plus trajectory-level feedback, and mask tool and user tokens. The paper gives no recipe for this.\n- Baselines: GRPO on the same scalar rubric reward, base model, and optionally SFT on successes. Use matched rollouts, wall-clock or generation budget, and 3 seeds as in the paper. Consider the SDPO+GRPO hybrid, since a judge-reward setting is noisier.\n- Eval: tau-bench pass^k on held-out tasks, plus per-dimension rates from the rubric (invalid tool-call rate, ungrounded-claim rate, clarification-before-action rate on ambiguous prompts). Also hold out a different judge or human spot-check to detect reward hacking against the training judge.\n- Model: the paper finds gains scale with model size (small gain at 0.6B, SDPO < GRPO at Qwen2.5-1.5B), so use a stronger base. Qwen3-8B or larger is the closest match to the paper, and a bigger Qwen3.5 or gpt-oss model would be reasonable. Thinking is off in the paper's runs.\n- Hyperparameters to start from (Table 12, with-feedback column): lr 1e-6, AdamW, 8 rollouts, EMA teacher rate 0.01, reverse-KL, top-K 20, rollout IS clip 2, temp 1.0. Full-parameter lr values will not transfer to LoRA (LoRA typically needs a much higher lr), so sweep.\n- Compute: SDPO adds one extra teacher forward pass per rollout (the paper reports small overhead). The paper's runs took about 6h on 4 GH200 GPUs per run for 8B full fine-tuning.\n\nWhat may not carry over to LoRA/Spindle:\n- The teacher is the live policy with a different prompt. With LoRA, the teacher forward pass uses the same adapter weights, which is fine. But an EMA of student parameters needs a second adapter copy, or the trust-region variant needs reference-adapter logprobs (base-model logprobs under the feedback prompt, which is cheap with LoRA by disabling the adapter). The paper found the trust-region variant slightly better (50.6 vs 49.3).\n- SDPO needs per-token top-K logits (or full distributions) from both student and teacher, not just sampled-token logprobs. A Tinker-compatible API typically exposes sampled-token logprobs, and possibly top-k logprobs. If Spindle only returns the logprob of the chosen token, you would have to use the single-token approximation, which the paper's ablation shows is weaker (token-level < logit-level). Check what Spindle supports, and whether a custom loss with an externally supplied per-token advantage or teacher logprobs is possible. A 'sequence-level SDPO' (average teacher/student log-ratio as a scalar advantage) can run in a standard importance-sampling RL loop and still beat GRPO in the paper.\n- The paper's trust-region teacher and JS divergence need teacher logits under the reference model, which may not be exposed.\n- All paper results use full fine-tuning, so whether LoRA capacity or optimization changes the picture is untested.",
        "title": "Reinforcement Learning via Self-Distillation"
      },
      "error": null,
      "full_text": true,
      "kind": "paper",
      "title": "Reinforcement Learning via Self-Distillation",
      "url": "https://arxiv.org/abs/2601.20802"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "measures": "model name reported by the created training client",
          "metric": "executed_base_model",
          "one_of": [
            "Qwen/Qwen3.5-4B"
          ]
        },
        {
          "measures": "loss_fn passed to forward_backward in both arms",
          "metric": "executed_loss_fn",
          "one_of": [
            "importance_sampling"
          ]
        },
        {
          "max": 0.95,
          "measures": "share of pilot rollouts where the judge flags at least one dimension, with non-empty critique text",
          "metric": "frac_rollouts_with_judge_failure_feedback",
          "min": 0.2
        },
        {
          "measures": "share of real assistant loss tokens with |log p_teacher - log p_student| > 0.05 nats",
          "metric": "frac_assistant_tokens_abs_sdpo_adv_gt_0.05",
          "min": 0.05
        },
        {
          "max": 10.0,
          "measures": "99th percentile |A_t| over assistant tokens before clipping; caps runaway teacher disagreement",
          "metric": "p99_abs_sdpo_adv_nats",
          "min": 0.1
        },
        {
          "measures": "share of GRPO groups whose 4 rubric rewards are not all equal",
          "metric": "frac_groups_nonzero_reward_variance",
          "min": 0.25
        }
      ],
      "reason": "SDPO only differs from GRPO if the judge produces failures with critiques, and if the teacher conditioned on that feedback actually shifts token probabilities. GRPO needs nonzero within-group reward variance. Executed model, loss and advantage source must match the spec."
    },
    "arms": [
      {
        "advantage": "group-centered scalar rubric reward",
        "lr": 4e-05,
        "name": "grpo",
        "seed": 0
      },
      {
        "advantage": "per-token log p_teacher - log p_student, frozen initial teacher",
        "lr": 4e-05,
        "name": "sdpo_token",
        "seed": 0
      }
    ],
    "base_model": "Qwen/Qwen3.5-4B",
    "baseline": "GRPO with the same env, user simulator, judge, tasks, task order, rollouts, LoRA config, lr, steps and eval. Only the advantage estimator differs.",
    "batch_size": 32,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 120
    },
    "command": "Custom tinker SDK script (rx_sdpo_tau.py), written against tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']).\n\nClients:\n- tc = create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32).\n- Frozen samplers for the user simulator and judge: Qwen/Qwen3.5-9B base sampling client.\n- Frozen teacher: Qwen3.5-4B base sampling client.\n\nEach of 20 steps:\n1. Take the next 8 train tasks (same fixed order for both arms).\n2. Sample 4 multi-turn rollouts per task via save_weights_for_sampler + sampling client (temperature 1.0, max 1024 tokens per turn, max 30 turns).\n3. Call the judge once per rollout, cached and shared logic for both arms.\n4. Compute advantages. GRPO uses reward minus group mean. SDPO (the custom part) calls teacher compute_logprobs on the feedback-conditioned prompt over the student's assistant tokens, then A_t = clip(lp_teacher - lp_student, -2, 2).\n5. tc.forward_backward(datums, loss_fn='importance_sampling') with the student sampling logprobs, then tc.optim_step(tinker.AdamParams(learning_rate=4e-5, beta1=0.9, beta2=0.95, eps=1e-8)).\n\nEval: at step 20, run 1 trial on each of the 115 retail test tasks, scored by the tau-bench ground-truth reward. Run the held-out gpt-oss-120b judge for per-dimension rates. Save per-task outcomes for the paired analysis.\n\nActivation-failure plan, in order:\n(a) Offline first. Forced activation: inject a known failure critique and check A_t is nonzero. Forced non-activation: give the teacher an empty-feedback, identical prompt and check A_t is about 0. Gradient check: the SDPO path with advantages set to GRPO values reproduces the stock loss and gradients within the correctness_preflight tolerances.\n(b) A 3-step treatment pilot is logged against activation_gate and preserved even if it fails.\n(c) If the gate fails, the only pre-specified calibration is one change to the judge rubric prompt's strictness, selected on 16 train tasks (never test tasks). Cap: 3 updates and 15 wall-minutes. Any change is recorded as a revised spec/deviation before the full run. Gates are never relaxed.",
    "correctness_preflight": {
      "paths": [
        {
          "baseline_revision": "UNRESOLVED: pin tinker-cookbook commit SHA (stock rl loop GRPO advantages + importance_sampling loss)",
          "id": "sdpo_token_advantage_into_importance_sampling",
          "kind": "loss",
          "reference_revision": "UNRESOLVED: pin SDPO authors' released code commit for arXiv 2601.20802 (token-level advantage); fallback is a float64 numpy reimplementation of the paper's equation, reviewed before running",
          "tolerances": {
            "baseline_gradient": {
              "atol": 0.0001,
              "min_cosine": 0.999,
              "rtol": 0.01
            },
            "baseline_loss": {
              "atol": 0.0001,
              "rtol": 0.001
            },
            "reference_gradient": {
              "atol": 0.001,
              "min_cosine": 0.995,
              "rtol": 0.02
            },
            "reference_loss": {
              "atol": 0.001,
              "rtol": 0.005
            }
          }
        }
      ],
      "required": true,
      "status": "unresolved: commit SHAs for tinker-cookbook and the SDPO reference code must be pinned before any full run; no full sweep until pinned"
    },
    "dataset": {
      "hf_path": null,
      "n_examples": 160,
      "name": "tau-bench retail tasks (github.com/sierra-research/tau-bench, pinned commit; not on HF)",
      "split": "train (training) / test (115 tasks, eval)"
    },
    "decision_rule": {
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "primary",
          "margin": 0.03,
          "metric": "tau_bench_retail_pass1",
          "weights": {
            "baseline": -1,
            "treatment": 1
          }
        }
      ],
      "estimand": "Mean per-task difference in tau-bench retail test ground-truth success (pass^1), sdpo_token minus grpo, at the fixed step-20 checkpoints; larger is better",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "tau-bench retail test",
      "metric": "tau_bench_retail_pass1 (ground-truth task success, 1 trial per task)",
      "n_examples": 115
    },
    "experiment_features": {
      "adaptive_selection": false,
      "checkpoint_restore": false,
      "custom_loss": true,
      "custom_optimizer": false,
      "generated_training_data": false,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": true,
      "warmup_adaptation": false
    },
    "feasibility": null,
    "feasibility_check": {
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Measured feasibility not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "categorical_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": false
    },
    "hypothesis": "With the same rollouts, judge calls and steps, token-level SDPO raises tau-bench retail test pass^1 over GRPO by at least 3 points. Its gain is largest on invalid-tool-call rate.",
    "lora_rank": 32,
    "max_seq_len": 8192,
    "method": "Both arms train a LoRA (rank 32) on Qwen/Qwen3.5-4B, thinking off, in a client-side multi-turn tau-bench retail environment built from the public tau-bench repo (pinned commit). The retail train tasks are used for training and the 115 retail test tasks are held out.\n\nShared rollout setup:\n- A frozen user simulator (Qwen/Qwen3.5-9B base sampling client) plays the customer.\n- A frozen rubric judge (Qwen/Qwen3.5-9B, fixed prompt) reads each trajectory and the policy doc. It returns text feedback plus 0/1 scores on three dimensions: invalid tool use, ungrounded policy/fact claims, and clarification-before-action when the request is ambiguous.\n- The scalar reward is the mean of the three dimension scores.\n- Each step uses 8 tasks x 4 rollouts, temperature 1.0, AdamW at lr 4e-5, for 20 steps.\n- Only assistant tokens get loss. User and tool tokens are masked.\n\nThe arms differ in exactly one thing, the advantage estimator fed to the stock importance_sampling loss:\n- Baseline: GRPO. Group-centered scalar judge reward with no std normalization, broadcast to all assistant tokens.\n- Treatment: token-level SDPO, the custom part. A teacher context is built from the conversation prefix, the judge feedback text, a successful sibling rollout from the same group if one exists, and 'Respond correctly to the customer.' The student's own trajectory is not shown to the teacher. The teacher is the frozen initial model (paper Table 4: frozen teacher 48.8 vs trust-region 50.6), scored via compute_logprobs on the student's sampled assistant tokens. The per-token advantage is A_t = log p_teacher(y_t) - log p_student(y_t), clipped to [-2, 2].\n\nDeviations from the paper:\n- Logit-level top-K KL is replaced by token-level, since only sampled-token logprobs are assumed. The paper ranks token-level below logit-level but above GRPO.\n- The model is 4B rather than 8B, to fit the 2 GPU-hour budget.\n\nThe primary eval is tau-bench's own ground-truth reward (database state plus required outputs), so it is independent of the training judge. Per-dimension rates are secondary and scored by a different held-out judge (openai/gpt-oss-120b) to expose judge gaming.",
    "optimizer": {
      "lr": 4e-05,
      "name": "adam"
    },
    "preflight": {
      "probe": "Create training clients for both arms with the exact adam config (lr 4e-5, rank 32). Run one forward_backward(importance_sampling) + optim_step on one recorded tau-bench rollout with GRPO advantages, then with SDPO advantages from teacher compute_logprobs. Confirm compute_logprobs returns per-token logprobs aligned to assistant tokens, and that the activation_gate token-advantage fraction is nonzero on that rollout.",
      "riskiest_assumption": "A 4B model conditioned on rubric-judge critique assigns meaningfully different token probabilities than the student, so SDPO advantages carry signal rather than noise. The paper shows SDPO weakens at small scale.",
      "tuning": "The arms share the optimizer and differ only in the advantage estimator, so both use the same fixed lr 4e-5 (a common LoRA scale-up of the paper's full-FT 1e-6/1e-5). No per-arm LR grid fits the 2 GPU-hour budget. A follow-up with more budget should give each arm a 3-point grid selected on held-out train tasks, with the grid-edge rule applied."
    },
    "primary_endpoint": "tau_bench_retail_pass1",
    "restoration_preflight": {
      "applicable": true,
      "fixed_batch_required": true,
      "groups": [
        {
          "arms": [
            {
              "name": "grpo",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05,
                "lora_rank": 32,
                "seed": 0
              }
            },
            {
              "name": "sdpo_token",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05,
                "lora_rank": 32,
                "seed": 0
              }
            }
          ]
        }
      ],
      "reason": "Both trained arms start from the same base LoRA initialization. Identity is verified by hashing the initial adapter state (or a shared immutable init checkpoint plus an observed load acknowledgment) and by matching logprobs on a fixed tokenized batch before the first update. The optimizer config is checked after creation."
    },
    "spec_preflight": {
      "findings": [],
      "status": "passed"
    },
    "steps": 20,
    "success_criterion": "Validated if the 95% paired Hoeffding interval for (sdpo_token minus grpo) tau_bench_retail_pass1 over 115 test tasks has its lower bound above +0.03. Failed if the upper bound is below +0.03. Otherwise inconclusive; at this size the pilot is expected to settle only large effects. Per-dimension rates (held-out gpt-oss-120b judge) are reported descriptively, not as the verdict.",
    "title": "Token-level SDPO vs GRPO, Qwen3.5-4B LoRA, tau-bench retail, rubric-judge reward",
    "training_type": "rl"
  },
  "summary": "SDPO turns a rubric judge's text critique into per-token advantages. We compare it with GRPO on tau-bench retail pass^1 and on per-dimension failure rates.",
  "tags": [
    "rl",
    "paper",
    "eval",
    "lora",
    "agents"
  ],
  "title": "Does token-level SDPO beat GRPO on tau-bench retail with a rubric judge?",
  "value_estimate": 4,
  "value_rationale": "Whether judge text critique beats scalar judge reward for LoRA agent RL is a question Spindle users building support agents would act on, and the paper never tested multi-turn or judge feedback. A 4B, 20-step pilot will only detect large effects."
}