🔬Modal Research Exchange

Forwarded from Sumit Sanghai (Intology): We have essentially reached the AutomationBench benchmaxxing target for our first milestone. At checkpoint 427, Qwen-27B achieves 49.17% held-out pass@1 across 120 tasks, up from 32.50% initially, and 52.33% across all 600 tasks, up from 39.33%. Held-out partial reward increases from 56.07% to 78.58%, approaching Gemini-Argon's 78.95% SOTA. Our next focus is synthetic enterprise-environment generation using Locus, followed by RL on those environments, to improve transfer to AutomationBench. We plan to use Modal for sampling and pass@k measurements to score and filter synthetic environments, then RL on the most promising ones. Evaluation: temperature 1.0...
didn't expect to actually get results from them! from a skim this looks promising, but would love for y'all to look at this critically

[kailash] Their RL setup all looks reasonable, seems like a standard one-step off policy dapo recipe that seems to work. it would've been nice to see a clear overfitting run with a smaller subset of tasks before this to validate the trainability of automation-bench style environments. Right now the improvement mostly seems to be in the partial reward, which is an incremental reward for intermediate steps; this denser reward can lend itself to reward hacking. The full success plot's gain seems to also sit within the existing noise intervals. I'd be curious if they've analyzed traces to see evidence of reward hacking. In my mind a convincing experiment would look like choosing a small subset of homogeneous automationbench tasks (<=10), have Locus generate a set of synthetic environments which target the same capabilities (same tool calls/skills) without referencing the tasks, and do a short overfitting RL run that shows training on the synthetic envs directly improves eval scores on the small held out set. Or run a control by training on a "random" selection from a broad pool of generated envs and show targeted synthetic env generation has better eval improvements.
[micah] Their harness includes some Spindle components but doesn't run through the Spindle service.

{
  "adjacent_experiment": "Short RL overfitting run on about 10 public tool-use tasks, targeted versus random training tasks, tracking full-success and partial reward separately to detect partial-reward hacking.",
  "reproducibility_reason": "The core claim depends on Intology's Locus-generated synthetic environments and their AutomationBench harness. Neither is available to us, so the targeted-versus-random environment comparison cannot be built.",
  "reproducible_on_spindle": false,
  "spec": null,
  "summary": "RL on synthetic enterprise environments aimed at a small AutomationBench task subset, compared against randomly chosen environments, with held-out pass@1 and traces checked for partial-reward hacking.",
  "tags": [
    "rl",
    "eval",
    "data",
    "not-reproducible"
  ],
  "title": "Does targeted synthetic-env RL beat random envs on AutomationBench?",
  "value_estimate": 3,
  "value_rationale": "Knowing whether targeted environment generation transfers, and whether dense partial rewards get hacked, matters to agentic-RL customers. It is tied to a partner's private tooling and benchmark harness."
}