🔬Modal Research Exchange

https://github.com/browser-use/rl-env-use

[jason.mancuso] https://x.com/browser_use/status/2112767614468694286
[peyton] feels too good to be true
[peyton] but we should try
[jason.mancuso] yeah not sure how it wouldn't just become ungrounded noise
[peyton] like this is literally RSI if it works lol
[peyton] btw i am bullish on environment gen tho

{
  "adjacent_experiment": "",
  "reproducibility_reason": "Fits math_rl LoRA RL on a served Qwen model. The task generator is the frozen base model, sampled through the sampling client with code-computed labels. The generated set plugs in as a client-side dataset in the cookbook RL loop.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": {
        "caveats": "(1) The headline Qwen result is prompt optimization on 35 forecasts from 29 trials; the CI (+0.07 to +0.45) is wide. The README says most of the gain is calibration, and a constant 50% baseline already scores 0.75, so 0.78 is barely above it. (2) No RL weight training is reported, so it says nothing about whether RL on generated environments transfers. (3) The mining table counts 'verified facts' with no definition of 'mission', how verification works, or why Claude Code scores 0; it may compare a general agent against a purpose-built one, and the second row's 'same model both sides' is not explained further. (4) There is no ablation of the frontier-panel filter, the harder-rewrite step, or the leak and memorization rejection. (5) The model names (GPT-6 Astra, Opus 5.5, Qwen 3.6 35B-A3B) and the 'since July 2026' dates could not be verified by me, so treat the figures as the repo's own claims. (6) The task type, forecasting binary outcomes, differs from math reasoning, and gold answers from real-world outcomes are inherently noisy. (7) The linked tweet may contain additional claims not captured here. (8) Self-generated math tasks risk wrong gold labels, difficulty collapse, and distribution narrowness, none of which this source addresses.",
        "claim": "From a one-line skill description, a pipeline can build a verified RL environment of real, web-mined cases that frontier models do not saturate, and can hillclimb a system prompt for a smaller target model. The shipped example claims Qwen 3.6 35B-A3B goes from 0.53 to 0.78 on held-out clinical-trial forecasts, level with Opus 5.5. The claim is about prompt optimization on mined tasks, not about RL weight updates.",
        "context": "It builds on Anthropic's 'Automating eval design and hillclimbing' workflow and is a Browser Use product plug (cloud API key, utm link). Its distinction from LLM-synthesized tasks is that cases are real and the outcomes are grounded in later-published results, which is the opposite of the 'self-generated math tasks' framing in the idea title. The Modal discussion reflects this mismatch: jason.mancuso worries self-generated tasks would 'become ungrounded noise', and peyton calls it 'literally RSI if it works' but is bullish on environment generation. The repo is better evidence for grounding via real-world mining than for self-generation. Modal's team may care because automated environment generation is a route to RL-with-verifiable-rewards data in domains without existing benchmarks. The repo can also be a source of environments for Spindle, and the paired-CI hillclimbing protocol is a useful model for small-n evaluation.",
        "evidence": "Mining comparison table (Claude Code vs Browser Use; 'missions' is not defined in the README): 8 x clinical trial results: 0 of 86 facts verified vs 17 verified and 4 complete cases. 6 x clinical trial results, same model both sides: 0 of 43 vs 3 verified and 2 complete cases. 8 x human genetic variants (UniProt, gnomAD, GTEx): 6 of 82 vs 40 verified. Shipped example `envs/trial-forecast`: 35 forecasts from 29 ClinicalTrials.gov trials whose results were posted 'since July 2026', after the models' training data; built for $72, of which $17 was Browser Use. Held-out test scores from the chart alt text: GPT-6 Astra 0.81, climbed Qwen 0.78, Opus 5.5 0.77. Climbed Qwen 3.6 35B-A3B goes from 0.53 to 0.78 (+0.25, 95% CI +0.07 to +0.45). Always answering 50% scores 0.75, so the README itself says most of the gain is calibration, because the model stopped answering 0 or 1. The scoring metric is not named; a 0.75 score for always-50% is consistent with 1 minus Brier score, but this is an inference. There is no RL training result, no GSM8K or math result, and no comparison to a self-generated-task baseline. The linked tweet (x.com/browser_use/status/2112767614468694286) was not available.",
        "method": "The repo is built on Anthropic's 'Automating eval design and hillclimbing' workflow, with one change: tasks are real cases mined from the web by Browser Use agents rather than synthesized by an LLM. (1) Mining: agents find the materials (e.g. a trial protocol or a 10-Q) and the outcome published later. (2) Task writing: gold answers are computed by code, and any task a frontier panel solves is rewritten harder. (3) Filtering: tasks with leaked answers, memorized outcomes or ambiguous prompts are rejected. (4) Hillclimbing: the pipeline proposes edits to the target model's system prompt and evaluates them with a train/test split, a noise floor and paired confidence intervals. In the shipped run it kept 1 of 12 proposed edits. (5) Export: the dataset can be exported for an RL trainer or served in code via an `Env` class. The CLI is `rl-env-use create \"<skill>\" --budget 40` followed by `rl-env-use climb envs/<name> --model qwen-35b`. It needs Browser Use and OpenRouter API keys. The README does not describe the prompts, the panel composition, the rewrite loop, the scoring function or the verification procedure in reproducible detail; those live in SKILL.md and the code, which were not read.",
        "reproduction": "The idea asks whether RL on self-generated math tasks matches RL on GSM8K. This source does not test that. It covers web-mined real-world forecasting and fact tasks, and the only 'training' is prompt hillclimbing, with no weight updates. A faithful test of the idea would be a new experiment that borrows only the environment-generation concept. Smallest version: (a) Baseline: LoRA RL (GRPO-style, verifiable exact-match reward) on GSM8K train with a Qwen3.5 or similar small open model; evaluate on the GSM8K test set plus a held-out set such as MATH-500 or a fresh set. (b) Treatment: the same model, or a stronger generator, writes N synthetic grade-school-style problems with code-computed gold answers. Filter by agreement across several samples or a programmatic check, and drop trivial or ambiguous items. Run the same RL with matched problem count, steps, learning rate and LoRA rank. (c) Controls: base model, and a mix of the two datasets. Use several seeds, because LoRA RL variance on GSM8K is large. Use a pass@1 delta with confidence intervals. Hyperparameters, compute and model sizes are not given by the source for any RL run, so everything must be chosen fresh. The synthetic task generation, verification and RL are all doable in Spindle with SFT/RL primitives. What does not carry over: Browser Use web mining, the ClinicalTrials.gov, UniProt, gnomAD and GTEx forecasting tasks, the 'post-training-cutoff' contamination control (which relies on the dates in the README), and the system-prompt hillclimbing loop, which is not a LoRA method. The `Env` export format is not documented here, so adapting it to Spindle would need a look at the repo.",
        "title": "rl-env-use: RL environments for anyone and any domain (browser-use/rl-env-use, GitHub README)"
      },
      "error": null,
      "full_text": true,
      "kind": "github",
      "title": "rl-env-use: RL environments for anyone and any domain (browser-use/rl-env-use, GitHub README)",
      "url": "https://github.com/browser-use/rl-env-use"
    },
    {
      "brief": {
        "caveats": "The source is a promotional tweet with no evidence, so the idea rests entirely on the unread repo. The meaning of 'verified' is undefined: LLM-judged or LLM-generated answer keys can contain wrong labels that RL will exploit or be harmed by. Generated tasks may be easier, narrower, or more templated than GSM8K, so reward may saturate quickly without real capability gain. If the generator is the same model being trained, the setup could amplify its own errors; there is no ablation separating grounding from self-generation. GSM8K is also near-saturated for modern Qwen-class models, so a base model might have little headroom and any difference between conditions could be noise. Contamination between generated tasks and GSM8K test items is possible. The matched-compute comparison is sensitive to difficulty filtering (for example the pass-rate band of useful prompts). Finally, a null or positive result on math does not show that the tool works for 'any domain', which is the tweet's actual claim.",
        "claim": "The tweet claims that rl-env-use is an open-source tool that turns a one-line skill description into a 'verified environment built from real cases', ready for RL 'hillclimbing', in any domain. It makes no claim about math, GSM8K, or matching human-curated datasets. That comparison is the Modal team's own framing.",
        "context": "This is part of the environment and task generation line of work: synthetic verifiable tasks for RLVR, self-play or self-instruct-style curricula, and LLM-written environments, as opposed to hand-curated sets like GSM8K or MATH. The team's discussion links it to recursive self-improvement ('literally RSI if it works'). Jason's skepticism, that self-generated tasks may drift into 'ungrounded noise', is the central risk. The tweet's 'built from real cases' wording suggests the tool grounds tasks in real data, which would partly address that risk, but this is unconfirmed. Modal's team might care because cheap, verified RL environments would remove the dataset bottleneck for RLVR on LoRA adapters and could make domain-specific post-training on Spindle much easier. Peyton is bullish on environment generation generally.",
        "evidence": "None. The tweet is a launch announcement with no numbers, models, datasets, baselines, or training curves, and it reports no results showing that RL on the generated environments improves any model. The only substantive claims are 'verified', 'built from real cases', and '100% open source'. The idea thread also contains no data, only opinion (Peyton: 'feels too good to be true'; Jason: it might 'become ungrounded noise'). Any evidence would have to come from the repo, which was not available here.",
        "method": "Not stated in the tweet. The only description is: 'Describe a skill in one line. Get a verified environment built from real cases.' This implies a pipeline of skill description -> collected or generated cases -> some verification step -> an RL environment. The tweet does not say how cases are sourced (scraped, LLM-generated, or user-supplied), what 'verified' means (programmatic checker, LLM judge, or human review), which model does the generation, or what the environment interface is. The GitHub repo was not read, so none of this can be reimplemented from the source text. A triage agent should read the repo README before committing to a design that depends on these details.",
        "reproduction": "Note that the idea title ('RL on self-generated math tasks vs GSM8K') is a test of the general premise, not a reproduction of anything the tweet reports. Smallest experiment:\n(1) Generate a math environment with rl-env-use, or with a simple stand-in if the tool is hard to run. The input would be a one-line skill such as 'grade-school arithmetic word problems with integer answers'. Cap it at roughly GSM8K's size, 7,473 train problems, for a fair comparison. Record how the tool verifies answers.\n(2) Audit a random sample of about 100 generated tasks by hand, or against a strong model, for answer correctness and for duplication or near-duplication of GSM8K test items. Report the error rate before any training.\n(3) Run two matched RL runs of LoRA GRPO-style RL with verifiable reward (exact-match on the final answer) on the same small open model (for example a small Qwen3.5 variant): one on GSM8K train, one on the generated set. Use the same steps, batch size, group size, learning rate, and LoRA rank. Add a no-training baseline. Specific hyperparameters are not given by the source, so choose and fix them yourself. Use 2-3 seeds if budget allows.\n(4) Evaluate on the GSM8K test set (1,319 problems) with greedy or fixed-temperature sampling, plus a held-out harder set such as MATH-500 to check transfer. Track reward curves and response length to catch reward hacking.\nCompute is modest: a small model with LoRA fits a single GPU per run through Spindle's Tinker-style sampling and training loop.\nCaveats on carry-over: if the repo's environments are browser or agentic (the tweet does not say), they may not map to Spindle's text-only RL loop without a custom environment wrapper. The 'self-generated' part only tests the premise if the generator is the model being trained or a comparable model, and the tweet does not specify that.",
        "title": "Introducing rl-env-use: RL environments for anyone, in any domain (@browser_use tweet, linked to github.com/browser-use/rl-env-use)"
      },
      "error": null,
      "full_text": true,
      "kind": "tweet",
      "title": "Introducing rl-env-use: RL environments for anyone, in any domain (@browser_use tweet, linked to github.com/browser-use/rl-env-use)",
      "url": "https://x.com/browser_use/status/2112767614468694286"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "max": 1.0,
          "measures": "share of rl_selfgen training problems whose ids are in the audited generated manifest",
          "metric": "treatment_fraction_batches_from_generated_set",
          "min": 1.0
        },
        {
          "max": 1.0,
          "measures": "share of rl_gsm8k training problems whose ids are in the GSM8K train manifest",
          "metric": "baseline_fraction_batches_from_gsm8k_train",
          "min": 1.0
        },
        {
          "measures": "fraction of rl_selfgen groups in the first 2 pilot steps with mixed rewards",
          "metric": "selfgen_frac_groups_nonzero_reward_variance_pilot",
          "min": 0.3
        },
        {
          "max": 0.85,
          "measures": "mean reward on rl_selfgen pilot steps",
          "metric": "selfgen_mean_train_reward_pilot",
          "min": 0.15
        },
        {
          "max": 0.0,
          "measures": "normalized exact matches between accepted tasks and the GSM8K test questions",
          "metric": "generated_eval_exact_overlaps",
          "min": 0.0
        }
      ],
      "reason": "The treatment is a change of training data. The pilot confirms three things: (1) each arm actually executes on its intended data; (2) generated tasks give non-degenerate GRPO groups, so RL has signal (all-0 or all-1 groups give zero advantage); (3) the generated set is not contaminated with eval items."
    },
    "arms": [
      {
        "data": "none",
        "name": "base",
        "seed": 0,
        "trained": false
      },
      {
        "data": "640 GSM8K train problems",
        "lr": 4e-05,
        "name": "rl_gsm8k",
        "optimizer": "adam",
        "seed": 0,
        "steps": 40,
        "trained": true
      },
      {
        "data": "640 audited Qwen3.5-2B-generated tasks",
        "lr": 4e-05,
        "name": "rl_selfgen",
        "optimizer": "adam",
        "seed": 0,
        "steps": 40,
        "trained": true
      }
    ],
    "base_model": "Qwen/Qwen3.5-2B",
    "baseline": "rl_gsm8k is the same stock math_rl LoRA RL on 640 GSM8K train problems, with identical model, rank, lr, group size, batch, steps, seed, renderer, reward and eval. The untrained base model is evaluated as a reference for gain recovery.",
    "batch_size": 16,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 120
    },
    "command": "Step 1, generation, filtering and audit (custom script):\n  python gen_tasks.py --base_url $TINKER_BASE_URL --generator Qwen/Qwen3.5-2B --n_accept 640 --filter_samples 8 --keep_band 1-6 --overlap_ref openai/gsm8k:main/test --out /tmp/rx-selfgen/tasks.jsonl\n  This uses only tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']).create_sampling_client plus sandboxed exec of solve().\n  python audit_tasks.py --base_url $TINKER_BASE_URL --solver Qwen/Qwen3.5-397B-A17B --sample 100 --compare numeric_exact --max_label_errors 0 --out /tmp/rx-selfgen/audit.json\n  The run stops if the audit fails.\n\nStep 2, timing pilot: 2 steps per arm plus 100 eval items. Publish feasibility before continuing.\n\nStep 3, baseline:\n  python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=16 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=40 seed=0 log_path=/tmp/rx-run/rl_gsm8k\n  Restrict to the 640-problem seed-0 shuffled subset via the dataset builder.\n\nStep 4, treatment:\n  The same command with a custom client-side dataset builder (custom part) that serves /tmp/rx-selfgen/tasks.jsonl through the same math env and reward, with log_path=/tmp/rx-run/rl_selfgen.\n\nStep 5, eval: sample greedily from base and both save_weights_for_sampler checkpoints on all 1,319 GSM8K test problems with max_tokens=512, score exact match per item, and apply the paired decision rule.",
    "correctness_preflight": {
      "reason": "Both arms use the unchanged math_rl importance-sampling loss and the stock AdamW backend. Only the training-problem source differs, and that is client-side data code with no custom loss or optimizer.",
      "required": false
    },
    "dataset": {
      "hf_path": "openai/gsm8k",
      "n_examples": 640,
      "name": "gsm8k (baseline train / eval) and self-generated tasks (treatment train)",
      "split": "main/train for baseline, main/test for eval"
    },
    "decision_rule": {
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "noninferiority_vs_gsm8k_rl",
          "margin": -0.03,
          "metric": "accuracy",
          "weights": {
            "rl_gsm8k": -1,
            "rl_selfgen": 1
          }
        },
        {
          "bounds": [
            0,
            1
          ],
          "id": "recovered_half_gain",
          "margin": 0.0,
          "metric": "accuracy",
          "weights": {
            "base": -0.5,
            "rl_gsm8k": -0.5,
            "rl_selfgen": 1
          }
        }
      ],
      "estimand": "GSM8K test greedy exact-match accuracy of fixed seed-0 checkpoints of Qwen3.5-2B LoRA RL on self-generated tasks, relative to RL on GSM8K train and to the untrained base; larger is better",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "GSM8K test (openai/gsm8k main/test)",
      "metric": "exact-match accuracy, greedy, max_tokens=512",
      "n_examples": 1319
    },
    "experiment_features": {
      "adaptive_selection": false,
      "checkpoint_restore": false,
      "custom_loss": false,
      "custom_optimizer": false,
      "generated_training_data": true,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": true,
      "warmup_adaptation": false
    },
    "feasibility": null,
    "feasibility_check": {
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Measured feasibility not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "numeric_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": true
    },
    "hypothesis": "Two things should hold on the 1,319 GSM8K test problems: (1) LoRA RL on 640 audited tasks written by frozen Qwen3.5-2B reaches GSM8K test exact-match accuracy no more than 3 points below the same RL on 640 GSM8K train problems; (2) it recovers more than half of that run's gain over the untrained base model.",
    "lora_rank": 32,
    "max_seq_len": 1024,
    "method": "The linked repo (browser-use/rl-env-use) does web-mining and system-prompt hillclimbing with no RL weight training. This experiment borrows only the environment-generation premise and tests it directly with RL.\n\nGeneration (custom, client-side):\n- Frozen base Qwen/Qwen3.5-2B, through the sampling client at T=1.0, is prompted with one fixed instruction: 'Write one grade-school math word problem with a single integer answer, then a short Python function solve() that returns the answer.'\n- Each solve() runs in a sandbox with a 2 s timeout. The label is its integer return value.\n- Filters, fixed before any training:\n  - parse and execution succeed;\n  - the problem text does not contain the answer;\n  - the problem is 20 to 120 words long;\n  - normalized exact match against all GSM8K test questions finds no overlap;\n  - the base model solves the problem from text alone in 1 to 6 of 8 samples at max_tokens=512, so the task is learnable and not trivial.\n- The first 640 accepted tasks, in generation order, form the training set. No selection is made on eval results.\n\nGeneration audit, before any training:\n- Draw 100 random accepted tasks.\n- Each is re-solved blind from the problem text alone by Qwen/Qwen3.5-397B-A17B, which is used for sampling only. Record solver identity and worked solutions.\n- Compare answers numeric_exact.\n- Any label error, or any eval overlap, stops the run for repair and re-audit. Do not train.\n- Save the generated/accepted/rejected/execution-error counts and the data and audit manifests as evidence.\n\nTraining:\n- Both RL arms start from fresh LoRA rank 32 on Qwen3.5-2B and use the identical stock math_rl loop:\n  - group_size=8, groups_per_batch=16, lr=4e-5 (AdamW), max_tokens=512, 40 steps, seed 0;\n  - the same renderer and the same answer-extraction exact-match reward.\n- Baseline arm: 640 GSM8K train problems (first 640 after a fixed seed-0 shuffle).\n- Treatment arm: the 640 generated tasks. This is the only difference between the arms.\n- A third, untrained arm: base model, eval only.\n\nEval:\n- All three arms on the full GSM8K test split (1,319 problems), greedy decoding, max_tokens=512, the same prompt and extractor.\n- Results are paired per item.\n\nPreflight: no custom loss or optimizer. The only custom code is the dataset builder and the generation/audit script.",
    "optimizer": {
      "lr": 4e-05,
      "name": "adam"
    },
    "primary_endpoint": "gsm8k_test_exact_match_accuracy_greedy_1319",
    "restoration_preflight": {
      "applicable": true,
      "fixed_batch_required": true,
      "groups": [
        {
          "arms": [
            {
              "name": "rl_gsm8k",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05,
                "lora_rank": 32,
                "seed": 0
              }
            },
            {
              "name": "rl_selfgen",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05,
                "lora_rank": 32,
                "seed": 0
              }
            }
          ]
        }
      ],
      "reason": "Two RL arms are trained from a shared starting point. Initial identity must be shown by matching logprobs on a fixed tokenized batch, from both freshly created LoRA clients, before the first update. The base model name alone is insufficient. The active optimizer and its config are verified after creation."
    },
    "steps": 40,
    "success_criterion": "Validated if both criteria hold, with simultaneous 95% paired Hoeffding intervals over the 1,319 GSM8K test items:\n(1) rl_selfgen minus rl_gsm8k accuracy has a lower bound above -0.03 (noninferiority);\n(2) rl_selfgen - 0.5*rl_gsm8k - 0.5*base has a lower bound above 0 (more than half the gain recovered).\nFailed if either upper bound falls below its margin. Otherwise inconclusive. A one-seed pilot may be inconclusive, and it says nothing about training-seed variance.",
    "title": "Self-generated math tasks vs GSM8K train for LoRA RL on Qwen3.5-2B",
    "training_type": "rl"
  },
  "summary": "Qwen3.5-2B writes its own checkable math word problems for RL training; we compare GSM8K test accuracy against the same RL run on real GSM8K problems.",
  "tags": [
    "rl",
    "data",
    "environment-generation",
    "lora",
    "eval"
  ],
  "title": "Does RL on Qwen3.5-2B's own math tasks match GSM8K RL?",
  "value_estimate": 4,
  "value_rationale": "Whether cheap self-generated verifiable tasks can stand in for curated RL data directly affects how Spindle users source RLVR data. The linked repo only shows prompt hillclimbing on web-mined forecasts, so this experiment tests the team's premise rather than reproducing a reported result."
}