Modal Research Exchange

you are

Self-generated math tasks vs GSM8K train for LoRA RL on Qwen3.5-2B

Failed no result Devin Qwen/Qwen3.5-2B RL 2.0 GPU-h budget started 6h ago 35m 07s
Redo
Pick where the agent works.
View session in Devin ↗

Launch instructions verified (sha256 d371b1ef...) and fully read. Generated 5,120 candidates with frozen Qwen3.5-2B (temp 1.0): 640 accepted, 3,220 rejected, 1,260 execution errors.

Full summary

Launch instructions verified (sha256 d371b1ef...) and fully read. Generated 5,120 candidates with frozen Qwen3.5-2B (temp 1.0): 640 accepted, 3,220 rejected, 1,260 execution errors. Exact overlap with the 1,319 GSM8K test items is 0. Blind audit of 100 deterministically selected accepted tasks by Qwen3.5-397B-A17B (greedy, 8,192 max tokens, numeric_exact): 56 gave a boxed answer and 25 of those disagree with the stored solve() label. Examples: selfgen_000098 is labelled 455 but the text gives 5 pages/day x 7 days = 35; selfgen_002765 is labelled 8 but the problem gives 2x3 = 6. The other 44 hit the token limit without a boxed answer. generation_audit.validate returned status=failed (label errors 25 > 0, plus incomplete re-solve sample). As the spec requires, the run stopped before any optimizer update: no restoration probe of trained arms, no activation pilot, no RL training, no full eval. Measured otherwise: shared init LoRA checkpoint created and probed, then deleted at cleanup. Frozen evaluator smoke test on base model: 0.78 on the first 100 GSM8K test items. That is a pilot measurement, not the primary endpoint. No Modal apps or environments were created.

No result No result: insufficient evidence.

Result

No arm results yet.

generated_candidates
5,120
accepted_tasks
640
rejected_candidates
3,220
execution_error_candidates
1,260
eval_overlap_count
0
audit_sample_size
100
audit_resolved
56
audit_label_errors
25
pilot_base_accuracy_first100
0.78
Changes from the spec 0 · original hypothesis not tested

None reported: ran as specified.

Original hypothesis tested: no

Tested instead: Only the pre-training generation stage ran: generation, filtering and the blind 397B audit. The audit failed, so neither RL arm was trained and the primary endpoint was never measured.

Devin's verdict on the changed experiment: inconclusive

Checks · activation unverified · evaluator unknown · feasibility unknown
Custom implementation correctness: not_applicable
spec.correctness_preflight opts out and no feature it covers is true
Validates executor-reported evidence; autonomous training calls are not blocked by this server.
Executor reported inconclusive; shown as no result.
Treatment activation: unverified
Not reported: treatment_fraction_batches_from_generated_set, baseline_fraction_batches_from_gsm8k_train, selfgen_frac_groups_nonzero_reward_variance_pilot, selfgen_mean_train_reward_pilot, generated_eval_exact_overlaps.
treatment_fraction_batches_from_generated_set
not reported needs >= 1 and <= 1 · missing
baseline_fraction_batches_from_gsm8k_train
not reported needs >= 1 and <= 1 · missing
selfgen_frac_groups_nonzero_reward_variance_pilot
not reported needs >= 0.3 · missing
selfgen_mean_train_reward_pilot
not reported needs >= 0.15 and <= 0.85 · missing
generated_eval_exact_overlaps
not reported needs >= 0 and <= 0 · missing
Initialization / optimizer verification: unverified
rl_gsm8k: missing or duplicate arm evidence.
Research decision: uncertain
Decision evidence was not recorded.
Inference unit: fixed_checkpoint_items · 1 scheduled seed(s). Bounded Hoeffding confidence interval.
Evaluator comparability: unknown
Evaluator provenance was not recorded.
Feasibility projection: unknown
Projected costs, not measured GPU usage.
Projected wall time
unknown
Projected GPU allocation
unknown
  • Re-projection required: lora_rank changed or was not recorded
  • Re-projection required: dataset changed or was not recorded
  • Re-projection required: steps changed or was not recorded
  • Re-projection required: batch_size changed or was not recorded
  • Re-projection required: max_seq_len changed or was not recorded
  • Re-projection required: max_tokens changed or was not recorded
  • Re-projection required: arms changed or was not recorded
  • Re-projection required: n_examples changed or was not recorded
  • Re-projection required: n_eval changed or was not recorded
  • Representative training timing not recorded
  • Positive recovery reserve not recorded
Timing, sample counts and allocation evidence
{
  "generation_applicability": {
    "check": "generation_audit",
    "enforce": true,
    "features": {
      "generated_training_data": true
    },
    "reasons": [
      "experiment_features: generated_training_data is true"
    ],
    "status": "required"
  },
  "generation_counts": {
    "candidates": 5120,
    "filter_samples": 40960,
    "yield": 0.125
  },
  "gpu_hours": null,
  "known_gpu_hours": 0,
  "phases": {
    "audit": {
      "gpu_hours": null,
      "planned_samples": 100.0,
      "planned_units": 100,
      "wall_seconds": 1253.3
    },
    "evaluation": {
      "gpu_hours": null,
      "planned_samples": 4257.0,
      "planned_units": 4257,
      "wall_seconds": 700.2764999999999
    },
    "filter": {
      "gpu_hours": null,
      "planned_samples": 40960.0,
      "planned_units": 40960,
      "wall_seconds": 288.7494051346274
    },
    "generation": {
      "gpu_hours": null,
      "planned_samples": 5120.0,
      "planned_units": 5120,
      "wall_seconds": 132.65
    },
    "setup": {
      "gpu_hours": null,
      "planned_samples": 0.0,
      "planned_units": 1,
      "wall_seconds": 12.99
    }
  },
  "problems": [
    "Re-projection required: lora_rank changed or was not recorded",
    "Re-projection required: dataset changed or was not recorded",
    "Re-projection required: steps changed or was not recorded",
    "Re-projection required: batch_size changed or was not recorded",
    "Re-projection required: max_seq_len changed or was not recorded",
    "Re-projection required: max_tokens changed or was not recorded",
    "Re-projection required: arms changed or was not recorded",
    "Re-projection required: n_examples changed or was not recorded",
    "Re-projection required: n_eval changed or was not recorded",
    "Representative training timing not recorded",
    "Positive recovery reserve not recorded"
  ],
  "record": {
    "config": {
      "audit_sample": 100,
      "audit_solver": "Qwen/Qwen3.5-397B-A17B",
      "base_model": "Qwen/Qwen3.5-2B",
      "filter_samples": 8,
      "generation_chunk": 1024,
      "pilot_eval_items": 100
    },
    "effective_config": {
      "arms": [
        "base",
        "rl_gsm8k",
        "rl_selfgen"
      ],
      "base_model": "Qwen/Qwen3.5-2B",
      "batch_size": "16 groups x 8",
      "max_tokens": 512,
      "n_eval": 1319,
      "steps": 40,
      "training_executed": false
    },
    "generation": {
      "accepted": 640,
      "attempted": 5120,
      "filter_samples_per_candidate": 8,
      "target_accepted": 640
    },
    "measurements": {
      "audit": {
        "allocation_source": null,
        "gpu_count": null,
        "planned_units": 100,
        "samples": 100,
        "seconds": 1253.3,
        "units": 100
      },
      "evaluation": {
        "allocation_source": null,
        "gpu_count": null,
        "planned_units": 4257,
        "samples": 100,
        "seconds": 16.45,
        "units": 100
      },
      "filter": {
        "allocation_source": null,
        "gpu_count": null,
        "planned_units": 25552,
        "samples": 25552,
        "seconds": 180.13,
        "units": 25552
      },
      "generation": {
        "allocation_source": null,
        "gpu_count": null,
        "planned_units": 5120,
        "samples": 5120,
        "seconds": 132.65,
        "units": 5120
      },
      "setup": {
        "allocation_source": null,
        "gpu_count": null,
        "planned_units": 1,
        "samples": 0,
        "seconds": 12.99,
        "units": 1
      }
    },
    "primary_endpoint": "gsm8k_test_exact_match_accuracy_greedy_1319",
    "recovery": {
      "allocation_source": null,
      "gpu_count": null,
      "seconds": 0
    },
    "source": "generation_summary.json, audit_report.json, init_checkpoint_preflight.json, run-metrics.json"
  },
  "recorded_at": 1793743561.4038932,
  "status": "unknown",
  "wall_seconds": null
}
Generated task audit: unverified
Generation audit artifacts missing, inconsistent or unreadable.
Answer agreement with recorded independent solutions does not establish semantic correctness. Normalized exact overlap does not detect paraphrases.
Spec exp_c2bf4b4ffc
Experiment version
exp_c2bf4b4ffc
Execute the selected saved experiment version.

Hypothesis

Two things should hold on the 1,319 GSM8K test problems: (1) LoRA RL on 640 audited tasks written by frozen Qwen3.5-2B reaches GSM8K test exact-match accuracy no more than 3 points below the same RL on 640 GSM8K train problems; (2) it recovers more than half of that run's gain over the untrained base model.

Success criterion

Validated if both criteria hold, with simultaneous 95% paired Hoeffding intervals over the 1,319 GSM8K test items: (1) rl_selfgen minus rl_gsm8k accuracy has a lower bound above -0.03 (noninferiority); (2) rl_selfgen - 0.5*rl_gsm8k - 0.5*base has a lower bound above 0 (more than half the gain recovered). Failed if either upper bound falls below its margin. Otherwise inconclusive. A one-seed pilot may be inconclusive, and it says nothing about training-seed variance.

gsm8k (baseline train / eval) and self-generated tasks (treatment train) 40 steps batch 16 LoRA r32 lr 4e-05 120 min wall budget
Method
The linked repo (browser-use/rl-env-use) does web-mining and system-prompt hillclimbing with no RL weight training. This experiment borrows only the environment-generation premise and tests it directly with RL. Generation (custom, client-side): - Frozen base Qwen/Qwen3.5-2B, through the sampling client at T=1.0, is prompted with one fixed instruction: 'Write one grade-school math word problem with a single integer answer, then a short Python function solve() that returns the answer.' - Each solve() runs in a sandbox with a 2 s timeout. The label is its integer return value. - Filters, fixed before any training: - parse and execution succeed; - the problem text does not contain the answer; - the problem is 20 to 120 words long; - normalized exact match against all GSM8K test questions finds no overlap; - the base model solves the problem from text alone in 1 to 6 of 8 samples at max_tokens=512, so the task is learnable and not trivial. - The first 640 accepted tasks, in generation order, form the training set. No selection is made on eval results. Generation audit, before any training: - Draw 100 random accepted tasks. - Each is re-solved blind from the problem text alone by Qwen/Qwen3.5-397B-A17B, which is used for sampling only. Record solver identity and worked solutions. - Compare answers numeric_exact. - Any label error, or any eval overlap, stops the run for repair and re-audit. Do not train. - Save the generated/accepted/rejected/execution-error counts and the data and audit manifests as evidence. Training: - Both RL arms start from fresh LoRA rank 32 on Qwen3.5-2B and use the identical stock math_rl loop: - group_size=8, groups_per_batch=16, lr=4e-5 (AdamW), max_tokens=512, 40 steps, seed 0; - the same renderer and the same answer-extraction exact-match reward. - Baseline arm: 640 GSM8K train problems (first 640 after a fixed seed-0 shuffle). - Treatment arm: the 640 generated tasks. This is the only difference between the arms. - A third, untrained arm: base model, eval only. Eval: - All three arms on the full GSM8K test split (1,319 problems), greedy decoding, max_tokens=512, the same prompt and extractor. - Results are paired per item. Preflight: no custom loss or optimizer. The only custom code is the dataset builder and the generation/audit script.
Baseline
rl_gsm8k is the same stock math_rl LoRA RL on 640 GSM8K train problems, with identical model, rank, lr, group size, batch, steps, seed, renderer, reward and eval. The untrained base model is evaluated as a reference for gain recovery.
Full spec
{
  "activation_gate": {
    "applicable": true,
    "checks": [
      {
        "max": 1.0,
        "measures": "share of rl_selfgen training problems whose ids are in the audited generated manifest",
        "metric": "treatment_fraction_batches_from_generated_set",
        "min": 1.0
      },
      {
        "max": 1.0,
        "measures": "share of rl_gsm8k training problems whose ids are in the GSM8K train manifest",
        "metric": "baseline_fraction_batches_from_gsm8k_train",
        "min": 1.0
      },
      {
        "measures": "fraction of rl_selfgen groups in the first 2 pilot steps with mixed rewards",
        "metric": "selfgen_frac_groups_nonzero_reward_variance_pilot",
        "min": 0.3
      },
      {
        "max": 0.85,
        "measures": "mean reward on rl_selfgen pilot steps",
        "metric": "selfgen_mean_train_reward_pilot",
        "min": 0.15
      },
      {
        "max": 0.0,
        "measures": "normalized exact matches between accepted tasks and the GSM8K test questions",
        "metric": "generated_eval_exact_overlaps",
        "min": 0.0
      }
    ],
    "reason": "The treatment is a change of training data. The pilot confirms three things: (1) each arm actually executes on its intended data; (2) generated tasks give non-degenerate GRPO groups, so RL has signal (all-0 or all-1 groups give zero advantage); (3) the generated set is not contaminated with eval items."
  },
  "arms": [
    {
      "data": "none",
      "name": "base",
      "seed": 0,
      "trained": false
    },
    {
      "data": "640 GSM8K train problems",
      "lr": 4e-05,
      "name": "rl_gsm8k",
      "optimizer": "adam",
      "seed": 0,
      "steps": 40,
      "trained": true
    },
    {
      "data": "640 audited Qwen3.5-2B-generated tasks",
      "lr": 4e-05,
      "name": "rl_selfgen",
      "optimizer": "adam",
      "seed": 0,
      "steps": 40,
      "trained": true
    }
  ],
  "base_model": "Qwen/Qwen3.5-2B",
  "baseline": "rl_gsm8k is the same stock math_rl LoRA RL on 640 GSM8K train problems, with identical model, rank, lr, group size, batch, steps, seed, renderer, reward and eval. The untrained base model is evaluated as a reference for gain recovery.",
  "batch_size": 16,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Step 1, generation, filtering and audit (custom script):\n  python gen_tasks.py --base_url $TINKER_BASE_URL --generator Qwen/Qwen3.5-2B --n_accept 640 --filter_samples 8 --keep_band 1-6 --overlap_ref openai/gsm8k:main/test --out /tmp/rx-selfgen/tasks.jsonl\n  This uses only tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']).create_sampling_client plus sandboxed exec of solve().\n  python audit_tasks.py --base_url $TINKER_BASE_URL --solver Qwen/Qwen3.5-397B-A17B --sample 100 --compare numeric_exact --max_label_errors 0 --out /tmp/rx-selfgen/audit.json\n  The run stops if the audit fails.\n\nStep 2, timing pilot: 2 steps per arm plus 100 eval items. Publish feasibility before continuing.\n\nStep 3, baseline:\n  python -m tinker_cookbook.recipes.math_rl.train model_name=Qwen/Qwen3.5-2B base_url=$TINKER_BASE_URL env=gsm8k group_size=8 groups_per_batch=16 learning_rate=4e-5 lora_rank=32 max_tokens=512 max_steps=40 seed=0 log_path=/tmp/rx-run/rl_gsm8k\n  Restrict to the 640-problem seed-0 shuffled subset via the dataset builder.\n\nStep 4, treatment:\n  The same command with a custom client-side dataset builder (custom part) that serves /tmp/rx-selfgen/tasks.jsonl through the same math env and reward, with log_path=/tmp/rx-run/rl_selfgen.\n\nStep 5, eval: sample greedily from base and both save_weights_for_sampler checkpoints on all 1,319 GSM8K test problems with max_tokens=512, score exact match per item, and apply the paired decision rule.",
  "correctness_preflight": {
    "reason": "Both arms use the unchanged math_rl importance-sampling loss and the stock AdamW backend. Only the training-problem source differs, and that is client-side data code with no custom loss or optimizer.",
    "required": false
  },
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 640,
    "name": "gsm8k (baseline train / eval) and self-generated tasks (treatment train)",
    "split": "main/train for baseline, main/test for eval"
  },
  "decision_rule": {
    "alpha": 0.05,
    "criteria": [
      {
        "bounds": [
          0,
          1
        ],
        "id": "noninferiority_vs_gsm8k_rl",
        "margin": -0.03,
        "metric": "accuracy",
        "weights": {
          "rl_gsm8k": -1,
          "rl_selfgen": 1
        }
      },
      {
        "bounds": [
          0,
          1
        ],
        "id": "recovered_half_gain",
        "margin": 0.0,
        "metric": "accuracy",
        "weights": {
          "base": -0.5,
          "rl_gsm8k": -0.5,
          "rl_selfgen": 1
        }
      }
    ],
    "estimand": "GSM8K test greedy exact-match accuracy of fixed seed-0 checkpoints of Qwen3.5-2B LoRA RL on self-generated tasks, relative to RL on GSM8K train and to the untrained base; larger is better",
    "method": "paired_hoeffding",
    "seeds": [
      0
    ],
    "unit": "fixed_checkpoint_items",
    "version": 1
  },
  "eval": {
    "aggregation": null,
    "benchmark": "GSM8K test (openai/gsm8k main/test)",
    "metric": "exact-match accuracy, greedy, max_tokens=512",
    "n_examples": 1319
  },
  "experiment_features": {
    "adaptive_selection": false,
    "checkpoint_restore": false,
    "custom_loss": false,
    "custom_optimizer": false,
    "generated_training_data": true,
    "heldout_loss": false,
    "learned_generator": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "warmup_adaptation": false
  },
  "feasibility": null,
  "feasibility_check": {
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {},
    "problems": [
      "Measured feasibility not recorded"
    ],
    "status": "unknown",
    "wall_seconds": null
  },
  "feasibility_required": true,
  "generation_audit": {
    "answer_comparison": "numeric_exact",
    "max_eval_overlaps": 0,
    "max_label_errors": 0,
    "min_sample_size": 100,
    "required": true
  },
  "hypothesis": "Two things should hold on the 1,319 GSM8K test problems: (1) LoRA RL on 640 audited tasks written by frozen Qwen3.5-2B reaches GSM8K test exact-match accuracy no more than 3 points below the same RL on 640 GSM8K train problems; (2) it recovers more than half of that run's gain over the untrained base model.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "The linked repo (browser-use/rl-env-use) does web-mining and system-prompt hillclimbing with no RL weight training. This experiment borrows only the environment-generation premise and tests it directly with RL.\n\nGeneration (custom, client-side):\n- Frozen base Qwen/Qwen3.5-2B, through the sampling client at T=1.0, is prompted with one fixed instruction: 'Write one grade-school math word problem with a single integer answer, then a short Python function solve() that returns the answer.'\n- Each solve() runs in a sandbox with a 2 s timeout. The label is its integer return value.\n- Filters, fixed before any training:\n  - parse and execution succeed;\n  - the problem text does not contain the answer;\n  - the problem is 20 to 120 words long;\n  - normalized exact match against all GSM8K test questions finds no overlap;\n  - the base model solves the problem from text alone in 1 to 6 of 8 samples at max_tokens=512, so the task is learnable and not trivial.\n- The first 640 accepted tasks, in generation order, form the training set. No selection is made on eval results.\n\nGeneration audit, before any training:\n- Draw 100 random accepted tasks.\n- Each is re-solved blind from the problem text alone by Qwen/Qwen3.5-397B-A17B, which is used for sampling only. Record solver identity and worked solutions.\n- Compare answers numeric_exact.\n- Any label error, or any eval overlap, stops the run for repair and re-audit. Do not train.\n- Save the generated/accepted/rejected/execution-error counts and the data and audit manifests as evidence.\n\nTraining:\n- Both RL arms start from fresh LoRA rank 32 on Qwen3.5-2B and use the identical stock math_rl loop:\n  - group_size=8, groups_per_batch=16, lr=4e-5 (AdamW), max_tokens=512, 40 steps, seed 0;\n  - the same renderer and the same answer-extraction exact-match reward.\n- Baseline arm: 640 GSM8K train problems (first 640 after a fixed seed-0 shuffle).\n- Treatment arm: the 640 generated tasks. This is the only difference between the arms.\n- A third, untrained arm: base model, eval only.\n\nEval:\n- All three arms on the full GSM8K test split (1,319 problems), greedy decoding, max_tokens=512, the same prompt and extractor.\n- Results are paired per item.\n\nPreflight: no custom loss or optimizer. The only custom code is the dataset builder and the generation/audit script.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "primary_endpoint": "gsm8k_test_exact_match_accuracy_greedy_1319",
  "restoration_preflight": {
    "applicable": true,
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "rl_gsm8k",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "learning_rate": 4e-05,
              "lora_rank": 32,
              "seed": 0
            }
          },
          {
            "name": "rl_selfgen",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "learning_rate": 4e-05,
              "lora_rank": 32,
              "seed": 0
            }
          }
        ]
      }
    ],
    "reason": "Two RL arms are trained from a shared starting point. Initial identity must be shown by matching logprobs on a fixed tokenized batch, from both freshly created LoRA clients, before the first update. The base model name alone is insufficient. The active optimizer and its config are verified after creation."
  },
  "steps": 40,
  "success_criterion": "Validated if both criteria hold, with simultaneous 95% paired Hoeffding intervals over the 1,319 GSM8K test items:\n(1) rl_selfgen minus rl_gsm8k accuracy has a lower bound above -0.03 (noninferiority);\n(2) rl_selfgen - 0.5*rl_gsm8k - 0.5*base has a lower bound above 0 (more than half the gain recovered).\nFailed if either upper bound falls below its margin. Otherwise inconclusive. A one-seed pilot may be inconclusive, and it says nothing about training-seed variance.",
  "title": "Self-generated math tasks vs GSM8K train for LoRA RL on Qwen3.5-2B",
  "training_type": "rl"
}
Evidence 14

Partial · Download manifest. Recorded files are retained copies; checkpoint references do not include model weights.

Not recorded: predictions, checkpoint references.

All metrics
generated_candidates
5,120
accepted_tasks
640
rejected_candidates
3,220
execution_error_candidates
1,260
eval_overlap_count
0
audit_sample_size
100
audit_resolved
56
audit_label_errors
25
pilot_base_accuracy_first100
0.78
Interim updates from Devin (2, first Nov 3, 21:38 UTC)
Artifact references
  • script · gen_tasks.py sha256 b9bd7f6185bc
  • script · audit_tasks.py sha256 cab6077b8b59
  • script · common.py sha256 80bf83e1e4ee
  • script · preflight_init.py sha256 b1257d93d1f6
  • script · eval_arm.py sha256 69d2381c3688
  • script · cleanup.py sha256 ffad2e7c46ef
  • effective_config · effective-config.json sha256 98700339afb4
  • code_revision · code-revision.txt sha256 79994ecbc3e1
  • generated_audit · audit_validation.json sha256 b1c64fb48819
  • metrics · generation_summary.json sha256 3095aa795505
  • metrics · init_checkpoint_preflight.json sha256 98524ddf748d
  • metrics · run-metrics.json sha256 7aaf9894051e
  • generated_tasks · generated_candidates.jsonl sha256 33deb38267f2
  • generated_audit · audit_report.json sha256 a5ee29594095
  • generated_tasks · gsm8k_test_eval.jsonl sha256 f0c09bd3e50c
  • predictions · pilot_eval_base_100.jsonl sha256 380a77d45aa3
  1. Nov 3, 22:06 UTC · failed · Launch instructions verified (sha256 d371b1ef...) and fully read. Generated 5,120 candidates with frozen Qwen3.5-2B (temp 1.0): 640 accepted, 3,220 rejected, 1,260 execution errors. Exact overlap wit…
  2. Nov 3, 21:38 UTC · running · Launch instructions downloaded, SHA256 verified, fully read. Spindle connectivity confirmed. Shared init LoRA checkpoint created (tinker://5e5bad216c1d64fa63bcce94880992f7/weights/rx-init-run_4125735…
Artifacts
correctness_validation
{'status': 'not_applicable', 'required': False, 'checks': [], 'reason': 'spec.correctness_preflight opts out and no feature it covers is true', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'not_applicable', 'reported_verdict': 'inconclusive'}
decision_validation
{'status': 'uncertain', 'issues': ['Decision evidence was not recorded.'], 'criteria': [], 'outcome_reason': 'insufficient_evidence', 'unit': 'fixed_checkpoint_items', 'n_seeds': 1, 'alpha': 0.05, 'method': 'paired_hoeffding', 'applicability': {'check': 'decision_rule', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: paired_comparison is true'], 'features': {'paired_comparison': True, 'adaptive_selection': False}}}
feasibility
{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: max_tokens changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Re-projection required: n_eval changed or was not recorded', 'Representative training timing not recorded', 'Positive recovery reserve not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 12.99, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 700.2764999999999, 'planned_units': 4257, 'planned_samples': 4257.0, 'gpu_hours': None}, 'generation': {'wall_seconds': 132.65, 'planned_units': 5120, 'planned_samples': 5120.0, 'gpu_hours': None}, 'filter': {'wall_seconds': 288.7494051346274, 'planned_units': 40960, 'planned_samples': 40960.0, 'gpu_hours': None}, 'audit': {'wall_seconds': 1253.3, 'planned_units': 100, 'planned_samples': 100.0, 'gpu_hours': None}}, 'generation_counts': {'yield': 0.125, 'candidates': 5120, 'filter_samples': 40960}, 'generation_applicability': {'check': 'generation_audit', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: generated_training_data is true'], 'features': {'generated_training_data': True}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.5-2B', 'generation_chunk': 1024, 'filter_samples': 8, 'audit_solver': 'Qwen/Qwen3.5-397B-A17B', 'audit_sample': 100, 'pilot_eval_items': 100}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['base', 'rl_gsm8k', 'rl_selfgen'], 'steps': 40, 'batch_size': '16 groups x 8', 'max_tokens': 512, 'n_eval': 1319, 'training_executed': False}, 'primary_endpoint': 'gsm8k_test_exact_match_accuracy_greedy_1319', 'source': 'generation_summary.json, audit_report.json, init_checkpoint_preflight.json, run-metrics.json', 'measurements': {'setup': {'seconds': 12.99, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 16.45, 'units': 100, 'planned_units': 4257, 'samples': 100, 'gpu_count': None, 'allocation_source': None}, 'generation': {'seconds': 132.65, 'units': 5120, 'planned_units': 5120, 'samples': 5120, 'gpu_count': None, 'allocation_source': None}, 'filter': {'seconds': 180.13, 'units': 25552, 'planned_units': 25552, 'samples': 25552, 'gpu_count': None, 'allocation_source': None}, 'audit': {'seconds': 1253.3, 'units': 100, 'planned_units': 100, 'samples': 100, 'gpu_count': None, 'allocation_source': None}}, 'generation': {'attempted': 5120, 'accepted': 640, 'target_accepted': 640, 'filter_samples_per_candidate': 8}, 'recovery': {'seconds': 0, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793743561.4038932}
feasibility_required
True
generation_audit
{'version': 1, 'stage': 'before_training', 'counts': {'generated': 5120, 'accepted': 640, 'rejected': 3220, 'execution_errors': 1260}, 'samples': [{'item_id': 'selfgen_002047', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004283', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000837', 'independent_answer': '10', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001584', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000386', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004140', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000071', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001049', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000253', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001720', 'independent_answer': '2', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002530', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000112', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003245', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001338', 'independent_answer': '9', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001706', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004457', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004833', 'independent_answer': '7', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003118', 'independent_answer': '20', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004705', 'independent_answer': '14', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000056', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000895', 'independent_answer': '2', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002765', 'independent_answer': '6', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002711', 'independent_answer': '27', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003007', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003254', 'independent_answer': '255', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003776', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000098', 'independent_answer': '35', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003463', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004787', 'independent_answer': '15', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002786', 'independent_answer': '21', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002348', 'independent_answer': '0', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002929', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004566', 'independent_answer': '585', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002508', 'independent_answer': '15', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001104', 'independent_answer': '5', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000529', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000799', 'independent_answer': '0', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002019', 'independent_answer': '1000', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004098', 'independent_answer': '10', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002403', 'independent_answer': '50', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000934', 'independent_answer': '28', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000633', 'independent_answer': '29', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000030', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004619', 'independent_answer': '90', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001783', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004005', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001736', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000496', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000548', 'independent_answer': '10', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003845', 'independent_answer': '13', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001580', 'independent_answer': '3', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001911', 'independent_answer': '10', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002338', 'independent_answer': '5', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002971', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002708', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000509', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003303', 'independent_answer': '9', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002659', 'independent_answer': '2', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003361', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002104', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003343', 'independent_answer': '5', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002383', 'independent_answer': '2', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002872', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002684', 'independent_answer': '360', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004709', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000716', 'independent_answer': '247', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001691', 'independent_answer': '6', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000593', 'independent_answer': '14', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004380', 'independent_answer': '30', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003796', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004035', 'independent_answer': '8', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002226', 'independent_answer': '1', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001769', 'independent_answer': '32', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003475', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001198', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004983', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004598', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003279', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004514', 'independent_answer': '7', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002560', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003298', 'independent_answer': '17', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_002363', 'independent_answer': '0', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001208', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004769', 'independent_answer': '5/3', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003620', 'independent_answer': '20', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003304', 'independent_answer': '8.5', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003625', 'independent_answer': 'number', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003131', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001159', 'independent_answer': '25', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000103', 'independent_answer': '9', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004461', 'independent_answer': '3', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001983', 'independent_answer': '1', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000553', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000443', 'independent_answer': '43', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003563', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001653', 'independent_answer': '1', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_004663', 'independent_answer': '5', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_001843', 'independent_answer': '8', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_003513', 'independent_answer': '', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}, {'item_id': 'selfgen_000752', 'independent_answer': '8', 'resolver_id': 'Qwen/Qwen3.5-397B-A17B renderer=qwen3_5 greedy max_tokens=8192', 'method': 'independent_resolve', 'blind_to_label': True}], 'sample_seed': 0, 'dataset_file': 'generated_candidates.jsonl', 'evaluation_file': 'gsm8k_test_eval.jsonl', 'audit_file': 'audit_report.json', 'overlap_count': 0, 'label_error_count': 25}
generation_audit_validation
{'status': 'unverified', 'reasons': ['Generation audit artifacts missing, inconsistent or unreadable.'], 'applicability': {'check': 'generation_audit', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: generated_training_data is true'], 'features': {'generated_training_data': True}}}
initialization_provenance
{'version': 1, 'arms': [{'arm': 'shared_init', 'base_model': 'Qwen/Qwen3.5-2B', 'init_checkpoint_uri': 'tinker://5e5bad216c1d64fa63bcce94880992f7/weights/rx-init-run_4125735d30', 'lora_rank': 32, 'optimizer_config': 'adamw', 'fixed_batch_sha256': '849a425584079298badce3b742f6e53d7d7f86553205e38c3f9b9cbed08d2a90', 'creator_output_sha256': '4c113a6292ebb82f3097fb5a71711de8845c498688636abae0144a2de3d02591', 'before_first_update': True, 'restored_by_arms': 'none (training not started; checkpoint deleted at cleanup)', 'optimizer_state': 'unverified', 'evidence_ref': 'init_checkpoint_preflight.json'}]}
launch
{'experiment_id': 'exp_c2bf4b4ffc', 'context_status': 'snapshotted', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}
launch_instructions
{'name': 'launch-instructions-d371b1efe487.md', 'sha256': 'd371b1efe48796adc82334d4c4a5a247d71f8e84b97a20386453be217c23e03c', 'url': 'https://ta-01m6fva5db548f5skm214fepcr-8000-oah9l7x00jpo7186eb6cix5v0.w.modal.host/api/runs/run_4125735d30/evidence/d371b1efe48796adc82334d4c4a5a247d71f8e84b97a20386453be217c23e03c'}
reported_outcome_reason
incomplete_execution
reported_verdict
inconclusive
restoration_validation
{'status': 'unverified', 'required': True, 'checks': [{'check': 'rl_gsm8k', 'status': 'unverified', 'reason': 'rl_gsm8k: missing or duplicate arm evidence.'}, {'check': 'rl_selfgen', 'status': 'unverified', 'reason': 'rl_selfgen: missing or duplicate arm evidence.'}], 'reason': 'rl_gsm8k: missing or duplicate arm evidence.', 'applicability': 'required'}
Agent trace full trace with commands in Devin ↗
Details
run
run_4125735d30
backend
Devin
session
devin-6133f7c067774368ac0741709c11746a
experiment
exp_c2bf4b4ffc
created
Nov 3, 21:12 UTC
started
Nov 3, 21:30 UTC
finished
Nov 3, 22:06 UTC
duration
35m 07s
GPU-h budget
2.0
wall budget
120 min
base model
Qwen/Qwen3.5-2B
training
RL
dataset
gsm8k (baseline train / eval) and self-generated tasks (treatment train)
result
final
outcome reason
insufficient evidence
Other runs of this idea

Submit an idea

Most ideas come in from Slack threads. Use this when yours is not in one yet; the triage agent picks it up.