harness_bench.h File Reference

Harness convergence benchmark — A/B/C feedback modes (issue #81). More...

#include <neograph/api.h>
#include <neograph/graph/types.h>
#include <neograph/graph/compiler.h>
#include <neograph/graph/validator.h>
#include <string>
#include <vector>

Include dependency graph for harness_bench.h:

Go to the source code of this file.

Classes
struct	neograph::graph::ConvergenceMetrics
	Metrics for one convergence attempt. More...
struct	neograph::graph::Feedback
	One round of feedback from the compiler to the LLM. More...
struct	neograph::graph::HarnessTask
	A benchmark task: natural language spec converted into a harness. More...

Functions
enum NEOGRAPH_API NEOGRAPH_API const char *	neograph::graph::feedback_mode_name (FeedbackMode mode)
	Human-readable name for each feedback mode.
NEOGRAPH_API Feedback	neograph::graph::format_feedback (const json &core, const std::string &compile_err, const ValidationReport &validation, FeedbackMode mode, int attempted_routes=-1)
	Format feedback for a given mode from a compile+validate result.
NEOGRAPH_API json	neograph::graph::generate_report (const std::vector< ConvergenceMetrics > &results)
	Generate a convergence report as JSON.
NEOGRAPH_API HarnessTask	neograph::graph::parse_task (const json &j)
	Parse a HarnessTask from a JSON fixture file.
NEOGRAPH_API ConvergenceMetrics	neograph::graph::run_simulation (const HarnessTask &task, const json &seed_harness, FeedbackMode mode, int max_turns=20, unsigned int seed=42)
	Run a simulated convergence benchmark against a single task.

Variables
enum NEOGRAPH_API	neograph::graph::FAIL_ONLY = 1
	B: "failed" only (baseline)
enum NEOGRAPH_API	neograph::graph::FeedbackMode
	A: full error + witness (agent-optimized)
enum NEOGRAPH_API	neograph::graph::RUNTIME_SYMPTOMS = 2
	C: runtime symptom simulation.

Detailed Description

Harness convergence benchmark — A/B/C feedback modes (issue #81).

Measures the real-world value of compiler diagnostics by comparing three feedback regimes when an LLM iteratively corrects a generated harness:

A. Full diagnostic: strict compiler error text + witness JSON + validator warnings. Every error carries a source coordinate and a machine-readable counterexample. B. Fail-only: "compile failed, try again" — no detail. Baseline. C. Runtime symptoms: compile succeeds, graph runs, outputs are wrong. Simulates the v0.1.x silent-drop era.

The framework does NOT call an LLM — it provides the feedback formatting, metrics collection, and convergence measurement. Run with an external script that feeds LLM-generated harnesses through compile() -> feedback -> iterate.

Definition in file harness_bench.h.

Function Documentation

format_feedback()

NEOGRAPH_API Feedback neograph::graph::format_feedback	(	const json &	core,
		const std::string &	compile_err,
		const ValidationReport &	validation,
		FeedbackMode	mode,
		int	attempted_routes = -1
	)

Format feedback for a given mode from a compile+validate result.

Parameters

core	The harness topology JSON submitted by the LLM.
compile_err	Empty if compile succeeded, else the exception text.
validation	Validation report (may be empty if compile failed).
mode	Which feedback regime to format.
attempted_routes	For C (runtime symptoms): number of routes that silently dropped or misrouted. -1 = unknown.

References neograph::graph::format_feedback().

Referenced by neograph::graph::format_feedback().

run_simulation()

NEOGRAPH_API ConvergenceMetrics neograph::graph::run_simulation	(	const HarnessTask &	task,
		const json &	seed_harness,
		FeedbackMode	mode,
		int	max_turns = 20,
		unsigned int	seed = 42
	)

Run a simulated convergence benchmark against a single task.

This does NOT call an LLM. Instead, it generates candidate harnesses by applying random mutations to the seed_harness, then measures how many "turns" each feedback mode takes to converge. This gives a lower-bound estimate of diagnostic value without LLM costs.

For real LLM experiments, the external script should call format_feedback() per turn and record metrics.

References neograph::graph::run_simulation().

Referenced by neograph::graph::run_simulation().