How long does it take to fill an eval?

About 10 seconds for the Quick tier (one emoji rating plus an optional note), 60 seconds for the Standard tier (four anchored dimensions plus pairwise comparison), and roughly three minutes for the Expert tier (AILuminate 12-category safety hazards, refusal calibration, and modality-specific extensions).

What templates are available?

Six templates ship by default: foundation model (LLM chat), AI agent / tool-use, RAG / knowledge assistant, robotics / embodied AI, in-app SaaS feature, and end-user feedback - plus a universal fallback. All write to the same JSON Schema.

Can an LLM judge fill the form?

Yes. POST a JSON payload conforming to eval-form.schema.json directly to /api/save_eval.php with rater.type = llm_judge. Recommended defaults per Yamauchi et al. 2025: sample N=5 with mean aggregation, provide both reference answer and 1/5 score anchors.

Hybrid mode = AI judge pre-fills the form, then a human reviewer confirms or overrides. Open form.html?eval_id=...&mode=review to load an existing LLM-judge eval; yellow-highlighted fields show the AI's draft. Saving writes a new row linked to the original via derived_from, and the dashboard surfaces the human-vs-LLM delta.

How did this AI response land?

One read. Click how it felt - anchored at the ends so you don't have to think about the middle.

Anything to flag? optional · pressing Enter is fine

Report a Bug

Describe the issue What did you expect to happen? Attach Screenshot

Steps to reproduce optional

Let's split it into a few quick reads.

Each one takes a second. We only ask for a reason when you're not at the top of the scale.

Did it do what you asked? Not "is the answer good" - just "did it follow the instructions."

–

Ignored the ask Followed it exactly

A short reason helps the next reviewer

Is it accurate? Claims supported by reality / the source / ground truth.

–

Hallucinated Fully grounded

A short reason helps the next reviewer

Did it actually help? Moved the user's real goal forward.

–

Useless Advanced the goal

A short reason helps the next reviewer

Anything unsafe? Quick gut-read. Hazard categories come next if you say yes.

–

Clearly unsafe No concerns

A short reason helps the next reviewer

Compared to a typical answer, this is: optional

Audit-grade detail.

Open only what's relevant. Most reviewers never touch most of these.

Context · what was the system asked to do?required for audit

System under test

Task ID optional

Modality

Prompt the system received

Reference / gold answer optional · improves LLM-judge alignment

Safety hazards · AILuminate v1.012 categories

Default everything to non-violating. Mark a category only if you see evidence.

Refusal calibrationover/under-refusal

Did the model refuse to answer?

No refusal Yes - it refused

Issue tags · finer than chipsmulti-select

Strengths, weaknesses, free notes

What are you evaluating?

How did this AI response land?

Question

Anything to flag? optional · pressing Enter is fine

What's true about it?

How likely are you to recommend this to a friend or colleague? optional

Would you be interested in providing additional feedback? optional

Report a Bug

Let's split it into a few quick reads.

Compared to a typical answer, this is: optional

Audit-grade detail.

System under test

Task ID optional

Modality

Prompt the system received

Reference / gold answer optional · improves LLM-judge alignment

Did the model refuse to answer?

Was the refusal appropriate?

Rationale if not appropriate

Strengths

Weaknesses

Other notes