Test Types
AnalysisEntry
Analysis entry for chat-level analysis content.
| Field | Type | Required | Description |
|---|---|---|---|
content | string | No | Analysis content text |
AnalyticsFacets
Resolved analytics facets — embeddable in any artifact response.
Contains filter field visibility, available options for dropdowns, and date range boundaries. Returned inline from artifact get/search responses so each page has its filter facets ready for SSR.
| Field | Type | Required | Description |
|---|---|---|---|
fields | AnalyticsFilterFields | Yes | Filter field visibility configuration |
department_options | AnalyticsFilterOption[] | No | Department dropdown options |
cohort_options | AnalyticsFilterOption[] | No | Cohort dropdown options |
role_options | AnalyticsRoleOption[] | No | Available role resource options |
attempt_options | string[] | No | Available attempt options |
date_range_earliest | string | No | Earliest available date for filtering |
date_range_latest | string | No | Latest available date for filtering |
AnalyticsFilterField
Visibility/disabled state for a single filter field.
| Field | Type | Required | Description |
|---|---|---|---|
visible | boolean | No | Whether the filter field is visible |
disabled | boolean | No | Whether the filter field is disabled |
AnalyticsFilterFields
Per-page filter field visibility configuration.
| Field | Type | Required | Description |
|---|---|---|---|
date_range | AnalyticsFilterField | No | Date range filter config |
departments | AnalyticsFilterField | No | Department filter config |
cohorts | AnalyticsFilterField | No | Cohort filter config |
roles | AnalyticsFilterField | No | Role filter config |
attempts | AnalyticsFilterField | No | Attempt filter config |
AnalyticsFilterOption
A single filter option for dropdown selectors.
| Field | Type | Required | Description |
|---|---|---|---|
value | string | Yes | Option value for the filter |
label | string | Yes | Human-readable option label |
AnalyticsRoleOption
Hydrated role resource option for analytics role filters.
| Field | Type | Required | Description |
|---|---|---|---|
value | string | Yes | Option value for the filter |
label | string | Yes | Human-readable option label |
id | string | Yes | Role resource UUID |
name | string | Yes | Role resource name |
description | string | No | Role description |
icon_id | string | No | Icon resource UUID |
color_id | string | No | Color resource UUID |
level | integer | Yes | Role privilege level |
BenchmarkDepartmentItem
Department resource for benchmark.
| Field | Type | Required | Description |
|---|---|---|---|
department_id | string | Yes | Department identifier |
name | string | No | Department display name |
description | string | No | Department description |
BenchmarkEvalOperational
Eval card for the benchmark page — analogous to ChatSimulationOperational.
| Field | Type | Required | Description |
|---|---|---|---|
eval_id | string | Yes | Eval identifier |
eval_name | string | No | Eval display name |
eval_description | string | No | Eval description |
model_ids | string[] | No | Associated model IDs |
total_tests | integer | No | Total number of tests |
archived_tests | integer | No | Number of archived tests |
total_invocations | integer | No | Total number of invocations |
completed_invocations | integer | No | Number of completed invocations |
highest_score | number | No | Highest score achieved |
has_passed | boolean | No | Whether eval has been passed |
status | string | No | Eval status |
infinite_mode | boolean | No | Whether eval uses infinite mode |
department_ids | string[] | No | Associated department IDs |
rubric_ids | string[] | No | Associated rubric IDs |
BenchmarkHistoryItem
Single test row in benchmark history list — mirrors HistoryItem shape.
| Field | Type | Required | Description |
|---|---|---|---|
test_id | string | Yes | Test identifier |
date | string | No | Formatted date string of the test |
profile_id | string | No | UUID of the profile who owns the test |
profile_name | string | No | Display name of the profile |
eval_id | string | No | Parent eval ID |
eval_name | string | No | Parent eval name |
rubric_id | string | No | Rubric ID for this test |
rubric_name | string | No | Rubric display name |
num_models | integer | No | Total number of models in the test |
num_models_completed | integer | No | Number of models completed |
model_ids | string[] | No | UUIDs of associated models |
model_names | string[] | No | Display names of associated models |
score | integer | No | Overall test score percent (0-100) |
score_status | string | No | Score status label (e.g. high, medium, low) |
pass_pct | integer | No | Pass percentage threshold from rubric |
show_view | boolean | No | Whether the view action is available |
show_continue | boolean | No | Whether the continue action is available |
is_archived | boolean | No | Whether the test is archived |
infinite_mode | boolean | No | Whether the test uses infinite mode |
department_ids | string[] | No | Associated department IDs |
BenchmarkHistoryResponse
Paginated history response.
| Field | Type | Required | Description |
|---|---|---|---|
data | BenchmarkHistoryItem[] | No | History items |
total_count | integer | No | Total number of matching records |
page | integer | No | Current page number |
page_size | integer | No | Items per page |
eval_options | FilterOption[] | No | Eval filter options |
model_options | FilterOption[] | No | Model filter options |
profile_options | FilterOption[] | No | Profile filter options |
rubric_options | FilterOption[] | No | Rubric filter options |
CallerPermissions
Evaluated permissions for the current caller on this artifact type.
| Field | Type | Required | Description |
|---|---|---|---|
can_create | boolean | Yes | Whether the caller can create new artifacts |
can_draft | boolean | Yes | Whether the caller can create/update drafts |
can_duplicate | boolean | Yes | Whether the caller can duplicate artifacts |
has_access | boolean | No | Whether the caller can view this entity |
can_edit | boolean | No | Whether the caller can edit this entity |
can_delete | boolean | No | Whether the caller can delete this entity |
disabled_reason | string | No | Human-readable reason if editing is disabled |
ColumnInfo
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Column name |
type | string | Yes | Column data type |
nullable | boolean | Yes | Whether the column is nullable |
DocsApiResponse
| Field | Type | Required | Description |
|---|---|---|---|
list | PageMetaItem | Yes | — |
detail | PageMetaItem | Yes | — |
new | PageMetaItem | Yes | — |
DocsResponse
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Resource or entry name |
type | string | Yes | Resource or entry type identifier |
description | string | Yes | Human-readable description |
materialized_view | MvInfo | No | Materialized view metadata |
tables | TableInfo[] | Yes | Related database tables |
operations | OperationInfo[] | Yes | Available operations |
EvalSetup
Run-level eval scaffold — first-class on the generate response.
Audit’s **output spread carries this onto
<artifact>.generate.completed. Null when no rubric-bearing
agent participated.
| Field | Type | Required | Description |
|---|---|---|---|
test_id | string | Yes | — |
invocations | InvocationSlot[] | Yes | — |
FeedbackEntry
Feedback by standard for grading state.
standard_group_id is derived from standards metadata lookup.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | UUID of the feedback entry |
standard_id | string | No | UUID of the associated standard |
standard_group_id | string | No | UUID of the standard group |
total | number | No | Total score for this standard |
feedback | string | No | Feedback text for this standard |
FilterOption
A single filter option for dropdown selectors.
| Field | Type | Required | Description |
|---|---|---|---|
value | string | Yes | Internal value for the filter option |
label | string | No | Display label for the filter option |
count | integer | No | Number of matching records |
GenerateConfig
Developer configuration — all optional with sensible defaults.
| Field | Type | Required | Description |
|---|---|---|---|
operations | string[] | No | — |
dangerous | boolean | No | — |
params | object | No | — |
group_id | string | No | — |
GenerationsTestListItem
Single generation group in the test generations response.
| Field | Type | Required | Description |
|---|---|---|---|
group_id | string | Yes | UUID of the generation group |
session_id | string | No | UUID of the parent session |
group_name | string | No | Name of the generation group |
created_at | string | No | Timestamp of the generation |
GetInvocationDraftResponse
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | UUID of the draft |
created_at | string | Yes | Creation timestamp |
generated | boolean | Yes | Whether this was AI-generated |
mcp | boolean | Yes | Whether MCP tooling was used |
active | boolean | Yes | Whether this draft is active |
session_id | string | Yes | Associated session UUID |
name | string | No | Immutable draft label set at create time |
department_ids | string[] | Yes | Associated department UUIDs |
description_ids | string[] | Yes | Associated description UUIDs |
flag_ids | string[] | Yes | Associated flag UUIDs |
key_ids | string[] | Yes | Associated key UUIDs |
modality_ids | string[] | No | Associated modality UUIDs |
quality_ids | string[] | No | Associated quality UUIDs |
model_flag_ids | string[] | Yes | Associated model flag UUIDs |
model_position_ids | string[] | Yes | Associated model position UUIDs |
model_rubric_ids | string[] | Yes | Associated model rubric UUIDs |
name_ids | string[] | Yes | Associated name UUIDs |
profile_ids | string[] | Yes | Associated profile UUIDs |
reasoning_level_ids | string[] | Yes | Associated reasoning level UUIDs |
temperature_level_ids | string[] | Yes | Associated temperature level UUIDs |
voice_ids | string[] | Yes | Associated voice UUIDs |
value_id | string | No | Associated value UUID |
pricing_ids | string[] | Yes | Associated pricing UUIDs |
endpoint_ids | string[] | Yes | Associated endpoint UUIDs |
pending_department_ids | string[] | No | Pending department UUIDs |
pending_description_ids | string[] | No | Pending description UUIDs |
pending_flag_ids | string[] | No | Pending flag UUIDs |
pending_key_ids | string[] | No | Pending key UUIDs |
pending_modality_ids | string[] | No | Pending modality UUIDs |
pending_quality_ids | string[] | No | Pending quality UUIDs |
pending_model_flag_ids | string[] | No | Pending model flag UUIDs |
pending_model_position_ids | string[] | No | Pending model position UUIDs |
pending_model_rubric_ids | string[] | No | Pending model rubric UUIDs |
pending_name_ids | string[] | No | Pending name UUIDs |
pending_reasoning_level_ids | string[] | No | Pending reasoning level UUIDs |
pending_temperature_level_ids | string[] | No | Pending temperature level UUIDs |
pending_voice_ids | string[] | No | Pending voice UUIDs |
pending_value_ids | string[] | No | Pending value UUIDs |
pending_pricing_ids | string[] | No | Pending pricing UUIDs |
pending_endpoint_ids | string[] | No | Pending endpoint UUIDs |
GetTestFeedbackResponse
| Field | Type | Required | Description |
|---|---|---|---|
feedback_id | string | Yes | — |
grade_id | string | Yes | — |
call_id | string | Yes | — |
tool_call_id | string | Yes | — |
standard_id | string | No | — |
total | integer | Yes | — |
feedback | string | Yes | — |
total_points | integer | Yes | — |
pass_points | integer | Yes | — |
created_at | string | Yes | — |
GetTestGradeResponse
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
invocation_id | string | Yes | — |
created_at | string | Yes | — |
updated_at | string | Yes | — |
passed | boolean | Yes | — |
score | integer | Yes | — |
time_taken | integer | Yes | — |
generated | boolean | Yes | — |
mcp | boolean | Yes | — |
active | boolean | Yes | — |
call_id | string | Yes | — |
GetTestInvocationResponse
| Field | Type | Required | Description |
|---|---|---|---|
invocation_id | string | Yes | — |
test_id | string | Yes | — |
group_id | string | Yes | — |
invocation_created_at | string | Yes | — |
invocation_title | string | Yes | — |
use_custom | boolean | Yes | — |
position | integer | Yes | — |
invocation_completed | boolean | Yes | — |
grade_id | string | Yes | — |
grade_score | number | Yes | — |
grade_passed | boolean | Yes | — |
grade_time_taken | number | Yes | — |
rubric_id | string | Yes | — |
agent_ids | string[] | No | — |
quality_id | string | No | — |
department_ids | string[] | No | — |
voice_id | string | No | — |
temperature_level_id | string | No | — |
reasoning_level_id | string | No | — |
modality_ids | string[] | No | — |
GetTestInvocationRunsResponse
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
test_invocation_id | string | Yes | — |
test_invocation_traces_id | string | No | — |
run_id | string | No | — |
created_at | string | Yes | — |
updated_at | string | Yes | — |
generated | boolean | Yes | — |
mcp | boolean | Yes | — |
active | boolean | Yes | — |
GetTestInvocationTracesResponse
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
test_invocation_id | string | Yes | — |
run_id | string | No | — |
created_at | string | Yes | — |
updated_at | string | Yes | — |
generated | boolean | Yes | — |
mcp | boolean | Yes | — |
active | boolean | Yes | — |
reasoning_level_ids | string[] | No | — |
temperature_level_ids | string[] | No | — |
voice_ids | string[] | No | — |
prompt_ids | string[] | No | — |
instruction_ids | string[] | No | — |
tool_ids | string[] | No | — |
quality_ids | string[] | No | — |
modality_ids | string[] | No | — |
GetTestResponse
| Field | Type | Required | Description |
|---|---|---|---|
test_id | string | Yes | — |
call_id | string | Yes | — |
eval_id | string | Yes | — |
profile_id | string | Yes | — |
department_ids | string[] | Yes | — |
test_name | string | Yes | — |
test_description | string | Yes | — |
num_invocations | integer | Yes | — |
infinite_mode | boolean | Yes | — |
is_dynamic | boolean | Yes | — |
archived | boolean | Yes | — |
test_created_at | string | Yes | — |
GradeData
Grade information for a chat (no id - not a resource).
| Field | Type | Required | Description |
|---|---|---|---|
score | number | No | Grade score achieved |
passed | boolean | No | Whether the grade is passing |
description | string | No | Grade description text |
time_taken | integer | No | Time taken in seconds |
total_points | number | No | Total available points |
pass_points | number | No | Points required to pass |
GradingStateData
Grading state for a chat in Record format.
All fields are Records keyed by standard_id strings. This is the exact format the client needs - no transformation required.
| Field | Type | Required | Description |
|---|---|---|---|
achieved_standards | object | No | Map of standard_id to achieved status |
passed_standards | object | No | Map of standard_id to passed status |
feedback_by_standard_id | object | No | Map of standard_id to feedback text |
GroupCall
Tool call referenced by a message.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
tool_name | string | No | — |
template_name | string | No | — |
tool | object | No | — |
ledger_status | string | No | — |
ledger_operation | string | No | — |
ledger_artifact | string | No | — |
ledger_artifact_id | string | No | — |
GroupMessage
Message within a run.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
role | string | Yes | — |
created_at | string | No | — |
text_ids | string[] | No | — |
audio_ids | string[] | No | — |
image_ids | string[] | No | — |
video_ids | string[] | No | — |
file_ids | string[] | No | — |
call_ids | string[] | No | — |
calls | GroupCall[] | No | — |
reasoning | boolean | No | True when this row is a chain-of-thought trace persisted alongside the assistant answer (rendered as a collapsed accordion). |
in_context | boolean | No | Whether this message is included in the LLM context for the next generation. Mirrors the dedup pass that builds chat history (see in_context_reason). |
in_context_reason | string | No | Why this message is in/out of LLM context. ‘kept’ = included; ‘deduped_read’ = older read-only call to a tool that has a fresher result later in the group; future values may include ‘trimmed_top_n’. |
GroupResource
Lightweight \{id, name\} for cross-referencing run-level ids
(model_id / agent_id / profile_id) against human-readable
names on the analytics panel. Names come from the canonical
get_models / get_agents / get_profiles black boxes.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
name | string | No | — |
GroupRun
Run within a group, with its messages.
Carries token / cost / model / agent / profile attribution so the
analytics view can render per-run cost + actor info without a
parallel detail shape. profile_id is the authoring profile
(human user), agent_id is the LLM-side actor, model_id is
the model used by that agent. All optional — runs predating these
columns or with unresolved attributions surface None.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | Yes | — |
created_at | string | No | — |
input_tokens | integer | No | — |
output_tokens | integer | No | — |
cached_input_tokens | integer | No | — |
cost | number | No | — |
model_id | string | No | — |
agent_id | string | No | — |
profile_id | string | No | — |
previous_context_start_index | integer | No | Index in messages where the current run’s own messages begin; earlier rows are previous-context replay. None when the run has no previous context attached. |
messages | GroupMessage[] | No | — |
InvocationDepartmentResource
| Field | Type | Required | Description |
|---|---|---|---|
department_id | string | No | Unique identifier |
name | string | No | Display name |
description | string | No | Description text |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationDescriptionResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
description | string | No | Description text |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationDetail
Per-invocation graded-view payload.
Mirrors ChatData from attempt — one of these per invocation,
each carrying its rubric structure and the list of runs that
executed against it. primary_run_id is the default local
selection (usually the most recent / row-summary grade).
| Field | Type | Required | Description |
|---|---|---|---|
invocation_id | string | Yes | UUID of the test_invocation_entry |
rubric_id | string | No | UUID of the rubric used to grade this invocation |
rubric_structure | RubricStructureData | No | Rubric structure for TableRubric (standards / groups / mappings) |
primary_run_id | string | No | Default selected run for this invocation |
agent_id | string | No | UUID of the agent under test |
model_id | string | No | UUID of the model the agent is set up with |
voice_id | string | No | UUID of the voice resource |
temperature_level_id | string | No | UUID of the temperature level |
reasoning_level_id | string | No | UUID of the reasoning level |
quality_id | string | No | UUID of the quality level |
modality_ids | string[] | No | Modality resource ids |
runs | InvocationRunDetail[] | No | Per-execution detail |
InvocationEndpointResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
base_url | string | No | Endpoint base URL |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationFlagResource
Flag option row — one per (name, type, value) flags_resource entry.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Flag resource identifier |
name | string | No | Flag display name |
type | string | No | Flag type |
value | boolean | No | Underlying bool value of this option |
description | string | No | Flag description text |
icon_id | string | No | Icon identifier for the flag |
icon | string | No | Resolved SVG markup for the icon (hydrated from icons_resource) |
generated | boolean | No | Whether the flag was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationKeyResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
key_id | string | No | Resource identifier alias for picker compatibility |
name | string | No | Display name |
description | string | No | Description text |
key_masked | string | No | Masked key preview |
masked_key | string | No | Masked key preview alias |
active | boolean | No | Whether this key is active |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationModalityResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
modality_id | string | No | Resource identifier alias for picker compatibility |
modality | string | No | Modality code |
name | string | No | Human-readable modality name |
description | string | No | Description text |
is_input | boolean | No | Whether this modality is input-facing |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationModelFlagOptionResource
Cross-product option row: one per (model_id, flag_type, value) tuple.
| Field | Type | Required | Description |
|---|---|---|---|
model_id | string | No | Model identifier |
flag_id | string | No | Flag resource identifier |
type | string | No | Flag type, e.g. ‘model_active’ |
value | boolean | No | Underlying flag value for this option |
name | string | No | Display name from the flags_resource row |
description | string | No | Description text from the flags_resource row |
icon | string | No | Icon SVG markup hydrated from icons_resource |
InvocationModelFlagResource
Per-model flag junction row for an invocation.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Model-flag resource identifier |
model_id | string | No | Associated model identifier |
flag_id | string | No | Associated flag identifier |
type | string | No | Flag type (e.g. ‘model_active’) of the linked flags_resource row |
value | boolean | No | Underlying bool value of the linked flags_resource row |
name | string | No | Display name |
description | string | No | Description text |
icon | string | No | Icon SVG markup |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationModelFlagValue
Denormalized per-(model, type) selection for draft requests.
| Field | Type | Required | Description |
|---|---|---|---|
model_id | string | Yes | Target model identifier |
type | string | Yes | Flag type, e.g. ‘model_active’ |
value | boolean | Yes | Desired flag value |
InvocationNameResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
name | string | No | Display name |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationPricingResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
pricing_id | string | No | Resource identifier alias for picker compatibility |
pricing_type | string | No | Pricing type |
price | number | No | Price amount |
unit_name | string | No | Unit name |
unit_category | string | No | Unit category |
unit_value | integer | No | Unit value |
name | string | No | Display label |
description | string | No | Description text |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationQualityResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
quality_id | string | No | Resource identifier alias for picker compatibility |
quality | string | No | Quality label |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationReasoningLevelResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
reasoning_level | string | No | Reasoning level label |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationRunDetail
Per-run grade + replay detail within an invocation.
One row per test_invocation_runs_entry binding. Carries the
TableRubric-ready grading_state plus the message/call ids the
client uses to slice entries.messages / entries.calls.
| Field | Type | Required | Description |
|---|---|---|---|
run_id | string | Yes | UUID of the runs_entry row this binding executed |
binding_id | string | Yes | UUID of the test_invocation_runs_entry binding row |
grade_id | string | No | UUID of the test_grade_entry, if graded |
created_at | string | No | When the binding was created |
completed | boolean | No | Whether the binding has a completion record |
grade | GradeData | No | Score / passed / time_taken summary |
grading_state | GradingStateData | No | Achieved/passed/feedback maps keyed by standard_id |
feedbacks | FeedbackEntry[] | No | Per-standard feedback rows |
analyses | AnalysisEntry[] | No | Chat-level analysis content (currently unused for tests) |
message_ids | string[] | No | Message ids belonging to this run |
call_ids | string[] | No | Tool-call ids belonging to this run |
InvocationSlot
One agent’s slot in a multi-agent generation pool.
Populated by setup_generation_test when an agent carries a
rubric. The client uses these IDs to drive the eval workflow:
review the candidate’s output, optionally fire a grader against
its invocation_id, and promote/reject by call_id via the
existing idempotency_key + accept pattern.
| Field | Type | Required | Description |
|---|---|---|---|
invocation_id | string | Yes | — |
agent_id | string | Yes | — |
rubric_id | string | No | — |
InvocationTemperatureLevelResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
temperature | number | No | Temperature numeric value |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationValueResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
value | string | No | Value text |
type | string | No | Value type |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
InvocationVoiceResource
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier |
voice | string | No | Voice label |
generated | boolean | No | Whether this was AI-generated |
suggested | boolean | No | Whether this is a suggested option |
selected | boolean | No | Whether this is currently selected |
pending | boolean | No | Whether this selection is pending acceptance |
ListFilterOption
Standardized option for list endpoint filter sections.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | No | Unique identifier for this filter option |
name | string | No | Display name |
count | integer | No | Number of matching records |
hex_code | string | No | Hex color code for display |
value | string | No | Internal value |
type | string | No | Option type discriminator |
ListFilterSection
Filter section with options and echoed request state.
| Field | Type | Required | Description |
|---|---|---|---|
options | ListFilterOption[] | No | Available filter options |
selected_ids | string[] | No | Currently selected filter option IDs |
search | string | No | Active search text for filtering |
MvInfo
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Materialized view name |
definition | string | Yes | SQL definition of the view |
columns | ColumnInfo[] | Yes | List of columns in the view |
OperationInfo
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Operation name |
description | string | Yes | Human-readable description of the operation |
params | ParamInfo[] | Yes | List of operation parameters |
returns | object | No | Return type schema |
OperationPrompts
Starter prompts keyed by operation name.
Each key is an operation (e.g. “create”, “search”, “draft”, “export”) and the value is a list of starter prompts for that operation. The client picks from the operations the caller has permission for and rotates through them.
| Field | Type | Required | Description |
|---|---|---|---|
prompts | object | No | Map of operation name to starter prompts |
PageMetaItem
| Field | Type | Required | Description |
|---|---|---|---|
title | string | Yes | — |
description | string | Yes | — |
ParamInfo
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Parameter name |
type | string | Yes | Parameter data type |
required | boolean | Yes | Whether the parameter is required |
default | any | No | Default value if not required |
ProducedMedia
One asset produced by a generation run.
resource_id is the canonical id the per-artifact download tools
accept (e.g. Scenario_Image_Download(image_id=resource_id) for
modality="image"). It maps to images_resource.id /
videos_resource.id / audios_resource.id depending on the
modality.
| Field | Type | Required | Description |
|---|---|---|---|
modality | "image" | "video" | "audio" | Yes | — |
resource_id | string | Yes | — |
upload_id | string | Yes | — |
mime_type | string | No | — |
file_size | integer | No | — |
ProfileSummary
Caller identity derived from JWT — who you are on this page.
Superset of the old six-field version: now carries everything the client
needs so that /\{artifact\}/context fully replaces /profiles/context
and the extra getLayoutContextData round-trip can be dropped.
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Display name of the authenticated user |
role | string | Yes | Role name (e.g. ‘Super Administrator’) |
role_level | integer | Yes | Role hierarchy level (0 = highest privilege) |
department_ids | string[] | Yes | Departments the user belongs to |
artifact_access | string[] | Yes | Artifact types this role can access (sidebar visibility) |
role_permissions | any[][] | Yes | Full (artifact, operation) permission tuples for granular page gating |
is_active | boolean | Yes | Whether the user’s profile is active |
id | string | Yes | Profile UUID (SocketProvider, ProfileProvider) |
theme | ThemeBundle | No | Resolved theme: hex primitives + derived oklch tokens + score thresholds |
session_id | string | No | Current session UUID |
is_emulation | boolean | No | Whether user is in emulation mode (ProfileProvider) |
role_resources | QGetProfileContextV4RoleResource[] | No | All role resources for emulation display (ProfileProvider) |
scoped_roles | string[] | No | Roles the user can emulate (ProfileProvider) |
active | boolean | No | Alias for is_active (ProfileProvider uses this name) |
QGetProfileContextV4RoleResource
| Field | Type | Required | Description |
|---|---|---|---|
role | string | No | — |
name | string | No | — |
description | string | No | — |
icon_value | string | No | — |
color_hex | string | No | — |
RubricStructureData
Rubric structure data in Record format.
All fields are Records keyed by standard_group_id or standard_id strings. This is the exact format the client needs - no transformation required.
| Field | Type | Required | Description |
|---|---|---|---|
standard_groups | object | No | Map of group_id to standard_id lists |
standard_groups_mapping | object | No | Map of group_id to group metadata |
standards_mapping | object | No | Map of standard_id to standard metadata |
RunStatus
Outcome of one watched run.
| Field | Type | Required | Description |
|---|---|---|---|
run_id | string | Yes | — |
status | "pending" | "completed" | "failed" | Yes | — |
modality | string | No | Modality of generated output (image, video, text…) |
resource_ids | string[] | No | Resource UUIDs produced by this run (e.g. images_resource ids) |
error | string | No | — |
SearchMessageResponse
| Field | Type | Required | Description |
|---|---|---|---|
message_id | string | Yes | UUID of the message |
run_id | string | Yes | UUID of the parent run |
role | string | Yes | Message role (e.g. user, assistant) |
message_created_at | string | Yes | Message creation timestamp |
text_ids | string[] | Yes | UUIDs of text resources |
audio_ids | string[] | Yes | UUIDs of audio resources |
image_ids | string[] | Yes | UUIDs of image resources |
video_ids | string[] | Yes | UUIDs of video resources |
file_ids | string[] | Yes | UUIDs of file resources |
call_ids | string[] | Yes | UUIDs of call resources |
reasoning | boolean | No | Chain-of-thought trace row |
SearchTestItem
Single test row in search results.
| Field | Type | Required | Description |
|---|---|---|---|
test_id | string | Yes | UUID of the test |
eval_id | string | No | UUID of the eval |
eval_name | string | No | Name of the eval |
eval_description | string | No | Description of the eval |
department_ids | string[] | No | UUIDs of associated departments |
test_name | string | No | Name of the test |
test_description | string | No | Description of the test |
num_invocations | integer | No | Number of invocations |
infinite_mode | boolean | No | Whether infinite mode is enabled |
is_dynamic | boolean | No | Whether the test is dynamic |
archived | boolean | No | Whether the test is archived |
created_at | string | No | ISO timestamp when test was created |
TableInfo
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Table name |
columns | ColumnInfo[] | Yes | List of columns in the table |
TestConfigGroup
A group bucket for the picker — used as the section header.
Renders one accordion section per row. run_count is the total
rows in the group (across the whole inner pagination universe,
not just the current expanded window). last_run_at drives the
outer ordering (most-recent-group first).
| Field | Type | Required | Description |
|---|---|---|---|
group_id | string | Yes | UUID of the group |
name | string | No | Human-readable group name (or null if unnamed) |
run_count | integer | No | Total run configs in this group |
last_run_at | string | No | ISO timestamp of the most recent run in this group |
TestConfigItem
A reusable run configuration the picker can queue.
Sources from runs_entry rows. Each row is a distinct config
(agent + model + bundle) that can be re-fired any number of times
into fresh trace executions. The bundle ids carried here come from
the historical run’s agent_resource — the picker passes them as
RunPanelState to /test/invocation/trace so the new trace records the same
prompt + tool + instruction set the original run executed against.
| Field | Type | Required | Description |
|---|---|---|---|
run_id | string | Yes | UUID of the runs_entry config |
group_id | string | No | UUID of the parent group (for grouping in the picker) |
agent_name | string | No | Display name of the agent |
model_name | string | No | Display name of the underlying model |
label | string | Yes | Human-readable picker label |
created_at | string | No | When this config was first created |
prompt_ids | string[] | No | Prompt resource ids from the historical agent |
tool_ids | string[] | No | Tool resource ids from the historical agent |
instruction_ids | string[] | No | Instruction resource ids from the historical agent |
model_id | string | No | Model id from the historical agent |
temperature | number | No | Temperature value from the historical agent |
reasoning | string | No | Reasoning level value from the historical agent |
quality | string | No | Quality value from the historical agent |
permissions | any[][] | No | Historical (artifact, operation) pairs this run executed |
TestEntries
Entry payloads grouped by type.
| Field | Type | Required | Description |
|---|---|---|---|
tests | GetTestResponse[] | No | Test entry payloads |
invocations | GetTestInvocationResponse[] | No | Invocation entry payloads |
runs | GetTestInvocationRunsResponse[] | No | Run entry payloads |
groups | GetTestInvocationTracesResponse[] | No | Group entry payloads |
grades | GetTestGradeResponse[] | No | Grade entry payloads |
feedback | GetTestFeedbackResponse[] | No | Feedback entry payloads |
messages | SearchMessageResponse[] | No | Message entry payloads |
calls | any[] | No | Tool call entries from original run |
TestResources
Resource maps keyed by ID string.
| Field | Type | Required | Description |
|---|---|---|---|
evals | object | No | Eval resources keyed by ID |
rubrics | object | No | Rubric resources keyed by ID |
agents | object | No | Agent resources keyed by ID |
models | object | No | Model resources keyed by ID |
voices | object | No | Voice resources keyed by ID |
temperature_levels | object | No | Temperature level resources keyed by ID |
reasoning_levels | object | No | Reasoning level resources keyed by ID |
modalities | object | No | Modality resources keyed by ID |
prompts | object | No | Prompt resources keyed by ID |
instructions | object | No | Instruction resources keyed by ID |
tools | object | No | Tool resources keyed by ID |
qualities | object | No | Quality resources keyed by ID |
standard_groups | object | No | Standard group resources keyed by ID |
standards | object | No | Standard resources keyed by ID |
TestRunItem
A single run row for the UI table, derived from a benchmark invocation.
| Field | Type | Required | Description |
|---|---|---|---|
chat_id | string | Yes | ID of the chat |
invocation_id | string | Yes | ID of the invocation |
run_id | string | No | ID of the run |
group_id | string | No | ID of the group |
suite_entry_id | string | No | ID of the suite entry |
model_name | string | No | Name of the model used |
agent_name | string | No | Name of the agent used |
status | string | No | Run status |
grade_score | number | No | Grade score for the run |
grade_passed | boolean | No | Whether the run passed grading |
TestStatusSummary
| Field | Type | Required | Description |
|---|---|---|---|
total | integer | No | Total number of invocations |
completed | integer | No | Number of completed invocations |
in_progress | integer | No | Number of in-progress invocations |
not_started | integer | No | Number of not-started invocations |
ThemeBundle
Full theme payload for a page bootstrap.
Riding along on every /\{artifact\}/context response via
ProfileSummary.theme. Layers:
primitives/dark_primitives— hex inputs the settings editor reads/writes (light + dark palettes).tokens/dark_tokens— oklch tokens the client paints with.ThemeStyleemits two<style>blocks: one scoped to:root:not(.dark)(light) and one to:root.dark(dark).thresholds— numeric score thresholds for analytics components. Empty-in → empty-out per token: missing values fall through to the matchingglobals.cssdefault.
| Field | Type | Required | Description |
|---|---|---|---|
primitives | ThemePrimitives | No | Hex inputs from the setting (light palette, for the theme editor) |
tokens | ThemeTokens | No | Derived oklch tokens for light mode (SSR CSS-var injection) |
dark_primitives | ThemePrimitives | No | Hex inputs from the setting (dark palette, for the theme editor) |
dark_tokens | ThemeTokens | No | Derived oklch tokens for dark mode (SSR CSS-var injection) |
thresholds | Thresholds | No | Score thresholds resolved from the setting |
ThemePrimitives
40 optional fields. The 17 essentials drive the rest; the other 23 are overrides for fine-tuning when derivation isn’t what you want.
Empty primitive → empty token → client falls back to globals.css.
| Field | Type | Required | Description |
|---|---|---|---|
background | string | No | — |
primary | string | No | — |
accent | string | No | — |
card | string | No | — |
sidebar | string | No | — |
muted_foreground | string | No | — |
ring | string | No | — |
border | string | No | — |
destructive | string | No | — |
success | string | No | — |
warning | string | No | — |
info | string | No | — |
chart1 | string | No | — |
chart2 | string | No | — |
chart3 | string | No | — |
chart4 | string | No | — |
chart5 | string | No | — |
foreground | string | No | — |
card_foreground | string | No | — |
popover | string | No | — |
popover_foreground | string | No | — |
primary_foreground | string | No | — |
secondary | string | No | — |
secondary_foreground | string | No | — |
muted | string | No | — |
accent_foreground | string | No | — |
destructive_foreground | string | No | — |
danger | string | No | — |
danger_foreground | string | No | — |
input | string | No | — |
success_foreground | string | No | — |
warning_foreground | string | No | — |
info_foreground | string | No | — |
sidebar_foreground | string | No | — |
sidebar_primary | string | No | — |
sidebar_primary_foreground | string | No | — |
sidebar_accent | string | No | — |
sidebar_accent_foreground | string | No | — |
sidebar_border | string | No | — |
sidebar_ring | string | No | — |
ThemeTokens
40 fully-resolved CSS variable values (snake_case 1:1 with vars).
| Field | Type | Required | Description |
|---|---|---|---|
background | string | No | — |
foreground | string | No | — |
card | string | No | — |
card_foreground | string | No | — |
popover | string | No | — |
popover_foreground | string | No | — |
primary | string | No | — |
primary_foreground | string | No | — |
secondary | string | No | — |
secondary_foreground | string | No | — |
muted | string | No | — |
muted_foreground | string | No | — |
accent | string | No | — |
accent_foreground | string | No | — |
destructive | string | No | — |
destructive_foreground | string | No | — |
danger | string | No | — |
danger_foreground | string | No | — |
border | string | No | — |
input | string | No | — |
ring | string | No | — |
success | string | No | — |
success_foreground | string | No | — |
warning | string | No | — |
warning_foreground | string | No | — |
info | string | No | — |
info_foreground | string | No | — |
chart1 | string | No | — |
chart2 | string | No | — |
chart3 | string | No | — |
chart4 | string | No | — |
chart5 | string | No | — |
sidebar | string | No | — |
sidebar_foreground | string | No | — |
sidebar_primary | string | No | — |
sidebar_primary_foreground | string | No | — |
sidebar_accent | string | No | — |
sidebar_accent_foreground | string | No | — |
sidebar_border | string | No | — |
sidebar_ring | string | No | — |
Thresholds
Numeric score thresholds resolved from the active setting.
Server pre-buckets dashboard metrics into success | warning | danger | neutral already, so most components don’t need these values. Surface
them for chart reference lines, tooltips, and any client-side bucketing.
| Field | Type | Required | Description |
|---|---|---|---|
success | integer | Yes | Score >= this counts as success |
warning | integer | Yes | Score >= this counts as warning |
danger | integer | Yes | Score < success threshold but >= this counts as danger; below is neutral/no-data |
app__infra__invocation__types__DraftFormState
Server-authoritative form state returned after draft save.
| Field | Type | Required | Description |
|---|---|---|---|
name_id | string | No | Saved name identifier |
name | string | No | Saved name value |
description_id | string | No | Saved description identifier |
description | string | No | Saved description value |
value_id | string | No | Saved value identifier |
flag_ids | string[] | No | Saved flag identifiers |
department_ids | string[] | No | Saved department identifiers |
key_id | string | No | Saved key identifier |
endpoint_id | string | No | Saved endpoint identifier |
modality_ids | string[] | No | Saved modality identifiers |
temperature_level_id | string | No | Saved temperature level identifier |
pricing_id | string | No | Saved pricing identifier |
reasoning_level_id | string | No | Saved reasoning level identifier |
quality_ids | string[] | No | Saved quality identifiers |
voice_ids | string[] | No | Saved voice identifiers |
model_flag_ids | string[] | No | Saved model flag junction-row identifiers |
model_flag_values | InvocationModelFlagValue[] | No | Denormalized (model_id, type, value) echo derived from model_flag_ids |
model_position_ids | string[] | No | Saved model position identifiers |
model_rubric_ids | string[] | No | Saved model rubric identifiers |
pending_ids | string[] | No | Pending resource identifiers |
app__infra__invocation__types__SectionFilter
Per-section filter options for GET requests.
| Field | Type | Required | Description |
|---|---|---|---|
search | string | No | Filter options by search text |
limit | integer | No | Max options to return |
selected | boolean | No | Only return selected items |
suggested | boolean | No | Only return suggested items |
include | boolean | No | Include this section in response (default true) |
parameter_ids | string[] | No | Reserved for compatibility with shared filter parsing |
Last updated on