<LLMSceneBuilder />

<LLMSceneBuilder /> adds an LLM Scene Builder panel to the visualizer dashboard. You type a natural-language instruction (“Move the arm 200 mm forward along X”), the plugin calls your onInfer callback with the current frame state, and presents a diff of every proposed field change before anything is applied. The user confirms or cancels — no frame is mutated until confirmation.

The plugin is model-agnostic: you wire in whatever LLM backend you prefer via the onInfer prop.

Usage

<script lang="ts">
	import { Visualizer } from '@viamrobotics/visualization'
	import { LLMSceneBuilder } from '@viamrobotics/visualization/plugins'
	import type { ComponentFrameInfo, FrameDelta } from '@viamrobotics/visualization/plugins'

	async function handleInfer(
		prompt: string,
		components: ComponentFrameInfo[]
	): Promise<{ updates: FrameDelta[]; explanation?: string; refusal?: string }> {
		const response = await fetch('/api/infer-frames', {
			method: 'POST',
			headers: { 'Content-Type': 'application/json' },
			body: JSON.stringify({ prompt, components }),
		})
		return response.json()
	}
</script>

<div class="h-screen w-screen">
	<Visualizer>
		<LLMSceneBuilder onInfer={handleInfer} />
	</Visualizer>
</div>

A robot-outline button appears in the dashboard. Clicking it opens the LLM Scene Builder floating panel. Enter a prompt and press Submit (or Enter) — the panel shows a diff table while the LLM responds, then lets the user confirm or cancel.

Props

Prop	Type	Default	Description
onInfer	InferCallback	—	Required. Called with the prompt and current frame state; must return proposed deltas and an explanation.

InferCallback

type InferCallback = (
	prompt: string,
	components: ComponentFrameInfo[]
) => Promise<{ updates: FrameDelta[]; explanation?: string; refusal?: string }>

The plugin passes every component that has a frame defined:

interface ComponentFrameInfo {
	name: string
	frame: {
		parent: string | undefined
		translation: { x?: number; y?: number; z?: number } | undefined
		orientation: { roll: number; pitch: number; yaw: number } // degrees
		geometry?:
			| { type: 'none' }
			| { type: 'box'; x: number; y: number; z: number }
			| { type: 'sphere'; r: number }
			| { type: 'capsule'; r: number; l: number }
	}
}

Your callback should return:

• updates — an array of FrameDelta objects describing changes (see below). Omit any field that should remain unchanged.
• explanation — a human-readable summary shown above the diff table.
• refusal — optional. When set, the plugin skips the diff and shows the message in its error state instead. Use it for requests the plugin cannot fulfil (e.g. adding a new component).

FrameDelta

interface FrameDelta {
	componentName: string
	translation?: { x?: number; y?: number; z?: number } // mm, absolute
	orientation?: { roll?: number; pitch?: number; yaw?: number } // degrees, delta applied to current
	// resize (omit `type`), change shape (set `type`), or remove (`type: 'none'`); send only changed dims
	geometry?: {
		type?: 'none' | 'box' | 'sphere' | 'capsule'
		x?: number
		y?: number
		z?: number
		r?: number
		l?: number
	}
	parent?: string
	explanation?: string // per-component note shown in the diff
}

Caution

translation values are absolute — they replace the current translation field directly. orientation values are deltas added to the current Euler angles. geometry is like translation/orientation: send only the changed dims (in millimeters). Omit type to resize the current shape (unspecified dims keep their value); set type to change shape and send that type’s dims (box → x/y/z, sphere → r, capsule → r/l). If the component has no geometry, send type + dims to add one; send type: 'none' to remove an existing geometry.

The plugin validates every delta before showing the diff: unknown component names, self-referential parent assignments, non-finite numbers, and geometry with missing or non-positive dimensions are surfaced as errors without blocking the rest of the update batch.

The scene-builder contract

Getting good deltas out of a model is mostly prompt work: millimeters, Euler degrees wrapped to [-180, 180], deltas rather than whole frames, one entry per component, and a refusal policy for “add a component”. That prompt and its schemas ship with the package so you don’t have to rebuild them:

import {
	SCENE_BUILDER_SYSTEM_PROMPT,
	SceneBuilderRequestSchema,
	SceneBuilderResponseSchema,
	sceneBuilderSystemMessage,
} from '@viamrobotics/visualization/scene-builder'

This is a separate entry point from /plugins on purpose. /plugins is a Svelte barrel — importing it from a Node or edge handler pulls in every plugin’s components. /scene-builder is pure TypeScript with no Svelte, three.js, or DOM dependency, so it is safe to import server-side.

Nothing here is required. It is a reference implementation of the contract onInfer is built around: bringing your own model means changing the call, not rewriting the prompt.

Example: using the Anthropic SDK

SceneBuilderResponseSchema is a Zod schema, so it can drive structured outputs directly — the model is constrained to the response shape rather than asked to produce JSON and hoped at:

// src/routes/api/infer-frames/+server.ts  (SvelteKit)
import Anthropic from '@anthropic-ai/sdk'
import { zodOutputFormat } from '@anthropic-ai/sdk/helpers/zod'
import {
	SceneBuilderRequestSchema,
	SceneBuilderResponseSchema,
	sceneBuilderSystemMessage,
} from '@viamrobotics/visualization/scene-builder'

const client = new Anthropic() // reads ANTHROPIC_API_KEY from the environment

export async function POST({ request }) {
	const { prompt, components } = SceneBuilderRequestSchema.parse(await request.json())

	const message = await client.messages.parse({
		model: 'claude-opus-5',
		max_tokens: 16000,
		system: sceneBuilderSystemMessage(components),
		messages: [{ role: 'user', content: prompt }],
		output_config: { format: zodOutputFormat(SceneBuilderResponseSchema) },
	})

	return Response.json(message.parsed_output)
}

Tip

sceneBuilderSystemMessage(components) appends the serialized component context to SCENE_BUILDER_SYSTEM_PROMPT. Keeping the components in the system message rather than the user turn means the prompt prefix is stable per scene, so repeated edits to the same scene hit the prompt cache.

Any provider works — the prompt and schemas are provider-agnostic. Use whichever SDK you prefer and validate the result with SceneBuilderResponseSchema before returning it.