<LLMSceneBuilder />

<LLMSceneBuilder /> adds a Frame Builder panel to the visualizer dashboard. You type a natural-language instruction (“Move the arm 200 mm forward along X”), the plugin calls your onInfer callback with the current frame state, and presents a diff of every proposed field change before anything is applied. The user confirms or cancels — no frame is mutated until confirmation.

The plugin is model-agnostic: you wire in whatever LLM backend you prefer via the onInfer prop.

Usage

<script lang="ts">
	import { Visualizer } from '@viamrobotics/motion-tools'
	import { LLMSceneBuilder } from '@viamrobotics/motion-tools/plugins'
	import type { ComponentFrameInfo, FrameDelta } from '@viamrobotics/motion-tools/plugins'

	async function handleInfer(
		prompt: string,
		components: ComponentFrameInfo[]
	): Promise<{ updates: FrameDelta[]; explanation: string }> {
		const response = await fetch('/api/infer-frames', {
			method: 'POST',
			headers: { 'Content-Type': 'application/json' },
			body: JSON.stringify({ prompt, components }),
		})
		return response.json()
	}
</script>

<div class="h-screen w-screen">
	<Visualizer>
		<LLMSceneBuilder onInfer={handleInfer} />
	</Visualizer>
</div>

A robot-outline button appears in the dashboard. Clicking it opens the Frame Builder floating panel. Enter a prompt and press Submit (or Enter) — the panel shows a diff table while the LLM responds, then lets the user confirm or cancel.

Props

Prop	Type	Default	Description
onInfer	InferCallback	—	Required. Called with the prompt and current frame state; must return proposed deltas and an explanation.

InferCallback

type InferCallback = (
	prompt: string,
	components: ComponentFrameInfo[]
) => Promise<{ updates: FrameDelta[]; explanation: string }>

The plugin passes every component that has a frame defined:

interface ComponentFrameInfo {
	name: string
	frame: {
		parent: string | undefined
		translation: { x?: number; y?: number; z?: number } | undefined
		orientation: { roll: number; pitch: number; yaw: number } // degrees
	}
}

Your callback should return:

• updates — an array of FrameDelta objects describing changes (see below). Omit any field that should remain unchanged.
• explanation — a human-readable summary shown above the diff table.

FrameDelta

interface FrameDelta {
	componentName: string
	translation?: { x?: number; y?: number; z?: number } // mm, absolute
	orientation?: { roll?: number; pitch?: number; yaw?: number } // degrees, delta applied to current
	parent?: string
	explanation?: string // per-component note shown in the diff
}

Caution

translation values are absolute — they replace the current translation field directly. orientation values are deltas added to the current Euler angles.

The plugin validates every delta before showing the diff: unknown component names, self-referential parent assignments, and non-finite numbers are surfaced as errors without blocking the rest of the update batch.

Example: using the Anthropic SDK

Set ANTHROPIC_API_KEY in your environment (see .env.example). A minimal server-side route using the Anthropic SDK:

// src/routes/api/infer-frames/+server.ts  (SvelteKit)
import Anthropic from '@anthropic-ai/sdk'

export async function POST({ request }) {
	const { prompt, components } = await request.json()

	const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY })

	const message = await client.messages.create({
		model: 'claude-sonnet-4-6',
		max_tokens: 1024,
		messages: [
			{
				role: 'user',
				content: [
					{
						type: 'text',
						text: `Current robot frame configuration (JSON):\n${JSON.stringify(components, null, 2)}\n\nUser request: ${prompt}\n\nRespond with JSON matching the schema: { updates: FrameDelta[], explanation: string }`,
					},
				],
			},
		],
	})

	const text = message.content.find((b) => b.type === 'text')?.text ?? '{}'
	return new Response(text, { headers: { 'Content-Type': 'application/json' } })
}