Images, video and models

A picture, a video clip, or a glTF model can be an entity of its own, drawn inside the canvas so nodes can sit in front of it or behind it. A node can also show what one of its outputs holds, in a band inside its own body.

People can also paste or drop media onto the canvas to make these entities, as Dropping and pasting media describes.

Move the pointer over any of these to see its controls:

'use client';import { useEffect, useRef } from 'react';import { KookieFlow, useGraph, type Entity, type KookieFlowInstance } from '@kushagradhawan/kookie-flow';/** * A picture, a clip and a model, each an entity of its own. Move the pointer over one to see its * controls: every one has the expand button, the clip has a play bar, and the model turns. */const initialEntities: Entity[] = [  {    id: 'picture', type: 'image', position: { x: 0, y: 0 }, width: 280, height: 186,    data: { src: '/image-1.jpg', objectFit: 'cover' },  },  {    id: 'clip', type: 'video', position: { x: 320, y: 0 }, width: 320, height: 180,    data: { src: '/video.mp4', autoplay: true, objectFit: 'cover' },  },  {    id: 'model', type: 'mesh', position: { x: 680, y: 0 }, width: 240, height: 240,    data: { src: '/model.glb' },  },];export default function MediaEntitiesExample() {  const { entities, edges, onEntitiesChange, onEdgesChange, onConnect } = useGraph({ initialEntities });  const flowRef = useRef<KookieFlowInstance>(null);  useEffect(() => {    flowRef.current?.fitView({ padding: 40, maxZoom: 1 });  }, []);  return (    <div style={{ width: '100%', height: 360 }}>      <KookieFlow        ref={flowRef}        entities={entities}        edges={edges}        onEntitiesChange={onEntitiesChange}        onEdgesChange={onEdgesChange}        onConnect={onConnect}      />    </div>  );}

Media shows its controls while the pointer is over it, and hides them when the pointer leaves. The controls are the same on a media entity and in a node's band.

• Every picture, clip and model has an expand button in its top-right corner. It opens the media in a viewer over the page. Press Escape, the close button, or the dark area around the media to close it.
• A clip also has a play bar along its bottom edge. In the viewer, the clip starts from the same frame and can play with sound.
• A model turns when you drag it. In the viewer, it turns at full size.

Media smaller than 64 × 48 has no expand button. A clip smaller than 120 × 80 has no play bar. Set controls: false to turn the controls off.

Give the entity type: 'image', a width, a height, and a data.src.

const cover: Entity = {  id: 'cover', type: 'image', position: { x: 0, y: 0 }, width: 320, height: 200,  data: { src: '/uploads/cover.jpg', objectFit: 'cover' },};

ImageEntityData is the type of this data, and ImageEntity is Entity<ImageEntityData> with type: 'image'.

The canvas fetches src from the browser, so if your pictures live on another origin, that server must allow cross-origin requests (CORS). The entity shows a flat placeholder while the picture loads, and a flat error colour if it fails, with a warning in the console.

When a picture first decodes, the entity keeps its width and takes the picture's own proportions, down to a height of 40. The canvas reports the new height to onEntitiesChange as a dimensions change. A new src reshapes the entity again. Its corners follow the same radius as node bodies.

Give the entity type: 'video', a width, a height, and a data.src.

const clip: Entity = {  id: 'clip', type: 'video', position: { x: 360, y: 0 }, width: 320, height: 180,  data: { src: '/uploads/walkthrough.mp4', autoplay: true },};

VideoEntityData is the type of this data, and VideoEntity is Entity<VideoEntityData> with type: 'video'.

These rules decide what plays:

• Clips are always muted. Browsers refuse to start a clip with sound until the person has interacted with the page, so the canvas plays every clip without sound.
• A clip plays only while it's on screen. A clip that leaves the view, or sits inside a collapsed frame, pauses, and it carries on when it's visible again.
• At most four clips play at once. Any others show their current frame until a playing clip pauses or leaves the view.
• Entities with the same src share one clip. Pausing one of them pauses them all.

Moving the pointer over a clip at least 120 × 80 shows a bar along its bottom edge. The button at the left plays and pauses, and pressing or dragging along the track jumps through the clip. A press on the button outranks autoplay. When you use a clip as decoration, set controls: false so the pointer never raises a bar on it.

The canvas loads a clip from its URL without credentials, so a clip on another origin must be served with CORS headers. When the clip's size is known, the entity keeps its width and takes the clip's proportions, and the canvas reports the change as it does for a picture.

Give the entity type: 'mesh', a width, a height, and a data.src that points at a .glb or .gltf file.

const chair: Entity = {  id: 'chair', type: 'mesh', position: { x: 720, y: 0 }, width: 240, height: 240,  data: { src: '/models/chair.glb', cameraPosition: { x: 1, y: 0.6, z: 1 }, autoRotate: true },};

MeshEntityData is the type of this data, and MeshEntity is Entity<MeshEntityData> with type: 'mesh'.

The camera stands far enough back that the whole model fits the box, whatever size the model is, so cameraPosition only picks the angle you see it from. The background is transparent, and the canvas lights the model, because most glTF files carry no lights of their own. A model that can't load, because of a missing file, a parse error, or a refused cross-origin request, shows the error colour.

Dragging a model turns it. To move the entity, drag the strip along its top edge, which appears under the pointer. A model shorter than 72 units has no strip, so every drag on it turns it. Set orbit: false to let a drag anywhere on the model move it instead.

A turn someone makes isn't written to the entity, so it isn't saved. A reload shows the model from cameraPosition again. A model is drawn at the width and height you give it. Unlike a picture or a clip, it doesn't change shape to match the file.

Select an image, a video, or a model and drag a handle. A resize keeps the entity's proportions, as they are when the drag starts. Hold Shift to resize freely instead. When you set aspectLocked: false, the two swap: the resize is free, and Shift keeps the proportions. Hold Alt to resize from the centre rather than from the opposite edge.

When the drag ends, the canvas reports a dimensions change, and a position change as well if the resize moved the entity's top-left corner.

When the image entry of entityTypes sets toolbar: true, the toolbar's image controls set objectFit and aspectLocked on a selected picture. The Selection toolbar chapter covers them.

Set preview on a node to draw a band inside its body that shows what one of its output sockets holds. The band sits below the sockets, and the node grows taller to make room for it.

const generate: Entity = {  id: 'generate', type: 'default', position: { x: 0, y: 0 }, width: 260,  data: { label: 'Generate' },  inputs: [{ id: 'prompt', name: 'Prompt', type: 'string' }],  outputs: [{ id: 'image', name: 'Image', type: 'image' }],  preview: { socket: 'image', height: 180, fit: 'contain' },};

EntityPreview is the type of preview. An entity type can give every node of that type a band through its own preview, as Entity types describes.

The band is drawn only on a node whose width is stated, either on the node itself or through its type's defaultWidth. A node that states neither still grows to make room for the band, but leaves it empty. If you give the node a height too short for the band, the band shrinks to fit inside the node.

The value comes from the evaluation engine, not from the node's data. You put it there by returning it from onEvaluate, or by calling setSocketValue on the ref. The Evaluation chapter covers both. This example draws an image entity beside a node whose band shows a picture placed with setSocketValue:

'use client';import { useEffect, useRef } from 'react';import { KookieFlow, useGraph, type Entity, type KookieFlowInstance } from '@kushagradhawan/kookie-flow';const initialEntities: Entity[] = [  // An image entity: the picture is the whole entity.  {    id: 'photo', type: 'image', position: { x: 0, y: 0 }, width: 300, height: 200,    data: { src: '/image-1.jpg', objectFit: 'cover' },  },  // A node with a preview band that shows whatever its `image` output holds.  {    id: 'render', type: 'default', position: { x: 360, y: 0 }, width: 240,    data: { label: 'Render' },    outputs: [{ id: 'image', name: 'Image', type: 'image' }],    preview: { socket: 'image', height: 140 },  },];export default function MediaExample() {  const { entities, edges, onEntitiesChange, onEdgesChange, onConnect } = useGraph({ initialEntities });  const flowRef = useRef<KookieFlowInstance>(null);  useEffect(() => {    const flow = flowRef.current;    if (!flow) return;    // No evaluation here, so put a value on the output directly. The band draws it.    flow.setSocketValue('render', 'image', '/image-3.jpg');    flow.fitView({ padding: 48, maxZoom: 1 });  }, []);  return (    <div style={{ width: '100%', height: 360 }}>      <KookieFlow        ref={flowRef}        entities={entities}        edges={edges}        onEntitiesChange={onEntitiesChange}        onEdgesChange={onEdgesChange}        onConnect={onConnect}      />    </div>  );}

The band decides what to draw from the value:

• A string ending in .mp4, .webm, .ogv, .mov, or .m4v plays as a clip.
• A string ending in .glb or .gltf shows as a model. For clips and models, a query string or a hash after the extension is ignored.
• Any other string that starts with http://, https://, /, ./, ../, or blob: shows as a picture, so a link with no extension works.
• A data: URL shows as a picture, a clip, or a model, from its MIME type.
• An ImageBitmap, an HTMLImageElement, or an HTMLCanvasElement shows as a picture, with no download.
• Anything else, such as a number, an object, a plain word, or no value yet, leaves the band empty.

All three kinds sit side by side below. Generate is a manual node: press the button, watch the progress ring, and the picture it returns lands in its own band. The clip and the model never evaluate — their values were placed with setSocketValue — which is the same band fed the other way.

'use client';import { useCallback, useEffect, useRef } from 'react';import { Button, Flex, Stack, Text } from '@kushagradhawan/kookie-ui-react';import {  KookieFlow,  useGraph,  type Entity,  type EntityTypeDefinition,  type KookieFlowInstance,  type OnEvaluate,} from '@kushagradhawan/kookie-flow';/** * Three nodes, three kinds of band. Each names one of its own output sockets, and the canvas * draws whatever value is sitting on it: a picture, a clip, or a model. */const entityTypes: Record<string, EntityTypeDefinition> = {  'ai/generate': {    type: 'ai/generate',    label: 'Generate',    defaultWidth: 260,    // Manual: it waits to be asked rather than running when its inputs change.    evaluation: 'manual',    inputs: [{ id: 'prompt', name: 'Prompt', type: 'string' }],    outputs: [{ id: 'image', name: 'Image', type: 'image' }],    preview: { socket: 'image', height: 150, fit: 'cover' },  },  'media/clip': {    type: 'media/clip',    label: 'Clip',    defaultWidth: 260,    outputs: [{ id: 'video', name: 'Video', type: 'video' }],    preview: { socket: 'video', height: 150, fit: 'cover' },  },  'media/model': {    type: 'media/model',    label: 'Model',    defaultWidth: 260,    outputs: [{ id: 'mesh', name: 'Mesh', type: 'mesh' }],    preview: { socket: 'mesh', height: 150, fit: 'contain' },  },};/** * `video` is not one of the built-in socket types, so without this the Clip node's socket would * draw in the grey of `any`. Your entries are merged over the defaults. */const socketTypes = {  video: { name: 'Video', color: '--purple-10' },};const initialEntities: Entity[] = [  { id: 'generate', type: 'ai/generate', position: { x: 0, y: 0 }, data: {} },  { id: 'clip', type: 'media/clip', position: { x: 300, y: 0 }, data: {} },  { id: 'model', type: 'media/model', position: { x: 600, y: 0 }, data: {} },];const sleep = (ms: number, signal: AbortSignal) =>  new Promise<void>((resolve, reject) => {    const timer = setTimeout(resolve, ms);    signal.addEventListener('abort', () => { clearTimeout(timer); reject(new Error('aborted')); }, { once: true });  });/** * The application's logic. A real image model would be a fetch; this waits, reports progress, and * answers with a URL — which is all the band ever needs. */const onEvaluate: OnEvaluate = async (id, type, inputs, ctx) => {  if (type !== 'ai/generate') return;  for (let step = 1; step <= 20; step++) {    await sleep(90, ctx.signal);    ctx.progress(step / 20);  }  return { image: '/preview-render.jpg' };};export default function PreviewBandExample() {  const { entities, edges, onEntitiesChange, onEdgesChange, onConnect } = useGraph({ initialEntities });  const flowRef = useRef<KookieFlowInstance>(null);  const generate = useCallback(() => flowRef.current?.evaluate('generate'), []);  useEffect(() => {    const flow = flowRef.current;    if (!flow) return;    // The clip and the model never evaluate: a value put on the socket fills the band just as a    // computed one does.    flow.setSocketValue('clip', 'video', '/video.mp4');    flow.setSocketValue('model', 'mesh', '/model.glb');    flow.fitView({ padding: 40, maxZoom: 1 });  }, []);  return (    <Stack gap="3">      <Flex gap="3" align="center">        <Button size="2" emphasis="loud" onClick={generate}>Generate</Button>        <Text size="2">The picture lands in the node's own band when the run finishes.</Text>      </Flex>      <div style={{ width: '100%', height: 400 }}>        <KookieFlow          ref={flowRef}          entities={entities}          edges={edges}          entityTypes={entityTypes}          socketTypes={socketTypes}          onEntitiesChange={onEntitiesChange}          onEdgesChange={onEdgesChange}          onConnect={onConnect}          onEvaluate={onEvaluate}        />      </div>    </Stack>  );}

A clip in a band plays muted and loops whenever the band is on screen, until someone pauses it with the play bar. A model in a band turns when you drag the band; drag anywhere else on the node to move it. A bitmap, image element or canvas is uploaded once, when it arrives, so drawing on the same canvas later doesn't change the band. When you want a new picture, set the value again with a new object.

A still picture costs nothing per frame once it's on the GPU. A playing clip and a turning model cost work on every frame, and the canvas limits how many of each run at once.

Each src is downloaded and decoded once, and every entity that uses it shares the result. The decoding happens off the main thread where the browser allows it. The canvas first uploads a copy no larger than 256 pixels on its longest side, and decodes the full picture only when the entity is wider than 256 pixels on screen. When more than 32 full-size pictures are held, the one drawn least recently goes back to its small copy until it's drawn large again.

maxImageTextureSize caps the full-size copy of each picture entity along its longest side. A decoded picture uses 4 bytes for each pixel, so a 2048 × 2048 copy takes about 16 MB. Lower the cap if your board holds many large pictures, and raise it if your users zoom in close on detailed ones.

A clip loads only enough to know its size and show its first frame until it plays. While it plays, the canvas uploads a new frame to the GPU on every frame, so the cost grows with the number of clips playing, not the number on the board.

Entities with the same src share one loaded model.

Each model is drawn into an image of its own, sized to how large the entity is on screen, rounded up to a power of two from 128 to 1024 pixels. That image is drawn again only when something changes: the model finishes loading, the entity crosses a size step, someone turns it, or autoRotate is on. A board of still models therefore costs nothing per frame.

Eight models hold an image at once. When a ninth needs one, the model that has been off screen longest gives its image up, and draws a new one when it comes back into view. If all eight are on screen, the ninth shows the placeholder until one of them leaves the view. Model bands work the same way, with four images of up to 512 pixels, except that a band left without an image stays empty.