Object Tracking

Detect and pose registered 3D objects using camera frames

Object tracking lets you detect registered 3D objects in the camera feed and get their 6-DOF pose. Unlike VPS map localization, it does not require scanning a space — you register individual objects in the Multiset portal and query them by their object codes.

Setup

Set mapType to 'object-tracking' and pass an array of object codes. You can query up to 10 objects in a single session.

import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';
import * as THREE from 'three';

const client = new MultisetClient({
  clientId: 'YOUR_CLIENT_ID',
  clientSecret: 'YOUR_CLIENT_SECRET',
  mapType: 'object-tracking',
  code: ['OBJECT_CODE_1', 'OBJECT_CODE_2'],
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
renderer.xr.enabled = true;
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
scene.background = null;
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 100);

const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  { client }
);

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showObjectMeshes: true,  // download and display 3D outline meshes for detected objects
  onObjectMeshLoaded: (objectCode) => {
    console.log(`Mesh loaded for object: ${objectCode}`);
  },
});

adapter.initialize();

Triggering Detection

Object tracking does not run automatically by default. Trigger it from a button or gesture:

document.getElementById('detect-btn').addEventListener('click', async () => {
  const result = await adapter.trackObjects();

  if (result?.poseFound) {
    console.log('Detected objects:', result.objectCodes);
    console.log('Position:', result.position);
    console.log('Rotation:', result.rotation);
    console.log('Confidence:', result.confidence);
  } else {
    console.log('No object detected');
  }
});

Auto Tracking

Enable autoTracking to trigger a detection attempt automatically when the session starts:

const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  {
    client,
    autoTracking: true,
    restartTracking: true,            // re-attempt tracking after AR tracking loss recovery
    trackingCaptureDelayMs: 500,      // wait 500 ms after session start before capturing
    onObjectTrackingSuccess: (result) => {
      console.log('Tracking result:', result);
    },
    onObjectTrackingFailure: (reason) => {
      console.warn('Tracking failed:', reason);
    },
  }
);

Tracking Result

The result returned by trackObjects() or passed to onObjectTrackingSuccess:

Field	Type	Description
poseFound	boolean	Whether an object was detected
position	{x, y, z}	Object position in tracker space
rotation	{x, y, z, w}	Object orientation as a quaternion
confidence	number	Detection confidence (0 to 1)
objectCodes	string[]	Codes of detected objects

Object Tracking in Needle Engine

Set Map Type to ObjectTracking in the MultisetVPS Inspector and enter one or more object codes as a comma-separated string in Map Code:

OBJECT_CODE_1,OBJECT_CODE_2

Enable Auto Tracking and Show Object Meshes in the Inspector for the same behavior as autoTracking: true and showObjectMeshes: true.

Using the Core Entry Point

Without Three.js, use the adapter result handler on the session manager:

session.setAdapterObjectTrackingHandler((result, trackerSpace) => {
  const { poseFound, position, rotation, objectCodes } = result;
  if (poseFound) {
    // Apply position and rotation to your scene graph
  }
});

// Trigger manually
const result = await session.trackObjects();