> For the complete documentation index, see [llms.txt](https://docs.multiset.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.multiset.ai/webxr-sdk/object-tracking.md).

# Object Tracking

Object tracking lets you detect registered 3D objects in the camera feed and get their 6-DOF pose. Unlike VPS map localization, it does not require scanning a space — you register individual objects in the Multiset portal and query them by their object codes.

### Setup

Set `mapType` to `'object-tracking'` and pass an array of object codes. You can query up to 10 objects in a single session.

```typescript
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';
import * as THREE from 'three';

const client = new MultisetClient({
  clientId: 'YOUR_CLIENT_ID',
  clientSecret: 'YOUR_CLIENT_SECRET',
  mapType: 'object-tracking',
  code: ['OBJECT_CODE_1', 'OBJECT_CODE_2'],
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
renderer.xr.enabled = true;
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
scene.background = null;
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 100);

const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  { client }
);

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showObjectMeshes: true,  // download and display 3D outline meshes for detected objects
  onObjectMeshLoaded: (objectCode) => {
    console.log(`Mesh loaded for object: ${objectCode}`);
  },
});

adapter.initialize();
```

### Triggering Detection

Object tracking does not run automatically by default. Trigger it from a button or gesture:

```typescript
document.getElementById('detect-btn').addEventListener('click', async () => {
  const result = await adapter.trackObjects();

  if (result?.poseFound) {
    console.log('Detected objects:', result.objectCodes);
    console.log('Position:', result.position);
    console.log('Rotation:', result.rotation);
    console.log('Confidence:', result.confidence);
  } else {
    console.log('No object detected');
  }
});
```

### Auto Tracking

Enable `autoTracking` to trigger a detection attempt automatically when the session starts:

```typescript
const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  {
    client,
    autoTracking: true,
    restartTracking: true,            // re-attempt tracking after AR tracking loss recovery
    trackingCaptureDelayMs: 500,      // wait 500 ms after session start before capturing
    onObjectTrackingSuccess: (result) => {
      console.log('Tracking result:', result);
    },
    onObjectTrackingFailure: (reason) => {
      console.warn('Tracking failed:', reason);
    },
  }
);
```

### Tracking Result

The result returned by `trackObjects()` or passed to `onObjectTrackingSuccess`:

| Field         | Type           | Description                        |
| ------------- | -------------- | ---------------------------------- |
| `poseFound`   | `boolean`      | Whether an object was detected     |
| `position`    | `{x, y, z}`    | Object position in tracker space   |
| `rotation`    | `{x, y, z, w}` | Object orientation as a quaternion |
| `confidence`  | `number`       | Detection confidence (0 to 1)      |
| `objectCodes` | `string[]`     | Codes of detected objects          |

### Object Tracking in Needle Engine

Set **Map Type** to `ObjectTracking` in the `MultisetVPS` Inspector and enter one or more object codes as a comma-separated string in **Map Code**:

```
OBJECT_CODE_1,OBJECT_CODE_2
```

Enable **Auto Tracking** and **Show Object Meshes** in the Inspector for the same behavior as `autoTracking: true` and `showObjectMeshes: true`.

### Using the Core Entry Point

Without Three.js, use the adapter result handler on the session manager:

```typescript
session.setAdapterObjectTrackingHandler((result, trackerSpace) => {
  const { poseFound, position, rotation, objectCodes } = result;
  if (poseFound) {
    // Apply position and rotation to your scene graph
  }
});

// Trigger manually
const result = await session.trackObjects();
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.multiset.ai/webxr-sdk/object-tracking.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.