Three.js Integration

Full end-to-end VPS integration with Three.js using ThreeAdapter

ThreeAdapter wires XRSessionManager to a Three.js renderer. It manages the XR render loop, synchronizes the camera, downloads the map mesh, and gives you a worldFromMap matrix in onLocalizationSuccess so you can place content at any map-local coordinate.

Installation

npm install @multisetai/vps three

Three.js 0.169.0 or higher is required.

Complete Working Example

The following sets up a full AR session with auto-localization. Place this in your entry file and serve over HTTPS.

import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

// --- Scene setup ---
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
scene.background = null; // transparent so the camera feed shows through

const camera = new THREE.PerspectiveCamera(
  70,
  window.innerWidth / window.innerHeight,
  0.01,
  100
);
scene.add(camera);

scene.add(new THREE.AmbientLight(0xffffff, 1));

window.addEventListener('resize', () => {
  camera.aspect = window.innerWidth / window.innerHeight;
  camera.updateProjectionMatrix();
  renderer.setSize(window.innerWidth, window.innerHeight);
});

// --- Multiset client ---
const client = new MultisetClient({
  clientId: import.meta.env.VITE_MULTISET_CLIENT_ID,
  clientSecret: import.meta.env.VITE_MULTISET_CLIENT_SECRET,
  mapType: 'map',
  code: import.meta.env.VITE_MAP_CODE,
});

await client.authorize();

// --- XR session ---
const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  {
    client,
    autoLocalize: true,
    onLocalizationFailure: (reason) => console.warn('Localization failed:', reason),
    onError: (error) => console.error('Session error:', error),
  }
);

// --- Three.js adapter ---
const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showMesh: true,    // download and display the map mesh after localization
  showGizmo: true,   // show transform gizmo at map origin

  onLocalizationSuccess: (result, worldFromMap) => {
    // worldFromMap is a THREE.Matrix4 that converts map-local to world coordinates.
    // Use it to place objects at specific map-local positions.
    const box = new THREE.Mesh(
      new THREE.BoxGeometry(0.3, 0.3, 0.3),
      new THREE.MeshStandardMaterial({ color: 0x0099ff })
    );

    // Place the box at map origin
    box.applyMatrix4(worldFromMap);
    scene.add(box);

    console.log('Confidence:', result.localizeData.confidence);
  },
});

// --- Start ---
// initialize() starts the preview loop and mounts the AR button
adapter.initialize();

Placing Content at Map Coordinates

The worldFromMap matrix in onLocalizationSuccess transforms any map-local position to Three.js world space. Use it to position objects at known coordinates within your map.

onLocalizationSuccess: (result, worldFromMap) => {
  // Example: place a marker at map-local position (2, 0, -3)
  const marker = new THREE.Mesh(
    new THREE.SphereGeometry(0.1),
    new THREE.MeshBasicMaterial({ color: 0xff0000 })
  );

  const localPosition = new THREE.Vector3(2, 0, -3);
  localPosition.applyMatrix4(worldFromMap);
  marker.position.copy(localPosition);
  scene.add(marker);
}

MapSet Mode

Switch mapType to 'map-set' and use your map set code. Everything else stays the same.

const client = new MultisetClient({
  clientId: '...',
  clientSecret: '...',
  mapType: 'map-set',
  code: 'MAPSET_XXXXXXXXXX',
});

Manual Localization (No Auto-Localize)

Disable autoLocalize and trigger localization from your own UI:

const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  {
    client,
    autoLocalize: false,
  }
);

const adapter = new ThreeAdapter({ session, renderer, scene, camera });
adapter.initialize();

document.getElementById('localize-btn').addEventListener('click', async () => {
  const result = await adapter.localizeFrame();
  if (result) {
    console.log('Localized with confidence:', result.localizeData.confidence);
  } else {
    console.log('No pose found');
  }
});

Custom AR Button

Disable the built-in button and build your own:

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  useDefaultButton: false,
});

adapter.initialize();

myStartButton.addEventListener('click', () => adapter.startSession());
myStopButton.addEventListener('click', () => adapter.stopSession());

Frame-Level Access

Use onXRFrame to run code every frame during an active AR session, after camera matrices are synced but before rendering:

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  onXRFrame: ({ frame, pose, deltaSeconds }) => {
    // Update animations, physics, etc.
    myObject.rotation.y += deltaSeconds;
  },
});

Cleanup

adapter.dispose(); // ends session, removes listeners, releases GL resources