Three.js Integration

Full end-to-end VPS integration with Three.js using ThreeAdapter

ThreeAdapter wires XRSessionManager to a Three.js renderer. It manages the XR render loop, synchronizes the camera, downloads the map mesh, and gives you a worldFromMap matrix in onLocalizationSuccess so you can place content at any map-local coordinate.

Installation

npm install @multisetai/vps three

Three.js 0.169.0 or higher is required.

Complete Working Example

The following sets up a full AR session with auto-localization. Place this in your entry file and serve over HTTPS.

import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

// --- Scene setup ---
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
scene.background = null; // transparent so the camera feed shows through

const camera = new THREE.PerspectiveCamera(
  70,
  window.innerWidth / window.innerHeight,
  0.01,
  100
);
scene.add(camera);

scene.add(new THREE.AmbientLight(0xffffff, 1));

window.addEventListener('resize', () => {
  camera.aspect = window.innerWidth / window.innerHeight;
  camera.updateProjectionMatrix();
  renderer.setSize(window.innerWidth, window.innerHeight);
});

// --- Multiset client ---
const client = new MultisetClient({
  clientId: import.meta.env.VITE_MULTISET_CLIENT_ID,
  clientSecret: import.meta.env.VITE_MULTISET_CLIENT_SECRET,
  mapType: 'map',
  code: import.meta.env.VITE_MAP_CODE,
});

await client.authorize();

// --- XR session ---
const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  {
    client,
    autoLocalize: true,
    onLocalizationFailure: (reason) => console.warn('Localization failed:', reason),
    onError: (error) => console.error('Session error:', error),
  }
);

// --- Three.js adapter ---
const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showMesh: true,    // download and display the map mesh after localization
  showGizmo: true,   // show transform gizmo at map origin

  onLocalizationSuccess: (result, worldFromMap) => {
    // worldFromMap is a THREE.Matrix4 that converts map-local to world coordinates.
    // Use it to place objects at specific map-local positions.
    const box = new THREE.Mesh(
      new THREE.BoxGeometry(0.3, 0.3, 0.3),
      new THREE.MeshStandardMaterial({ color: 0x0099ff })
    );

    // Place the box at map origin
    box.applyMatrix4(worldFromMap);
    scene.add(box);

    console.log('Confidence:', result.localizeData.confidence);
  },
});

// --- Start ---
// initialize() starts the preview loop and mounts the AR button
adapter.initialize();

Placing Content at Map Coordinates

The worldFromMap matrix in onLocalizationSuccess transforms any map-local position to Three.js world space. Use it to position objects at known coordinates within your map.

onLocalizationSuccess: (result, worldFromMap) => {
  // Example: place a marker at map-local position (2, 0, -3)
  const marker = new THREE.Mesh(
    new THREE.SphereGeometry(0.1),
    new THREE.MeshBasicMaterial({ color: 0xff0000 })
  );

  const localPosition = new THREE.Vector3(2, 0, -3);
  localPosition.applyMatrix4(worldFromMap);
  marker.position.copy(localPosition);
  scene.add(marker);
}

MapSet Mode

Switch mapType to 'map-set' and use your map set code. Everything else stays the same.

const client = new MultisetClient({
  clientId: '...',
  clientSecret: '...',
  mapType: 'map-set',
  code: 'MAPSET_XXXXXXXXXX',
});

Manual Localization (No Auto-Localize)

Disable autoLocalize and trigger localization from your own UI:

const session = new XRSessionManager(
  renderer.getContext() as WebGL2RenderingContext,
  {
    client,
    autoLocalize: false,
  }
);

const adapter = new ThreeAdapter({ session, renderer, scene, camera });
adapter.initialize();

document.getElementById('localize-btn').addEventListener('click', async () => {
  const result = await adapter.localizeFrame();
  if (result) {
    console.log('Localized with confidence:', result.localizeData.confidence);
  } else {
    console.log('No pose found');
  }
});

Customizing the AR button

The built-in button is a single element that toggles the session: it reads START AR, becomes STOP AR while a session is running, and swaps its CSS class from multiset-ar-inactive to multiset-ar-active. It always carries the class multiset-ar-button.

There are three levels of customization.

Restyle the built-in button

onButtonCreated hands you the element the SDK created:

const adapter = new ThreeAdapter({
  session, renderer, scene, camera,
  buttonContainer: myOverlayDiv,          // defaults to overlayRoot, then document.body
  onButtonCreated: (button) => {
    button.style.cssText = '';            // clear the inline styles first, see below
    button.classList.add('my-ar-button');
  },
});

createButton() sets its appearance through inline style.cssText, and inline styles beat an external stylesheet. A rule targeting .multiset-ar-button in your CSS will appear to do nothing. Either clear button.style.cssText first as above, mutate button.style directly, or mark your CSS !important.

Replace it entirely

const adapter = new ThreeAdapter({
  session, renderer, scene, camera,
  useDefaultButton: false,
});

adapter.initialize();                     // returns null when useDefaultButton is false

myStartButton.addEventListener('click', () => adapter.startSession());
myStopButton.addEventListener('click', () => adapter.stopSession());

Your own button has to track session state itself, since nothing updates its label. Use addSessionStartListener and addSessionEndListener, which is also what the built-in button does:

adapter.addSessionStartListener(() => { myStartButton.textContent = 'STOP AR'; });
adapter.addSessionEndListener(() => { myStartButton.textContent = 'START AR'; });

startSession() must be the first call in your click handler. Any await before it consumes the browser's user activation token and the session request is rejected. This is the most common reason a custom button fails while the built-in one works.

Build the button yourself from the session

session.createButton() returns the pre-styled element without the adapter mounting it, so you can place it wherever you like:

const button = session.createButton();
myToolbar.appendChild(button);

Frame-Level Access

Use onXRFrame to run code every frame during an active AR session, after camera matrices are synced but before rendering:

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  onXRFrame: ({ frame, pose, deltaSeconds }) => {
    // Update animations, physics, etc.
    myObject.rotation.y += deltaSeconds;
  },
});

Cleanup

adapter.dispose(); // ends session, removes listeners, releases GL resources