Needle Engine Integration

Integrate Multiset VPS into Needle Engine projects using Inspector components or the NeedleAdapter API

The SDK includes three Needle Engine components that cover the full VPS workflow. You can use them via the Unity Inspector (no code required for basic setups) or programmatically via NeedleAdapter for custom logic.

Installation

npm install @multisetai/vps

Needle Engine ships its own Three.js fork, so do not install three separately.

Inspector-Driven Setup (Recommended for Unity Developers)

This path lets you configure VPS entirely from the Unity Inspector and export to web with Needle Engine.

Step 1: Copy the Templates

Copy the three template files into your Needle Engine web project's scripts folder:

cp node_modules/@multisetai/vps/templates/MultisetVPS.ts web/src/scripts/
cp node_modules/@multisetai/vps/templates/MapAnchor.ts web/src/scripts/
cp node_modules/@multisetai/vps/templates/MapType.ts web/src/scripts/
cp node_modules/@multisetai/vps/templates/MapType.cs Assets/Scripts/

MapType.cs is a C# file for Unity — it exposes the MapType enum in the Inspector dropdown. MapType.ts is the TypeScript counterpart used by Needle Engine at runtime.

Step 2: Add MultisetVPS to a GameObject

In Unity, add the MultisetVPS component to any GameObject. Fill in the Inspector fields:

Field	Description
Client Id	From the Multiset Developer Portal
Client Secret	From the Multiset Developer Portal
Map Code	Map code, MapSet code, or comma-separated object codes
Map Type	SingleMap, MapSet, or ObjectTracking
Auto Localize	Start localization automatically when the AR session begins
Show Mesh	Download and display the map mesh after localization
Show Gizmo	Display a transform gizmo at the map origin
Relocalization	Re-localize when AR tracking is lost
Background Localization	Continuously re-localize to reduce drift
Confidence Check	Reject results below the confidence threshold
Confidence Threshold	Minimum accepted confidence (0.2 to 0.8)

Step 3: Add MapAnchor to Content GameObjects

Add the MapAnchor component to any GameObject you want anchored to the VPS map.

Field	Description
Offset	Position offset from the map origin in metres
Match Orientation	Align object rotation to the map orientation
Rotation Offset	Additional rotation applied after matching orientation
Hide Until Localized	Keep the object hidden until the first successful localization
Is Right Handed	Enable if entering Three.js coordinates directly. Leave off for Unity coordinates (auto-converts)

MapAnchor hides the entire GameObject when hideUntilLocalized is on. If the object has sibling components that need to run before localization, move them to a separate always-visible GameObject.

Step 4: Export and Run

Export the scene to web via Needle Engine. The AR button appears automatically on the exported page. No additional JavaScript is required.

---

Programmatic Setup (Custom Logic)

Use NeedleAdapter directly when you need to add listeners, react to localization events in other Behaviours, or build custom UI.

import { Behaviour, serializable } from '@needle-tools/engine';
import { MultisetClient } from '@multisetai/vps/core';
import { NeedleAdapter } from '@multisetai/vps/needle';

export class MyVPSController extends Behaviour {

  @serializable()
  clientId: string = '';

  @serializable()
  clientSecret: string = '';

  @serializable()
  mapCode: string = '';

  private adapter: NeedleAdapter | null = null;

  async start() {
    const client = new MultisetClient({
      clientId: this.clientId,
      clientSecret: this.clientSecret,
      mapType: 'map',
      code: this.mapCode,
    });

    await client.authorize();

    this.adapter = new NeedleAdapter({
      client,
      sessionOptions: {
        autoLocalize: true,
        confidenceCheck: true,
        confidenceThreshold: 0.6,
      },
      showMesh: true,
      onLocalizationSuccess: (result, worldFromMap) => {
        console.log('Localized! Confidence:', result.localizeData.confidence);
        // worldFromMap is a THREE.Matrix4
      },
    });
  }

  onDestroy() {
    this.adapter?.dispose();
  }
}

Listening to Localization Events from Another Behaviour

NeedleAdapter supports multiple listeners, so other components can subscribe without coupling to MultisetVPS:

import { Behaviour } from '@needle-tools/engine';
import { NeedleAdapter } from '@multisetai/vps/needle';
import type { ILocalizeAndMapDetails } from '@multisetai/vps/core';
import * as THREE from 'three';

export class MyContentPlacer extends Behaviour {

  start() {
    // Find the adapter from MultisetVPS on the scene
    const vps = this.gameObject.scene.getComponentInChildren(MultisetVPS);
    if (!vps?.adapter) return;

    vps.adapter.addLocalizationListener(this.onLocalized);
    vps.adapter.addSessionEndListener(this.onSessionEnd);
  }

  private onLocalized = (result: ILocalizeAndMapDetails, worldFromMap: THREE.Matrix4) => {
    // Place content at a map-local position
    const localPos = new THREE.Vector3(1.5, 0, -2);
    localPos.applyMatrix4(worldFromMap);
    this.gameObject.position.copy(localPos);
  };

  private onSessionEnd = () => {
    // Reset content visibility
    this.gameObject.visible = false;
  };

  onDestroy() {
    const vps = this.gameObject.scene.getComponentInChildren(MultisetVPS);
    vps?.adapter?.removeLocalizationListener(this.onLocalized);
    vps?.adapter?.removeSessionEndListener(this.onSessionEnd);
  }
}

Runtime-Spawned MapAnchors

If you instantiate a GameObject containing a MapAnchor after the scene loads, register it with the adapter so it receives the last localization result immediately:

import { instantiate } from '@needle-tools/engine';
import { MapAnchor } from './MapAnchor';

const prefab = ...; // your prefab reference
const instance = instantiate(prefab);
const anchor = instance.getComponent(MapAnchor);

if (anchor && vps.adapter) {
  vps.adapter.registerAnchor(anchor); // replays last localization if available
}

Customizing the buttons

MultisetVPS does not use the SDK's built-in AR button. It passes useDefaultButton: false and builds its own set in _createUI(), so every button you see on screen lives in the template file you copied into src/scripts/. That is the one place to edit.

Button	Appears when	Does
START AR / STOP AR	always	Toggles the XR session. The label is swapped in _onSessionStart() and _onSessionEnd().
CAPTURE FRAME	VPS modes (SingleMap, MapSet)	Runs one localization. Shows LOCALIZING... and disables itself while the request is in flight, then resets.
TRACK OBJECT	ObjectTracking mode	Runs one object detection pass. Shows TRACKING... while running.
CLEAR MESHES	ObjectTracking mode	Removes the object outline meshes already placed in the scene.

To change the layout, styling, or which buttons exist, edit _createUI(). To change what the capture button reads while it is working, look for LOCALIZING... in the same file and at _resetCaptureButton(), which restores the default label.

Whatever you build, startSession() must be the first call inside the click handler. Any await before it consumes the browser's user activation token and the session request is rejected. The template notes this in a comment directly above _createUI().

The UI is mounted on the canvas's parent element so it shares the same stacking context. During an immersive-ar session the WebGL canvas receives no pointer events, so the session requests the dom-overlay feature to make HTML tappable. A Unity world-space Canvas still renders for signage, but it cannot be tapped while a session is presenting.

Manual Session Control

// Check support before showing the AR button
const supported = await NeedleAdapter.isSupported();

// Start and stop from your own UI
await adapter.startSession(); // must be inside a user gesture handler
adapter.stopSession();

// Trigger localization manually (when autoLocalize is false)
const result = await adapter.localizeFrame();