How It Works

This page explains how a MultiSet application thinks about space, independent of which SDK, language, or hardware you use. The vocabulary here (pose, session space, map space, localization, anchor) shows up on every other page, in every SDK API, and in every API response.

Read it top to bottom if you are new to spatial computing. If you have shipped AR or robot navigation before, three coordinate spaces and keeping content valid are where a VPS changes the picture.

Any camera moving through physical space

MultiSet VPS answers one question: where is this camera right now, inside a space that has already been scanned? The answer is a 6-DoF pose, accurate to centimeters, computed from an ordinary camera frame.

That question does not change with the thing holding the camera. A phone in someone's hand, a headset on their head, glasses on their face, a robot crossing a warehouse, a drone inspecting a facade, a camera bolted to a machine: all of them are cameras moving through physical space, and all of them need the same answer. The platform draws no distinction between them. There is no AR mode and no robotics mode. There is one map, one query, and one pose, and what happens next belongs to the client.

Every agent sends the same request and gets back the same pose

Agent	What it is doing	What it does with the pose
Phone, tablet, headset	Running an AR session	Draws content registered to the real room, every frame
AI glasses and wearables	Worn hands-free, often with no display	Fires audio, haptic, or HUD events at the right place
Mobile robot or AMR	Driving a route indoors	Corrects the drift in its own odometry
Drone or UAV	Flying an inspection or a survey	Holds an absolute position where GPS is unusable
Fixed or IoT camera	Bolted to a wall, a machine, or a cart	Reports precise location with no AR framework involved

Three things hold whatever the agent is:

• The map is shared. One scan serves phones, glasses, robots, and drones together, in a single coordinate frame. A robot and a technician's glasses report positions that can be compared directly, with no extra alignment step.

• The query is the same. Camera frame, intrinsics, and optional hints in. poseFound, position, and rotation out. Any client that can make an HTTPS request can call it, down to an ESP32 class board: see Integration with IoT Devices.

• The pose is absolute. It carries no accumulated error, however long the agent has been running.

One constraint shapes every integration. A query is a network round trip of roughly two seconds, so no agent can put it inside its render loop or its control loop. Each one runs its own local motion estimate in between and uses the VPS for absolute correction. The rest of this page is that pattern in detail: first the pose and the spaces it lives in, then each agent in turn.

One map, however you capture it

Nothing above says how the space got scanned, and the map does not care. A site can be flown with a 360° camera drone, walked with a handheld 360 camera, or scanned with a phone or a survey scanner, and those captures merge into a single coordinate frame rather than staying separate files.

Captures from the air and the ground fold into one model that every device queries

Two things follow:

• Aerial and ground combine. A MapSet holds the relative transform between captures, so a facade flown from the air and a corridor walked on foot end up in the same frame. Query the set and the pose resolves across all of it.

• You do not have to query from where you captured. Localization matches a live camera view against the map, not against a recorded path, so a device can localize from a position and angle no capture ever occupied, as long as it is looking at mapped surfaces.

See Third Party Scans for every supported input, and 360 Videos for the ground and aerial video routes.

---

What goes in, what comes out

Input	Where it comes from	Why it matters
Camera frame	The agent's camera	The only thing the VPS actually looks at
Camera intrinsics	ARKit, ARCore, the camera driver, or the Camera Intrinsics endpoint	Required for an accurate pose
Camera and inertial motion	Whatever the agent uses to track its own movement	Keeps the pose alive between queries
GPS fix (optional)	The device, passed as geoHint	Narrows the search on a georeferenced map
Prior knowledge (optional)	hintPosition, hintFloorHeight, hintMapCodes	Narrows the search, making queries faster and more reliable

Output	Shape	What it is for
poseFound	boolean	Whether the query succeeded at all
position, rotation	x, y, z in meters and a quaternion	The camera pose in map space
confidence	number	How much to trust this result before applying it
GeoPose (optional)	latitude, longitude, altitude, quaternion	Global handoff, see GeoPose Support
Map assets (optional)	mesh or splat	Occlusion, physics, and visualization

---

Spatial state

Pose

A pose is a position and an orientation together: six degrees of freedom. MultiSet returns it as a position in meters and a rotation quaternion, both expressed in the map's coordinate space.

Two systems write the pose, everything else reads it

Almost every bug in a spatial application is a pose bug. Content in the wrong place, content that slides, a robot that thinks it is one aisle over, a result that is correct on one device and not another: all of these are the pose being wrong, not the thing reading it.

By default poses come back in a left-handed, Y-up frame (the Unity convention). Set isRightHanded: true to get the right-handed Y-up frame that ARKit, ARCore, WebXR, and Three.js use. See Coordinate Systems.

Tracking, and why it drifts

Every agent has some way of estimating its own motion: ARKit, ARCore, WebXR or the Quest runtime on a consumer device, wheel encoders and LiDAR SLAM on a robot, visual-inertial odometry on a drone. All of them work the same way in the end, by measuring change from the last known state and adding it up. Each measurement carries a small error, and those errors compound. Go far enough and content that started perfectly placed slides away from where you put it, or the robot's reported position ends up in the next aisle. This is drift.

Tracking error accumulates. Each localization resets it.

Drift is worse with fast movement, poor lighting, reflective or featureless surfaces, and lower-end hardware. No agent can eliminate it on its own, which is exactly what a VPS is for: each successful localization replaces the accumulated estimate with a measured, absolute one. Practical guidance on when to trigger this lives in Drift Control.

Three coordinate spaces

The single most useful habit in a VPS application is knowing which space a set of numbers is in. There are three, and a coordinate is meaningless without knowing which one it belongs to.

The same physical point has a different set of numbers in each space

Session space is what an agent gets for free from its own motion estimate. A phone gets it from ARKit or ARCore, a robot from wheel or visual odometry (the odom frame in ROS), a drone from its VIO. These are all the same kind of frame: local, relative, and private to one run. The origin is wherever that particular session happened to start, so a phone and a robot standing side by side in the same room hold two completely different sets of coordinates for the same spot. It is precise near the origin and gets less trustworthy the further the agent travels. Nothing durable should ever be stored in this space.

Map space is the frame of a Map, a MapSet, or a Map Version. Its origin is tied to the physical space that was scanned, so it is the same today and next year, on a phone and on a robot. This is the frame you author in, store in, and share in.

Global space is WGS 84: latitude, longitude, altitude, and heading, optionally packaged as GeoPose. It only exists for a map that has been georeferenced. It is the right frame for handing position to a GIS, a fleet manager, or another platform, and the wrong frame for placing anything precisely, because GPS on its own is accurate to meters rather than centimeters.

Transforms between the spaces

A transform converts a pose from one space to another. Each of these has exactly one source.

Conversion	What provides it	Typical use
Session space → Map space	A successful localization	Rendering map-authored content, or placing the agent on a site plan
Map space → Session space	The inverse of the same transform	Reporting where the agent is, feeding hintPosition into the next query
Map space → Global space	Georeferencing the map	GeoPose output, GIS and robotics handoff
Global space → Map space	The same georeference, applied server side	Turning a device GPS fix into a geoHint filter
Map space → MapSet space	MapSet alignment	One continuous frame across several scans of a large venue
Old map space → New map space	Map Versioning	Keeping content authored on an earlier scan valid after a rescan

Hint parameters are always supplied in the map's native left-handed frame, whatever you set isRightHanded to. Passing a right-handed hintPosition straight back from a previous result is the most common cause of a query that mysteriously stops finding a pose. Negate x first. See Coordinate Systems.

---

Localization

What a query actually gives you

Before localizing, an agent knows how it has moved but has no idea where it is. A localization query answers that: send a camera frame, get back the camera's pose in map space. Once you have it, the session frame is pinned to the map frame, and map coordinates become directly usable.

Localization solves the offset between session space and map space

Reading the result

A query returns poseFound and, when it succeeds, a confidence value. Treat both as a gate rather than a formality: applying a low-confidence pose is worse than applying none, because it moves something that was previously correct. On a failed query, keep running on local tracking and try again from a different viewpoint.

There are two ways to ask:

• A single frame query sends one image and answers in about two seconds. Use it for frequent updates in well-textured spaces.

• A multi frame query sends four to six images with their tracking poses and answers in up to about five seconds. Use it when the space is repetitive, dim, or partly occluded.

When to localize

Localization is not a one-off setup step. Localize:

• On entry, to establish the transform in the first place.

• On a schedule, to correct drift before it becomes visible. Background relocalization can run every 15 seconds to 2 minutes.

• After tracking is lost, when fast motion or poor lighting breaks the local tracker.

• When the app returns from the background, since an interrupted session usually resumes with a broken frame.

The SDKs handle these triggers for you; with the REST API you own the policy.

---

Anchoring content

Author in map space, always

An anchor is a pose in map space that something of yours hangs from: a 3D model, a label, a navigation waypoint, an inspection point, a trigger zone. Because map space is stable, an anchor authored once is valid in every future session, on every device, and for every agent. Two people standing in the same room, localized against the same map, see the same object in the same physical place with no extra syncing, and a robot sent to those coordinates arrives at the same spot. That is what makes shared and multiplayer experiences straightforward on a VPS.

The practical rule: store coordinates in map space, never in session space. Session coordinates are valid for one run on one unit, and are meaningless the moment it restarts.

You can author anchors visually with Content Space in the MultiSet App, or place them in the editor against the map mesh in Unity. To find the map-space coordinates of a specific spot, see Find Hint Coordinates.

Keeping content valid when the space changes

Physical spaces change: furniture moves, a wing gets refitted, a scan ages. Rescanning would normally produce a brand new map with a brand new origin, invalidating everything you authored.

Map Versioning exists to prevent that. A new scan is aligned to the base map by the VPS and stored as a version of it rather than as a replacement, so the coordinate frame carries over and your anchors, routes, and overlays keep working without re-authoring, even when the new capture comes from different hardware.

A useful pairing to remember:

Use a MapSet to make a space bigger. Use a Map Version to make a space newer.

---

Agent by agent

Everything above is shared. What differs is the loop each agent runs, and what it does with a pose once it has one.

AR on phones, headsets, and tablets

The most demanding case, because the pose is consumed continuously. Camera frames arrive 30 to 60 times a second, and on each one the pose changes, content has to be re-projected, and a frame has to be drawn. Nothing waits for a tap.

The per-frame loop runs on the device. A VPS query happens occasionally alongside it.

Since the render loop cannot wait two seconds for a query, the division of labour is fixed: device tracking carries the pose frame to frame, and the VPS corrects it periodically. The SDKs do that plumbing, including when to relocalize, which leaves you placing content in map space and letting the SDK keep it aligned.

A continuous pose buys capabilities the other agents do not get:

• Registered rendering. Content sits on real geometry on every frame, not only at the moment of a fix.

• Occlusion. The map mesh lets real walls and machinery hide virtual objects. See Occlusion.

• Shared sessions. Two devices localized against one map agree on every coordinate, which is all multiplayer AR needs.

• Navigation. Paths authored once in map space drive turn-by-turn guidance for everyone. See NavMesh Navigation.

Robots

A mobile robot estimates its own motion from wheel encoders, an IMU, and often LiDAR or visual SLAM. Every one of those is dead reckoning: each estimate is relative to the last, so error grows with distance travelled. It is the same problem as drift on a phone, measured in a warehouse instead of a room.

The conventional cure is loop closure, which needs the robot to return somewhere it has already been and recognise it. A VPS query has no such requirement. It is an absolute measurement against a map made earlier, available at any point on the route, whether or not the robot has passed that way this run.

Each fix removes the error built up since the last one

Feed the result into the robot's existing state estimator as a global position measurement rather than overwriting the state with it. Odometry keeps running at its own high rate between fixes, and each VPS result corrects the offset that has built up since the previous one. In ROS terms, it is a correction to the map to odom transform.

Two things to plan for:

• ROS is right-handed and Z-up. Request the right-handed pose with isRightHanded: true, then apply your own axis swap on top of it. See Coordinate Systems.

• Fixes are sparse next to odometry. A query is a round trip of roughly two seconds, so treat the result as a measurement to fuse rather than a position to snap to, and gate it on confidence before it reaches a motion planner.

Drones

GPS is the default answer outdoors, and it degrades in exactly the places drones earn their keep: close to a facade, under a canopy or a bridge deck, inside a warehouse, between racking. Signals bounce off structures or disappear altogether, and the error is measured in meters at the moment you most need centimeters.

GPS quality depends on where the aircraft is. Visual positioning does not.

Visual positioning is unaffected by all of that, because it only needs to see the space that was mapped. The architecture matches the robot case: onboard visual-inertial odometry flies the aircraft between queries, and VPS fixes stop that estimate drifting. If the map is georeferenced, the pose also comes back as GeoPose, so an existing waypoint stack that speaks latitude and longitude carries on speaking it with a visually derived position underneath.

A drone can also be what builds the map in the first place. See Antigravity A1 (Drone) for capturing a site from the air.

AI glasses and wearables

The most interesting case, because the assumptions from the AR section do not hold. Many pairs, Meta Ray-Ban Smart Glasses among them, have a camera but no 6-DoF tracking, and often no display either. The camera streams to a paired phone, and the phone runs the query. See Meta Ray-Ban Smart Glasses for the sample apps.

The consequence that shapes everything else: there is no pose between queries. With nothing dead reckoning in the gaps, there is no continuous frame to register content into, so anchoring a rendered object stops being possible. What you have instead is a series of accurate, absolute position readings.

Each fix is evaluated against zones and points of interest, and triggers an action

That turns out to be plenty, because it changes the question from "where do I draw this" to "where is the wearer now, and what should happen there". Each fix is checked against zones and points of interest, and what it triggers is audio, haptics, a notification, or a line of text on a HUD:

• Zone entry and exit. Fire when a fix lands inside a region the previous fix was outside of.

• Proximity. Compare the fix against a list of points of interest and act on the nearest one in range.

• Heading. The pose carries rotation as well as position, so you know which way the wearer is facing and can say "on your left" instead of only "nearby".

• Turn-by-turn. Chain zones along a route and announce each leg as the wearer reaches it.

Two design consequences follow from having no tracking:

• Cadence is resolution. The interval between queries decides how finely you can react. At walking pace, a fix every three seconds is roughly one every four meters, so a two-meter trigger zone will be missed regularly. Size zones to the cadence, or raise the cadence.

• Debounce every trigger. Two consecutive fixes either side of a boundary can toggle a zone on and off. Require a zone change to hold before acting on it, and never re-announce the same event on a repeat fix inside the same zone.

Glasses with a display, such as the Ray-Ban Display models, sit between the two worlds: still no 6-DoF tracking, but a HUD to put the answer on, which suits a heading arrow and a distance that refresh once per fix.