Multiplayer AR

Creating Multiplayer AR Experiences with MultiSet VPS SDK for Unity

The MultiSet Visual Positioning System (VPS) SDK for Unity empowers developers to build immersive, multiplayer Augmented Reality experiences. A core component of this is the ability for all users who localize within the same physical map to share a unified coordinate system. This synchronization allows for the seamless interaction of users and virtual objects in a shared AR space. Once this shared coordinate system is established, developers can leverage existing Unity networking solutions, such as Netcode for GameObjects or third-party assets like Photon, to broadcast and stream player coordinates across a local or global network, depending on the application's requirements.

Working sample: the SDK ships a complete shared-AR scene at Assets/MultiSet/Scenes/MultiplayerSample/MultiPlayerSample.unity, including networking, avatars and occlusion. See Multiplayer Sample. The page you are reading explains the coordinate model behind it.

Establishing a Shared Coordinate System

The MultiSet SDK achieves a shared coordinate system through its "Map Space" GameObject. When a user's device successfully localizes within a pre-scanned map, the MultiSet SDK sets the position and rotation of the MapSpace GameObject so that it aligns with the physical environment. This means that for every user localized in the same map, their individual MapSpace will have the exact same position and orientation in the real world.

This MapSpace then acts as the anchor for all AR content. By placing all shared virtual objects and player representations as children of the MapSpace GameObject, developers can ensure that these elements appear in the same real-world location for all users.

When you need a coordinate conversion, and when you do not

This is the part that most often trips people up, so it is worth stating plainly.

If content can be parented under MapSpace, you need no maths at all. Parent it, set its localPosition, and it will land in the same real-world spot on every device. Unity's scene graph does the work.

You only need to convert coordinates when a position has to leave the Unity scene graph. That means over a network connection, into a database, or into a saved file. A raw Vector3 on the wire is meaningless on the receiving device because each device's AR session starts at its own arbitrary origin. Converting it into MapSpace's frame first makes it portable.

Navigation does not need this conversion. A NavMesh Surface parented under MapSpace is automatically re-placed by Unity when MapSpace moves, so NavMeshAgent destinations work directly in world space with no InverseTransformPoint and no runtime re-bake. See NavMesh Navigation.

Player Coordinate Transformation

Each player's AR camera has its own unique world coordinates inside their own Unity scene. To share a pose across the network, convert it into the shared MapSpace frame before sending, then convert it back into world space on the receiving device.

The conversion uses two pairs of Unity operations. Positions use Transform.InverseTransformPoint and Transform.TransformPoint. Rotations must be converted too, otherwise every remote avatar will face the wrong way:

// Before sending: Unity world space -> shared MapSpace frame
Vector3    mapSpacePosition = mapSpace.transform.InverseTransformPoint(arCamera.position);
Quaternion mapSpaceRotation = Quaternion.Inverse(mapSpace.transform.rotation) * arCamera.rotation;

// After receiving: shared MapSpace frame -> this device's Unity world space
Vector3    worldPosition = mapSpace.transform.TransformPoint(mapSpacePosition);
Quaternion worldRotation = mapSpace.transform.rotation * mapSpaceRotation;

mapSpacePosition and mapSpaceRotation are what you put on the wire. They describe where the player is inside the map, which is identical for every participant, regardless of where each of them started their AR session.

A note on naming. Unity calls the result of InverseTransformPoint a local position, because it is local to the transform you passed in. The shipped MultiplayerManager.cs sample calls the same value globalPos, because it is global to the shared map. These are the same thing described from two directions. This page uses mapSpacePosition and worldPosition to avoid the ambiguity entirely.

Requirements and gotchas

• Only read MapSpace after localization succeeds. Before the first successful localization, MapSpace is inactive and sits at the scene origin, so any conversion against it returns meaningless values. Gate your networking on the LocalizationSuccess event.

• MapSpace moves again on every re-localization. If background localization or re-localization on tracking loss is enabled, MapSpace is re-aligned each time. Convert on demand rather than caching a converted Vector3, and prefer parenting to caching wherever you can.

• Keep MapSpace at a uniform scale of 1. InverseTransformPoint includes scale in its conversion, so a scaled MapSpace will silently distort shared positions. The SDK never changes MapSpace's scale.

• All devices must use the same mapCode or mapsetCode. Devices localized against different maps do not share a frame, even in the same physical room.

• Cross-platform handedness. If you exchange poses with a non-Unity client (for example a native ARKit app), you must also convert handedness. The shipped MultiplayerManager.cs shows this under its applyHandednessConversion flag.

Sample Script: PlayerPositionManager

The following script shows the complete round trip: waiting for localization, converting the local player's pose into the shared frame for broadcast, and placing a remote player from a pose received off the network.

Instructions:

1. Create a new C# script named PlayerPositionManager.

2. Attach it to a GameObject in your scene, for example a "GameManager" or the player's root object.

3. In the Inspector, assign your AR Camera, the MapSpace GameObject, the MapLocalizationManager in the scene, and a prefab to represent remote players.

using System.Collections.Generic;
using UnityEngine;
using MultiSet;

/// <summary>
/// Converts the local player's pose into the shared MapSpace frame for broadcast,
/// and places remote players from poses received over the network.
///
/// Positions expressed in the MapSpace frame are identical on every device that
/// localized against the same map, which is what makes them safe to send.
/// </summary>
public class PlayerPositionManager : MonoBehaviour
{
    [Tooltip("The main AR camera representing the user's viewpoint.")]
    public Camera arCamera;

    [Tooltip("The GameObject that is moved and oriented by the MultiSet SDK upon successful localization.")]
    public GameObject mapSpace;

    [Tooltip("The MapLocalizationManager in your scene.")]
    public MapLocalizationManager localizationManager;

    [Tooltip("Prefab used to represent a remote player.")]
    public GameObject remotePlayerPrefab;

    [Tooltip("How many pose updates to broadcast per second.")]
    public float sendRate = 10f;

    private readonly Dictionary<string, Transform> remotePlayers = new Dictionary<string, Transform>();
    private bool isLocalized;
    private float sendTimer;

    void OnEnable()
    {
        localizationManager.LocalizationSuccess.AddListener(OnLocalized);
    }

    void OnDisable()
    {
        localizationManager.LocalizationSuccess.RemoveListener(OnLocalized);
    }

    void OnLocalized()
    {
        // MapSpace is now aligned to the physical space, so conversions are valid.
        isLocalized = true;
    }

    void Update()
    {
        if (!isLocalized || arCamera == null || mapSpace == null)
        {
            return;
        }

        sendTimer += Time.deltaTime;

        if (sendTimer >= 1f / sendRate)
        {
            sendTimer = 0f;
            BroadcastPlayerPose();
        }
    }

    /// <summary>
    /// Converts the AR camera pose from this device's Unity world space into the
    /// shared MapSpace frame, then hands it to your networking layer.
    /// </summary>
    void BroadcastPlayerPose()
    {
        Transform cam = arCamera.transform;

        Vector3 mapSpacePosition = mapSpace.transform.InverseTransformPoint(cam.position);
        Quaternion mapSpaceRotation = Quaternion.Inverse(mapSpace.transform.rotation) * cam.rotation;

        // Replace with your networking solution, for example Netcode for GameObjects
        // or Photon. Only these two values need to travel over the wire.
        // YourNetworkingSolution.SendPose(mapSpacePosition, mapSpaceRotation);
    }

    /// <summary>
    /// Call this from your networking layer when a remote player's pose arrives.
    /// The incoming pose is in the shared MapSpace frame and is converted back
    /// into this device's Unity world space before being applied.
    /// </summary>
    /// <param name="remotePlayerId">A unique identifier for the remote player.</param>
    /// <param name="mapSpacePosition">Remote player position in the shared MapSpace frame.</param>
    /// <param name="mapSpaceRotation">Remote player rotation in the shared MapSpace frame.</param>
    public void UpdateRemotePlayerPose(string remotePlayerId, Vector3 mapSpacePosition, Quaternion mapSpaceRotation)
    {
        if (!isLocalized)
        {
            // Nothing to anchor against yet, so drop the update.
            return;
        }

        Transform remotePlayer;

        if (!remotePlayers.TryGetValue(remotePlayerId, out remotePlayer))
        {
            remotePlayer = Instantiate(remotePlayerPrefab, mapSpace.transform).transform;
            remotePlayer.name = "RemotePlayer_" + remotePlayerId;
            remotePlayers.Add(remotePlayerId, remotePlayer);
        }

        // Because the avatar is parented under MapSpace, the received pose can be
        // applied directly as a local pose. No conversion is needed on this path.
        remotePlayer.localPosition = mapSpacePosition;
        remotePlayer.localRotation = mapSpaceRotation;
    }

    /// <summary>
    /// Equivalent placement for an avatar that is NOT parented under MapSpace.
    /// Here the received pose must be converted into world space explicitly.
    /// </summary>
    public void UpdateUnparentedRemotePlayer(Transform remotePlayer, Vector3 mapSpacePosition, Quaternion mapSpaceRotation)
    {
        remotePlayer.SetPositionAndRotation(
            mapSpace.transform.TransformPoint(mapSpacePosition),
            mapSpace.transform.rotation * mapSpaceRotation);
    }

    /// <summary>
    /// Call this from your networking layer when a remote player disconnects.
    /// </summary>
    public void RemoveRemotePlayer(string remotePlayerId)
    {
        Transform remotePlayer;

        if (remotePlayers.TryGetValue(remotePlayerId, out remotePlayer))
        {
            Destroy(remotePlayer.gameObject);
            remotePlayers.Remove(remotePlayerId);
        }
    }
}

Note the two placement paths at the end. UpdateRemotePlayerPose parents the avatar under MapSpace and assigns the received pose directly, with no conversion. UpdateUnparentedRemotePlayer keeps the avatar at the scene root and converts explicitly. Both produce the same result on screen. Prefer the parented version wherever your networking library allows it, because it stays correct automatically when MapSpace is re-aligned by a later localization.

Related pages

• Multiplayer Sample for the complete shipped scene, including transport, avatars and occlusion.

• NavMesh Navigation for pathfinding, which needs no coordinate conversion.

• MapLocalizationManager for the full list of localization events.