ARObjectTrackingView

Overview

The ARObjectTrackingView provides AR-based object tracking using the MultiSetSDK framework. This view captures frames from the AR camera and sends them to the MultiSet backend to detect and locate specific tracked objects, placing animated outline meshes over them in the real world.

Description

The object tracking view handles:

  • AR session management using ARKit and RealityKit

  • Frame capture and processing for object detection

  • Object tracking API requests (per configured objectCodes)

  • Outline mesh rendering on detected objects

  • Auto-tracking on view appearance

  • Background re-tracking at configurable intervals

  • Re-tracking when AR tracking is lost

Key Features

Auto-Tracking

When MultiSetConfig.autoObjectTracking is true, tracking starts automatically after the AR session reaches normal tracking state (with a 2-second initial delay and up to 30 seconds of polling).

Background Re-Tracking

When MultiSetConfig.backgroundObjectTracking is true, the SDK periodically re-sends tracking requests to refine object positions at the interval specified by bgObjectTrackingDurationSeconds.

Re-Tracking on AR Loss

When MultiSetConfig.restartObjectTracking is true, object tracking is automatically restarted when AR tracking state becomes paused or stopped.

Capture Delay

MultiSetConfig.objectTrackingCaptureDelay adds a short pause before capturing the tracking frame. This gives the user time to aim the camera at the object after tapping Track.

Outline Mesh Visualization

Each tracked object gets an animated outline mesh rendered over it — a two-layer RealityKit entity with an occlusion pass (depth-only) and an animated outline shader with spark/glow effects.

Gizmo Management

The localization gizmo (RGB axes) is hidden on onAppear and restored on onDisappear, since object tracking operates independently from the localization anchor.


SDK Methods Used

Object Tracking Control

Method
Description

MultiSet.shared.startObjectTracking()

Triggers a tracking request for all configured object codes

MultiSet.shared.stopObjectTracking()

Stops ongoing tracking

MultiSet.shared.clearObjectMeshes()

Removes all outline meshes from the scene

Gizmo Control

Method
Description

MultiSet.shared.setGizmoVisible(false)

Hides the localization gizmo during object tracking

State Properties

Property
Type
Description

MultiSet.shared.isObjectTrackingActive

Bool

Whether a tracking request is in progress

MultiSet.shared.hasTrackedObject

Bool

Whether tracking has succeeded at least once

MultiSet.shared.config

MultiSetConfig?

Current SDK configuration

MultiSet.shared.isAuthenticated

Bool

Whether the SDK is authenticated


State Management

View State Properties

Property
Type
Description

isTrackingNormal

Bool

Whether AR tracking state is normal

hasTrackedObject

Bool

Whether an object has been successfully tracked

isObjectTrackingActive

Bool

Polled from SDK — tracking request in progress

trackedObjectCode

String

Code of the last successfully tracked object

autoTrackingTask

Task?

Reference to the auto-tracking polling task

showCloseConfirmation

Bool

Controls close confirmation alert

showToast

Bool

Controls toast message visibility

showFailureAlert

Bool

Controls retry alert on failure


Callbacks

onObjectTrackingSuccess

Called when an object is successfully detected and positioned:

onObjectTrackingFailure

Called when object detection fails. Shows a retry alert with a user-friendly message:

User-friendly error mapping:

Server error
Displayed message

"object not found"

Point the camera at the object and try again

"low confidence"

Try a different angle or move closer

"api error" / "http"

Check internet connection and try again

onTrackingChanged

Handles AR tracking state changes to enable/disable the Track button:

onObjectMeshLoaded

Called when the outline mesh for a tracked object has been downloaded and placed:


Actions

startTracking()

Manually triggers object tracking:

resetTracking()

Clears all tracked object meshes and resets state. Restarts auto-tracking if enabled:

cleanup()

Cleans up resources when the view disappears:


Configuration

The view reads object tracking settings from MultiSet.shared.config:

Overlay Components

APILoadingOverlay

Shows while a tracking request is in progress:

  • Semi-transparent black background

  • Circular progress spinner

  • "Tracking object..." status text


Last updated