ARObjectTrackingView

Overview

The ARObjectTrackingView provides AR-based object tracking using the MultiSetSDK framework. This view captures frames from the AR camera and sends them to the MultiSet backend to detect and locate specific tracked objects, placing animated outline meshes over them in the real world.

Description

The object tracking view handles:

• AR session management using ARKit and RealityKit

• Frame capture and processing for object detection

• Object tracking API requests (per configured objectCodes)

• Outline mesh rendering on detected objects

• Auto-tracking on view appearance

• Background re-tracking at configurable intervals

• Re-tracking when AR tracking is lost

Key Features

Auto-Tracking

When MultiSetConfig.autoObjectTracking is true, tracking starts automatically after the AR session reaches normal tracking state (with a 2-second initial delay and up to 30 seconds of polling).

Background Re-Tracking

When MultiSetConfig.backgroundObjectTracking is true, the SDK periodically re-sends tracking requests to refine object positions at the interval specified by bgObjectTrackingDurationSeconds.

Re-Tracking on AR Loss

When MultiSetConfig.restartObjectTracking is true, object tracking is automatically restarted when AR tracking state becomes paused or stopped.

Capture Delay

MultiSetConfig.objectTrackingCaptureDelay adds a short pause before capturing the tracking frame. This gives the user time to aim the camera at the object after tapping Track.

Outline Mesh Visualization

Each tracked object gets an animated outline mesh rendered over it: a two-layer RealityKit entity with an occlusion pass (depth-only) and an animated outline shader with spark/glow effects.

Gizmo Management

The localization gizmo (RGB axes) is hidden on onAppear and restored on onDisappear, since object tracking operates independently from the localization anchor.

---

SDK Methods Used

Object Tracking Control

Method	Description
MultiSet.shared.startObjectTracking()	Triggers a tracking request for all configured object codes
MultiSet.shared.stopObjectTracking()	Stops ongoing tracking
MultiSet.shared.clearObjectMeshes()	Removes all outline meshes from the scene

Gizmo Control

Method	Description
MultiSet.shared.setGizmoVisible(false)	Hides the localization gizmo during object tracking

State Properties

Property	Type	Description
MultiSet.shared.isObjectTrackingActive	Bool	Whether a tracking request is in progress
MultiSet.shared.hasTrackedObject	Bool	Whether tracking has succeeded at least once
MultiSet.shared.config	MultiSetConfig?	Current SDK configuration
MultiSet.shared.isAuthenticated	Bool	Whether the SDK is authenticated

---

State Management

View State Properties

Property	Type	Description
isTrackingNormal	Bool	Whether AR tracking state is normal
hasTrackedObject	Bool	Whether an object has been successfully tracked
isObjectTrackingActive	Bool	Polled from SDK, tracking request in progress
trackedObjectCode	String	Code of the last successfully tracked object
autoTrackingTask	Task?	Reference to the auto-tracking polling task
showCloseConfirmation	Bool	Controls close confirmation alert
showToast	Bool	Controls toast message visibility
showFailureAlert	Bool	Controls retry alert on failure

---

Callbacks

onObjectTrackingSuccess

Called when an object is successfully detected and positioned:

sdkDelegate.onObjectTrackingSuccess = { objectCode, confidence in
    trackedObjectCode = objectCode
    hasTrackedObject = true
    if MultiSet.shared.config?.showAlerts == true {
        showToastMessage("Object tracked: \(objectCode)", success: true)
    }
}

onObjectTrackingFailure

Called when object detection fails. Shows a retry alert with a user-friendly message:

sdkDelegate.onObjectTrackingFailure = { error in
    if MultiSet.shared.config?.showAlerts == true {
        failureAlertMessage = userFriendlyTrackingMessage(for: error)
        showFailureAlert = true
    }
}

User-friendly error mapping:

Server error	Displayed message
"object not found"	Point the camera at the object and try again
"low confidence"	Try a different angle or move closer
"api error" / "http"	Check internet connection and try again

onTrackingChanged

Handles AR tracking state changes to enable/disable the Track button:

sdkDelegate.onTrackingChanged = { state in
    isTrackingNormal = (state == .tracking)
}

onObjectMeshLoaded

Called when the outline mesh for a tracked object has been downloaded and placed:

sdkDelegate.onObjectMeshLoaded = { objectCode in
    // Mesh is now visible in scene
}

---

Actions

startTracking()

Manually triggers object tracking:

private func startTracking() {
    MultiSet.shared.startObjectTracking()
}

resetTracking()

Clears all tracked object meshes and resets state. Restarts auto-tracking if enabled:

private func resetTracking() {
    autoTrackingTask?.cancel()
    autoTrackingTask = nil
    MultiSet.shared.stopObjectTracking()
    MultiSet.shared.clearObjectMeshes()
    hasTrackedObject = false
    trackedObjectCode = ""
    showToastMessage("Tracking reset", success: true)
    startAutoTrackingIfNeeded()
}

cleanup()

Cleans up resources when the view disappears:

private func cleanup() {
    autoTrackingTask?.cancel()
    autoTrackingTask = nil
    MultiSet.shared.stopObjectTracking()
}

---

Configuration

The view reads object tracking settings from MultiSet.shared.config:

config.objectCodes = ["OBJ_CODE_1", "OBJ_CODE_2"]  // Objects to track (max 10)
config.autoObjectTracking = true                     // Auto-start tracking
config.backgroundObjectTracking = true               // Re-track at intervals
config.bgObjectTrackingDurationSeconds = 15.0        // Interval between re-tracks (5-30s)
config.restartObjectTracking = true                  // Re-track on AR loss
config.objectTrackingCaptureDelay = 1.0              // Delay before capture (0.5-3.0s)
config.firstObjectTrackingUntilSuccess = true        // Retry silently until first success
config.showAlerts = true                             // Show toast/alert feedback

Overlay Components

APILoadingOverlay

Shows while a tracking request is in progress:

• Semi-transparent black background

• Circular progress spinner

• "Tracking object..." status text

---

Related

• LandingView

• ARLocalizationView

• MultiSetConfig