ObjectTrackingActivity

Overview

The ObjectTrackingActivity is a dedicated AR activity for detecting and tracking pre-registered physical objects using the MultiSet SDK. Once an object is successfully tracked, its 3D mesh is fetched from the MultiSet platform and rendered in the AR scene with an animated glowing outline shader.

Description

This activity is responsible for:

1. Establishing an AR session with ARCore and Sceneform

2. Fetching and rendering 3D object meshes from the MultiSet platform

3. Sending camera frames to the MultiSet Object Tracking API

4. Applying an animated outline-only shader to tracked object meshes

5. Managing background tracking and auto-retry behavior

6. Forwarding tracking results to the host application via MultiSetSDKCallback

---

Launching the Activity

Object codes can be passed either directly via an intent extra, or pre-configured on ObjectTrackingConfig.

// Option A: pass codes via Intent extra
val objectCodes = arrayOf("OBJ_001", "OBJ_002")
val intent = Intent(this, ObjectTrackingActivity::class.java)
intent.putExtra(ObjectTrackingActivity.EXTRA_OBJECT_CODES, objectCodes)
startActivity(intent)

// Option B: configure via ObjectTrackingConfig (codes are read automatically)
ObjectTrackingConfig.objectCodes = arrayOf("OBJ_001", "OBJ_002")
ObjectTrackingConfig.validate()
startActivity(Intent(this, ObjectTrackingActivity::class.java))

Intent Extras

Extra	Type	Description
EXTRA_OBJECT_CODES	Array<String>	Array of object codes to track (max 10)

---

Prerequisites

The SDK must be initialized and authenticated before launching this activity. If the SDK is not authenticated, the activity will finish immediately with a toast message.

// Ensure SDK is initialized before launching
MultiSetSDK.initialize(context, config, callback)

// Launch only after onAuthenticationSuccess()
override fun onAuthenticationSuccess() {
    val intent = Intent(this, ObjectTrackingActivity::class.java)
    intent.putExtra(ObjectTrackingActivity.EXTRA_OBJECT_CODES, arrayOf("OBJ_001"))
    startActivity(intent)
}

---

Key Features

Auto-Tracking

When ObjectTrackingConfig.autoTracking is true, tracking starts automatically once the AR session is ready and the device camera is in a stable tracking state. A 1-second stabilization delay ensures a good initial frame.

Background Tracking

When ObjectTrackingConfig.backgroundTracking is true, the activity continues sending tracking requests at the configured interval (bgTrackingDurationSeconds) after the first successful track. This keeps the object pose updated over time.

Manual Tracking Trigger

A capture button is visible in the AR view. Tapping it manually triggers a new tracking request regardless of auto or background settings.

Animated Outline Mesh

Once an object is tracked, its 3D GLB mesh is fetched from the MultiSet platform (with local caching) and placed in the AR scene. A custom Filament shader renders only the mesh outline with:

• Rim-lit edge glow

• Animated spark particles (dual-layer noise)

• Traveling wave along the Y axis

• Pulse modulation

• Configurable glow intensity and color

Confidence Filtering

When ObjectTrackingConfig.confidenceCheck is true, tracking results below confidenceThreshold are discarded and the activity retries automatically.

Re-tracking on AR Loss

When ObjectTrackingConfig.restartTracking is true, the activity detects AR tracking state changes (PAUSED/STOPPED) and automatically re-initiates tracking when the device recovers.

---

Configuration

The activity reads all settings from ObjectTrackingConfig at session startup:

ObjectTrackingConfig.objectCodes = arrayOf("OBJ_001", "OBJ_002")
ObjectTrackingConfig.autoTracking = true
ObjectTrackingConfig.backgroundTracking = true
ObjectTrackingConfig.bgTrackingDurationSeconds = 15f
ObjectTrackingConfig.restartTracking = true
ObjectTrackingConfig.firstTrackingUntilSuccess = true
ObjectTrackingConfig.captureDelayMs = 1000L
ObjectTrackingConfig.confidenceCheck = true
ObjectTrackingConfig.confidenceThreshold = 0.3f
ObjectTrackingConfig.showAlerts = true
ObjectTrackingConfig.imageQuality = 80
ObjectTrackingConfig.validate()

See ObjectTrackingConfig for full property details.

---

Session Wiring

The activity builds an ArFrameSource, creates an ObjectTrackingSession from the SDK, applies ObjectTrackingConfig to it, and attaches callbacks. The SDK owns capture scheduling, retry and background re-tracking.

val frameSource = ArFrameSource(arFragment, this, ImageProcessor())

val session = MultiSetSDK.objectTrackingSession(frameSource, objectCodes.toList()).apply {
    captureDelayMs = ObjectTrackingConfig.captureDelayMs
    backgroundTracking = ObjectTrackingConfig.backgroundTracking
    confidenceCheck = ObjectTrackingConfig.confidenceCheck
    confidenceThreshold = ObjectTrackingConfig.confidenceThreshold
    firstTrackingUntilSuccess = ObjectTrackingConfig.firstTrackingUntilSuccess
    imageQuality = ObjectTrackingConfig.imageQuality

    onTrackingRequested = { /* show scanning overlay */ }
    onTrackingSuccess = { result -> /* anchor and render the object mesh */ }
    onTrackingFailure = { error -> /* error.kind classifies the failure */ }
}

As with localization, capture is deferred until ARCore reports TRACKING, and session.stop() cancels both the scheduled capture and any request already in flight.

---

Callbacks (MultiSetSDKCallback Interface)

Results are forwarded to the callback registered via MultiSetSDK.initialize().

onObjectTrackingSuccess(result: ObjectTrackingResult)

Called when an object is successfully tracked.

Parameter	Type	Description
result	ObjectTrackingResult	Contains object code, pose, and confidence

override fun onObjectTrackingSuccess(result: ObjectTrackingResult) {
    Log.d(TAG, "Tracked: ${result.objectCode}")
    Log.d(TAG, "Position: ${result.position.joinToString()}")
    Log.d(TAG, "Rotation: ${result.rotation.joinToString()}")
    Log.d(TAG, "Confidence: ${result.confidence}")
}

onObjectTrackingFailure(error: String)

Called when tracking fails (after all retries are exhausted, if firstTrackingUntilSuccess is false).

Parameter	Type	Description
error	String	Error message describing the failure

override fun onObjectTrackingFailure(error: String) {
    Log.e(TAG, "Tracking failed: $error")
}

onTrackingStateChanged(state: TrackingState)

Called when the underlying AR tracking state changes.

Parameter	Type	Description
state	TrackingState	Current state: TRACKING, PAUSED, or STOPPED

---

ObjectTrackingResult

Property	Type	Description
objectCode	String	Code of the tracked object
objectCodes	List<String>	All object codes returned by the API
position	FloatArray	XYZ position in ARCore world space
rotation	FloatArray	XYZW quaternion orientation
confidence	Float	Tracking confidence score (0.0–1.0)

---

AR Session Setup

The activity configures the ARCore session with settings optimized for object tracking:

Setting	Value
Update Mode	LATEST_CAMERA_IMAGE
Focus Mode	AUTO
Light Estimation	DISABLED
Depth Mode	AUTOMATIC
Plane Finding	DISABLED

Plane rendering is disabled to keep the view clean during object tracking.

---

Lifecycle

Event	Behavior
onCreate	Validates SDK auth, sets up AR, fetches mesh, starts animation
onDestroy	Stops animation, releases tracking manager and mesh handler

The activity does not persist the SDK instance. This is managed by the host MainActivity.

---

Related

• FrameSource and Sessions

• MainActivity

• ObjectTrackingConfig

• MultiSetLocalizationActivity