> For the complete documentation index, see [llms.txt](https://docs.multiset.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.multiset.ai/native-sdk/android-native/sample-activities/objecttrackingactivity.md).

# ObjectTrackingActivity

The Android AR activity that detects registered objects, renders each one's outline mesh on its own anchor and re-tracks once AR tracking recovers.

## Overview

The `ObjectTrackingActivity` is a dedicated AR activity for detecting and tracking pre-registered physical objects using the MultiSet SDK. Once an object is successfully tracked, its 3D mesh is fetched from the MultiSet platform and rendered on that object's own anchor with an animated glowing outline.

## Description

This activity is responsible for:

1. Establishing an AR session with ARCore and Sceneform
2. Sending camera frames to the MultiSet Object Tracking API through an `ObjectTrackingSession`
3. Fetching each tracked object's 3D mesh and placing it on that object's own anchor
4. Drawing an animated outline around each tracked object
5. Reporting ARCore tracking to the SDK, and hiding and re-tracking objects after a tracking break or a return from the background
6. Showing run progress: the capture and server-wait overlays and the round status card

***

## Launching the Activity

Object codes can be passed either directly via an intent extra, or pre-configured on `ObjectTrackingConfig`.

```kotlin
// Option A: pass codes via Intent extra
val objectCodes = arrayOf("OBJ_001", "OBJ_002")
val intent = Intent(this, ObjectTrackingActivity::class.java)
intent.putExtra(ObjectTrackingActivity.EXTRA_OBJECT_CODES, objectCodes)
startActivity(intent)

// Option B: configure via ObjectTrackingConfig (codes are read automatically)
ObjectTrackingConfig.objectCodes = arrayOf("OBJ_001", "OBJ_002")
ObjectTrackingConfig.validate()
startActivity(Intent(this, ObjectTrackingActivity::class.java))
```

### Intent Extras

| Extra                | Type            | Description                             |
| -------------------- | --------------- | --------------------------------------- |
| `EXTRA_OBJECT_CODES` | `Array<String>` | Array of object codes to track (max 10) |

***

## Prerequisites

The SDK must be initialized and authenticated before launching this activity. If the SDK is not authenticated, the activity will finish immediately with a toast message.

```kotlin
// Ensure SDK is initialized before launching
MultiSetSDK.initialize(context, config, callback)

// Launch only after onAuthenticationSuccess()
override fun onAuthenticationSuccess() {
    val intent = Intent(this, ObjectTrackingActivity::class.java)
    intent.putExtra(ObjectTrackingActivity.EXTRA_OBJECT_CODES, arrayOf("OBJ_001"))
    startActivity(intent)
}
```

***

## Key Features

### Auto-Tracking

When `ObjectTrackingConfig.autoTracking` is `true`, tracking starts automatically once ARCore reports `TRACKING`. Each capture waits `captureDelayMs` first, so the camera can settle; the capture overlay shows that hold.

### Background Tracking

When `ObjectTrackingConfig.backgroundTracking` is `true`, the session re-tracks at the configured interval (`bgTrackingDurationSeconds`) after each run. Background runs show only on the status card, and the capture button stays in place, disabled, while one runs.

### Manual Tracking Trigger

A capture button is visible in the AR view. Tapping it triggers a new tracking request regardless of auto or background settings.

### One Anchor per Object

Each tracked object gets its own anchor node, keyed by its object code. A result moves only that object's outline, so tracking a second object never moves the first one's outline onto it.

### Animated Outline Mesh

Once an object is tracked, its 3D GLB mesh is fetched from the MultiSet platform (with local caching) and placed on its anchor. A custom Filament material, ported from the iOS SDK, draws a glowing animated rim around the object's edges. A depth-only occluder hides the inside of the shell, so the camera feed is drawn first (`cameraStream.renderPriority = Renderable.RENDER_PRIORITY_FIRST`). A toggle at the bottom left hides or shows the outlines, and a loading strip takes its place while a mesh downloads.

### Status and Toasts

The top left shows the tracked object's code and the round status card, level with Reset and Close. A status chip shows above them only until an object is tracked. With `showAlerts` on, a successful track shows an "Object Tracked" toast.

### Confidence Filtering

When `ObjectTrackingConfig.confidenceCheck` is `true`, tracking results below `confidenceThreshold` are discarded and the activity retries automatically.

### Re-tracking after a Tracking Break

When `ObjectTrackingConfig.restartTracking` is `true`, the session re-tracks once AR tracking has held for 1 s after a loss, without an alert. An interruption hides every outline until tracking is back. A return from the background is treated as a world-origin reset: each outline stays hidden until **that** object is tracked again, since it may have moved.

***

## Configuration

The activity reads all settings from `ObjectTrackingConfig` at session startup:

```kotlin
ObjectTrackingConfig.objectCodes = arrayOf("OBJ_001", "OBJ_002")
ObjectTrackingConfig.autoTracking = true
ObjectTrackingConfig.backgroundTracking = true
ObjectTrackingConfig.bgTrackingDurationSeconds = 15f
ObjectTrackingConfig.restartTracking = true
ObjectTrackingConfig.firstTrackingUntilSuccess = true
ObjectTrackingConfig.captureDelayMs = 1000L
ObjectTrackingConfig.confidenceCheck = true
ObjectTrackingConfig.confidenceThreshold = 0.3f
ObjectTrackingConfig.showAlerts = true
ObjectTrackingConfig.imageQuality = 80
ObjectTrackingConfig.validate()
```

See [ObjectTrackingConfig](/native-sdk/android-native/api-reference/objecttrackingconfig.md) for full property details.

***

## Session Wiring

The activity builds an `ArFrameSource`, creates an `ObjectTrackingSession` from the SDK, applies `ObjectTrackingConfig` to it, and attaches callbacks. The SDK owns capture scheduling, retry, background re-tracking and the restart after a tracking break.

```kotlin
val frameSource = ArFrameSource(arFragment, this, ImageProcessor())

val session = MultiSetSDK.objectTrackingSession(frameSource, objectCodes.toList()).apply {
    captureDelayMs = ObjectTrackingConfig.captureDelayMs
    backgroundTracking = ObjectTrackingConfig.backgroundTracking
    bgTrackingDurationMs = (ObjectTrackingConfig.bgTrackingDurationSeconds * 1000).toLong()
    restartTracking = ObjectTrackingConfig.restartTracking
    confidenceCheck = ObjectTrackingConfig.confidenceCheck
    confidenceThreshold = ObjectTrackingConfig.confidenceThreshold
    firstTrackingUntilSuccess = ObjectTrackingConfig.firstTrackingUntilSuccess
    imageQuality = ObjectTrackingConfig.imageQuality

    onStageChanged = { stage, isBackground -> /* overlays and the status card */ }
    onPlacementVisibilityChanged = { objectCode, visible -> /* show or hide that object's outline */ }
    onTrackingSuccess = { result ->
        val anchor = objectAnchors.getOrPut(result.objectCode) {
            Node().also { arFragment.arSceneView.scene.addChild(it) }
        }
        anchor.localPosition = Vector3(result.position[0], result.position[1], result.position[2])
        anchor.localRotation = Quaternion(result.rotation[0], result.rotation[1], result.rotation[2], result.rotation[3])
        meshRenderer.fetchAndRender(result.objectCode, anchor)
    }
    onTrackingFailure = { error -> /* error.kind classifies the failure */ }
}
```

As with localization, capture is deferred until ARCore reports `TRACKING`. The activity reports ARCore's tracking state to the session on the first frame and on every change (`reportArTracking` in `ar/ArTrackingSignals.kt`), and in `onPause`, before `super.onPause()`, it marks the frames taken so far as stale, reports `PAUSED` and calls `notifyWorldOriginReset()`.

**Reset** calls `session.resetSession()`, which forgets every placement and re-arms the silent first track, then removes the outlines and their anchors.

***

## Callbacks (MultiSetSDKCallback Interface)

Sessions built by `MultiSetSDK` also report to the callback registered via `MultiSetSDK.initialize()`.

### onObjectTrackingSuccess(result: ObjectTrackingResult)

Called when an object is successfully tracked.

| Parameter | Type                   | Description                                |
| --------- | ---------------------- | ------------------------------------------ |
| `result`  | `ObjectTrackingResult` | Contains object code, pose, and confidence |

```kotlin
override fun onObjectTrackingSuccess(result: ObjectTrackingResult) {
    Log.d(TAG, "Tracked: ${result.objectCode}")
    Log.d(TAG, "Position: ${result.position.joinToString()}")
    Log.d(TAG, "Rotation: ${result.rotation.joinToString()}")
    Log.d(TAG, "Confidence: ${result.confidence}")
}
```

### onObjectTrackingFailure(error: String)

Called when a tracking run fails and is reported. With `firstTrackingUntilSuccess` on, a failed first track is reported only after its silent retries, and a restart's miss is never reported.

| Parameter | Type     | Description                          |
| --------- | -------- | ------------------------------------ |
| `error`   | `String` | Error message describing the failure |

```kotlin
override fun onObjectTrackingFailure(error: String) {
    Log.e(TAG, "Tracking failed: $error")
}
```

### onObjectTrackingStageChanged(stage, isBackground)

Called when a tracking run moves to another stage. See [Run stages](/native-sdk/android-native/api-reference/framesource.md#run-stages).

### onTrackingStateChanged(state: TrackingState)

Called when a session is told the AR tracking state through `reportTrackingState`.

| Parameter | Type            | Description                                       |
| --------- | --------------- | ------------------------------------------------- |
| `state`   | `TrackingState` | Current state: `TRACKING`, `PAUSED`, or `STOPPED` |

***

## ObjectTrackingResult

| Property      | Type           | Description                          |
| ------------- | -------------- | ------------------------------------ |
| `objectCode`  | `String`       | Code of the tracked object           |
| `objectCodes` | `List<String>` | All object codes returned by the API |
| `position`    | `FloatArray`   | XYZ position in ARCore world space   |
| `rotation`    | `FloatArray`   | XYZW quaternion orientation          |
| `confidence`  | `Float`        | Tracking confidence score (0.0–1.0)  |

***

## AR Session Setup

The activity configures the ARCore session with settings optimized for object tracking:

| Setting          | Value                 |
| ---------------- | --------------------- |
| Update Mode      | `LATEST_CAMERA_IMAGE` |
| Focus Mode       | `AUTO`                |
| Light Estimation | `DISABLED`            |
| Depth Mode       | `DISABLED`            |
| Plane Finding    | `DISABLED`            |

Depth is off because nothing on this screen uses it, and requesting it made ARCore reject the session on devices without the Depth API. Plane rendering is disabled to keep the view clean during object tracking.

***

## Lifecycle

| Event       | Behavior                                                                                                                 |
| ----------- | ------------------------------------------------------------------------------------------------------------------------ |
| `onCreate`  | Validates SDK auth, sets up AR and the session                                                                           |
| `onPause`   | Marks frames stale, reports `PAUSED` and a world-origin reset, so outlines stay hidden after the return until re-tracked |
| `onDestroy` | Stops the session, releases the mesh renderer and clears the anchors                                                     |

The activity does not persist the SDK instance. This is managed by the host `MainActivity`.

***

## Related

* [FrameSource and Sessions](/native-sdk/android-native/api-reference/framesource.md)
* [MainActivity](/native-sdk/android-native/sample-activities/mainactivity.md)
* [ObjectTrackingConfig](/native-sdk/android-native/api-reference/objecttrackingconfig.md)
* [MultiSetLocalizationActivity](/native-sdk/android-native/sample-activities/multisetlocalizationactivity.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.multiset.ai/native-sdk/android-native/sample-activities/objecttrackingactivity.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.