MultiFrameARActivity

Overview

The MultiFrameARActivity provides AR-based localization using multiple camera frames. This activity captures a sequence of frames from the AR camera, along with their camera poses, and sends them to the MultiSet backend for more accurate pose estimation.

Description

Multi-frame localization is ideal for scenarios where:

• Higher accuracy is required

• The environment has repetitive or sparse visual features

• The user can move the device slightly during capture

• More robust localization is needed

The activity handles:

• AR session management using ARCore and Sceneform

• Sequential multi-frame capture with configurable intervals

• Camera pose tracking for each captured frame

• Localization API requests with multiple images

• Pose calculation and gizmo positioning

• Mesh visualization

• Background localization

• GPS hint integration

---

How It Works

1. Initialization Flow

onCreate()
  └─> setupAR()
        └─> configureSession()
              └─> addGizmoToScene()
                    └─> initializeMeshComponents()
                          └─> initializeLocalization()

2. Localization Flow

localizeFrame()
  └─> startLocalization()
        └─> captureFrames() [loop]
              └─> captureAndProcessFrame()
        └─> sendLocalizationRequest()
              └─> handleLocalizationResponse()
                    └─> poseHandler()
                          └─> updateGizmoPosition()
                                └─> loadMeshAfterLocalization()

---

Key Features

Multi-Frame Capture

Captures multiple frames (configurable via LocalizationConfig.numberOfFrames) with intervals between captures (configurable via LocalizationConfig.frameCaptureIntervalMs). Each frame includes:

• Image data (JPEG compressed)

• Camera position (X, Y, Z)

• Camera rotation (quaternion)

Visual Feedback

An animated phone icon guides users to move their device during frame capture, ensuring diverse viewpoints for better localization.

Auto-Localization

When LocalizationConfig.autoLocalize is enabled, localization starts automatically after the AR session is ready.

Background Localization

When LocalizationConfig.backgroundLocalization is enabled, the activity periodically sends localization requests to refine positioning.

Relocalization

When LocalizationConfig.relocalization is enabled, automatic relocalization is triggered when AR tracking state becomes PAUSED or STOPPED.

---

Public Methods

localizeFrame()

Manually triggers the multi-frame localization process.

Declaration:

fun localizeFrame()

Description: Initiates the multi-frame capture sequence. Captures the configured number of frames at the configured interval, then sends all frames to the MultiSet API for localization.

Usage:

binding.localizeButton.setOnClickListener {
    localizeFrame()
}

---

resetWorldOrigin()

Resets the AR world origin and clears the current localization.

Declaration:

fun resetWorldOrigin()

Description:

• Pauses and reconfigures the AR session

• Cancels any ongoing localization or background jobs

• Clears captured images

• Removes the mesh visualization

• Resets the gizmo position

• Optionally restarts auto-localization

---

Multi-Frame Capture

Captures a sequence of frames:

1. Loops for numberOfFrames iterations

2. Waits for frameCaptureIntervalMs between captures

3. Captures and processes each frame

4. Updates UI with capture progress

5. Proceeds to API request when all frames are captured

---

Data Structures

ImageData

Property	Type	Description
imageBytes	ByteArray	JPEG compressed image data
metadata	ImageMetadata	Camera pose information

ImageMetadata

Property	Type	Description
x	Float	Camera X position
y	Float	Camera Y position
z	Float	Camera Z position
qx	Float	Quaternion X component
qy	Float	Quaternion Y component
qz	Float	Quaternion Z component
qw	Float	Quaternion W component

UploadData

Property	Type	Description
width	Int	Image width in pixels
height	Int	Image height in pixels
px	Float	Principal point X
py	Float	Principal point Y
fx	Float	Focal length X
fy	Float	Focal length Y

---

API Request

Sends all captured frames to the MultiSet API:

• Includes camera intrinsics (from first frame)

• Includes all captured images with their poses

• Includes map code or map set code

• Optionally includes GPS hint

• Optionally requests geo-coordinates in response

---

Response Handling

Processes the multi-frame localization response:

1. Checks if pose was found

2. Validates confidence threshold (if enabled)

3. Calculates final pose using tracking pose data

4. Updates gizmo position

5. Loads mesh visualization

6. Notifies callback

Pose Calculation

Calculates the final world-space pose:

1. Extracts estimated pose from response

2. Extracts tracking pose from response

3. Computes inverse of estimated pose matrix

4. Multiplies tracking pose by inverse estimated pose

5. Returns final position and rotation

---

State Management

Property	Type	Description
isLocalizing	Boolean	Whether localization is in progress
isFirstLocalization	Boolean	Whether this is the first localization attempt
isBackgroundLocalizationRequest	Boolean	Whether current request is from background
lastTrackingState	TrackingState	Previous AR tracking state
isSessionConfigured	Boolean	Whether AR session has been configured
capturedImages	MutableList<ImageData>	List of captured frames
uploadData	UploadData?	Camera intrinsics for upload

---

Configuration

The activity reads configuration from LocalizationConfig:

LocalizationConfig.autoLocalize = true
LocalizationConfig.backgroundLocalization = true
LocalizationConfig.backgroundLocalizationIntervalSeconds = 30f
LocalizationConfig.relocalization = true
LocalizationConfig.numberOfFrames = 4
LocalizationConfig.frameCaptureIntervalMs = 500L
LocalizationConfig.confidenceCheck = false
LocalizationConfig.confidenceThreshold = 0.3f
LocalizationConfig.firstLocalizationUntilSuccess = true
LocalizationConfig.showAlerts = true
LocalizationConfig.enableMeshVisualization = true
LocalizationConfig.enableGeoHint = false
LocalizationConfig.includeGeoCoordinatesInResponse = false
LocalizationConfig.imageQuality = 90

---

Usage Example

Launching the Activity

// From MainActivity after authentication
val intent = Intent(this, MultiFrameARActivity::class.java)
startActivity(intent)

Configuring for Multi-Frame Localization

// Customize multi-frame settings
LocalizationConfig.numberOfFrames = 5
LocalizationConfig.frameCaptureIntervalMs = 600L
LocalizationConfig.autoLocalize = true
LocalizationConfig.backgroundLocalization = true
LocalizationConfig.enableMeshVisualization = true
LocalizationConfig.validate()

// Launch activity
val intent = Intent(this, MultiFrameARActivity::class.java)
startActivity(intent)

---

Lifecycle

onCreate()

• Validates SDK authentication

• Inflates layout

• Loads and validates configuration

• Sets up AR and UI

• Initializes phone animation

onDestroy()

• Stops GPS status updates

• Cancels localization and background jobs

• Cancels phone animation

• Stops GPS handler

• Removes mesh visualization

---

Related

• MainActivity

• SingleFrameARActivity

• LocalizationConfig