FrameSource and Sessions

Overview

From version 1.16.0 the MultiSet Android SDK contains no ARCore or Sceneform code. It never talks to an AR runtime directly. Instead, your app converts each camera frame into a plain CameraFrame and supplies it through the FrameSource interface. The SDK owns the loop: capture scheduling, retry, background re-localization, confidence gating, image preparation, networking, and pose maths.

This is the whole integration surface. Implement one interface, then start a session.

---

FrameSource

interface FrameSource {
    suspend fun acquire(): CameraFrame?
}

acquire() is called by the SDK whenever it needs a frame. Return null when no frame is available right now, for example when the tracker has not converged or the runtime has no new CPU image this tick. Returning null is not an error. The SDK polls briefly (up to 20 attempts, 100 ms apart) before giving up on a capture, which absorbs transient unavailability.

CameraFrame

data class CameraFrame(
    val bitmap: Bitmap?,
    val cameraPosition: Vec3,
    val cameraRotation: Quat,
    val intrinsics: CameraIntrinsics,
    val orientation: DeviceOrientation
)

Property	Description
bitmap	Raw RGB camera image. May be null for runtimes that supply pre-encoded frames.
cameraPosition	World position of the camera at capture time, as Vec3.
cameraRotation	World rotation of the camera at capture time, as Quat.
intrinsics	CameraIntrinsics(fx, fy, cx, cy, imageWidth, imageHeight) for the source resolution of bitmap.
orientation	DeviceOrientation.PORTRAIT or LANDSCAPE.

Supply the raw, uncorrected pose and intrinsics. The SDK applies portrait orientation correction exactly once, internally. If you rotate the bitmap or correct the quaternion yourself, the correction is applied twice and the resulting pose is wrong.

Example: an ARCore frame source

class ArFrameSource(
    private val arFragment: ArFragment,
    private val context: Context,
    private val imageProcessor: ImageProcessor,
) : FrameSource {

    override suspend fun acquire(): CameraFrame? = withContext(Dispatchers.Main) {
        val frame = arFragment.arSceneView.arFrame ?: return@withContext null
        val camera = frame.camera
        if (camera.trackingState != TrackingState.TRACKING) return@withContext null

        val pose = camera.pose
        val intr = camera.imageIntrinsics

        val image = try {
            frame.acquireCameraImage()
        } catch (e: Exception) {
            return@withContext null   // NotYetAvailableException is transient
        }

        val imgWidth = image.width
        val imgHeight = image.height

        val orientation =
            if (context.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE)
                DeviceOrientation.LANDSCAPE else DeviceOrientation.PORTRAIT

        val bitmap = try {
            withContext(Dispatchers.Default) { imageProcessor.yuvToBitmap(image) }
        } finally {
            image.close()
        }

        CameraFrame(
            bitmap = bitmap,
            cameraPosition = Vec3(pose.tx(), pose.ty(), pose.tz()),
            cameraRotation = Quat(pose.qx(), pose.qy(), pose.qz(), pose.qw()),
            intrinsics = CameraIntrinsics(
                fx = intr.focalLength[0],
                fy = intr.focalLength[1],
                cx = intr.principalPoint[0],
                cy = intr.principalPoint[1],
                imageWidth = imgWidth,
                imageHeight = imgHeight,
            ),
            orientation = orientation,
        )
    }
}

Read the AR frame on the main thread, then move the YUV to bitmap conversion off it, and close the image exactly once afterwards.

---

LocalizationSession

Create a session from the SDK, configure it, attach callbacks, then start it.

val session = MultiSetSDK.localizationSession(frameSource, LocalizationMode.MULTI_FRAME)

Properties

Property	Type	Default	Description
backgroundLocalization	Boolean	true	Keep re-localizing periodically after the first success.
bgLocalizationDurationMs	Long	15000L	Interval between background localizations.
numberOfFrames	Int	5	Frames captured per multi-frame query. Valid range 4 to 6.
frameCaptureIntervalMs	Long	500L	Delay between captures within a multi-frame query.
confidenceCheck	Boolean	true	Reject results below confidenceThreshold.
confidenceThreshold	Float	0.3f	Minimum accepted server confidence.
firstLocalizationUntilSuccess	Boolean	true	Silently retry transient failures until the first success.
imageQuality	Int	80	JPEG quality for uploaded frames.
hintMapCodes	List<String>	emptyList()	Restrict a MapSet query to named maps.
queryMode	QueryMode	VPS1	Search strategy. Single-frame only.
poseConsistencyCheck	Boolean	false	Discard results that contradict the device trajectory.
poseConsistencyThreshold	Float	10f	Tolerance in metres. Valid range 3 to 30.

Callbacks

Callback	Signature	Fires when
onLocalizationRequested	() -> Unit	A capture has started. Use it to show scanning UI.
onLocalizationSuccess	(LocalizationResult) -> Unit	A pose was accepted.
onLocalizationFailure	(MultiSetError) -> Unit	The query failed.
onLocalizationFalsePositive	(FalsePositiveInfo) -> Unit	A pose was discarded as inconsistent.

Methods

Method	Description
start()	Begin capturing.
stop()	Cancel the scheduled capture and any request already in flight.
notifyTrackingInterrupted()	Tell the pose gate that AR tracking broke, so it stops trusting its reference blindly.
resetPoseReference()	Drop the pose reference. The next accepted result becomes the new one.

stop() cancels in-flight work as well as scheduled work. Without that, a response arriving after you tore your scene down would re-apply the old pose. Call stop() in onDestroy and whenever you reset the scene.

---

ObjectTrackingSession

val session = MultiSetSDK.objectTrackingSession(frameSource, listOf("OBJ_001", "OBJ_002"))

Property	Type	Default	Description
captureDelayMs	Long	1000L	Delay before the first capture.
backgroundTracking	Boolean	true	Keep re-tracking after the first success.
bgTrackingDurationMs	Long	15000L	Interval between background tracking attempts.
confidenceCheck	Boolean	true	Reject results below confidenceThreshold.
confidenceThreshold	Float	0.3f	Minimum accepted server confidence.
firstTrackingUntilSuccess	Boolean	true	Silently retry transient failures until the first success.
imageQuality	Int	80	JPEG quality for uploaded frames.

Callbacks are onTrackingRequested, onTrackingSuccess(ObjectTrackingResult) and onTrackingFailure(MultiSetError). start() and stop() behave as they do for localization.

---

MultiSetError

Session failures are classified rather than passed as free text.

data class MultiSetError(
    val kind: Kind,
    val message: String,
    val statusCode: Int? = null,
    val rawBody: String? = null,
)

Kind	Meaning	Suggested handling
SERVER	The API rejected the request. Retrying unchanged will not help.	Surface to the developer. Check map codes and parameters.
AUTH	Credentials rejected (401 or 403).	Check client ID and secret.
NETWORK	The API was unreachable.	Ask the user to check connectivity.
TRANSIENT	Scan-time failure such as no pose found or low confidence.	Expected. Prompt the user to move and scan again.

error.isTransient is shorthand for kind == TRANSIENT.

---

FalsePositiveInfo

Delivered when poseConsistencyCheck is enabled and a response contradicts the device's own trajectory. The request succeeded; the answer was wrong, most often a visually similar location elsewhere in the map.

Property	Type	Description
jumpMeters	Float	How far the discarded pose sat from the last accepted one.
thresholdMeters	Float	The tolerance in force.
consecutiveCount	Int	Consecutive false positives since the last accepted fix. Starts at 1.
mapCodes	List<String>	Map codes the server reported for the discarded response.
confidence	Float?	Server confidence of the discarded response, when reported.
reason	String	Short machine-readable explanation, for logs.
summary	String	One-line summary suitable for logging.

The scene is left untouched when this fires. The gizmo and mesh keep the last accepted pose. If consecutiveCount keeps climbing, the reference itself may be wrong; call MultiSetSDK.resetPoseConsistencyReference() to re-bootstrap it.

---

Raw Stateless APIs

For full control over capture timing, skip sessions and call the suspend APIs directly:

suspend fun trackObjects(frame: CameraFrame): ObjectTrackingResult
suspend fun localizeSingleFrame(frame: CameraFrame, queryMode: QueryMode = QueryMode.VPS1): LocalizationResult
suspend fun localizeMultiFrame(frames: List<CameraFrame>): LocalizationResult

These perform one request each and return the computed world pose. Scheduling, retry and gating are yours to implement.

---

Related

• Android Native Overview

• LocalizationConfig

• ObjectTrackingConfig