> For the complete documentation index, see [llms.txt](https://docs.multiset.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.multiset.ai/native-sdk/android-native/api-reference/framesource.md). # FrameSource and Sessions Supply camera frames to the Android SDK through FrameSource and CameraFrame, run a LocalizationSession or ObjectTrackingSession, report AR tracking and follow each run as it happens. ## Overview The MultiSet Android SDK contains no ARCore, Sceneform or Jetpack XR code. It never talks to an AR runtime directly. Instead, your app converts each camera frame into a plain `CameraFrame` and supplies it through the `FrameSource` interface. The SDK owns the loop: capture scheduling, retry, background re-localization, recovery after a tracking break, confidence gating, image preparation, networking, and pose maths. This is the whole integration surface. Implement one interface, start a session, and report the AR tracking state to it. The same boundary runs the ARCore sample app on phones and the XREAL Aura sample on Android XR glasses. *** ## FrameSource ```kotlin interface FrameSource { suspend fun acquire(): CameraFrame? val isHostActive: Boolean get() = true } ``` `acquire()` is called by the SDK whenever it needs a frame. Return `null` when no frame is available right now, for example when the tracker has not converged or the runtime has no new CPU image this tick. Returning `null` is not an error. The SDK polls briefly (up to 20 attempts, 100 ms apart) before giving up on a capture, which absorbs transient unavailability. `isHostActive` should be `false` while your app is in the background. Scheduled runs wait for it rather than capture from a paused camera: a background refresh re-arms its interval, a silent retry waits without using up an attempt, and a recovery waits for the next tracking report. ### CameraFrame ```kotlin data class CameraFrame( val bitmap: Bitmap?, val cameraPosition: Vec3, val cameraRotation: Quat, val intrinsics: CameraIntrinsics, val orientation: DeviceOrientation, val timestampSeconds: Double = System.nanoTime() / 1_000_000_000.0, ) ``` | Property | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `bitmap` | Raw RGB camera image. May be `null` for runtimes that supply pre-encoded frames. | | `cameraPosition` | World position of the camera at capture time, as `Vec3`. | | `cameraRotation` | World rotation of the camera at capture time, as `Quat`. | | `intrinsics` | `CameraIntrinsics(fx, fy, cx, cy, imageWidth, imageHeight)` for the source resolution of `bitmap`. | | `orientation` | `DeviceOrientation.PORTRAIT` or `LANDSCAPE`. | | `timestampSeconds` | Capture time on a monotonic clock (ARCore: `frame.timestamp / 1e9`). The pose consistency check uses it as its only clock. | {% hint style="warning" %} Supply the **raw, uncorrected** pose and intrinsics. The SDK applies portrait orientation correction exactly once, internally. If you rotate the bitmap or correct the quaternion yourself, the correction is applied twice and the resulting pose is wrong. {% endhint %} The SDK scales every frame uniformly so that its longer side is `imageLongSide` pixels (default 960, at most 1280), keeping its aspect ratio, and scales the intrinsics to match. Nothing is cropped or stretched. ARCore's 640 × 480 image goes out at 960 × 720 (720 × 960 in portrait). ### Example: an ARCore frame source ```kotlin class ArFrameSource( private val arFragment: ArFragment, private val context: Context, private val imageProcessor: ImageProcessor, ) : FrameSource { override val isHostActive: Boolean get() = arFragment.lifecycle.currentState.isAtLeast(Lifecycle.State.RESUMED) override suspend fun acquire(): CameraFrame? = withContext(Dispatchers.Main) { val frame = arFragment.arSceneView.arFrame ?: return@withContext null val camera = frame.camera if (camera.trackingState != TrackingState.TRACKING) return@withContext null val pose = camera.pose val intr = camera.imageIntrinsics val image = try { frame.acquireCameraImage() } catch (e: Exception) { return@withContext null // NotYetAvailableException is transient } val imgWidth = image.width val imgHeight = image.height val orientation = if (context.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE) DeviceOrientation.LANDSCAPE else DeviceOrientation.PORTRAIT val bitmap = try { withContext(Dispatchers.Default) { imageProcessor.yuvToBitmap(image) } } finally { image.close() } CameraFrame( bitmap = bitmap, cameraPosition = Vec3(pose.tx(), pose.ty(), pose.tz()), cameraRotation = Quat(pose.qx(), pose.qy(), pose.qz(), pose.qw()), intrinsics = CameraIntrinsics( fx = intr.focalLength[0], fy = intr.focalLength[1], cx = intr.principalPoint[0], cy = intr.principalPoint[1], imageWidth = imgWidth, imageHeight = imgHeight, ), orientation = orientation, timestampSeconds = frame.timestamp / 1_000_000_000.0, ) } } ``` Read the AR frame on the main thread, then move the YUV to bitmap conversion off it, and close the image exactly once afterwards. *** ## LocalizationSession Create a session from the SDK, attach callbacks, then start it. ```kotlin val session = MultiSetSDK.localizationSession(frameSource) // mode defaults to the config's ``` The session starts with every setting of the `MultiSetSDKConfig` passed to `MultiSetSDK.initialize`. Set a property to override it for this session only. ### Properties | Property | Type | Default | Description | | --------------------------------- | -------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `backgroundLocalization` | `Boolean` | `true` | Re-localize at intervals after the first fix. A change on a running session takes effect at once. | | `bgLocalizationDurationMs` | `Long` | `30000L` | Interval between background runs, from `backgroundLocalizationIntervalSeconds` (10 to 180 s). | | `relocalization` | `Boolean` | `true` | Re-localize after a tracking break that follows a fix. See [Tracking breaks](#tracking-breaks-and-the-return-from-the-background). | | `firstLocalizationUntilSuccess` | `Boolean` | `true` | Retry a failed first fix silently, up to 5 times, before reporting it. See [First localization until success](#first-localization-until-success). | | `numberOfFrames` | `Int` | `4` | Frames captured per multi-frame query. Valid range 4 to 6. | | `frameCaptureIntervalMs` | `Long` | `200L` | Delay between captures within a multi-frame query. | | `confidenceCheck` | `Boolean` | `false` | Reject results below `confidenceThreshold`. | | `confidenceThreshold` | `Float` | `0.3f` | Minimum accepted server confidence. | | `imageQuality` | `Int` | `90` | JPEG quality of the query image. | | `imageLongSide` | `Int` | `960` | Longer side of the query image in pixels, at most 1280. | | `queryMode` | `QueryMode` | `VPS1` | `VPS1` or `VPS2`, sent with single-frame and multi-frame queries. | | `hintMapCodes` | `List` | empty | Restrict a MapSet query to named maps. Sent only for a MapSet. | | `hintPosition` | `String` | `""` | Approximate position, `"x,y,z"`. | | `hintFloorHeight` | `String` | `""` | Floor and ceiling height band, `"floor,ceiling"`, for example `"0,5"`. | | `hintRadius` | `Int` | `25` | Search radius in metres, 1 to 100. Sent only with a geo or position hint. | | `use2DFiltering` | `Boolean` | `false` | Filter on horizontal distance only. Sent only with a geo or position hint. | | `enableGeoHint` | `Boolean` | `false` | Send the GPS fix from `geoHintProvider` as `geoHint`. | | `geoHintProvider` | `GeoHintProvider?` | `MultiSetSDK.geoHintProvider` | Where the GPS fix comes from. See [GPS hint](#gps-hint). | | `includeGeoCoordinatesInResponse` | `Boolean` | `false` | Ask for the fix in geographic terms, filling `geoCoordinates` and `geoPose` on the result. | | `poseConsistencyCheck` | `Boolean` | `false` | Discard results that contradict the device trajectory. | | `poseConsistencyThreshold` | `Float` | `4f` | Distance limit from the anchor, in metres. Valid range 1.5 to 15. | | `poseConsistencyYawThreshold` | `Float` | `25f` | Heading limit from the anchor, in degrees. Valid range 10 to 60. | | `poseConsistencyTuning` | `GateTuning?` | `null` | Overrides for the check's internal values. `null` uses the shipped ones. | | `mapAliasingRisk` | `Map` | empty | Per-map multiplier on the viewpoints needed to move the anchor, for spaces with repeated structure. | ### State | Property | Type | Description | | ------------------ | ------------------- | --------------------------------------------------------------------------------------- | | `stage` | `LocalizationStage` | The current run's stage. See [Run stages](#run-stages). | | `isBusy` | `Boolean` | A run is in progress or a silent retry is due. `start()` is refused while it is `true`. | | `isAwaitingRetry` | `Boolean` | A silent first-fix retry is due. Keep the user's progress UI up. | | `recoveryPhase` | `RecoveryPhase` | `NONE`, `WAITING_FOR_TRACKING` or `REALIGNING`. | | `needsRealignment` | `Boolean` | The scene is out of place after a tracking break. Hide map content while it is `true`. | | `anchorTrust` | `AnchorTrust` | Standing of the pose consistency anchor: `PROVISIONAL`, `CORROBORATED` or `TRUSTED`. | ### Callbacks | Callback | Signature | Fires when | | ----------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `onLocalizationRequested` | `() -> Unit` | A run has started. | | `onLocalizationSuccess` | `(LocalizationResult) -> Unit` | A pose was accepted. | | `onLocalizationFailure` | `(MultiSetError) -> Unit` | A run failed. Silent retries and recovery runs report no failure. | | `onLocalizationFalsePositive` | `(FalsePositiveInfo) -> Unit` | A pose was discarded as inconsistent. | | `onLocalizationCorrected` | `(from: AnchorPose, to: AnchorPose, trust: AnchorTrust) -> Unit` | Agreeing fixes out-voted the anchor and the map moved. Fires before the success that carries the new pose. | | `onAnchorTrustChanged` | `(AnchorTrust) -> Unit` | The anchor's standing changed. | | `onStageChanged` | `(stage: LocalizationStage, isBackground: Boolean) -> Unit` | The run moved to another stage. | | `onRecoveryPhaseChanged` | `(phase: RecoveryPhase, needsRealignment: Boolean) -> Unit` | The recovery after a tracking break moved on. | A session built by `MultiSetSDK` also reports its results, stages and recovery to the `MultiSetSDKCallback` passed to `initialize`. If you handle both, handle each event in only one of them. ### Methods | Method | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `start()` | Run now. Refused while `isBusy`. | | `stop()` | Cancel the scheduled run and drop the one in flight. | | `reportTrackingState(state)` | Tell the session the AR tracking state. Report the first state and every change. | | `notifyTrackingInterrupted()` | The camera was lost. The scene stays out of place until the next fix. | | `notifyWorldOriginReset()` | The AR world frame is gone (ARCore `BAD_STATE` or `STOPPED`, or a return from the background). Drops the run in flight and every offset measured in the old frame. | | `resetPoseReference()` | Drop the pose consistency anchor. The next accepted fix becomes the new one. | | `resetSession()` | Start over: clears the anchor and the recovery and re-arms the silent first-fix retry. Runs your `worldOriginResetHandler` if you set one. | {% hint style="info" %} `stop()` drops a request already in flight as well as the scheduled one, so a response arriving after you tore your scene down cannot re-apply the old pose. A `stop()`, `start()` or `resetSession()` called from inside a result callback wins: the session does not schedule its next run over it. {% endhint %} *** ## Run stages Both sessions report each run step by step through `onStageChanged(stage, isBackground)`: | Stage | Meaning | | ------------------------------- | ----------------------------------------------------------------------------------------- | | `Capturing(captured, required)` | Frames are being taken. For object tracking this includes the capture delay. | | `Uploading` | The query is going out. A fixed step of up to 2 s, since the request reports no progress. | | `AwaitingResponse` | The server is searching. | | `Idle` | The run is over. `Idle` arrives **before** the result callback. | `isBackground` is `true` for a background refresh, a recovery and an automatic re-track. Show those without covering the scene. While a silent first-fix retry is due the stage is back at `Idle` but `isAwaitingRetry` is `true`, so keep the user's overlay up until it ends. *** ## First localization until success With `firstLocalizationUntilSuccess` on, a failed **first** fix of the session is retried silently up to 5 times, 0.5 s apart. After a timeout, an HTTP 5xx or a 429 the waits grow to 4, 8 and then 12 s, so a busy server is not handed a second search at once. The failure is then reported once and the background loop takes over. This works the same with background localization on or off. It applies only to the first fix, never to a background refresh or a recovery. A rejected request (HTTP 4xx other than 408 and 429, including 401 and 403) is reported at once. `resetSession()` re-arms it. Object tracking's `firstTrackingUntilSuccess` follows the same rules. *** ## Tracking breaks and the return from the background Report **every** AR tracking change to the session, the return to tracking included, and the two kinds of break: ```kotlin session.reportTrackingState(TrackingState.TRACKING) // the first state, and every change session.reportTrackingState(TrackingState.PAUSED) // a limited spell: 2 s grace before it counts session.notifyTrackingInterrupted() // the camera was lost session.notifyWorldOriginReset() // BAD_STATE, STOPPED, back from the background ``` The sample app maps ARCore to these calls in `ar/ArTrackingSignals.kt`: | ARCore | Calls | | ---------------------------------------- | ----------------------------------------------------------------- | | `TRACKING` | `reportTrackingState(TRACKING)` | | `PAUSED` with `CAMERA_UNAVAILABLE` | `reportTrackingState(PAUSED)`, then `notifyTrackingInterrupted()` | | `PAUSED` with any other reason | `reportTrackingState(PAUSED)` | | `STOPPED`, or failure reason `BAD_STATE` | `reportTrackingState(...)`, then `notifyWorldOriginReset()` | With `relocalization` on, a break after a fix arms a **recovery**: once tracking is back and has held for 1 s, the session re-localizes, and keeps retrying every second while the scene is out of place. These are background runs and report no failures. A background refresh that falls due meanwhile gives way to them, and a tap replaces a scheduled one. Hide the map mesh and any world-anchored content while `needsRealignment` is `true`. {% hint style="warning" %} Nothing is recovered until you have called `reportTrackingState` at least once. A host that never reports tracking keeps the 2.0.0 behaviour: no recovery, and the refresh loop untouched. Report the first state too, and report the state before each break. {% endhint %} A world-origin reset also drops the run in flight, a tapped one included, because its frames belong to the world frame that is gone. The session then re-arms what that run would have done: the silent retry for a first fix, otherwise the recovery or the refresh. An interruption drops only a background run. ARCore cannot reset its world frame, and a session resumed after a trip to the background may no longer match the room. The sample app therefore treats the return as a world-origin reset, from `onPause` and **before** `super.onPause()`: ```kotlin override fun onPause() { frameSource.markFramesStale() // refuse frames taken before the pause session.reportTrackingState(TrackingState.PAUSED) session.notifyWorldOriginReset() super.onPause() } ``` *** ## ObjectTrackingSession ```kotlin val session = MultiSetSDK.objectTrackingSession(frameSource) // codes default to the config's ``` | Property | Type | Default | Description | | --------------------------- | --------- | -------- | -------------------------------------------------------------------------------------------------------------------- | | `captureDelayMs` | `Long` | `1000L` | Hold before each capture, from `objectTrackingCaptureDelayMs` (500 to 3000 ms). | | `backgroundTracking` | `Boolean` | `true` | Re-track at intervals, from `backgroundObjectTracking`. | | `bgTrackingDurationMs` | `Long` | `15000L` | Interval between background re-tracks, from `backgroundObjectTrackingIntervalSeconds` (5 to 30 s). | | `restartTracking` | `Boolean` | `true` | Re-track once tracking has held for 1 s after a loss. That run's miss is not reported. From `restartObjectTracking`. | | `firstTrackingUntilSuccess` | `Boolean` | `true` | Retry a failed first track silently, up to 5 times, from `firstObjectTrackingUntilSuccess`. | | `confidenceCheck` | `Boolean` | `false` | Reject results below `confidenceThreshold`. | | `confidenceThreshold` | `Float` | `0.3f` | Minimum accepted server confidence. | | `imageQuality` | `Int` | `90` | JPEG quality of the query image. | | `imageLongSide` | `Int` | `960` | Longer side of the query image in pixels, at most 1280. | | Callback | Signature | Fires when | | ------------------------------ | ----------------------------------------------------------- | ----------------------------------- | | `onTrackingRequested` | `() -> Unit` | A run has started. | | `onTrackingSuccess` | `(ObjectTrackingResult) -> Unit` | An object was tracked. | | `onTrackingFailure` | `(MultiSetError) -> Unit` | A run failed. | | `onStageChanged` | `(stage: LocalizationStage, isBackground: Boolean) -> Unit` | The run moved to another stage. | | `onPlacementVisibilityChanged` | `(objectCode: String, visible: Boolean) -> Unit` | Show or hide that object's content. | The session has the same `stage`, `isBusy` and `isAwaitingRetry` state, and the same `start()`, `stop()`, `reportTrackingState()`, `notifyTrackingInterrupted()` and `notifyWorldOriginReset()` methods as `LocalizationSession`. `resetSession()` forgets every placement and re-arms the silent first track. An interruption hides every placed object through `onPlacementVisibilityChanged(code, false)` until tracking is back. A world-origin reset keeps each one hidden until **that** object is tracked again, so content never shows where its object no longer is. {% hint style="warning" %} Give each tracked object its **own** anchor. A result moves one object only, so a single anchor shared by every object moves the first object's content onto the second one when the second is tracked. {% endhint %} *** ## MultiSetError Session failures are classified rather than passed as free text. ```kotlin data class MultiSetError( val kind: Kind, val message: String, val statusCode: Int? = null, val rawBody: String? = null, ) ``` | Kind | Meaning | Suggested handling | | ----------- | --------------------------------------------------------------- | --------------------------------------------------------- | | `SERVER` | The API rejected the request. Retrying unchanged will not help. | Surface to the developer. Check map codes and parameters. | | `AUTH` | Credentials rejected (401 or 403). | Check client ID and secret. | | `NETWORK` | The API was unreachable. | Ask the user to check connectivity. | | `TRANSIENT` | Scan-time failure such as no pose found or low confidence. | Expected. Prompt the user to move and scan again. | `error.isTransient` is shorthand for `kind == TRANSIENT`. *** ## FalsePositiveInfo Delivered when `poseConsistencyCheck` is on and a response contradicts the device's own trajectory. The request succeeded; the answer was wrong, most often a visually similar location elsewhere in the map. The scene is left untouched, and neither success nor failure is reported. | Property | Description | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `jumpMeters`, `yawDeltaDeg` | How far the discarded response placed the map from the anchor. | | `thresholdMeters`, `thresholdDegrees` | The limits in force: exactly `poseConsistencyThreshold` and `poseConsistencyYawThreshold`. | | `disagreement` | `DISTANCE`, `HEADING` or `BOTH`: which limit forced the contest. | | `cause` | `INSUFFICIENT_VIEWPOINTS`, `ANCHOR_HELD_VOTE` or `INLIER_MARGIN`. Only `INSUFFICIENT_VIEWPOINTS` means "move and ask again". | | `challengerSupport`, `requiredSupport`, `incumbentSupport` | Viewpoints behind the response, needed to move the anchor, and behind the anchor. | | `consecutiveCount` | Consecutive rejections since the last accepted fix. Starts at 1. | | `trust` | `AnchorTrust` of the anchor that was kept. | | `rejection` | `FALSE_POSITIVE`, `REESTABLISHING` (retried quietly) or `SUSPECTED_ALIASING` (localizing again from this spot will not help). | | `mapCodes`, `confidence`, `reason`, `summary` | The rejected response's map codes and confidence, and log-ready text. | The first response sets the anchor: a multi-frame one as `TRUSTED`, a single frame as `PROVISIONAL`. Fixes that agree refine it; fixes that disagree must be backed by enough separate viewpoints to move it, and `onLocalizationCorrected` fires when they do. After two consecutive rejections the session sends one multi-frame request to settle it (5 s cooldown, never for `SUSPECTED_ALIASING`). A bad **first** fix cannot be out-voted by the check, because that fix set the anchor. Offer `MultiSetSDK.resetLocalizationSession()` (or `session.resetSession()`) from your false-positive UI so the user can start over, and use `MultiSetSDK.resetPoseConsistencyReference()` when only the reference needs re-bootstrapping. *** ## GPS hint The SDK has no location stack of its own: your app supplies the position and owns the location permission. ```kotlin MultiSetSDK.geoHintProvider = GeoHintProvider { latestFix } // GeoCoordinates?, null sends nothing ``` Sessions built by `MultiSetSDK` and the raw APIs read it when each run starts, while `enableGeoHint` is on. A fix of all zeroes is never sent, and a fix without altitude is filtered horizontally. Send only a recent fix: the sample app sends one at most 2 minutes old. A geo hint needs a [geo-referenced map](/fundamentals/georeferencing-maps.md). *** ## Raw Stateless APIs For full control over capture timing, skip sessions and call the suspend APIs directly: ```kotlin suspend fun trackObjects(frame: CameraFrame): ObjectTrackingResult suspend fun localizeSingleFrame(frame: CameraFrame, queryMode: QueryMode? = null, geoHint: GeoCoordinates? = null): LocalizationResult suspend fun localizeMultiFrame(frames: List, queryMode: QueryMode? = null, geoHint: GeoCoordinates? = null): LocalizationResult ``` These perform one request each and return the computed world pose. They send the configured hints; `queryMode` defaults to the config's, and `geoHint` to `MultiSetSDK.geoHintProvider` while `enableGeoHint` is on. Scheduling, retry and gating are yours to implement. *** ## Related * [Android Native Overview](/native-sdk/android-native.md) * [LocalizationConfig](/native-sdk/android-native/api-reference/localizationconfig.md) * [ObjectTrackingConfig](/native-sdk/android-native/api-reference/objecttrackingconfig.md) * [Map Scanning (Non-LiDAR)](/native-sdk/android-native/api-reference/map-scanning.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.multiset.ai/native-sdk/android-native/api-reference/framesource.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.