> For the complete documentation index, see [llms.txt](https://docs.multiset.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.multiset.ai/native-sdk/android-native/api-reference/framesource.md). # FrameSource and Sessions Supply camera frames to the Android SDK through FrameSource and CameraFrame, then run a LocalizationSession or ObjectTrackingSession. ## Overview From version 1.16.0 the MultiSet Android SDK contains no ARCore or Sceneform code. It never talks to an AR runtime directly. Instead, your app converts each camera frame into a plain `CameraFrame` and supplies it through the `FrameSource` interface. The SDK owns the loop: capture scheduling, retry, background re-localization, confidence gating, image preparation, networking, and pose maths. This is the whole integration surface. Implement one interface, then start a session. *** ## FrameSource ```kotlin interface FrameSource { suspend fun acquire(): CameraFrame? } ``` `acquire()` is called by the SDK whenever it needs a frame. Return `null` when no frame is available right now, for example when the tracker has not converged or the runtime has no new CPU image this tick. Returning `null` is not an error. The SDK polls briefly (up to 20 attempts, 100 ms apart) before giving up on a capture, which absorbs transient unavailability. ### CameraFrame ```kotlin data class CameraFrame( val bitmap: Bitmap?, val cameraPosition: Vec3, val cameraRotation: Quat, val intrinsics: CameraIntrinsics, val orientation: DeviceOrientation ) ``` | Property | Description | | ---------------- | -------------------------------------------------------------------------------------------------- | | `bitmap` | Raw RGB camera image. May be `null` for runtimes that supply pre-encoded frames. | | `cameraPosition` | World position of the camera at capture time, as `Vec3`. | | `cameraRotation` | World rotation of the camera at capture time, as `Quat`. | | `intrinsics` | `CameraIntrinsics(fx, fy, cx, cy, imageWidth, imageHeight)` for the source resolution of `bitmap`. | | `orientation` | `DeviceOrientation.PORTRAIT` or `LANDSCAPE`. | {% hint style="warning" %} Supply the **raw, uncorrected** pose and intrinsics. The SDK applies portrait orientation correction exactly once, internally. If you rotate the bitmap or correct the quaternion yourself, the correction is applied twice and the resulting pose is wrong. {% endhint %} ### Example: an ARCore frame source ```kotlin class ArFrameSource( private val arFragment: ArFragment, private val context: Context, private val imageProcessor: ImageProcessor, ) : FrameSource { override suspend fun acquire(): CameraFrame? = withContext(Dispatchers.Main) { val frame = arFragment.arSceneView.arFrame ?: return@withContext null val camera = frame.camera if (camera.trackingState != TrackingState.TRACKING) return@withContext null val pose = camera.pose val intr = camera.imageIntrinsics val image = try { frame.acquireCameraImage() } catch (e: Exception) { return@withContext null // NotYetAvailableException is transient } val imgWidth = image.width val imgHeight = image.height val orientation = if (context.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE) DeviceOrientation.LANDSCAPE else DeviceOrientation.PORTRAIT val bitmap = try { withContext(Dispatchers.Default) { imageProcessor.yuvToBitmap(image) } } finally { image.close() } CameraFrame( bitmap = bitmap, cameraPosition = Vec3(pose.tx(), pose.ty(), pose.tz()), cameraRotation = Quat(pose.qx(), pose.qy(), pose.qz(), pose.qw()), intrinsics = CameraIntrinsics( fx = intr.focalLength[0], fy = intr.focalLength[1], cx = intr.principalPoint[0], cy = intr.principalPoint[1], imageWidth = imgWidth, imageHeight = imgHeight, ), orientation = orientation, ) } } ``` Read the AR frame on the main thread, then move the YUV to bitmap conversion off it, and close the image exactly once afterwards. *** ## LocalizationSession Create a session from the SDK, configure it, attach callbacks, then start it. ```kotlin val session = MultiSetSDK.localizationSession(frameSource, LocalizationMode.MULTI_FRAME) ``` ### Properties | Property | Type | Default | Description | | ------------------------------- | -------------- | ------------- | ---------------------------------------------------------- | | `backgroundLocalization` | `Boolean` | `true` | Keep re-localizing periodically after the first success. | | `bgLocalizationDurationMs` | `Long` | `15000L` | Interval between background localizations. | | `numberOfFrames` | `Int` | `5` | Frames captured per multi-frame query. Valid range 4 to 6. | | `frameCaptureIntervalMs` | `Long` | `500L` | Delay between captures within a multi-frame query. | | `confidenceCheck` | `Boolean` | `true` | Reject results below `confidenceThreshold`. | | `confidenceThreshold` | `Float` | `0.3f` | Minimum accepted server confidence. | | `firstLocalizationUntilSuccess` | `Boolean` | `true` | Silently retry transient failures until the first success. | | `imageQuality` | `Int` | `80` | JPEG quality for uploaded frames. | | `hintMapCodes` | `List` | `emptyList()` | Restrict a MapSet query to named maps. | | `queryMode` | `QueryMode` | `VPS1` | Search strategy. Single-frame only. | | `poseConsistencyCheck` | `Boolean` | `false` | Discard results that contradict the device trajectory. | | `poseConsistencyThreshold` | `Float` | `10f` | Tolerance in metres. Valid range 3 to 30. | ### Callbacks | Callback | Signature | Fires when | | ----------------------------- | ------------------------------ | -------------------------------------------------- | | `onLocalizationRequested` | `() -> Unit` | A capture has started. Use it to show scanning UI. | | `onLocalizationSuccess` | `(LocalizationResult) -> Unit` | A pose was accepted. | | `onLocalizationFailure` | `(MultiSetError) -> Unit` | The query failed. | | `onLocalizationFalsePositive` | `(FalsePositiveInfo) -> Unit` | A pose was discarded as inconsistent. | ### Methods | Method | Description | | ----------------------------- | -------------------------------------------------------------------------------------- | | `start()` | Begin capturing. | | `stop()` | Cancel the scheduled capture **and** any request already in flight. | | `notifyTrackingInterrupted()` | Tell the pose gate that AR tracking broke, so it stops trusting its reference blindly. | | `resetPoseReference()` | Drop the pose reference. The next accepted result becomes the new one. | {% hint style="info" %} `stop()` cancels in-flight work as well as scheduled work. Without that, a response arriving after you tore your scene down would re-apply the old pose. Call `stop()` in `onDestroy` and whenever you reset the scene. {% endhint %} *** ## ObjectTrackingSession ```kotlin val session = MultiSetSDK.objectTrackingSession(frameSource, listOf("OBJ_001", "OBJ_002")) ``` | Property | Type | Default | Description | | --------------------------- | --------- | -------- | ---------------------------------------------------------- | | `captureDelayMs` | `Long` | `1000L` | Delay before the first capture. | | `backgroundTracking` | `Boolean` | `true` | Keep re-tracking after the first success. | | `bgTrackingDurationMs` | `Long` | `15000L` | Interval between background tracking attempts. | | `confidenceCheck` | `Boolean` | `true` | Reject results below `confidenceThreshold`. | | `confidenceThreshold` | `Float` | `0.3f` | Minimum accepted server confidence. | | `firstTrackingUntilSuccess` | `Boolean` | `true` | Silently retry transient failures until the first success. | | `imageQuality` | `Int` | `80` | JPEG quality for uploaded frames. | Callbacks are `onTrackingRequested`, `onTrackingSuccess(ObjectTrackingResult)` and `onTrackingFailure(MultiSetError)`. `start()` and `stop()` behave as they do for localization. *** ## MultiSetError Session failures are classified rather than passed as free text. ```kotlin data class MultiSetError( val kind: Kind, val message: String, val statusCode: Int? = null, val rawBody: String? = null, ) ``` | Kind | Meaning | Suggested handling | | ----------- | --------------------------------------------------------------- | --------------------------------------------------------- | | `SERVER` | The API rejected the request. Retrying unchanged will not help. | Surface to the developer. Check map codes and parameters. | | `AUTH` | Credentials rejected (401 or 403). | Check client ID and secret. | | `NETWORK` | The API was unreachable. | Ask the user to check connectivity. | | `TRANSIENT` | Scan-time failure such as no pose found or low confidence. | Expected. Prompt the user to move and scan again. | `error.isTransient` is shorthand for `kind == TRANSIENT`. *** ## FalsePositiveInfo Delivered when `poseConsistencyCheck` is enabled and a response contradicts the device's own trajectory. The request succeeded; the answer was wrong, most often a visually similar location elsewhere in the map. | Property | Type | Description | | ------------------ | -------------- | --------------------------------------------------------------------- | | `jumpMeters` | `Float` | How far the discarded pose sat from the last accepted one. | | `thresholdMeters` | `Float` | The tolerance in force. | | `consecutiveCount` | `Int` | Consecutive false positives since the last accepted fix. Starts at 1. | | `mapCodes` | `List` | Map codes the server reported for the discarded response. | | `confidence` | `Float?` | Server confidence of the discarded response, when reported. | | `reason` | `String` | Short machine-readable explanation, for logs. | | `summary` | `String` | One-line summary suitable for logging. | The scene is left untouched when this fires. The gizmo and mesh keep the last accepted pose. If `consecutiveCount` keeps climbing, the reference itself may be wrong; call `MultiSetSDK.resetPoseConsistencyReference()` to re-bootstrap it. *** ## Raw Stateless APIs For full control over capture timing, skip sessions and call the suspend APIs directly: ```kotlin suspend fun trackObjects(frame: CameraFrame): ObjectTrackingResult suspend fun localizeSingleFrame(frame: CameraFrame, queryMode: QueryMode = QueryMode.VPS1): LocalizationResult suspend fun localizeMultiFrame(frames: List): LocalizationResult ``` These perform one request each and return the computed world pose. Scheduling, retry and gating are yours to implement. *** ## Related * [Android Native Overview](/native-sdk/android-native.md) * [LocalizationConfig](/native-sdk/android-native/api-reference/localizationconfig.md) * [ObjectTrackingConfig](/native-sdk/android-native/api-reference/objecttrackingconfig.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.multiset.ai/native-sdk/android-native/api-reference/framesource.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.