> For the complete documentation index, see [llms.txt](https://docs.multiset.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.multiset.ai/fundamentals/rest-api-docs/map-query.md). # Map Query The three VPS query endpoints that return a 6-DoF pose, JSON body, form data and multi image, plus every optional query parameter. The Map Query API returns a device's 6-DoF pose inside a map you have already scanned. Three endpoints do this, and they share the same response shape and the same optional parameters. | Endpoint | Body | Images | Use it for | | --------------------------------- | ------------------- | --------------------- | ---------------------------------------------------------------- | | `POST /vps/map/query` | JSON | 1, base64 encoded | The standard single-frame query, around 1 second | | `POST /vps/map/query-form` | multipart form-data | 1, as a file part | The same query when uploading a file is easier than encoding one | | `POST /vps/map/multi-image-query` | multipart form-data | 4 to 6, as file parts | A more robust pose in repetitive or difficult scenes | The endpoints come first below, then the [parameters](#query-parameters) that apply across all of them. ### VPS Query API Query your map to get the device position with respect to Map local origin. The query API takes query image (encoded as base64 string) and other image metadata. {% hint style="warning" %} The maximum query image resolution is **1280** pixels in either width or height. {% endhint %} {% openapi src="/files/ajrrlVmhtUvgT9mQ0Qyq" path="/vps/map/query" method="post" %} [VPSquery.yaml](https://3163433004-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FokTDI7QVY04Zvb1pQ8Ry%2Fuploads%2Fgit-blob-952bcb82d339a222cd42d081ed93024ffe52bd1b%2FVPSquery.yaml?alt=media) {% endopenapi %} #### Sample Response (JSON Body) ```json { "poseFound": true, "position": { "x": -5.89516855615433, "y": 1.225031596452081, "z": 2.2112895596804227 }, "rotation": { "x": -0.007873432249486393, "y": 0.8212519784928444, "z": 0.03204363652415735, "w": 0.5696107462509017 }, "confidence": 0.46875, "mapIds": ["67e12d4bff7ecf561f2f8a0c"], "mapCodes": ["MAP_RJFKKWQ1787J"], "responseTime": 2572 } ``` {% openapi src="/files/4THolmFT4ct9qovYeocP" path="/vps/map/query-form" method="post" %} [VPSqueryform.yaml](https://3163433004-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FokTDI7QVY04Zvb1pQ8Ry%2Fuploads%2Fgit-blob-5db84e26875054ecbe763c1e562e2b9cc73acdee%2FVPSqueryform.yaml?alt=media) {% endopenapi %} #### Sample Response (Form Data) ```json { "poseFound": true, "position": { "x": -2.8765866867008025, "y": 1.4018881843419664, "z": 7.677072632098843 }, "rotation": { "x": -0.0033476528966347096, "y": 0.6750939967230872, "z": -0.00310829178339555, "w": 0.7377175796541121 }, "confidence": 0.9175769612711023, "mapIds": ["67e12d4bff7ecf561f2f8a0c"], "mapCodes": ["MAP_RJFKKWQ1787J"], "responseTime": 2669 } ``` {% hint style="warning" %} **FormData Content-Type:** When using fetch or similar HTTP clients with FormData, do not manually set the Content-Type header. The browser automatically sets it to multipart/form-data with the required boundary string. Manually setting it will strip the boundary and cause a parse error. {% endhint %} #### Code Example (Python) ```python import json import requests BASE_URL = "https://api.multiset.ai/v1" TOKEN = "YOUR_M2M_TOKEN" with open("query.jpg", "rb") as f: response = requests.post( f"{BASE_URL}/vps/map/query-form", headers={"Authorization": f"Bearer {TOKEN}"}, data={ "mapCode": "MAP_RJFKKWQ1787J", "isRightHanded": "false", "width": "720", "height": "960", "fx": "670.4620971679688", "fy": "670.4620971679688", "px": "478.838623046875", "py": "365.346618652343", # Optional localization hints, all as strings "hintPosition": "-3.452,0.252,-0.531", "hintRadius": "15", "hintFloorHeight": "[1.5, 4.5]", }, files={"queryImage": ("query.jpg", f, "image/jpeg")}, ) print(response.status_code, response.json()) ``` ### VPS Multi Image Query API Multi-image query API requires a minimum of 4 images in each request. Up to 6 images can be passed. Higher image counts increase localization robustness and accuracy but may compromise latency. {% hint style="info" %} **About image#\_data properties:** The `image1_data`, `image2_data`, etc. fields contain the local SLAM tracking data (position and rotation) for when each image was captured. This is the device pose in the local coordinate system from ARKit (iOS) or ARCore (Android) at the moment the image was taken. {% endhint %} #### Tracking poses are optional The `imageN_data` parts are optional, which lets the multi-image query be used outside a SLAM session, for example from a set of stills or a camera rig with no tracking. **Optional does not mean unnecessary: send them whenever you have them.** A live ARKit or ARCore session already produces a pose per frame, and passing it gives the most accurate result. Two rules: * **Send a pose for every image, or for none.** Without poses the images are matched against each other instead, so they must visually overlap. A partial set is not rejected, but it is treated exactly as if you had sent none, so it is worth making the choice explicit in your client. * **A pose that you do send must be complete.** All seven fields (`x`, `y`, `z`, `qx`, `qy`, `qz`, `qw`) must be present together, and the value must be valid JSON. Either problem fails with `400`. The `trackingPose` in the response is your input echoed back, so it is absent whenever the poses were not used. {% openapi src="/files/oOKFI0YnlAkR5K578gJk" path="/vps/map/multi-image-query" method="post" %} [vps-maps-api.yaml](https://3163433004-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FokTDI7QVY04Zvb1pQ8Ry%2Fuploads%2Fgit-blob-f3ade18c75772f96d316a426adb56bd5552a6d22%2Fvps-maps-api.yaml?alt=media) {% endopenapi %} #### Sample Response (Multi Image) ```json { "poseFound": true, "estimatedPose": { "position": { "x": 3.9023228363353613, "y": 2.2199969114542415, "z": 7.684019738005462 }, "rotation": { "x": 0.0008889080606250241, "y": 0.7454695784085924, "z": -0.022682644709609446, "w": 0.6661529967948453 } }, "trackingPose": { "position": { "x": 0.0022450201213359833, "y": 2.471872329711914, "z": -10.018059730529785 }, "rotation": { "x": 0.030655404552817345, "y": 0.14812710881233215, "z": -0.00305502163246274, "w": -0.9884883761405945 } }, "imageId": "image4", "mapIds": ["67e12d4bff7ecf561f2f8a0c"], "confidence": 0.34274043817304123, "frames": [ { "imageId": "image1", "poseFound": true, "position": { "x": 3.8404791227185512, "y": 2.2358714094377391, "z": 7.5311847290014732 }, "rotation": { "x": 0.0104227310558921, "y": 0.7218904471209736, "z": -0.0248105549134402, "w": 0.6914338805274115 }, "confidence": 0.29881204871834762 }, { "imageId": "image2", "poseFound": true, "position": { "x": 3.8697402118840365, "y": 2.2281455037103951, "z": 7.6015528841019182 }, "rotation": { "x": 0.0061140925713855, "y": 0.7331886201553348, "z": -0.0235510983247741, "w": 0.6795726114402537 }, "confidence": 0.31204558839107514 }, { "imageId": "image3", "poseFound": false }, { "imageId": "image4", "poseFound": true, "position": { "x": 3.9023228363353613, "y": 2.2199969114542415, "z": 7.684019738005462 }, "rotation": { "x": 0.0008889080606250241, "y": 0.7454695784085924, "z": -0.022682644709609446, "w": 0.6661529967948453 }, "confidence": 0.34274043817304123 } ], "mapCodes": ["MAP_RJFKKWQ1787J"], "responseTime": 4919 } ``` #### Per-image poses (`frames`) Alongside the single `estimatedPose`, the response carries a `frames` array with one entry per image you sent, in the order the images were numbered. Each entry reports whether that individual image localized on its own, and if it did, the 6-DoF pose computed from it. | Field | Type | Description | | ------------ | ------- | --------------------------------------------------------------------------------------------- | | `imageId` | string | Which image this entry describes (`image1` to `image6`), matching the file part you uploaded. | | `poseFound` | boolean | Whether this individual image produced a pose. | | `position` | object | `x`, `y`, `z` in map-local coordinates. Present only when `poseFound` is `true`. | | `rotation` | object | Quaternion `x`, `y`, `z`, `w`. Present only when `poseFound` is `true`. | | `confidence` | number | Confidence score for this image alone. May be absent on a frame that did not localize. | **How `frames` relates to the top-level pose:** * `estimatedPose` remains the answer to use. It is the consensus pose across all images and is more robust than any single frame. * The top-level `imageId` names the winning frame, so the entry with that `imageId` carries the same `position`, `rotation`, and `confidence` as `estimatedPose`. * Every pose in `frames` is in the same coordinate frame as `estimatedPose`: it follows the `isRightHanded` value you sent, and on a versioned map it is expressed in the base map's frame. * Individual frames can fail while the overall query succeeds. A frame with `"poseFound": false` has no `position` or `rotation`, which is normal when one image in the set is blurred, occluded, or pointed at a featureless surface. * `frames` is informational. It is useful for diagnosing capture quality, spotting outlier viewpoints, and understanding which frames drove the result, but you do not need it for standard localization. {% hint style="warning" %} **FormData Content-Type:** When using fetch or similar HTTP clients with FormData, do not manually set the Content-Type header. The browser automatically sets it to multipart/form-data with the required boundary string. Manually setting it will strip the boundary and cause a parse error. {% endhint %} #### Code Example (Python) Split the request in two parts: the image binaries go in `files`, and every other field, including each `imageN_data` tracking pose, goes in `data` as a string. ```python import json import requests from contextlib import ExitStack BASE_URL = "https://api.multiset.ai/v1" TOKEN = "YOUR_M2M_TOKEN" image_paths = ["image1.jpg", "image2.jpg", "image3.jpg", "image4.jpg"] # Local SLAM tracking pose (ARKit / ARCore) for each image, in the same order tracking_poses = [ {"x": -5.1772, "y": 0.2936, "z": -2.6439, "qx": -0.0185, "qy": 0.9949, "qz": -0.0691, "qw": 0.0703}, {"x": -4.8210, "y": 0.3011, "z": -2.1094, "qx": -0.0201, "qy": 0.9932, "qz": -0.0805, "qw": 0.0812}, {"x": -4.3945, "y": 0.2874, "z": -1.7723, "qx": -0.0173, "qy": 0.9951, "qz": -0.0664, "qw": 0.0699}, {"x": -3.9012, "y": 0.2952, "z": -1.3388, "qx": -0.0190, "qy": 0.9940, "qz": -0.0731, "qw": 0.0764}, ] # Text fields, all sent as strings text_fields = { "mapCode": "MAP_RJFKKWQ1787J", # or "mapSetCode", exactly one of the two "isRightHanded": "false", "width": "720", "height": "960", "fx": "670.4620971679688", "fy": "670.4620971679688", "px": "478.838623046875", "py": "365.346618652343", # Optional localization hints "hintPosition": "-3.452,0.252,-0.531", } # Each imageN file part needs a matching imageN_data text part for index, pose in enumerate(tracking_poses, start=1): text_fields[f"image{index}_data"] = json.dumps(pose) with ExitStack() as stack: files = [ ( f"image{index}", (f"image{index}.jpg", stack.enter_context(open(path, "rb")), "image/jpeg"), ) for index, path in enumerate(image_paths, start=1) ] response = requests.post( f"{BASE_URL}/vps/map/multi-image-query", headers={"Authorization": f"Bearer {TOKEN}"}, data=text_fields, files=files, ) print(response.status_code, response.json()) ``` {% hint style="info" %} **Rules for the multipart body:** * File parts must be named `image1` to `image6`, and each one needs a matching `imageN_data` text part with the same number. A missing `imageN_data` returns `400` with `Missing metadata for imageN`. * Each `imageN_data` value is a JSON-encoded string containing `x`, `y`, `z`, `qx`, `qy`, `qz` and `qw`. All seven keys are required. * Send between 4 and 6 images. Fewer returns `At least 4 image file is required!`, more returns `Maximum 6 image files allowed!`. * Pass exactly one of `mapCode` or `mapSetCode`. * Do not set the `Content-Type` header yourself. `requests` adds `multipart/form-data` with the boundary when you pass `files`. {% endhint %} ### Query Parameters Every parameter below is optional, and all of them apply to the query endpoints above. Omit them and the query still runs. For the remaining localization parameters, `geoHint`, `hintRadius`, `use2DFiltering` and `hintMapCodes`, see [Localization](/fundamentals/localization.md). ### Spatial Hint: hintPosition The `hintPosition` parameter restricts the localization search to a radius around a given point in the map, speeding up the query and improving accuracy in visually repetitive scenes. | Parameter | Type | Description | | -------------- | ----------- | ----------------------------------------------------------------------- | | `hintPosition` | `[x, y, z]` | Hint point in the map's **left-handed (LHS / Unity) coordinate system** | | `hintRadius` | `number` | Search radius in meters (default `25`, range `5`–`100`) | {% hint style="warning" %} **`hintPosition` is in LHS (Unity) coordinates**, the same coordinate system returned in the `position` field of a successful localization response (`isRightHanded: false`). A previous localization result's `position` can be passed directly as the hint. If your application works in right-handed space (e.g. ROS, Three.js), convert to LHS before sending. {% endhint %} On form-data endpoints, pass `hintPosition` as a JSON-encoded string, e.g. `"[2.5, 0.1, 8.0]"`. See [Pose Prior : HintPosition](/fundamentals/localization/pose-prior-hintposition.md) for the full guide. ### Floor-Level Search: hintFloorHeight The `hintFloorHeight` parameter limits the search to a specific vertical band, enabling floor-level localization in multi-floor buildings. | Parameter | Type | Description | | ----------------- | ---------------- | ---------------------------------------- | | `hintFloorHeight` | `[y_min, y_max]` | Vertical band in map-local Y coordinates | **Rules:** * Both values are in map-local coordinates (or global MapSet coordinates for MapSet queries) * Order does not matter: `[-3, -1]` and `[-1, -3]` both select images with y ∈ \[-3, -1] * Can be combined with `hintPosition`: spatial filter runs first, height filter applies on the result **Examples:** ```json // Floor 2 of a building where floor is at y=3.0, ceiling at y=6.0 "hintFloorHeight": [3.0, 6.0] // Basement (negative Y) "hintFloorHeight": [-3.0, -0.5] // Form-data endpoints, pass as JSON string "hintFloorHeight": "[1.5, 4.5]" ``` ### Query Mode: queryMode The `queryMode` parameter selects which localization engine runs your query, letting you trade latency for accuracy. The request and response format stay exactly the same on both engines. | Value | Engine | Description | | ------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `vps-1` | Standard (default) | The fast standard localization engine, roughly 1 second for a single-frame query and 2 to 4 seconds for a multi-image query. Best for real-time, interactive localization. | | `vps-2` | Deep Search | A deep-search engine with up to **15% higher recall** and improved accuracy, at higher latency (around 2 seconds for a single-frame query, around 5 seconds for a multi-image query). Best for offline or background workflows and challenging, visually repetitive scenes. | **Rules:** * Optional. When omitted, the query runs on `vps-1`, so existing integrations are unaffected. * Supported on the single-image endpoints, `/vps/map/query` (JSON) and `/vps/map/query-form` (form-data), including `imageType=equirect`, and on the multi-image endpoint, `/vps/map/multi-image-query`. * Can be combined with all localization parameters (`hintPosition`, `hintRadius`, `hintFloorHeight`, `geoHint`, `hintMapCodes`). * Works with Maps, MapSets, and versioned maps. **Examples:** ```json // JSON body: run this query on the Deep Search engine "queryMode": "vps-2" // Form-data endpoints: pass as a plain string field "queryMode": "vps-2" ``` {% hint style="info" %} Use `vps-2` when accuracy matters more than latency, for example when placing persistent content or localizing in repetitive corridors. For high-frequency relocalization during an AR session, `vps-1` remains the recommended default. You can also try both engines on your own datasets from the Developer Portal simulation viewer. See [Query Mode](/fundamentals/localization/query-mode.md) for guidance on choosing between the two. {% endhint %} ### Search Depth: depth `depth` widens how far the query searches inside a map before it settles on a pose. Raise it on large maps and wide open spaces, where the right part of the space is harder to find. | Value | Search | | ---------- | -------------------------------------------- | | `1` | Default. Suits most maps. | | `2` to `5` | Progressively wider. Useful on large venues. | **Rules:** * Optional. An integer from `1` to `5`, defaulting to `1`, so existing integrations are unaffected. * Supported on the single-image endpoints, `/vps/map/query` (JSON) and `/vps/map/query-form` (form-data), and on the multi-image endpoint, `/vps/map/multi-image-query`. * Applies to both `pinhole` and `equirect` queries, and to both `queryMode` values. * A wider search takes longer, so raise it only as far as you need. * Works with Maps, MapSets, and versioned maps, and combines with the other localization parameters. **Example:** ```json // JSON body: widen the search on a large venue "depth": 2 ``` {% hint style="info" %} Start at the default. Raise `depth` only if a large map returns no pose, or a pose you do not expect. On small and medium maps it makes no practical difference and only costs time. {% endhint %} ### Localize by Coordinate: geoLocation `geoLocation` localizes **without naming a map**. Send a position and the nearest map whose [geo-fence](/fundamentals/georeferencing-maps/geo-fence.md) is in range is resolved and queried for you. {% hint style="warning" %} **This only works on geo-referenced maps.** A map resolves by coordinate only once it has a geo-fence, and it only gets a fence once it has real-world coordinates and a heading. If a map you expect is never matched, that is almost always why. See [Geo-fence](/fundamentals/georeferencing-maps/geo-fence.md) for how to geo-reference one. {% endhint %} ```json // JSON body "geoLocation": [37.794853, -122.394573] ``` ``` // form-data geoLocation=37.794853,-122.394573 ``` | Parameter | Meaning | | ----------------- | ------------------------------------------------------------------------------------------------------------------------ | | `geoLocation` | `[latitude, longitude]` in WGS 84 degrees. Mutually exclusive with `mapId`, `mapCode`, `mapSetId` and `mapSetCode`. | | `geoSearchRadius` | Overrides the search radius in metres for this request only. Ignored unless `geoLocation` is supplied. Defaults to 25 m. | **Rules:** * Only **active** maps in your account with `geoFenceReady: true` are considered. * Distance is measured to the fence edge, so a point inside a fence is `0` m away. * If the resolved map belongs to a MapSet, the **whole set** is queried, which is what you want indoors. * When nothing is in range the request returns `404`. A successful response carries `resolvedFrom`, naming the map the coordinates landed on: ```json "resolvedFrom": { "mapCode": "MAP_63ERQ0V9XSJD", "distanceMeters": 0, "searchRadiusMeters": 25 } ``` {% hint style="info" %} **`geoLocation` doubles as a `geoHint`.** If you supply neither `geoHint` nor `hintPosition`, the coordinate is reused as the localization prior, and `hintRadius` then defaults to **100 m** rather than the usual 25 m, because a raw GPS fix is less precise than a hint you calculated yourself. {% endhint %} {% hint style="warning" %} Resolving by coordinate costs a spatial lookup on every request. For anything running at frame rate, call `GET /v1/vps/map/nearby` once, cache the code it returns, and query that instead. See [Geo-fence](/fundamentals/georeferencing-maps/geo-fence.md). {% endhint %} ### Image Type: imageType `imageType` tells the query which projection your image uses. It applies to `/vps/map/query-form` only. | Value | Image | Camera intrinsics | | ---------- | -------------------------------------- | ------------------------------------------------------------- | | `pinhole` | A standard perspective photo (default) | **Required.** Send `fx`, `fy`, `px`, `py`, `width`, `height`. | | `equirect` | A full 360° equirectangular panorama | **Not required.** Send them and they are ignored. | A pinhole query needs intrinsics before a pixel can become a direction. An equirectangular image carries that already, so there is nothing to send. You get back one pose: the pose of the **forward-looking view**, meaning the direction the centre column of the image faces. **Rules:** * Optional. When omitted the query runs as `pinhole`, so existing integrations are unaffected. * Works with both `queryMode` values. `queryMode=vps-2` runs the panorama on the deep-search engine. * `queryImage` must be **2048 to 16384 px wide** with an aspect ratio of at least **1.9:1**, and 30 MB or smaller. Outside that, the request fails with `400`, or `413` if it is only the file size. * Works with Maps, MapSets, and versioned maps, and with all localization parameters. * REST only. The Unity, Quest, iOS and Android SDKs capture perspective frames and do not expose `imageType`. **Example:** ```bash curl -X POST "$BASE_URL/v1/vps/map/query-form" \ -H "Authorization: Bearer $TOKEN" \ -F "imageType=equirect" \ -F "mapCode=MAP_XXXXXXXXXXXX" \ -F "isRightHanded=true" \ -F "queryImage=@panorama.jpg" ``` The response has the same shape as a pinhole query: `poseFound`, `position`, `rotation`, `confidence`, and the matched `mapIds` and `mapCodes`. The `rotation` describes the forward-looking view, so orient your panorama with its centre column along your direction of travel if you want the returned heading to mean that. See [Equirect Query](/fundamentals/localization/equirect-query.md). {% hint style="info" %} Because one panorama covers the whole horizon, an equirect query does not depend on the user facing a mapped surface, which makes it a good fit for tripod, trolley, and robot captures. For a hand-held AR session, a `pinhole` frame from the device camera is still the right choice. {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.multiset.ai/fundamentals/rest-api-docs/map-query.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.