Summary
On three Femto Bolt cameras, the depth-to-color rotation returned by the device is not orthonormal. The first element is close to the cosine of the tilt between the two cameras, where a rotation matrix needs a value close to 1. The matrix therefore scales one axis by about 0.5%.
I get the same values through two independent host-side paths, so I believe the value comes from the device and is not computed on the host. I could not determine whether it was written at factory calibration or is generated by the firmware.
Devices
| Model | Femto Bolt |
| Firmware | 1.1.3 (all three units, read from the devices) |
| Hardware | 1.1 |
| Serial numbers | CL86541038L, CL8F25300JZ, CL86541006P |
| pyorbbecsdk2 | 2.1.3 (bundles OrbbecSDK 2.7.6) |
| OS | Windows 11 x64 |
What I see
| Unit | R[0][0] reported |
R[0][0] required |
Norm of row 0 | Norm of rows 1, 2 |
|---|---|---|---|---|
CL86541038L |
0.9947249 | 0.999968 | 0.994757 | 1.000000 |
CL8F25300JZ |
0.9946091 | 0.999974 | 0.994635 | 1.000000 |
CL86541006P |
0.9942945 | 0.999987 | 0.994308 | 1.000000 |
“Required” is sqrt(1 - R[0][1]^2 - R[0][2]^2). The determinant equals the norm of row 0. The other eight elements are consistent with a proper rotation: a tilt of about 6 degrees between the depth and color cameras.
Full matrix for CL86541038L, row-major, as stored in the color camera’s Rt.Rotation:
0.9947248697 0.0080308039 0.0005517029
-0.0080450317 0.9946922660 0.1025793031
0.0002750196 -0.1025804207 0.9947246313
In all three units R[0][0] is nearly equal to R[2][2] (they differ in the seventh digit).
Where the value comes from
Both paths return identical values:
- Calibration JSON stored on the device, read with
k4a_device_get_raw_calibration(OrbbecSDK-K4A-Wrapper): color camera,"Rt"→"Rotation". - OrbbecSDK through pyorbbecsdk2,
pipeline.get_camera_param().transform.rot. In the current OrbbecSDK_v2 source (2.10.6) this value is copied without computation from theOB_RAW_DATA_ALIGN_CALIB_PARAMblock read from the device (TOFDeviceCommonAlgParamManager::fetchParamFromDevice,AlgParseHelper::alignCalibParamParse).
How to reproduce
import numpy as np
import pyorbbecsdk as ob
pipeline = ob.Pipeline()
config = ob.Config()
config.enable_stream(pipeline.get_stream_profile_list(ob.OBSensorType.COLOR_SENSOR).get_video_stream_profile(1920, 1080, ob.OBFormat.BGRA, 15))
config.enable_stream(pipeline.get_stream_profile_list(ob.OBSensorType.DEPTH_SENSOR).get_video_stream_profile(1024, 1024, ob.OBFormat.Y16, 15))
pipeline.start(config)
rotation = np.asarray(pipeline.get_camera_param().transform.rot, dtype=np.float64).reshape(3, 3)
pipeline.stop()
print(rotation)
print("row norms:", np.linalg.norm(rotation, axis=1))
print("max |R R^T - I|:", np.abs(rotation @ rotation.T - np.eye(3)).max())
Output on CL86541038L: row norms [0.994757 1. 1.], max |R R^T - I| = 0.0105. A rotation matrix gives row norms of 1 and an error near 1e-7.
Impact
Any code that treats this matrix as rigid is affected, for example by inverting it with a transpose. In my multi-camera calibration (ChArUco boards, 1920x1080 color), replacing the matrix with the nearest rotation (SVD) reduced the board reprojection error from 1.3-4.7 px to 0.25-0.42 px.
Questions
- Is this a known issue for Femto Bolt?
- Is the fix in the firmware, in the factory calibration data, or should the SDK correct the matrix when reading it?
- Do other firmware versions return a correct matrix?
Attachments
Raw calibration JSON of the three units, as returned by k4a_device_get_raw_calibration:
CL8F25300JZ_raw_calibration.txt (4.0 KB)
CL86541006P_raw_calibration.txt (4.0 KB)
CL86541038L_raw_calibration.txt (4 KB)