Challenges in Vision-Based Robotic Grasping
Vision-driven robotic grasping has advanced across healthcare, manufacturing, and logistics, yet home environments remain challenging due to drastic lighting changes, limited textures, and cluttered backgrounds.
Object detection has evolved from handcrafted features to deep learning and transformer-based matchers such as LoFTR and LightGlue, the latter balancing speed and accuracy through adaptive computation.
For pose estimation, end-to-end deep methods reduce reliance on handcrafted features but demand large training datasets and heavy computation, limiting real-time deployment.
Traditional geometric approaches such as perspective-n-point (PnP) remain lightweight but are sensitive to matching noise. This paper addressed these gaps by proposing a lightweight framework that couples LightGlue-based graph matching with geometric vector mapping for 6D pose estimation, requiring only a single reference image without training, and employs Kalman filtering with an exponential moving average to stabilize detection under dynamic conditions.
Want to save for later? Click here.
Detection, Tracking, and Pose Estimation
The proposed method consists of two stages, namely, object detection and 6D pose estimation. For detection, the lightweight transformer-based matcher LightGlue matches features between a single reference image of the target's planar surface and the live camera frame.
From these correspondences, a homography transformation is computed using random sample consensus (RANSAC) to robustly map the object's position despite potential matching errors. However, detection bounding boxes often jitter due to lighting changes and motion blur.
To address this, a two-stage smoothing strategy is applied, where a Kalman filter first suppresses abrupt outliers by modeling the box's motion over time, followed by an exponential moving average to dampen high-frequency fluctuations, producing stable detections.
For pose estimation, the homography-mapped corners are used to locate keypoints. These points are back-projected into three-dimensional (3D) space using depth data, with robust filtering strategies to handle sensor noise near depth discontinuities.
From the 3D positions, two direction vectors along the object's surface are derived, and their cross product yields the surface normal. All three vectors are orthogonalized, normalized to unit vectors, and assembled into a rotation matrix representing the object's orientation.
This matrix is then converted to a quaternion for compact pose representation, with numerical fallback paths to ensure stability. Together with the object center's 3D coordinates, this provides the complete 6D pose for robotic grasping.
Validation of Robotic Grasping Framework
The experimental platform uses a high-performance host computer paired with a Jetson Nano controller and an Intel RealSense D435 camera. Red-green-blue (RGB) and depth images are transmitted via transmission control protocol (TCP), with dual-threaded parallel encoding to optimize throughput. Detection and pose estimation run on the host, and results are sent to the robotic arm over asynchronous robot operating system (ROS) communication.
For object detection, LightGlue-based template matching was compared against you-only-look-once (YOLO)v5 under zero-shot conditions with no task-specific training. LightGlue successfully detected objects at viewing angles up to 36° in either direction, doubling the 18° range achieved by YOLOv5 with generic pre-trained weights.
Further comparisons with YOLOv8 and Faster region-based convolutional neural network (R-CNN) across diverse household objects showed that general detectors may mislabel unfamiliar objects, whereas template matching reliably localizes targets using only a reference image.
A supplementary quantitative evaluation on approximately 1,000 medicine package images confirmed robust matching performance under varying illumination, occlusion, and viewpoint conditions.
For pose estimation, the impact of the Kalman filter and exponential moving average smoothing was evaluated. Without smoothing, bounding boxes exhibited noticeable drift and distortion due to unstable keypoint localization. Applying the dual smoothing strategy significantly improved stability, maintaining alignment even under harsh lighting.
Pose variance was then compared across three methods. The proposed geometric approach achieved lower variance than EPnP with RANSAC across all quaternion components, while Iterative PnP suffered from longer computation times. The proposed method balanced accuracy and speed, achieving high solver throughput.
Finally, the complete system was validated on a Realman RM65 robotic arm. The 6D pose was transformed into a grasp pose with a fixed offset, followed by trajectory generation with inverse kinematics and singularity checks.
Grasping tests under cluttered backgrounds, illumination changes, and varied camera viewpoints demonstrated successful adaptive grasping. Success was verified through visual confirmation of object lifting and gripper force feedback.
Towards Robust Robotic Grasping
This paper presented a lightweight framework for real-time 6D pose estimation and robotic grasping of planar objects in dynamic environments. By integrating LightGlue-based feature matching with geometric orthogonal vector mapping, the method requires only a single reference image.
A dual smoothing strategy combining Kalman filtering and exponential moving average stabilizes detection against illumination changes and motion blur. Experimental results confirmed the framework's real-time performance, robustness under wide viewing angles, and successful integration into a closed-loop robotic grasping system.
However, the approach remains limited to planar objects and depends on depth sensors. Future work will explore multimodal feature fusion and lightweight depth estimation to broaden applicability.
Journal Reference
Cui et al. (2026). Fast and lightweight 6D object pose estimation for efficient robotic grasping. Scientific Reports. DOI:10.1038/s41598-026-63003-2, https://www.nature.com/articles/s41598-026-63003-2
Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.