Robotics & Autonomous Systems

We describe the software components of a robotics system designed to autonomously grasp objects and perform dexterous manipulation tasks with only high-level supervision. The system is centered on the tight integration of several core functionalities, including perception, planning and control, with the logical structuring of tasks driven by a Behavior Tree architecture. The advantage of the implementation is to reduce the execution time while integrating advanced algorithms for autonomous manipulation. We describe our approach to 3-D perception, real-time planning, force compliant motions, and audio processing. Performance results for object grasping and complex manipulation tasks of in-house tests and of an independent evaluation team are presented.

Robotic grasping is one of the most important fields in robotics, in which great progress has been made in recent years with the help of convolutional neural network (CNN). However, including multiple objects in one scene can invalidate the existing CNN-based grasp detection algorithms, because manipulation relationships among objects are not considered, which are required to guide the robot to grasp things in the right order. This paper presents a new CNN architecture called Visual Manipulation Relationship Network (VMRN) to help robots detect targets and predict the manipulation relationships in real time, which ensures that the robot can complete tasks in a safe and reliable way. To implement end-to-end training and meet real-time requirements in robot tasks, we propose the Object Pairing Pooling Layer (OP2L) to help to predict all manipulation relationships in one forward process. Moreover, in order to train VMRN, we collect a dataset named Visual Manipulation Relationship Dataset (VMRD) consisting of 5185 images with more than 17000 object instances and the manipulation relationships between all possible pairs of objects in every image, which is labeled by the manipulation relationship tree. The experimental results show that the new network architecture can detect objects and predict manipulation relationships simultaneously and meet the real-time requirements in robot tasks.

Manipulating in the micro- or even nano world still poses a great challenge to robotics. Conventional (stationary) systems suffer from drawbacks regarding integration into process supervision and multi-robot approaches, which become highly relevant to fight scaling effects. This paper describes work currently being carried out which aims to make automated manipulation of micrometer-scaled objects possible by robots with nanometer precision. The goal is to establish a small cluster of (up to five) micro robots equipped with on-board electronics, sensors and wireless power supply. Power autonomy has been reached using inductive energy transmission from an external wireless power supply system or a battery based system. Electronics requirements are fulfilled in the electronic module with the full custom integrated circuit design for the robot locomotion control and the closed loop force control for AFM tool in cell manipulation applications. The maximum velocity obtained is about 0.4 mm/s with a saw tooth voltage signals of 20Vpp and 2500 Hz. In order to keep a AFM tool on micro-robot a specific tip with integrated piezoresistance, instead of the classical laser beam methodology, is validated for force measurement.

, which involve manipulation actions. Particularly, for complex manipulation tasks which are subject to geometric constraints, spatial information and a rich semantic knowledge about objects, types, and functionality are required, together with the way in which these objects can be manipulated. In this line, this paper presents an ontological-based reasoning framework called Perception and Manipulation Knowledge (PMK) that includes: (1) the modeling of the environment in a standardized way to provide common vocabularies for information exchange in human-robot or robot-robot collaboration, (2) a sensory module to perceive the objects in the environment and assert the ontological knowledge, (3) an evaluation-based analysis of the situation of the objects in the environment, in order to enhance the planning of manipulation tasks. The paper describes the concepts and the implementation of PMK, and presents an example demonstrating the range of information the framework can provide for autonomous robots.

A six degree-of-freedom computer-controlled manipulator is examined, and the relationships between the arm's joint variables and 3-space are derived. Arm trajectories using sequences of third-degree polynomials to describe the time history of each joint variable are presented and two approaches to the avoidance of obstacles are given. The equations of motion for the arm are derived and then decomposed into time-dependent factors and time-independent coefficients. Several new and simplifying relationships among the coefficients are proven. Two sample trajectories are analyzed in detail for purposes of determining the most important contributions to total force in order that relatively simple approximations to the equations of motion can be used.

A six-degree-of-freedom industrial robot to which was added a number of sensors-vision, range, sound, proximity, force/torque, and touch-to enhance its inspection and manipulation capabilities is described. The work falls under the scope of partial autonomy. In teleoperation mode, the human operator prepares the robotic system to perform the desired task. Using its sensory cues, the system maps the workspace and performs its operations in a fully autonomous mode. Finally, the system reports back to the human operator on the success or failure of the task and resumes its teleoperation mode. The feasibility of realistic autonomous robotic inspection and manipulation tasks using multisensory information cues is demonstrated. The focus is on the estimation of the three-dimensional position and orientation of the task panel and the use of other nonvision sensors for valve manipulation. The experiment illustrates the need for multisensory information to accomplish complex, autonomous robotic inspection and manipulation tasks.>

The autonomous and precise grasping operation of robots is considered challenging in situations where there are different objects with different shapes and postures. In this study, we proposed a method of 6-D target pose estimation for robot autonomous manipulation. The proposed method is based on: 1) a fully convolutional neural network for scene semantic segmentation and 2) fast global registration to achieve target pose estimation. To verify the validity of the proposed algorithm, we built a robot grasping operation system and used the point cloud model of the target object and its pose estimation results to generate the robot grasping posture control strategy. Experimental results showed that the proposed method can achieve a six-degree-of-freedom pose estimation for arbitrarily placed target objects and complete the autonomous grasping of the target. Comparative experiments demonstrated that the proposed target pose estimation method achieved a significant improvement in average accuracy and real-time performance compared with traditional methods.

This paper describes the development, mechanical configuration, and control system architecture of a lightweight, high performance, seven-degree-of-freedom manipulator at the Center for Engineering Systems Advanced Research (CESAR). Current activities focusing on modeling and parameter identification will provide a well-characterized manipulator for analytical and experimental research in manipulator dynamics and controls, coordinated manipulation, and autonomous mobile robotics.

Cooperative manipulation by autonomous intelligent robots promises fertile potential for improving the ability of robotic manipulation and expanding the domain where robots may be used. This paper focuses on the method for calculating the optimal force of each robot based on local sensor feedback and high-level communication. First, we propose adaptive force direction control, which is useful for monitoring the state of force equilibrium, and then describe the theoretical background of this control method. Second, we describe the strategy to find the proper force using our proposed control method. We also describe a control system architecture which is suitable for these operations. In this control system, each manipulator motion function, each sensor function, and each message-passing operation are programmed as primitive tasks which run in parallel. Third, we illustrate the usefulness of our proposed method with some experiments such as tumbling operation by two autonomous robots.

Cooperative manipulation by autonomous intelligent robots have good potential for improving the ability of robotic manipulation and expanding the domains where the robot may be used. This paper focuses on the method for searching proper desired force of each robot based on local sensor feedback and heigh level communication. First, we propose the adaptive force direction control, which is useful for monitoring the state of force equilibrium, and describe the theoretical background of this control method. Second, we describe the strategy for searching proper desired force using proposed control method and heigh level communication. We also describe a control system architecture which is suitable for these operations. In this control system, each manipulator motion function, each sensor function and each message passing operation are programmed as primitive tasks which run in parallel. Third, we illustrate the usefulness of proposed method with some experiments of tumbling operation by two autonomous robots.

In unstructured environments, autonomous robot manipulation suffers from high visual perception uncertainty, large control delays, and shallow vision-control fusion, resulting in low success rates and poor trajectory accuracy under disturbances. Existing visual servoing, Diffusion Policy, and vision-language-action (VLA) models mostly employ one-way or static fusion, lacking real-time bidirectional interaction. This study proposes a Bidirectional Vision-Control Fusion Framework (BVCFF). An Uncertainty-Aware Adaptive Fusion mechanism (UAAF) dynamically balances vision and control weights via visual entropy and Lyapunov gradients. A Graph Attention Temporal Fusion network (GAT-TF) captures multimodal long-term dependencies. An end-to-end differentiable joint optimization embeds Lyapunov stability into the composite loss for bidirectional error back-propagation. Gazebo simulation experiments and preliminary real-robot validation on a UR5e platform show superior performance: 94.8% grasping success, 87.6% insertion success, 80.3% dynamic success (simulation) and 87.2%, 76.4%, 68.7% (real-robot), 5.3 mm trajectory error, 43 FPS, and 0.92 robustness, outperforming seven benchmarks including Diffusion Policy and OpenVLA-inspired VLA. The deep bidirectional fusion provides an efficient, robust solution for embodied intelligence deployment.

At the core of a robotics system is the ability to acquire, inte-grate, and interpret multisensory data to generate appropriate actions in order to perform a given task. In this paper, we show the feasibility of a realistic autonomous manipulation task using multisensory information. The robotic system used here is a six-degree-of-freedom industrial robot to which we added a number of sensors: vision, range, sound, proximity, touch, force, and torque to enhance its inspection and manipulation capabilities. The cooperative use of sensors and the manipulation capabilities of this robotic system is shown through an experiment involving a task panel including a number of indicators (such as analog and digital meters), and controls (such as valves and push-buttons). The control of the robot, sensory data acquisition, processing, and interpretation, as well as global task supervision are performed by a dedicated VAX 11/785 computer using higher level languages. By integrating vision, range, proximity, touch, force, and torque sensors, many manipulation experiments are accomplished autonomously and successfully. The hardware and software modularity of this robotic system, the generic nature of its building blocks, as well as its reconfigurability make it an ideal tool for industrial multisensor robotic experimentation.