Dexterous and Autonomous Robotics setup

Research Platform

  1. DexMobile dual-arm mobile robot is a Husky-based platform equipped with two UR5e manipulators and anthropomorphic robotic hands for dexterous mobile manipulation research.
  2. Multimodal perception by RGB-D cameras and object-sensing modules support pose estimation, scene understanding, affordance recognition, and material-aware grasp planning.
  3. Simulation and learning environment based on PyBullet, ROS 2, and MoveIt enable reinforcement learning, motion planning, grasp evaluation, and sim-to-real experimentation.
  4. The platform supports task-oriented grasping, cooperative manipulation, object handover, contact-rich assembly, and human-informed robot learning.

Task-Oriented Grasping Using Reinforcement Learning with a Contextual Reward Machine


Motivation

  • Standard reinforcement learning often requires a large number of training samples.
  • Long, multi-stage grasping tasks are difficult to learn as a single policy.
  • Task-oriented grasping needs a more efficient, structured, and robust learning process.

Concept

  • A reinforcement-learning framework is integrated with a Contextual Reward Machine on the DexMobile platform.
  • The Contextual Reward Machine decomposes a complex grasping task into manageable subtasks.
  • Each subtask uses a stage-specific reward function, action space, and state-abstraction function.
Context-aware task-oriented grasping system structure
Context-aware task-oriented grasping framework based on the DexMobile platform.

Task-oriented grasping demonstration
Task-oriented grasping demonstrations: twist, press, pull, and wrap grasp.

Result

  • Achieved a 95% success rate across 1,000 simulated grasping tasks.
  • Achieved an 83.3% success rate across 60 real-world grasping tasks.
  • The evaluation covered six grasp affordances.

MagicHand: Context-Aware Dexterous Grasping Using an Anthropomorphic Robotic Hand


Motivation

  • A robot must understand material, rigidity, texture, dimensions, and fragility before selecting a grasp.
  • Many grasping systems rely on limited perception.
  • Limited perception prevents the robot from fully understanding an object's physical context before grasping.

Concept

  • MagicHand combines a Sawyer robotic arm, an AR10 anthropomorphic hand, an RGB-D camera, and a SCiO near-infrared sensor.
  • RGB-D data are used to estimate object dimensions.
  • Near-infrared spectra are used to recognize material properties.
  • The estimated physical characteristics are mapped to a suitable grasp strategy.
Context-aware task-oriented grasping system structure
The MagicHand platform combines a Sawyer arm, AR10 robotic hand, RGB-D camera, SCiO NIR sensor, and IMU for context-aware dexterous manipulation.

Task-oriented grasping demonstration
The PyBullet-Gym simulation environment supports reinforcement learning for object alignment, pick-and-place, and dexterous manipulation tasks.

Result

  • Achieved 99.64% material-recognition accuracy.
  • Kept dimension-estimation errors below 4 mm.
  • Completed 8 out of 10 robot grasping trials successfully.

Learning Task-Oriented Dexterous Grasping from Human Knowledge


Motivation

  • The same object may require different grasps for moving, holding, pressing, or turning.
  • Different tasks can require different grasp topologies.
  • The robot must select a grasp using both object affordances and task designations.

Concept

  • The framework learns task-oriented dexterous grasping from human grasping knowledge.
  • A deep-learning network predicts grasp topology from object affordances and task information.
  • Reinforcement learning adaptively deploys the selected grasp strategy in simulation.
Workflow and structure of the task-oriented grasp strategy generation system
Workflow and structure of the task-oriented grasp strategy generation system

Task-oriented grasping demonstration
Grasping simulations on arbitrary objects. Colored lines indicate trajectories of fingertips while grasping.

Result

  • Achieved a 100% grasp-strategy hit rate.
  • Achieved a 98.6% top-3 match rate.
  • Achieved an 85.6% success rate across 2,700 simulated grasping experiments.
  • The experiments included 17 objects.
Dexterous and Autonomous Robotics setup

Research Platform

  1. Interactive human-robot workspace: a shared experimental environment for studying communication, collaboration, object handover, and human-aware manipulation.
  2. Perception and intention sensing: RGB-D cameras, hand-pose estimation, and workspace monitoring support recognition of human motion, grasp intent, and interaction context.
  3. Projection and touch interaction: a projector-camera system displays robot intentions directly on surrounding surfaces and enables natural touch-based responses.
  4. Research capabilities: the platform supports adaptive handover, intention-aware planning, bidirectional communication, usability studies, and safe human-robot teaming experiments.

Omnisurface: Intention Awareness and Touch-Based Query for Human-Robot Teaming

Motivation

  • Human teammates need clear awareness of a robot's current state, planned actions, and intentions.
  • Screens, audio cues, and conventional projections provide limited spatial understanding.
  • Projection on curved, inclined, or deformable surfaces can become distorted.
  • Conventional systems usually do not support direct bidirectional interaction.

Concept

  • Omnisurface integrates a depth camera and projector.
  • The system reconstructs surface geometry and compensates for projection distortion.
  • Robot intentions are displayed directly in the shared workspace.
  • A depth-guided fingertip detector enables users to answer projected robot queries through natural touch.
Context-aware task-oriented grasping system structure
Omnisurface framework for distortion-compensated intention projection and touch-based query response.

Task-oriented grasping demonstration
Human-robot teaming scenarios using projected intention cues and direct touch interaction.

Result

  • Maintained average projection RMSE below 3 mm.
  • Operated at approximately 20–30 frames per second.
  • Maintained median latency below 50 ms.
  • Received higher user ratings than conventional GUI interaction.

Grasp Intention Interpretation in Object Handover for Human-Robot Teaming

Motivation

  • Human grasping habits vary across users and situations.
  • Different users may express the same intended grasp topology in different ways.
  • A robot must determine a suitable object position and orientation for smooth, safe, and comfortable handover.

Concept

  • The system extracts 21 three-dimensional hand keypoints from RGB images.
  • It classifies the grasp into one of six standard grasp topologies.
  • It estimates a suitable object pose using topology-specific keypoints.
  • A PPO policy moves and orients the object while penalizing unsafe contact with the human hand.
Workflow and structure of the task-oriented grasp strategy generation system
Grasp adaptation pipeline for recognizing human grasp topology, estimating object pose, and planning adaptive handover behavior.

Task-oriented grasping demonstration
Object-handover evaluation across six human grasp topologies in simulation.

Result

  • Achieved 93.3% training accuracy for grasp recognition.
  • Achieved 87.2% testing accuracy for grasp recognition.
  • Achieved an overall handover success rate of 83% across six grasp topologies.
FrameCRL cooperative robotics planning framework

Research Platform

  1. Cooperative Leo Rover fleet: multiple mobile robots provide a flexible testbed for coordinated navigation, distributed sensing, and collaborative field missions.
  2. Onboard perception and computation: cameras, localization sensors, and embedded processors support mapping, obstacle detection, object recognition, and autonomous decision-making.
  3. Multi-robot software stack: ROS-based communication, planning, and control modules enable task allocation, shared-world modeling, and synchronized execution.
  4. Research capabilities: the platform supports natural-language mission planning, cooperative exploration, rock-sampling scenarios, spatial constraints, and shared-resource coordination.

Controlled Robot Language with Frame Semantics (FrameCRL) for Autonomous Context-Aware High-Level Planning

Motivation

  • Natural-language instructions are intuitive but ambiguous and non-deterministic.
  • They are difficult to translate into complete formal plans.
  • LLM-generated plans often require post-processing.
  • LLM outputs provide limited verification and may be unreliable for direct robot execution.

Concept

  • FrameCRL combines Controlled Robot Language with FrameNet semantics.
  • Imperative verbs are anchored to semantic frames.
  • Object properties, spatial information, body-part references, and initial states are extracted from discourse representations.
  • The framework generates deterministic, context-aware PDDL plans.
  • A PDDL solver and behavior trees ground the plans into robot actions.
FrameCRL semantics extraction, plan generation, and action grounding framework
FrameCRL pipeline for semantic extraction, FrameNet-based verb anchoring, context-aware PDDL generation, and robot action grounding.

Dual-arm cooperative robot execution generated by FrameCRL
Dual-arm mobile robot executing cooperative pick-and-place plans generated from natural-language instructions.

Result

  • Produced almost fully executable plans whenever generation succeeded.
  • Outperformed LLM-based approaches on the HuRIC, CRC, and WikiHow corpora.
  • Achieved a 75% success rate across 20 dual-arm mobile robot simulation cases.

View Paper

HiP-CL: High-Level Planning Based on Common Language for Multi-Robot Teaming

Motivation

  • Natural-language multi-robot instructions are often ambiguous and incomplete.
  • Pure LLM planners can hallucinate or omit constraints.
  • Generated plans may be non-executable when spatial and shared-resource constraints must be satisfied.

Concept

  • HiP-CL combines a supervised fine-tuned language model with a formally defined Common Language.
  • Natural-language instructions are translated into unambiguous Common Language.
  • The framework generates complete PDDL planning problems.
  • Spatial and resource constraints are added before planning.
  • A formal planner produces deterministic multi-robot plans.
Context-aware task-oriented grasping system structure
HiP-CL framework for translating natural-language instructions into constrained, executable plans for multi-robot teams.

Task-oriented grasping demonstration
Multi-robot execution of generated plans under spatial and shared-resource constraints.

Result

  • Improved executability by 65.66% on the simple-task corpus.
  • Improved executability by 70.00% on the complex-task corpus.
  • Demonstrated collaborative rock-sampling missions with three Leo Rovers.
ezTwin digital twin framework for hydro-informatics

Research Platform

  1. ezTwin digital-twin framework: a scalable software platform for rapidly constructing hydrologic digital twins from limited geospatial and environmental inputs.
  2. Data and simulation integration: terrain, watershed, rainfall, and flow information are combined with hydrologic models to generate location-specific virtual environments.
  3. Visualization and synchronization: interactive flood maps and evolving environmental states connect simulated conditions with real-world observations.
  4. Research capabilities: the platform supports flood-scenario analysis, water-resource planning, rapid model deployment, and decision-support studies across different geographic regions.

ezTwin: A Scalable Framework for Rapid Building Digital Twin for Hydro-Informatics

Motivation

  • Hydrologic digital twins can improve flood analysis and water-resource management.
  • Their development often requires extensive geospatial data.
  • They also require specialized modeling expertise and significant setup time.
  • These requirements make rapid and consistent deployment across regions difficult.

Concept

  • ezTwin generates hydrologic digital twins for locations anywhere on Earth.
  • The framework requires only minimal geospatial inputs.
  • It simplifies deployment and supports flood-scenario planning.
  • It maintains synchronization between the digital representation and changing physical conditions.
ezTwin scalable hydrologic digital twin framework
ezTwin framework for rapidly constructing hydrologic digital twins from minimal geospatial inputs.

Hydrologic digital twin visualization and flood scenario result
Hydrologic digital-twin visualization supporting flood-scenario planning and synchronized environmental analysis.

Result

  • Enables faster and more accessible construction of hydrologic digital twins.
  • Supports real-time environmental visualization and decision-making.
  • The work was published in the World Environmental and Water Resources Congress 2026 proceedings.

View Paper