13 Jul

Gesture grab object at any contact point

Qiong Hu vr

Objective:

  • Grab an object when the hand is grabbing and near the object
  • Change object's position and rotation according to hand's position and rotation
  • Release the object when the hand is not grabbing

Easy solution (also the method in the previous version of the project):

  • When hand gesture is grab gesture and the hand center is close to the center of an object (within a threshold), attach the object center to the hand center (so maintain same position and rotation).
  • When hand gesture releases from grab gesture, break the attachment.
  • Pro: easy to implement; the attachment is stable when moving around (object won't slip)
  • Con: not intuitive, at init of grabbing, object is like "flying" towards the hand to jump through the threshold distance, and for some object, the center point is empty, so it looks kind of strange.

New method

Changes:

  • Use rigidbody collision to detect whether hand touches object, instead of hard-code distance threshold.
  • Create "Contact Point" for grabbing (think of it as a grabbing handle but it can appear at any location on the surface of the object upon grabbing and invisible), instead of only use the center to move.

Implementation Steps:

  1. Add rigidbody collider to every finger and palm of both hand models.
  2. When hand gesture is grab gesture, and hand collider has collision with an object (use tag to distinguish with other collider like ground and wall), record contact point position, hand position and rotation.
  3. Create invisible gameobject as "Contact Point", initialize in the contact point position, update its position and rotation to be the same as hand position and rotation.
  4. Create invisible gameobect "Object Representation" as a child object to "Contact Point", whose initial position and rotation is the same as those of the design object's center upon grabbing. So during grabbing and moving, position and rotation of "Object Representation" will change according to its parent gameobject "Contact Point" and indicate the supposed updated position and rotation of the design object.
  5. Update position and rotation of design object to be the same as "Object Representation".
  6. When hand gesture is released from grab gesture, destroy "Contact Point" and "Object Representation", grabbing ends.
  7. Tricky bug: During grabbing and moving, at contact point, the hand collider has constant collision with the design object which prevent from grabbing. Bug-fix solution: disable hand collider during grabbing, re-enable after grabbing ends.
  8. Other existing bug info: Unlike gloves, leap-motion-based hand can disappear from the frame, and drop the grabbed object, when hand re-appear at a different position as before, the grabbing "Contact Point" still record the previous position, so it will become distance-grabbing (look like some super power).

Demo video of grabbing an object

In this demo video, the hand moved the stool to a different location and flipped it over, by grabbing at different position on the surface of the stool.

Next Post Previous Post

Search

Popular Tags

foray origami robots blimp ducky vr edg SLAM computer vision computational design roco slam Origami rocolib arduino RoCo design Robot Compiler simulation Motion Planning autonomy edg-model mesh reinforcement learning; multi-agent; mean-field; logics create; kalman filter; motors imu friction mesh network tank esp32 networking contacts multi-agent pushing auxetic machine learning dynamics esp now visual processing wireless control communication webapp pcbs structure ROCO i2c path planning cars Robotica reinforcement learning computer skill openmv test motor dataset visual DARPA odometry Battery surface structure power consumption dash port-arrays modeling origami paper frequency diversity semantic infrastructure auxetics feedback gondola machine vision Consensus bistable auxetic controller gestural intuition mixed reality unit cell robot crawler gestural interface open house network create; kalman filter; EM; refactoring 4-wheeled darpa; create; icra2021 apriltags lidar logic control FORAY perception Quantum Information crazyflie 433mhz optimization propeller Foray structures sensors environmental design radio DER Kullback-Leibler Divergence lasercutter tutorial von Mises Fisher Distribution IR simulink legs project management localization distributed optimization blynk Auxetics painlessmesh experiment results offline reinforcement learning exam research proposal altimeter thrust PCB sensor fusion gate or webots multi-agent path planning UAV RL integration test arnhold joystick balloon test vision processing pushing problem modeling contacts mesh networking motor configuration nodemcu swarm object detection fusionbot ros2 rviz2 gazebo conference matlab arduino nano decentralized surfaces keyboard IROS Integrated Demo throughput peripheral device paper review reinforcement learning; motorized boats Writing slam; paper review rhino Short Post range distance surp origami robot nylon pvdf elastic polymers plastic hot plates reinforcement learning; multi-agent; mean-field; game theory; mean field; reinforcement learning current reinforcement learning; multi-agent; Xbee Zigbee robot compiler spi contact neural network dynamic modeling quasi-static modeling edg-examples Kalman Filter port-array von Mises Fisher von Mises-Fisher literature review multipack car Zero Elastic Energy modes swarm control solar cars system ID serial monitor edg-ide Discrete Elastic Rods ceres debug Consensus Algorithms coriolis force dynamic model proposal wtf deep learning pdms interpolation wireless signal strength reinforcement learning; multi-agent; decentralized misc reinforcement learning; multi-agent; mean-field; stationary demo video flytrap papers steering structured field pivoting differential power detectionrate color shape edg-refactoring pain python vision Machine Learning Q-Learning Model-Free release conferences quasi-static pushing liquid crystal elastomer nonholonomic systems iot grant feedback-controlled controls trajectory optimization surface contact Characterization tool ML multi-agent path finding Liquid Crystal Elastomers Learning kinematic_model reservoir computing 2.4 GHz concept 433 MHz Bayesian Network creative Computer skill internal silicone integrated system instance segmentation marginal cost pricing bootstrapping Bayesian networks ensemble methods offline Rl hardware manipulation COVID 19 paper cutter ESP32 kalman filter; multi-robot state space delta tolling buoyancy chiral square unit cell pressure distribution model prediction thrust configuration grid fabrication firmware protobuf #activecapturingmechanism #claw 3D-CHESS; kalman filter boxboat tracking model-based RL von Mises Distribution slam; flowchart; spring-mass 3D-CHESS; 433 rssi signal strength multirobot system writing in-situ design buoyancy equations boat deployment cell cameras der newton's method Barycenter Euler-Bernoulli beam sma-based actuator game theory Bessel functions von Mises-Fisher Distributions ide HDL board design battery current sensor ZMP center of friction Multi-Agent Systems code balloon semantic segmentation MIMO communications ICRA ICML guassian processes probabilistic neural networks graph generators MBRL batch learning ros edg-compiler multirobot optitrack adaptive control motor driver submission DNN kalman filtering point cloud 2.4ghz latency scenic codesign presentations machine training preliminary exam multiple devices inkscape material multi-object tracking ICLR Python actuation catching mechanism integration flapping color_segmentation lab_data result segmentation on multiple object and centers finding competition prelab multiple blimps pid defender Synthesizable Radio 3d printing kalman filter motor distribution barometer detection lift force depth estimation cad solidworks design tool pixracer pixhawk ardupilot flight controller regression fly test sensor selection