Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation

3D generation methods have shown visually compelling results powered by diffusion image priors. However, they often fail to produce realistic geometric details, resulting in overly smooth surfaces or geometric details inaccurately baked in albedo maps. To address this, we introduce a new method that incorporates touch as an additional modality to improve the geometric details of generated 3D assets. We design a lightweight 3D texture field to synthesize visual and tactile textures, guided by 2D diffusion model priors on both visual and tactile domains. We condition the visual texture generation on high-resolution tactile normals and guide
the patch-based tactile texture refinement with a customized TextureDreambooth. We further present a multi-part generation pipeline that enables us to synthesize different textures across various regions. To our knowledge, we are the first to leverage high-resolution tactile sensing to enhance geometric details for 3D generation tasks. We evaluate our method in both text-to-3D and image-to-3D settings. Our experiments demonstrate that our method provides customized and realistic fine geometric textures while maintaining accurate alignment between two modalities of vision and touch.

An Intelligent Robotic System for Perceptive Pancake Batter Stirring and Precise Pouring

Cooking robots have long been desired by the commercial market, while the technical challenge is significant. A major difficulty comes from the demand of perceiving and handling liquid with different properties. This paper presents a robot system that mixes batter and makes pancakes out of it, where understanding and handling the viscous liquid is an essential component. The system integrates advanced sensory and control algorithms to autonomously stir flour and water to achieve the desired batter uniformity, estimate the batter’s properties such as the water-flour ratio and liquid level, as well as perform precise manipulations to pour the batter into any specified shape. Experimental results show the system’s capability to always produce batter of desired uniformity, estimate water-flour ratio and liquid level precisely and accurately pour it into complex shapes. This research marks a significant stride towards automating culinary processes, showcasing the potential for robots to assist in domestic kitchens and revolutionize the process of food preparation.

Kitchen Artist: Precise Control of Liquid Dispensing for Gourmet Plating

Manipulating liquid is widely required for many tasks, especially in cooking. A common way to address this is extruding viscous liquid from a squeeze bottle. In this work, our goal is to create a sauce plating robot, which requires precise control of the thickness of squeezed liquids on a surface. Different liquids demand different manipulation policies. We command the robot to tilt the container and monitor the liquid response using a force sensor to identify liquid properties. Based on the liquid properties, we predict the liquid behavior with fixed squeezing motions in a data-driven way and calculate the required drawing speed for the desired stroke size. This open-loop system works effectively even without sensor feedback. Our experiments demonstrate accurate stroke size control across different liquids and fill levels. We show that understanding liquid properties can facilitate effective liquid manipulation. More importantly, our dish garnishing robot has a wide range of applications and holds significant commercialization potential.

RobotSweater: Scalable, Generalizable, and Customizable Machine-knitted Tactile Skins for Robots

RobotSweater is a machine-knitted pressure-sensitive low-cost tactile skin that is scalable, generalizable, and customizable. In this work, we design and fabricate a parameterized multi-layer tactile skin using off-the-shelf yarns with a programmable industrial knitting machine. We characterize our tactile skins to show their robust contact detection, multi-contact localization, and pressure sensing capability. Using our tactile skins, we demonstrate closed-loop control with tactile feedback for human lead-through control of a robot arm and human-robot interaction with a mobile robot.

Taxim: An Example-based Simulation Model for GelSight Tactile Sensors

Taxim is a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight. Our simulation framework is the first to incorporate marker motion field simulation together with the optical simulation. We simulate the optical response to the deformation with a polynomial lookup table. This table maps the deformed geometries to pixel intensity sampled by the embedded camera. We apply the linear elastic deformation theory and the superposition principle to simulate the surface markers’ motion that is caused by the surface stretch of the elastomer. The example-based approach requires less than 100 data points from a real sensor to calibrate the simulator and enables the model to easily migrate to other GelSight sensors or their variations.

Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing

Simulation is useful for data-driven robot manipulation tasks by providing prototyping platform and unlimited data. We integrate Taxim, a simulation model for GelSight tactile sensors into a physics-based robot simulator, and model the physics of contact as a bridge in between. We leverage it on a sim-to-real transfer learning to predict the grasp stability where we calibrate our physics simulator of robot dynamics, contact model, and tactile optical simulator with real-world data, and then demonstrate the effectiveness of our system on on various objects.

Understanding Dynamic Tactile Sensing for Liquid Property Estimation

We propose a new way of thinking about dynamic tactile sensing: by building a lightweight data-driven model based on the simplified physical principle. The liquid in a bottle will oscillate after a perturbation. We propose a simple physics-inspired model to explain this oscillation and use a high-resolution tactile sensor GelSight to sense it. Specifically, the viscosity and the height of the liquid determine the decay rate and frequency of the oscillation. We then train a Gaussian Process Regression model on a small amount of the real data to estimate the liquid properties. Experiments show that our model can classify three different liquids with 100% accuracy. The model can estimate volume with high precision and even estimate the concentration of sugar-water solution. It is data-efficient and can easily generalize to other liquids and bottles.

Tactile Optical Simulation

Tactile sensing has seen rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and robot-human interaction. However, the simulation of tactile sensors is still a challenge. In this project, we develop optical simulation techniques which can be used for novel sensor design and data-driven experiments using physically accurate material models and raytracing.

Improving Grasp Stability using Tactile Feedback

Grasping is one of the prime modality of manipulation for robotics. External vision sensors like RGBD cameras have traditionally been used to guide robots to perform manipulation tasks such as pick-and-place. However, these vision sensors often are positioned away from the point of grasp and provide less information about the success/failure of grasp and the mode of failure. In this work, we address one of such failures, namely failure due to the rotation of objects about the grasp point. If objects are grasped at points away from their center of gravity, they undergo rotation and this leads to grasping failure. As this rotation happens at the local region of the gripping point, it is challenging for vision sensors to detect and measure the rotation.

ShapeMap 3-D: Efficient shape mapping through dense touch and vision

Knowledge of 3-D object shape is important for robot manipulation, but may not be readily available in unstructured environments. We propose a framework that incrementally reconstructs tabletop 3-D objects from a sequence of tactile images and a noisy depth-map. Our contributions include: ( i ) recovering local shape from GelSight images, learned via tactile simulation ( ii ) incremental shape mapping through inference on our Gaussian process spatial graph (GP-SG). We demonstrate visuo-tactile mapping in both our simulated and real-world datasets

PoseIt: A Visual-Tactile Dataset of Holding Poses for Grasp Stability Analysis

When humans grasp objects in the real world, we often move our arm to hold the object in a different pose where we can use it. In contrast, typical lab settings only study the stability of the grasp immediately after lifting, without any subsequent re-positioning of the arm. However, an object’s stability could vary widely based on its holding pose, as the gravitational torque and gripper contact forces could change completely. To facilitate the study of how holding poses affect grasp stability, we present PoseIt, a novel multi-modal dataset that contains data collected from a full cycle of grasping an object, re-positioning the arm to one of the sampled poses, and shaking the object. Using data from PoseIt, we can formulate and tackle the task of predicting whether a grasped object is stable in a particular held pose. We train an LSTM classifier which achieves 85% accuracy on the proposed task, and our experimental results show that our classifiers can also generalize to unseen objects and poses. Finally, we compare different tactile sensors for the stability prediction task, demonstrating that the classifier performs better when trained on GelSight data than data collected from the WSG-DSA pressure array sensor PoseIt will be publicly released.

Soft Robotic Sensing

Soft bodies and robots provide a unique advantage for contact-rich tasks, due to their innate flexibility and compliance. However, due to the extensive contact that these bodies have with objects that they interact with, characterizing and measuring the deformation and contact force between them is critical to controlling and properly actuating soft robots. In this work, we aim to design and construct a soft robotic hand with fingers that will integrate the features of the GelSight tactile sensor, through components such as embedded cameras, patterned surfaces, and a new illumination system. The fingers of this hand will combine the unique features and structure of the GelSight tactile sensor, which measures surface normals and reconstructs height maps of contact surfaces in order to provide measurements and information on 3D geometry and contact force at high spatial resolutions, with the compliance and flexibility of soft bodies, which is beneficial for dexterous grasping and manipulation.