Continuous Image Generation From Low-Update-Rate Images and Physical Sensors Through a Conditional GAN for Robot Teleoperation

Abstract

When a robot is teleoperated, its operator control is based on transmitted images. Network limitations and/or a remote distance usually cause delays or interruptions of the image transmission, which is one of the reasons for the instability of teleoperation systems. In this article, we propose a high-update-rate image generation method using past low update image and current grip position and electrical motor current of gripper received by sensors during teleoperation via a conditional generative adversarial network. The main challenge is that such a network can generate current high-update-rate images from past low-update-rate one, the current high-update-rate grip force, and the grip angle. We equipped a robot gripper with a camera and a grip force sensor and collected a large data set of robot vision, grip force, and grip angle sequences; objects with deformation, including irregular deformation, and rigid objects were tested in the experiment to verify the possibility of high-update-rate image generation under various grip conditions. We found that the proposed network allows the generation of current images with high update rate.