SCI ADV-Tai Yanlong's research group proposed an artificial remote sensing haptic device with three-dimensional depth perception

The biosensorimotor system can obtain external information and transmit it to the internal nervous system through electrochemical signals, which in turn generates motor feedback and constructs interaction with the external environment [1].

On this basis, emerging biomimetic devices share many functional similarities with their biological counterparts and bridge the gap
between artificial and biological systems through a common computational and processing language.
Among them, the bionics of sensory functions are of great significance for human rehabilitation and the development of new artificial somatosensory systems with high-order functions/multimodalities, and are crucial for the development of prosthetics, wearable devices, intelligent robots and human-robot collaboration [2]
.
Among them, tactile electronics are very influential in enhancing human-robot collaboration (HMC), which mainly mimics the perception of mechanical movement of external objects by biological skin [3-6], including static recognition [location or distribution of contact points].
, the size of the contact area, the surface topography of the contact object, etc.
] and dynamic identification [vertical pressure, horizontal sliding or friction], etc
.
However, most of this sensing stays on two-dimensional (2D) skin surfaces involving contact/pressure, resulting in limited transmission of external information and inevitable contact damage and power consumption [7]
。 For contactless three-dimensional (3D) spatial perception, traditional methods such as laser, infrared, ultrasound, etc.
are used
.
However, due to the disadvantages of complex device layout, difficult function integration, and high cost, it is limited to its further application in the field of HMC [8].

On October 26, 2022, Tai Yanlong, researcher Li Guanglin, Shenzhen Institute of Advanced Technology-Neural Engineering Center, Chinese Academy of Sciences, and organic electronics research at Max Planck Institute for Polymer Research, Germany Team Prof.
Paschalis Gkoupidenis collaborated in Science Advances to present a presentation titled "An artificial remote tactile.
" device with 3D depth-of-field sensation", assistant researcher Shanshan Zhu is the first author
.
In this work, the research team proposed a stretchable self-powered three-dimensional remote tactile device (3D-RTD) that passes through a novel heterostructure Realize the perception of the mechanical movement of external objects in the depth-of-field (DOF) direction.

The device can establish an accurate logic relationship
with the DOF motion of external objects by sensing the positive/negative signal, frequency and amplitude of the signal.
Its perception mechanism is revealed by electrostatic field theory and multiphysics simulations, and its perceptual performance is verified
by a hybrid system of micro/macro interactions.
Finally, as a neural interface patch, the perception enhancement and auxiliary interaction functions of 3D-RTD were demonstrated in obstacle avoidance scenarios, that is, rats walked in a dim environment, felt potential dangers in a non-contact state and transmitted signals to the brain, and at the same time action feedback occurred to achieve sensation-perception- Interact with the whole process, which is not possible with traditional two-dimensional contact sensors
.
This work demonstrates the scene connection and logic recognition capabilities of 3D-RTD, which can be combined with bioperception, providing a new option
for multimodal neuromorphic devices and brain-like intelligence.

As a representative of human perceptual intelligence, the tactile feedback of organisms begins from the skin's perception of contact with external objects (Figure 1A), that is, the mechanical deformation of the skin in the contact area induces the tactile body to transmit the corresponding potential signals to the central nervous system, which in turn triggers the sensorimotor neural circuits to form tactile feedback
.
Based on similar signal transmission loops and principles, the authors propose a self-powered 3D-RTD and three-dimensional remote tactile system (3D-RTS).
, with enhanced perceptual performance
compared to 2D tactile sensing of living organisms.
The synergistic operation of biomimetic sensory systems and biolocomotion systems (Figure 1B), specifically, the non-contact depth-of-field mechanical movement of the hand (external object) triggers the virtual synapse 3D-RTD, which generates an electromotive force output
based on the response of the heterogeneous interface to electrostatic field polarization and electron transfer.
Subsequently, the generated potential changes are transmitted to the muscles of the organism, realizing the connection and information transmission
between the artificial and biological worlds.

Figure 1 A three-dimensional depth-of-field tactile device that transcends the function of the skin

(Source: SS.
Zhu.
et al.
, Sci Adv, 2022).

To evaluate the information transmission and connectivity function of 3D-RTD as a virtual synapse, it was integrated
with the skeletal muscle system of rats.
When the DOF movement of the external moving object triggers the 3D-RTD, the artificial system activates the hind limb muscles of the rat to produce movement
.
Two systems are presented: (I) 3D Remote Sensing Haptic System (3D-RTS) and (II) Stimulus and Force Measurement System (Figure 2).

Figure 2 Artificial 3D remote sensing haptic system (3D-RTS) is integrated into the neuromuscular system of
living organisms.

(Source: SS.
Zhu.
et al.
, Sci Adv, 2022).

To demonstrate the performance of 3D-RTS in sensory enhancement and interaction, this study presents an obstacle avoidance scenario
related to brain activity.
People who live in dimly lit environments or have visual impairments are at high
risk of collisions and falls.
Traditional mobility aids such as crutches still have limitations: obstacles on the ground can only be detected by contact (Figure 4A).

However, the danger of hanging in the air is easy to ignore, and the risk of collision cannot be completely eliminated
.
From another perspective, this group of people is more likely to develop perceptual skills/mapping
of surrounding non-contact objects.
Therefore, it is very important
to enhance their perception of the outside world with the help of artificial devices.
The rats were loaded with 3D-RTD neural interface patches on their backs and stimulation electrodes on their right hindlimbs to walk in a dimly lit runway with polyvinylidene fluoride (PVDF) film hanging from the ceiling in the center of the runway as a non-contact obstacle.

As the rat walked forward through a pre-set non-contact barrier, it sensed "danger" through 3D-RTS and generated real-time sensory feedback, gait changes, and brain-determined action feedback (Figure 4B-F).

。 A complete sensory-perceive-interactive process
is presented.

Figure 3 Application of three-dimensional depth-of-field tactile system (artificial-biological hybrid system) in obstacle avoidance, showing the complete sensory-perception-interaction process
.

(Source: SS.
Zhu.
et al.
, Science Advances, 2022).

In summary, the authors demonstrate a stretchable self-powered 3D-RTD, based on a novel heterostructure, which is capable of recognizing the depth of field motion
of external objects in three-dimensional space.
It can logically identify the approach-away of moving objects at a perceived distance of up to 20 cm and a frequency from 0.
0625 to 5 Hz
.
Based on electrostatic field theory and COMSOL finite element analysis, the relevant mechanism is revealed, and the perceived performance
of the material is optimized by adjusting the dielectric constant.
Based on the characteristics of 3D-RTD as a virtual synapse in signal transmission and connection function, the author's future work goal is to study the mechanism of heterostructured device interaction to achieve more functions
close to real synapses.
In addition, by constructing an artificial-biological integrated tactile system, this work synergistically realizes the microscopic interaction between virtual and biological synaptic electrons/ions, as well as the macroscopic interaction between the three-dimensional mechanical motion of external objects and the rat motor feedback, to verify the feasibility
of 3D-RTD as a neuromorphic device.
Finally, 3D-RTD, as a neural interface patch for sensory replacement and sensory enhancement, is applied in an obstacle avoidance scenario: rats equipped with 3D-RTDs produce a complete process
of sensory-perception-interaction with non-contact obstacles in a dimly lit runway.
The authors hope that this work can further promote the development of a series of applications related to human-computer fusion, especially in the field
of brain-like and hybrid intelligence-based interactions.