Li Dong's research group and collaborators developed a rationalized deep learning super-resolution microscopic imaging method

In recent years, computational superresolution methods represented by deep learning have been able to improve the resolution or signal-to-noise ratio of microscopic images without losing other imaging performance, showing great application prospects
.
However, in view of the image requirements of high fidelity and quantitative analysis required for biomedical research, there are three common problems in current deep learning microscopic imaging methods:

(1) Limited by the spectral-bias problem of deep learning, the output image resolution cannot reach the ground truth level;

(2) Limited by the hyper-resolution reconstruction and the ill-posed problem of denoising problems and the uncertainty of the neural network model (model-uncertainty), the authenticity of the reconstruction or prediction results cannot be guaranteed;

(3) The training of deep neural networks requires a large amount of data, and the collection of high-quality training data is extremely difficult or even impossible to achieve
in many application scenarios.

Due to the above bottlenecks, although the current research and development of deep learning microscopic imaging methods is in full swing and shows great potential to exceed the limits of traditional imaging performance, the above bottlenecks hinder the widespread use
of existing deep learning super-resolution or denoising methods in biological microscopic imaging experiments.

On October 6, 2022, Li Dong's research group at the Institute of Biophysics, Chinese Academy of Sciences, the Department of Automation of Tsinghua University, the Institute of Brain and Cognitive Sciences of Tsinghua University, and the Dai Qionghai Research Group of Tsinghua-IDG/McGovern Institute for Brain Sciences joined hands with Dr.
Jennifer Lippincott-Schwartz of the Howard Hughes Institute of Medicine (HHMI) in Nature Biotechnology The journal published a paper in Article form titled "Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes", proposing a set of rationalized deep learning.
rDL) microscopic imaging technology framework, the optical imaging model and physical priori and neural network structure design are integrated, the network training and prediction process are rationalized, so as to achieve high-performance, high-fidelity microscopic image denoising and super-resolution reconstruction, and combined with the multimodal structured light illumination microscope (Multi-SIM) and high-speed lattice light microscope (LLSM) independently developed and built by the laboratory, the traditional TIRF/GI-SIM, 3D-SIM, The imaging speed/timeline of LLS-SIM and LLSM has been increased by more than 30 times, achieving the current international fastest (684Hz) and the longest imaging time range (up to 3 hours, 60,000 time points or more) of living cell imaging performance, the first high-speed swing cilia (> 30Hz) transporter (IFT) transport behavior and the liquid-liquid phase of nucleolar fluid during complete cell division separation) process for fast, multicolored, long-range, super-resolution observations
.
Nature Biotechnology magazine also published a Research Briefing article on this work
.

Figure 1 Rationalization of deep learning super-resolution microscopy imaging neural network architecture

Specifically, the rationalized deep learning structured light super-resolution reconstruction architecture (rDL SIM) proposed by Li Dong/Dai Qionghai research team is different from the end-to-end (end-to-end) training mode of the existing super-resolved neural network model, but adopts a step-by-step reconstruction strategy, and first uses the proposed fusion imaging physical model and the structured light illumination priori neural network to denoiserate and enhance the original SIM image.
The SIM reconstruction is then performed by the classical parsing algorithm to obtain the final superresolved image
.
Compared with the super-resolved reconstruction neural network model DFCAN/DFGAN[1] proposed by the team in Nature Methods last year, rDL SIM can reduce the uncertainty of super-resolution reconstruction results by 3 to 5 times, and achieve higher fidelity and reconstruction quality.
Compared to other denoising algorithms such as CARE[2], the rDL SIM perfectly restores Moiré stripes modulated in the original image and enhances the high-frequency information by a factor of 10
.

In addition, for wide-field illumination or spot scanning imaging modes such as lattice photomicroscopy and confocal microscopy, the research team proposed a learnable Fourier domain noise suppression module (FNSM) that can use OTF information to adaptively filter out
noise in microscopic images 。 Then, they construct the channel attention denoising neural network architecture embedded in FNSM, and based on the spatio-temporal continuity of the microscopic imaging data itself, they propose a spatio-temporal interleaved sampling self-supervised training strategy (TiS/SiS-rDL), which can achieve the training of denoising neural networks comparable to supervised learning without additional acquisition of training data and no need to ensure that the time series data has time continuity, and solve the problem that high-quality training data in actual biological imaging experiments is difficult to obtain
。

Figure 2 Application overview of rationalized deep learning super-resolution microscopy imaging methods

The rationalized deep learning super-resolution microscopy imaging method can be applied to a variety of microscopic imaging modes including 2D-SIM, 3D-SIM, LLSM, etc.
, providing high-resolution, high-fidelity microscopic image reconstruction performance, compared with traditional methods can improve the imaging time range of up to 30 times and imaging speed
of 10 times 。 With the help of rDL imaging technology, the research team conducted many super-resolution in vivo imaging experiments that could not be carried out by past imaging methods, and together with HHMI, Dr.
Lippincott-Schwartz, Zhu Xueliang, researcher of the Center of Excellence in Molecular Cell Science of the Chinese Academy of Sciences, and He Kangmin, researcher of the Institute of Genetic Development of the Chinese Academy of Sciences, discussed its potential biological significance in depth, including:

(1) The adherent growth process of U2OS cells dripped on the slide was observed in two colors, long time (more than 1 hour), and super resolution (97nm resolution), which clearly and truly recorded the kinetic phenomena of cell adhesion and migration, without interfering with this long and fragile life process;

(2) Continuous superresolution observations of up to 60,000 time points at the fastest 684Hz imaging rate of high-speed swing cilia without significant photobleaching or cell activity damage in the process, and statistical analysis of ciliary swing mode and frequency;

(3) Ultrafast and super-resolution bichromatic imaging of swing cilia and ciliary transporters (IFT) was carried out, which for the first time revealed a variety of new behaviors such as collision, recombination and U-turn of IFT during travel;

(4) Through bicolor, long-term and super-resolution imaging of cCAS-DNA and ER, the directional movement, steering or diffusion of cCAS-DNA in the process of maintaining continuous contact with ER was observed, and the understanding of the interaction mechanism between membranous organelles and membraneless organelles was expanded;

(5) Three-dimensional super-resolution in vivo imaging of nucleolin phosphate protein (NPM1), RNA polymerase I subunit RPA49 and chromatin (H2B) in the process of HeLa cell division (12 sec acquisition interval, more than 2.
5 hours) was carried out, and the continuous observation of the three-dimensional super-resolution living body of NPM1 and RPA49 in the complete mitotic process was realized for the first time, revealing the nucleolar formation and NPM1 during the mitosis process.
Phase transition and interaction of RPA49 two membraneless subcellular structures;

(6) The Golgi body was continuously photographed at a frame rate of 10 Hz for up to 10,000 time points, and the three-color, high-speed (order of seconds) and ultra-long time range (order of hours, > 1000 time points) of the subcellular structures of the endoplasmic reticulum, lysosomes and mitochondria of the complete cell division process were realized, and the uniform distribution mechanism of
organelles in the progeny cells during the process of cell mitosis was deeply explored.

In summary, through the cross-innovation of artificial intelligence algorithms and optical microscopic imaging technology, the collaborative team of Li Dong and Dai Qionghai proposed a rationalized deep learning super-resolution microscopic imaging framework, which solved the problems of resolution loss, prediction uncertainty and training set of existing deep learning imaging methods, and can provide more than 30 times the improvement of imaging speed and time course for a variety of in vivo microscopic imaging modes, and provide important research tools for the development of cell biology, developmental biology, neuroscience and other fields.
The research ideas of cross-innovation and combination of soft and hard artificial intelligence algorithms and optical imaging principles adhered to and advocated by the research team have also opened up a new technical path
for the development of modern optical microscopic imaging.

Qiao Chang, a postdoctoral fellow in the Department of Automation of Tsinghua University, Li Di, a senior engineer at the Institute of Biophysics of the Chinese Academy of Sciences, and Liu Yong and Zhang Siwei, assistant researchers, are the co-first authors of the paper, and Li Dong, a researcher at the Institute of Biophysics of the Chinese Academy of Sciences, Dai Qionghai, a professor of the Department of Automation of Tsinghua University, and Jennifer Lippincott-Schwartz, a researcher at the Howard Hughes Institute of Medicine, are co-corresponding authors
.
This research has been funded
by the National Natural Science Foundation of China, the Ministry of Science and Technology, the Chinese Academy of Sciences, the China Postdoctoral Science Foundation, the Tencent Science Exploration Award, and the Shuimu Scholar Program of Tsinghua University.

Article link:

(Contributed by: Li Dong Research Group)