CSM-SR: Conditional Structure-Informed Multi-Scale GAN for Scientific Image Super-Resolution

The rapid advancements in scientific imaging, particularly in fields such as material science, medical imaging, and nanotechnology, have underscored the need for highly detailed images at the micro and nano scales. Traditional microscopic imaging techniques often face significant resolution limitations, leading to increased costs and insufficient detail for precise scientific analysis. Image Super-Resolution (SR) techniques have emerged as a promising solution, offering the ability to recover high-resolution images from low-resolution counterparts through advanced image processing methods. While deep learning-based SR models like VDSR--Very Deep Super-Resolution--, EDSR--Enhanced Deep Residual Networks for Single Image Super-Resolution--, GAN based SRGAN--Super-Resolution Generative Adversarial Network--, ESRGAN--Enhanced Super-Resolution Generative Adversarial Networks--, SPSR--Structure-Preserving Super Resolution with Gradient Guidance--, variational autoencoder based DSR-VAE--Deep Super-Resolution Variational Autoencoder-- and transformer based SwinIR--Swin Transformer for Image Restoration--, HMA-Net--, have demonstrated state-of-the-art performance in enhancing image resolution, they often fail to preserve the structural integrity crucial for accurate scientific analysis. 

To address this gap, the proposed approach integrates structural information using advanced conditional generative adversarial networks (cGANs) and a structure-informed convex loss function. This methodology is designed to improve both the visual quality and structural accuracy of super-resolved images. The research seeks to develop a super-resolution technique that not only enhances image quality but also preserves the structural integrity of image components, thereby facilitating more precise and realistic scientific analyses in fields such as material science and medical imaging.

Project Phases

1. Data Collection and Preprocessing:
Objective: Collect and preprocess a comprehensive set of scanning electron microscopy (SEM) images, including the SEM dataset, Hierarchical dataset, Majority dataset, and 100% dataset.

Figure: Electron microscope images showcasing various textures, patterns, and structures at microscopic scales.

Outcome: A clean and well-annotated dataset ready for training and evaluation purposes.

2. Evaluation of State-of-the-Art Models:
Objective: Conduct experiments using existing super-resolution models (SRGAN, ESRGAN, VDSR, SPSR, dSRVAE, SwinIR) on the collected datasets to establish baseline performance metrics.

Outcome: Experimental results providing insights into the strengths and limitations of current models.

3.1: Development of the Advance-Conditional Multi-Scale GAN Model:
1. Model Architecture Design
Generator Design: The generator model comprises advanced super-resolution residual blocks and attention blocks, coupled with multi-scale processing to enhance feature extraction capabilities.
Discriminator Design: The discriminator employs a combination of residual blocks and PatchGAN-style convolutions to effectively differentiate between real and generated images.

2. Integration of Structure-Informed Loss Function
Loss Components:

• Adversarial Loss: Utilizes binary cross-entropy to train the GAN, ensuring realistic image generation.
• Perceptual Loss: Employs a pre-trained VGG19 model to compare high-level features between the ground truth and generated images.
• Gradient Loss: Computes the difference in Sobel edges between the ground truth and generated images to maintain edge information.
• Second-Order Gradient Loss: Extends the gradient loss by considering second-order gradients, further enhancing edge preservation.
• Total Variation Loss: Encourages spatial smoothness in the generated images.
• Structural Similarity Loss: Measures the structural similarity between ground truth and generated images.

Total Loss Calculation: Combines all the above loss components into a single, comprehensive loss function for training.

4. Model Training and Evaluation
Objective: Train the SINSR model using the structure-informed loss function and evaluate its performance on the collected dataset.
Outcome: Detailed performance metrics, comparisons with baseline models, and qualitative analyses of the generated high-resolution images.

5. Advanced Model Development
Objective: Integrate additional architectural advancements, such as incorporating a Gradient Branch alongside the super-resolution branch, operating in parallel.
Outcome: An enhanced SINSR model architecture that significantly improves performance in maintaining structural integrity in high-resolution images.

6. Final Evaluation and Analysis
Objective: Conduct a comprehensive evaluation of the final SINSR model, incorporating both quantitative metrics and qualitative analyses.

1. Quantitative Metrics

• PSNR
• SSIM
• LPIPS

Quantitative comparison (average PSNR/SSIM/LPIPS) with state-of-the-artmethods for Scientific Image SR on the SEM Dataset.Best and second best performance are in red,and blue, colors, respectively. 

Comparison of super-resolution methods

Outcome: A fully developed and evaluated CSM-SR model, ready for application in downstream scientific analyses.

🎓 Role	👲 Name	🔗 GitHub	🔗 LinkedIn
Project Lead	Dr. Uthayasanker Thayasivam
Mentor	Brinthan, Vithurabhiman
Contributor	Randika Prabashwara
Contributor	Gayani Wickramarathna
Contributor	Oshadi Perera