Localization of Diffusion Model-Based Inpainting through Inter-Intra Similarity of Frequency Features
Image and Vision Computing
Recently, the enhanced abilities of diffusion models have led to more realistic inpainting results, which raises the potential for criminal activity through image forgery. In this study, we explore the detection of inpainted images generated by a diffusion model. We propose a method for inpainting localization using an inter-intra similarity (IIS) module based on image frequency features. The proposed IIS module learns the inter-patch relationship through the learnable frequency filter and subsequently covers the intra-patch relationship through the self-similarity operation. We provide the Diffusion Model Inpainting Dataset (DMID), a benchmark dataset comprising inpainted images using four different diffusion models and three types of masks. Additionally, a test dataset that includes three sampling steps is provided. We validated the effectiveness of our proposed approach by conducting comparative tests with existing forgery detectors using our dataset and testing the robustness of JPEG compression. Additionally, we tested our proposed method on datasets with different sampling step sizes. Our work provides a starting point for research on the detection of inpainting-based forgery using diffusion models. Additionally, by openly releasing the dataset, we offer an opportunity to advance future in-depth research related to forensics.
This work was supported in part by the National Research Foundation (NRF) of Korea funded by the Korean Government (MSIT)(NRF-2022R1A4A1033600).
If you want to cite our paper and code, you can use a BibTeX code here:
@article{lee2024localization,
title={Localization of diffusion model-based inpainting through the inter-intra similarity of frequency features},
author={Lee, Seung-Lee and Kang, Minjae and Hou, Jong-Uk},
journal={Image and Vision Computing},
pages={105138},
year={2024},
publisher={Elsevier}
}