Yuma Kinoshita

Hitoshi Kiya, Ryota Iijima, MaungMaung Aprilpyone et al.

In this paper, we propose a combined use of transformed images and vision transformer (ViT) models transformed with a secret key. We show for the first time that models trained with plain images can be directly transformed to models trained with encrypted images on the basis of the ViT architecture, and the performance of the transformed models is the same as models trained with plain images when using test images encrypted with the key. In addition, the proposed scheme does not require any specially prepared data for training models or network modification, so it also allows us to easily update the secret key. In an experiment, the effectiveness of the proposed scheme is evaluated in terms of performance degradation and model protection performance in an image classification task on the CIFAR-10 dataset.

Zheng Qi, AprilPyone MaungMaung, Yuma Kinoshita et al.

In this paper, we propose a privacy-preserving image classification method that is based on the combined use of encrypted images and the vision transformer (ViT). The proposed method allows us not only to apply images without visual information to ViT models for both training and testing but to also maintain a high classification accuracy. ViT utilizes patch embedding and position embedding for image patches, so this architecture is shown to reduce the influence of block-wise image transformation. In an experiment, the proposed method for privacy-preserving image classification is demonstrated to outperform state-of-the-art methods in terms of classification accuracy and robustness against various attacks.

Teruaki Akazawa, Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel template matching method with a white balancing adjustment, called N-white balancing, which was proposed for multi-illuminant scenes. To reduce the influence of lighting effects, N-white balancing is applied to images for multi-illumination color constancy, and then a template matching method is carried out by using adjusted images. In experiments, the effectiveness of the proposed method is demonstrated to be effective in object detection tasks under various illumination conditions.

Shinji Yamashita, Yuma Kinoshita, Hitoshi Kiya

This paper introduces a highly efficient algorithm capable of jointly estimating scale and rotation between two images with sub-pixel precision. Image alignment serves as a critical process for spatially registering images captured from different viewpoints, and finds extensive use in domains such as medical imaging and computer vision. Traditional phase-correlation techniques are effective in determining translational shifts; however, they are inadequate when addressing scale and rotation changes, which often arise due to camera zooming or rotational movements. In this paper, we propose a novel algorithm that integrates scale and rotation estimation based on the Fourier transform in log-polar coordinates with a cross-correlation maximization strategy, leveraging the auxiliary function method. By incorporating sub-pixel-level cross-correlation our method enables precise estimation of both scale and rotation. Experimental results demonstrate that the proposed method achieves lower mean estimation errors for scale and rotation than conventional Fourier transform-based techniques that rely on discrete cross-correlation.

Yuma Kinoshita, Hitoshi Kiya

We propose a novel scene-segmentation-based exposure compensation method for multi-exposure image fusion (MEF) based tone mapping. The aim of MEF-based tone mapping is to display high dynamic range (HDR) images on devices with limited dynamic range. To achieve this, this method generates a stack of differently exposed images from an input HDR image and fuses them into a single image. Our approach addresses the limitations of MEF-based tone mapping with existing segmentation-based exposure compensation, which often result in visually unappealing outcomes due to inappropriate exposure value selection. The proposed exposure compensation method first segments the input HDR image into subregions based on luminance values of pixels. It then determines exposure values for multi-exposure images to maximize contrast between regions while preserving relative luminance relationships. This approach contrasts with conventional methods that may invert luminance relationships or compromise contrast between regions. Additionally, we present an improved technique for calculating fusion weights to better reflect the effects of exposure compensation in the final fused image. In a simulation experiment to evaluate the quality of tone-mapped images, the MEF-based tone mapping with the proposed method outperforms three typical tone mapping methods including conventional MEF-based one, in terms of the tone mapped image quality index (TMQI).

Hitoshi Kiya, AprilPyone MaungMaung, Yuma Kinoshita et al.

This article presents an overview of image transformation with a secret key and its applications. Image transformation with a secret key enables us not only to protect visual information on plain images but also to embed unique features controlled with a key into images. In addition, numerous encryption methods can generate encrypted images that are compressible and learnable for machine learning. Various applications of such transformation have been developed by using these properties. In this paper, we focus on a class of image transformation referred to as learnable image encryption, which is applicable to privacy-preserving machine learning and adversarially robust defense. Detailed descriptions of both transformation algorithms and performances are provided. Moreover, we discuss robustness against various attacks.

Yuma Kinoshita, Hitoshi Kiya

We propose a novel intrinsic image decomposition network considering reflectance consistency. Intrinsic image decomposition aims to decompose an image into illumination-invariant and illumination-variant components, referred to as ``reflectance'' and ``shading,'' respectively. Although there are three consistencies that the reflectance and shading should satisfy, most conventional work does not sufficiently account for consistency with respect to reflectance, owing to the use of a white-illuminant decomposition model and the lack of training images capturing the same objects under various illumination-brightness and -color conditions. For this reason, the three consistencies are considered in the proposed network by using a color-illuminant model and training the network with losses calculated from images taken under various illumination conditions. In addition, the proposed network can be trained in a self-supervised manner because various illumination conditions can easily be simulated. Experimental results show that our network can decompose images into reflectance and shading components.

Teruaki Akazawa, Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel white balance adjustment, called "spatially varying white balancing," for single, mixed, and non-uniform illuminants. By using n diagonal matrices along with a weight, the proposed method can reduce lighting effects on all spatially varying colors in an image under such illumination conditions. In contrast, conventional white balance adjustments do not consider the correcting of all colors except under a single illuminant. Also, multi-color balance adjustments can map multiple colors into corresponding ground truth colors, although they may cause the rank deficiency problem to occur as a non-diagonal matrix is used, unlike white balancing. In an experiment, the effectiveness of the proposed method is shown under mixed and non-uniform illuminants, compared with conventional white and multi-color balancing. Moreover, under a single illuminant, the proposed method has almost the same performance as the conventional white balancing.

Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel method for separately estimating spectral distributions from images captured by a typical RGB camera. The proposed method allows us to separately estimate a spectral distribution of illumination, reflectance, or camera sensitivity, while recent hyperspectral cameras are limited to capturing a joint spectral distribution from a scene. In addition, the use of Bayesian inference makes it possible to take into account prior information of both spectral distributions and image noise as probability distributions. As a result, the proposed method can estimate spectral distributions in a unified way, and it can enhance the robustness of the estimation against noise, which conventional spectral-distribution estimation methods cannot. The use of Bayesian inference also enables us to obtain the confidence of estimation results. In an experiment, the proposed method is shown not only to outperform conventional estimation methods in terms of RMSE but also to be robust against noise.

Teruaki Akazawa, Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel multi-color balance adjustment for color constancy. The proposed method, called "n-color balancing," allows us not only to perfectly correct n target colors on the basis of corresponding ground truth colors but also to correct colors other than the n colors. In contrast, although white-balancing can perfectly adjust white, colors other than white are not considered in the framework of white-balancing in general. In an experiment, the proposed multi-color balancing is demonstrated to outperform both conventional white and multi-color balance adjustments including Bradford's model.

Teruaki Akazawa, Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel multi-color balance method for reducing color distortions caused by lighting effects. The proposed method allows us to adjust three target-colors chosen by a user in an input image so that each target color is the same as the corresponding destination (benchmark) one. In contrast, white balancing is a typical technique for reducing the color distortions, however, they cannot remove lighting effects on colors other than white. In an experiment, the proposed method is demonstrated to be able to remove lighting effects on selected three colors, and is compared with existing white balance adjustments.

Takayuki Osakabe, Miki Tanaka, Yuma Kinoshita et al.

In this paper, we propose a novel CycleGAN without checkerboard artifacts for counter-forensics of fake-image detection. Recent rapid advances in image manipulation tools and deep image synthesis techniques, such as Generative Adversarial Networks (GANs) have easily generated fake images, so detecting manipulated images has become an urgent issue. Most state-of-the-art forgery detection methods assume that images include checkerboard artifacts which are generated by using DNNs. Accordingly, we propose a novel CycleGAN without any checkerboard artifacts for counter-forensics of fake-mage detection methods for the first time, as an example of GANs without checkerboard artifacts.

Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel convolutional neural network (CNN) that never causes checkerboard artifacts, for image enhancement. In research fields of image-to-image translation problems, it is well-known that images generated by usual CNNs are distorted by checkerboard artifacts which mainly caused in forward-propagation of upsampling layers. However, checkerboard artifacts in image enhancement have never been discussed. In this paper, we point out that applying U-Net based CNNs to image enhancement causes checkerboard artifacts. In contrast, the proposed network that contains fixed convolutional layers can perfectly prevent the artifacts. In addition, the proposed network architecture, which can handle both local and global features, enables us to improve the performance of image enhancement. Experimental results show that the use of fixed convolutional layers can prevent checkerboard artifacts and the proposed network outperforms state-of-the-art CNN-based image-enhancement methods in terms of various objective quality metrics: PSNR, SSIM, and NIQE.

Hiroki Ito, Yuma Kinoshita, Hitoshi Kiya

We propose a transformation network for generating visually-protected images for privacy-preserving DNNs. The proposed transformation network is trained by using a plain image dataset so that plain images are transformed into visually protected ones. Conventional perceptual encryption methods have a weak visual-protection performance and some accuracy degradation in image classification. In contrast, the proposed network enables us not only to strongly protect visual information but also to maintain the image classification accuracy that using plain images achieves. In an image classification experiment, the proposed network is demonstrated to strongly protect visual information on plain images without any performance degradation under the use of CIFAR datasets. In addition, it is shown that the visually protected images are robust against a DNN-based attack, called inverse transformation network attack (ITN-Attack) in an experiment.

Ayana Kawamura, Yuma Kinoshita, Takayuki Nakachi et al.

We propose a privacy-preserving machine learning scheme with encryption-then-compression (EtC) images, where EtC images are images encrypted by using a block-based encryption method proposed for EtC systems with JPEG compression. In this paper, a novel property of EtC images is first discussed, although EtC ones was already shown to be compressible as a property. The novel property allows us to directly apply EtC images to machine learning algorithms non-specialized for computing encrypted data. In addition, the proposed scheme is demonstrated to provide no degradation in the performance of some typical machine learning algorithms including the support vector machine algorithm with kernel trick and random forests under the use of z-score normalization. A number of facial recognition experiments with are carried out to confirm the effectiveness of the proposed scheme.

Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a fixed convolutional layer with an order of smoothness not only for avoiding checkerboard artifacts in convolutional neural networks (CNNs) but also for enhancing the performance of CNNs, where the smoothness of its filter kernel can be controlled by a parameter. It is well-known that a number of CNNs generate checkerboard artifacts in both of two process: forward-propagation of upsampling layers and backward-propagation of strided convolutional layers. The proposed layer can perfectly prevent checkerboard artifacts caused by strided convolutional layers or upsampling layers including transposed convolutional layers. In an image-classification experiment with four CNNs: a simple CNN, VGG8, ResNet-18, and ResNet-101, applying the fixed layers to these CNNs is shown to improve the classification performance of all CNNs. In addition, the fixed layer are applied to generative adversarial networks (GANs), for the first time. From image-generation results, a smoother fixed convolutional layer is demonstrated to enable us to improve the quality of images generated with GANs.

Warit Sirichotedumrong, Yuma Kinoshita, Hitoshi Kiya

This paper aims to evaluate the safety of a pixel-based image encryption method, which has been proposed to apply images with no visual information to deep neural networks (DNN), in terms of robustness against ciphertext-only attacks (COA). In addition, we propose a novel DNN-based COA that aims to reconstruct the visual information of encrypted images. The effectiveness of the proposed attack is evaluated under two encryption key conditions: same encryption key, and different encryption keys. The results show that the proposed attack can recover the visual information of the encrypted images if images are encrypted under same encryption key. Otherwise, the pixel-based image encryption method has robustness against COA.

Ayana Kawamura, Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel privacy-preserving machine learning scheme with encrypted images, called EtC (Encryption-then-Compression) images. Using machine learning algorithms in cloud environments has been spreading in many fields. However, there are serious issues with it for end users, due to semi-trusted cloud providers. Accordingly, we propose using EtC images, which have been proposed for EtC systems with JPEG compression. In this paper, a novel property of EtC images is considered under the use of z-score normalization. It is demonstrated that the use of EtC images allows us not only to protect visual information of images, but also to preserve both the Euclidean distance and the inner product between vectors. In addition, dimensionality reduction is shown to can be applied to EtC images for fast and accurate matching. In an experiment, the proposed scheme is applied to a facial recognition algorithm with classifiers for confirming the effectiveness of the scheme under the use of support vector machine (SVM) with the kernel trick.

Chihiro Go, Yuma Kinoshita, Sayaka Shiota et al.

This paper proposes a novel multi-exposure image fusion (MEF) scheme for single-shot high dynamic range imaging with spatially varying exposures (SVE). Single-shot imaging with SVE enables us not only to produce images without color saturation regions from a single-shot image, but also to avoid ghost artifacts in the producing ones. However, the number of exposures is generally limited to two, and moreover it is difficult to decide the optimum exposure values before the photographing. In the proposed scheme, a scene segmentation method is applied to input multi-exposure images, and then the luminance of the input images is adjusted according to both of the number of scenes and the relationship between exposure values and pixel values. The proposed method with the luminance adjustment allows us to improve the above two issues. In this paper, we focus on dual-ISO imaging as one of single-shot imaging. In an experiment, the proposed scheme is demonstrated to be effective for single-shot high dynamic range imaging with SVE, compared with conventional MEF schemes with exposure compensation.

Chihiro Go, Yuma Kinoshita, Sayaka Shiota et al.

We proposes a novel single-shot high dynamic range imaging scheme with spatially varying exposures (SVE) considering hue distortion. Single-shot imaging with SVE enables us to capture multi-exposure images from a single-shot image, so high dynamic range images can be produced without ghost artifacts. However, SVE images have some pixels at which a range supported by camera sensors is exceeded. Therefore, generated images have some color distortion, so that conventional imaging with SVE has never considered the influence of this range limitation. To overcome this issue, we consider estimating the correct hue of a scene from raw images, and propose a method with the estimated hue information for correcting the hue of SVE images on the constant hue plain in the RGB color space.

Artit Visavakitcharoen, Yuma Kinoshita, Hitoshi Kiya

Multi-exposure image fusion is a method for producing an image with a wide dynamic range by fusing multiple images taken under various exposure values. In this paper, we discuss color distortion included in fused images, and propose a novel color compensation method for multi-exposure image fusion. In the proposed method, an inverse camera response function (CRF) is estimated by using multi-exposure images, and then a high dynamic range (HDR) radiance map is recovered. The color information of the radiance map is applied to images fused by conventional multi-exposure imaging to correct the color distortion. The proposed method can be applied to any existing fusion approaches for improving the quality of the fused images.

Yuma Kinoshita, Hitoshi Kiya

In this paper, we propose a novel convolutional neural network (CNN) architecture considering both local and global features for image enhancement. Most conventional image enhancement methods, including Retinex-based methods, cannot restore lost pixel values caused by clipping and quantizing. CNN-based methods have recently been proposed to solve the problem, but they still have a limited performance due to network architectures not handling global features. To handle both local and global features, the proposed architecture consists of three networks: a local encoder, a global encoder, and a decoder. In addition, high dynamic range (HDR) images are used for generating training data for our networks. The use of HDR images makes it possible to train CNNs with better-quality images than images directly captured with cameras. Experimental results show that the proposed method can produce higher-quality images than conventional image enhancement methods including CNN-based methods, in terms of various objective quality metrics: TMQI, entropy, NIQE, and BRISQUE.

Warit Sirichotedumrong, Takahiro Maekawa, Yuma Kinoshita et al.

We present a novel privacy-preserving scheme for deep neural networks (DNNs) that enables us not to only apply images without visual information to DNNs for both training and testing but to also consider data augmentation in the encrypted domain for the first time. In this paper, a novel pixel-based image encryption method is first proposed for privacy-preserving DNNs. In addition, a novel adaptation network is considered that reduces the influence of image encryption. In an experiment, the proposed method is applied to a well-known network, ResNet-18, for image classification. The experimental results demonstrate that conventional privacy-preserving machine learning methods including the state-of-the-arts cannot be applied to data augmentation in the encrypted domain and that the proposed method outperforms them in terms of classification accuracy.

Chien-Cheng Chien, Yuma Kinoshita, Hitoshi Kiya

A novel method of contrast enhancement is proposed for underexposed images, in which heavy noise is hidden. Under low light conditions, images taken by digital cameras have low contrast in dark or bright regions. This is due to a limited dynamic range that imaging sensors have. For these reasons, various contrast enhancement methods have been proposed so far. These methods, however, have two problems: (1) The loss of details in bright regions due to over-enhancement of contrast. (2) The noise is amplified in dark regions because conventional enhancement methods do not consider noise included in images. The proposed method aims to overcome these problems. In the proposed method, a shadow-up function is applied to adaptive gamma correction with weighting distribution, and a denoising filter is also used to avoid noise being amplified in dark regions. As a result, the proposed method allows us not only to enhance contrast of dark regions, but also to avoid amplifying noise, even under strong noise environments.

Yuma Kinoshita, Hitoshi Kiya

We propose a novel method for adjusting luminance for multi-exposure image fusion. For the adjustment, two novel scene segmentation approaches based on luminance distribution are also proposed. Multi-exposure image fusion is a method for producing images that are expected to be more informative and perceptually appealing than any of the input ones, by directly fusing photos taken with different exposures. However, existing fusion methods often produce unclear fused images when input images do not have a sufficient number of different exposure levels. In this paper, we point out that adjusting the luminance of input images makes it possible to improve the quality of the final fused images. This insight is the basis of the proposed method. The proposed method enables us to produce high-quality images, even when undesirable inputs are given. Visual comparison results show that the proposed method can produce images that clearly represent a whole scene. In addition, multi-exposure image fusion with the proposed method outperforms state-of-the-art fusion methods in terms of MEF-SSIM, discrete entropy, tone mapped image quality index, and statistical naturalness.

Yuma Kinoshita, Hitoshi Kiya

In this paper, a novel inverse tone mapping method using a convolutional neural network (CNN) with LDR based learning is proposed. In conventional inverse tone mapping with CNNs, generated HDR images cannot have absolute luminance, although relative luminance can. Moreover, loss functions suitable for learning HDR images are problematic, so it is difficult to train CNNs by directly using HDR images. In contrast, the proposed method enables us not only to estimate absolute luminance, but also to train a CNN by using LDR images. The CNN used in the proposed method learns a transformation from various input LDR images to LDR images mapped by Reinhard's global operator. Experimental results show that HDR images generated by the proposed method have higher-quality than HDR ones generated by conventional inverse tone mapping methods,in terms of HDR-VDP-2.2 and PU encoding + MS-SSIM.

Yuma Kinoshita, Hitoshi Kiya

In this paper, a novel image enhancement network is proposed, where HDR images are used for generating training data for our network. Most of conventional image enhancement methods, including Retinex based methods, do not take into account restoring lost pixel values caused by clipping and quantizing. In addition, recently proposed CNN based methods still have a limited scope of application or a limited performance, due to network architectures. In contrast, the proposed method have a higher performance and a simpler network architecture than existing CNN based methods. Moreover, the proposed method enables us to restore lost pixel values. Experimental results show that the proposed method can provides higher-quality images than conventional image enhancement methods including a CNN based method, in terms of TMQI and NIQE.

Chien Cheng Chien, Yuma Kinoshita, Sayaka Shiota et al.

This paper proposes a novel image contrast enhancement method based on both a noise aware shadow-up function and Retinex (retina and cortex) decomposition. Under low light conditions, images taken by digital cameras have low contrast in dark or bright regions. This is due to a limited dynamic range that imaging sensors have. For this reason, various contrast enhancement methods have been proposed. Our proposed method can enhance the contrast of images without not only over-enhancement but also noise amplification. In the proposed method, an image is decomposed into illumination layer and reflectance layer based on the retinex theory, and lightness information of the illumination layer is adjusted. A shadow-up function is used for preventing over-enhancement. The proposed mapping function, designed by using a noise aware histogram, allows not only to enhance contrast of dark region, but also to avoid amplifying noise, even under strong noise environments.

Yuma Kinoshita, Sayaka Shiota, Hitoshi Kiya

This paper proposes a novel pseudo multi-exposure image fusion method based on a single image. Multi-exposure image fusion is used to produce images without saturation regions, by using photos with different exposures. However, it is difficult to take photos suited for the multi-exposure image fusion when we take a photo of dynamic scenes or record a video. In addition, the multi-exposure image fusion cannot be applied to existing images with a single exposure or videos. The proposed method enables us to produce pseudo multi-exposure images from a single image. To produce multi-exposure images, the proposed method utilizes the relationship between the exposure values and pixel values, which is obtained by assuming that a digital camera has a linear response function. Moreover, it is shown that the use of a local contrast enhancement method allows us to produce pseudo multi-exposure images with higher quality. Most of conventional multi-exposure image fusion methods are also applicable to the proposed multi-exposure images. Experimental results show the effectiveness of the proposed method by comparing the proposed one with conventional ones.

Yuma Kinoshita, Taichi Yoshida, Sayaka Shiota et al.

This paper proposes a novel multi-exposure image fusion method based on exposure compensation. Multi-exposure image fusion is a method to produce images without color saturation regions, by using photos with different exposures. However, in conventional works, it is unclear how to determine appropriate exposure values, and moreover, it is difficult to set appropriate exposure values at the time of photographing due to time constraints. In the proposed method, the luminance of the input multi-exposure images is adjusted on the basis of the relationship between exposure values and pixel values, where the relationship is obtained by assuming that a digital camera has a linear response function. The use of a local contrast enhancement method is also considered to improve input multi-exposure images. The compensated images are finally combined by one of existing multi-exposure image fusion methods. In some experiments, the effectiveness of the proposed method are evaluated in terms of the tone mapped image quality index, statistical naturalness, and discrete entropy, by comparing the proposed one with conventional ones.

Yuma Kinoshita, Sayaka Shiota, Hitoshi Kiya

This paper proposes a novel luminance adjustment method based on automatic exposure compensation for multi-exposure image fusion. Multi-exposure image fusion is a method to produce images without saturation regions, by using photos with different exposures. In conventional works, it has been pointed out that the quality of those multi-exposure images can be improved by adjusting the luminance of them. However, how to determine the degree of adjustment has never been discussed. This paper therefore proposes a way to automatically determines the degree on the basis of the luminance distribution of input multi-exposure images. Moreover, new weights, called "simple weights", for image fusion are also considered for the proposed luminance adjustment method. Experimental results show that the multi-exposure images adjusted by the proposed method have better quality than the input multi-exposure ones in terms of the well-exposedness. It is also confirmed that the proposed simple weights provide the highest score of statistical naturalness and discrete entropy in all fusion methods.