Kenji Sugiyama

© 2020 IEEE. It is common in color imaging to use a single-sensor system with a color filter array (CFA), especially a Bayer CFA. Recently, color filter methods using the white component have been studied to improve both resolution and sensitivity. However, de-mosaicking performance is improved using a correlation of color planes, especially for a Bayer CFA. We propose a new imaging system with a pastel-color CFA. In this system, the color plane reconstruction is realized with an abstraction of the white-like component from RAW images. To improve performance, we discuss the use of the highfrequency component. However, the effect of sensor noise on the imaging performance has not been closely investigated in such imaging systems. In this paper, we evaluate these effects on various imaging systems. In our experiment, the optical color filter and sensor noise are simulated using computer calculations. With these results, it is recognized that the proposed method is less affected by sensor noise.

Currently, Y, Cb, Cr 4:2:0 system is widely used as the color format for the video system. We have proposed the frame sequential system as an alternative method which has one color plane in a frame. In this system, the conversion to usual RGB requires a motion compensated color interpolation. However, it is not easy in the case of high saturation color images, because of its difference between color planes. To avoid this problem, we propose the new color format which has dependent colors. The primary colors (RGB) are changed to pastel colors. The conversion from RGB to pastel colors is realized by the color matrix. Re-conversion is also possible by using the inverse matrix. To address the conversion to RGB, basic block matching is used in motion estimation. To avoid incorrect motion vectors, the antierror mechanism are strengthened. Finally, the performance of the pastel colored frame sequential and 4:2:0 are compared. From experimental results, the converted RGB pictures have higher PSNR than 4:2:0.

Very high-resolution video systems, such as 4K (3840x2160), enable a very close viewing distance that is almost the same as the picture height. This technology enables high-reality systems in homes to be realized. However, the very short distance causes significantly different views across the picture, and uniform processing for the picture may not always be the best choice. Here, we calculate the actual view size of a pixel, which depends on the pixel position in close viewing. We proposed the perceptual coding methods using pre/post-processing for the picture to be coded. It is a quantization control by a contrast change. However, the control of quantization should be decided based on the perceptual performance between pixel size and quantization. Therefore, we carried out subjective assessment about this. The bit rate reduction is checked for each method. The 4K resolution processing is realized by four patterns of HDTV picture coding with AVC. As the results of these experiments, it is recognized that the bit rate reduction 15 percent by the optimal coding.

High dynamic range (HDR) images that include large differences in brightness levels are studied to address the lack of knowledge on the quality estimation method for real HDR images. For this, we proposed the local peak signal-to-noise ratio (LPSNR), using the maximum value of neighboring pixels. Next, we proposed an objective estimation method that considers spatial frequency characteristics based on the actual brightness. In this method, we used 2D filter on an FFT for spatial frequency weighting. However, in the case of whole picture processing, the rational evaluation is difficult. To solve this problem, we divide a picture to small area, the weighting processing applied each area. Furthermore, we propose more simplified method without FFT/IFFT. In order to confirm performance of these objective estimation methods, we compared the results of the objective estimation with a subjective assessment. The results of the correlation of partial adaptive method processing does not give significant difference. The correlation of simplified method shows its usefulness.

High dynamic range (HDR) images that include large differences in brightness levels are studied to address the lack of knowledge on the quality estimation method for real HDR images. For this, we earlier proposed a new metric, the independent signal-to-noise ratio (ISNR), using the independent pixel value as the signal instead of the peak value (PSNR). Next, we proposed the local peak signal-to-noise ratio (LPSNR), using the maximum value of neighboring pixels, as an improved version. However, these methods did not sufficiently consider human perception. To address this issue, here we proposed an objective estimation method that considers spatial frequency characteristics based on the actual brightness. In this method, the approximated function for human characteristics is calculated and used as a 2D filter on an FFT for spatial frequency weighting. In order to confirm the usefulness of this objective estimation method, we compared the results of the objective estimation with a subjective assessment. We used the organic EL display which has a perfect contrast ratio for the subjective assessment. The results of experiments showed that perceptual weighting improves the correlation between the SNR and MOS of the subjective assessment. It is recognized that the weighted LPSNR gives the best correlation.

Very high-resolution video systems, such as 4K (4096x2160), enable a very close viewing distance that is almost the same as the picture height. This technology enables highreality systems in homes to be realized. However, the very short distance causes significantly different views across the picture, and uniform processing for the picture may not always be the best choice. Here, we calculate the actual view size of a pixel, which depends on the pixel position in close viewing. We propose perceptual coding methods using pre(post)-processing for the picture to be coded. The bit rate reduction is checked for each method, including contrast conversion, pre-filtering, and geometric image conversion. The 4K resolution processing is realized by four patterns of HDTV picture coding with H.264 and HEVC. As the results of these experiments, there is the possibility of a bit rate reduction of 5 to 25 percent on H.264 and 10 to 30 percent on HEVC.

The non-reference method is widely useful to estimation picture quality on the decoder side. In this paper, we discuss the estimation method for spatial blur that divides the frequency zones by the absolute value of 64 coefficients with an 8-by-8 DCT and compares them. It is recognized that absolute blur estimation is possible with the decoded picture only.

DCT encoding of images leads to block artifact and mosquito noise degradations in the decoded pictures. We propose an estimation to determine the mosquito noise block and level; however, this technique lacks sufficient linearity. To improve its performance, we use the sub-divided block for edge effect suppression. The subsequent results are mostly linear with the quantization.

The non-reference method is widely useful for picture quality estimation on the decoder side. In other work, we discussed pure non-reference estimation using only the decoded picture, and we proposed quantitative estimation methods for mosquito noise and block artifacts. In this paper, we discuss the estimation method as it applies to the degradation of the temporal domain. In the proposed method, motion compensated inter-picture differences and motion vector activity are the basic parameters of temporal degradation. To obtain these parameters, accurate but unstable motion estimation is used with a 1/16 reduction of processing power. Similar values of the parameters in the pictures can be seen in the stable original picture, but temporal degradation caused by the coding increases them. For intra-coded pictures, the values increase significantly. However, for inter-coded pictures, the values are the same or decrease. Therefore, by taking the ratio of the peak frame and other frames, the absolute value of the temporal degradation can be estimated. In this case, the peak frame may be intra-coded. Finally, we evaluate the proposed method using coded pictures with different quantization.

We have proposed a sequential color system for use as a video format. In this system, three color planes are used sequentially, and, it has a picture quality of 4:4:4 with half uncompressed data of 4:2:2, if suitable color interpolation is used. To realize this system, a conversion to whole RGB picture with the interpolation of missing color plane is important. For this, a motion compensated interpolation by the same color plane in the other frames is effective. However, the conventional methods of motion estimation and compensation are not useful directly because of the changing of color plane by a frame. We discuss the adaptive interpolation method using forward frame and backward frame. Further, a motion estimation method using the contour picture and the same color plane are proposed. Finally, it is shown that useful converted picture is given for the usual picture.

We have proposed a sequential color system for use as a video format. In this system, three color planes are used sequentially, and, it has a picture quality of 4:4:4 with half uncompressed data of 4:2:2, if suitable color interpolation is used. To realize this system, a high efficiency coding method is important. Before discussing the SC coding, R, G, B direct coding is compared to the coding for luminance and chrominance (Y, Cb, Cr) on view points of a coding efficiency and subjective picture quality. On the other hand, it seems that the optimizing for the perceptual quality is not easy in the direct coding. We discuss a quantization balance of R, G, B for the best subjective quality. Further, we propose the subjective quality improvement method by enhancing the PSNR of luminance component using quantization error diffusion between the color planes.

We have proposed a sequential color system for use as a video format. In this system, three color planes are used sequentially, and, it has a picture quality of 4:4:4 with half uncompressed data of 4:2:2, if suitable color interpolation is used. To realize this system, a conversion to whole RGB picture with the interpolation of missing color plane is important. For this, we use a motion compensated interpolation by the same color plane in the other frames, however, it does not work in very fast motion. Therefore, we should consider a color plane prediction method without motion compensation. In the case of one color lack, a value of lacked plane is predicted by existing planes using a color probability of reference picture. In the case of two colors lack, a clustering of color planes is used. Each cluster is predicted by the similarity of the cluster between target and reference frames.

Currently, the widely used video formats are 4:2:0 or 4:2:2, however, the picture quality is degraded; in particular with interlaced 4:2:0, the degradation of the luminance signal is significant with high saturation color. On the other hand, inter-picture processing with accurate motion compensation has been realized. This technique reduces the importance that the format contain all color components in a frame. On this basis, I have proposed the new video format called the sequential color system, in which three color planes appear sequentially. Sequential color retains the picture quality of 4:4:4, as long as the racked color is carefully interpolated. When it is used for a camera, the same resolution as three sensor system can be achieved with small optical loss and blur. For coding, this format has a half size of 4:2:2 for uncompressed data. For compression, only one plane exists to code in a frame, and, very high efficiency will be expected.

In the many kinds of video system, it is desired that the picture quality is controlled to provide the better picture. For this purpose, an objective picture quality estimation method without the reference picture is required. To respond this, a spatial blur estimation using DCT is proposed.

The local decoded picture is basically used as the reference for inter-picture prediction to avoid the mismatch between encoding and decoding. However, the coding efficiency is not necessarily the optimal in this processing. In this report, we try to use the original picture as the reference. In this case, the mismatch causes degradation of the picture quality. However, the bit amount will be reduced. To take optimal coding, we propose the adaptive method based on the bit amount and the decoded picture error. The original picture is used only in the block which has less bit amount and small error increasing. We apply this coding to MPEG-4 Main Profile. As the results of experiments, about 0.2dB gain in PSNR is given in a sequence. It is recognized that the adaptive method works successfully and coding performance can be improved.

As standard video encoder techniques have matured, their rate of improvement has slowed. Greater improvements are possible with purely new coding techniques however, these are incompatible with conventional methods. As an alternative, a new coding concept with semi-compatibility has been proposed, and an enhancement to I-pictures efficiency has been discussed. This applied method reduces the quantization error using motion compensated inter-picture processing. In this report, we apply this method to P-pictures to improve the efficiency of B-pictures. The quantization error component of the prediction signal is canceled by averaging the bi-directional prediction. Experiments using MPEG-4 show significant improvement in the coding efficiency using the proposed method. The maximum PSNR gain reaches 2.3 dB in a static sequence. At least 0.5 dB can be achieved in a high motion sequence. Furthermore, the proposed method is more useful for the larger P-picture period. © 2010 IEEE.

With DCT coding, block artifact and mosquito noise degradations appear in decoded pictures. The control of post filtering is important to reduce degradations without causing side effects. Decoding information is useful, if the filter is inside or close to the encoder; however. it is difficult to control with independent post filtering, such as in a display. In this case, control requires the estimation of the artifact from only the decoded Picture. In this work, we describe an estimation method that determines the mosquito noise block and level. In this method, the ratio of spatial activity is taken between the mosquito block and the neighboring flat block. We test the proposed method using the reconstructed pictures which are coded with different quantization scales. We recognize that the results are mostly reasonable with the different quantizations.

In a picture coding, DCT (Discrete Cosine Transform) and the block based motion compensation cause the degradations called block artifact and mosquito noise in the reconstructed pictures. These artifacts can be reduced by post-processing if the block position in the picture is determined. However, there are many kinds of picture format in actual video system. The picture in the display may be already converted from the other formats. In such case, the post-processing is difficult to do, because the block position of the original picture is not determined. To respond this, we propose the block position determination method by using the reported estimation method for block artifact level. In this, pixel continuities are checked. We test the proposed method by using the pictures which were converted from the other formats. As results of examination, the block position is determined at the pictures with un-noticeable degradation, such as the Q_scale equal to 3.

In video coding, H.264 (MPEG-4AVC) standard method are widely used as a high performance coding. In this, high PSNR value can be got by using rate distortion optimization. On the other hand, the subjective picture quality became important. However, it is difficult to consider the subjective picture quality because of 4x4 DCT which is instead of 8x8 DCT in MPEG-2 To realize the control of picture quality, we propose the method which use the pre and post processing. The picture before coding is filtered on human perceptual characteristics. Then, they are coded using rate distortion optimization. Finally, the decoded picture is filtered by the post-filter which has reverse characteristics to pre-filter. Further, we propose the adaptive control of the filter strength based on block activity. The proposed methods are evaluated by using JM13 CODEC. As results of them, the advantage of the subjective picture quality is recognized.

The rate of improvement to standard video encoder,, has slowed, hut improvement based on purely new coding techniques are incompatible with the conventional standard. As an alternative solution, a new coding concept with semi-compatibility is described. I-picture is used for access and refreshment. To increase total performance, it is desirable to enhance the efficiency of I-pictures. The quality changes at the border of a GOP also need to be reduced. In response. we propose coding that is applied to the inter-frame processing oil the border of a GOP. This applied method reduced the quantization error using motion compensated inter-picture processing. In this report, we verify the improvements in the efficiency and the compatibility of proposed method. As a result of our tests, we recognize that the total gain is maximally 1.6dB in the PSNR. In general. the degradation of performance in standard decoding is less than its gain. Also with the proposed method, we Find that the refreshment performance is sufficiently high.

To video signal recording for storage medium, high efficiency coding is used same as broadcast and communication. Currently, MPEG-2 is most widely used coding method and enhancement of efficiency is tried. About the MPEG-4 which achieves more effective, most newly method is Advanced Simple Profile, and Part 10 (H.264) is discussed now. Some non-standard methods are used mainly in USA not only for network, but also for storage media. I show technical items mainly about newest Part 10 and movement of new coding method and study. On the other hand, professional equipment that requires frame editing uses intra-frame coding. Improvement is tried in this type coding. In case of bit-stream recording, using of tans-coding will be increase.

In order to achieve better performance of 480i and 480p on different scanning type TV sets, a coding scheme, which includes a conversion from interlaced to progressive scanning of the P and I pictures, is newly proposed. In this method, the B picture is still interlaced, while all the pictures are predicted using progressive scanning. In decoding the progressive scanning pictures, intra-field interpolation of the residual is used. From experiments, the proposed coding scheme gives better picture quality than that of MPEG-2.

In digital broadcast, picture formats of progressive scanning and interlaced scanning are used. We have thus developed a broadcast system that uses common bit stream for the 480i and the 480p formats as a source picture. For this system, we take new coding which encode as interlaced scanning of B-picture and progressive scanning of P and I-picture. In this report, we discuss a progressive scanning decoding for this bit-stream. To realize this, we take intra-field interpolation for inter-picture predictive residual of B-picture. Experimental results showed that the proposed decoding scheme performs better than MPEG-2 interlaced coding with de-interlacing.

In digital broadcast, many kind of picture format are used. However, interlaced scanning TV receiver is still used in most homes. We have thus developed a broadcast system that uses common bit stream for the 480i and the 480p formats as a source picture and the 480i format for the receiver. We take new coding for the 480i pictures which include a conversion from interlaced scanning to progressive scanning of the P and I-picture only. We discuss a scanning conversion and use motion compensated adaptive interpolation. Experimental results showed that the proposed coding scheme performs better than interlaced scanning coding in MPEG-2.

Many TV broadcasters now use a progressive scanning format and an HDTV system to produce programs. However, standard TV receivers are still used in most homes. We have thus developed a broadcast system that uses the 480 p format for the source picture and the 480 i format for the receiver. We use progressive scanning coding because of its coding efficiency and room for improvement. The B-picture is not used as a prediction reference ; we use only the interlaced scanning lines in the B-picture to remove redundant information. We use 8 x 4 DCT instead of 8 x 8 DCT for coding the residual to realize this method. Experimental results showed that the proposed coding scheme performs better than other coding schemes such as two kinds of interlaced scanning coding and progressive scanning coding in MPEG-2.

In TV stations, progressive scanning and HDTV system are used recently. However, standard TV receiver is still used mainly at home. We suppose a broadcast system which use 480p format at source picture and 480i format at receiver. At first, we discuss about suitable coding for the supposed system, and show that progressive picture coding has advantages. Then, we propose efficient method that reject redundant of picture information for 480i picture decoding. Finally, we compare the efficiency of proposed method to MPEG-2 interlaced picture coding.