In this paper, Field-Programmable Gate Array (FPGA) implementation-based image demosaicing is carried out. The Newton Gregory interpolation algorithm is designed based on FPGA frame work. Interpolation is the method of assessing the value of a function for any in-between value of self-regulating variable, whereas the method of computing the value of the function outside the specified range is named extrapolation. The natural images are collected from Kodak image database and medical images are collected from UPOL (University of Phoenix Online) database. The proposed algorithm is executed on using Xilinx ISE (Integrated Synthesis Environment) Design Suite 14.2 and is confirmed on Xilinx Virtex-5 FPGA board. The main aim of this paper is to develop a demosaicing method based on the FPGA technique. Implementing image processing on hardware reduces the cost and make simpler for debugging and verification. The hardware implementation in this system gives the better quality of output image. Implementing image processing on hardware reduces the cost and makes simpler for debugging and verification. Newton- Gregory interpolation technique produces the full resolution of green channel with red and blue channels. The experimental result clearly shows that proposed Newton-Gregory interpolation-based image demosaicing outperformed other algorithms. FPGA implementation attains the maximum Peak Signal-To-Noise Ratio (PSNR) of 40.53 dB and MATLAB (Matrix Laboratory

The word “interpolation” denotes to interpolating some unidentified info from a specified set of well-known info. The method of interpolation is wide used as a valued tool in science and engineering [

The interpolation of a picture aims at estimating intermediate constituents between the well-known pixel values within the on the market low-resolution image. The image interpolation method is nothing however the image synthesis operation. This method is achieved row by row and so column by column. In digital image process, a method that converts a separate image outlined at one set of coordinate sites to a brand-new set of coordinate points is labeled as resampling. The method of resampling uses the interpolation purposes to seek out the pixel(s) value(s) at new coordinates as exposed in

There is several interpolation methods advanced to date which are separated into three key types described as:

Nearest neighbor interpolation

Bilinear interpolation

Bicubic interpolation

Maschal et al. (2013) have proposed a method to compare the quality of two demosaicing procedures by considering the two-color artifacts, fringing and zippering effects. The prior algorithm measures the reconstructed edges formed in the image whether it is perfectly matched or not by edge slope measure algorithm. The second algorithm is based on color component difference field which determines the deviations in the color plane and estimates the red-blue plane and red–green plane in the demosaicing algorithm and it is known to be false measure demosaicing scheme. The sharpness of the pixels and the deviations in the images are the two most criteria to be considered in the calculation of the accuracy scheme of the demosaicing algorithm.

Hsieh Hou et al. (2013) presented the work in cubic splines for image interpolation and digital filtering especially the B-spline (Basis Spline) interpolation and B-spline filtering. The sampling rate adaption is required in image processing. The lag-range interpolation is the basic concept in spline interpolation. The guarantee is not given and the process is inaccurate in this lag-range interpolation. This leads to the polynomial interpolation which runs in a local procedure. The cardinal spline can be used by sampling theorem. Truncations are needed in the cardinal spline and produce the smoothest results. In magnification and modification and smoothing process, the B-spline interpolation is considered as the best. But the computation of this interpolation technique is complex.

Venkadesh et al. (2014) presented a work on color filter array interpolation using the adaptive filter. This work states the color filter array is better than beam splitters. Here for demosaicing the color difference ratio is considered. The noise is removed by using the adaptive filter since it is well performed in edges to eliminate the error that occurs due to interpolation. This method produces a highly robust and high PSNR output. The image quality should be improved.

Pekkucuksen et al. (2013) have presented a work on demosaicing the image on color filter array model by reconstructing the missing sample in interpolation by taking multicolor gradient technique. This algorithm won’t need threshold value and it is not a repeatable procedure and does not need to make any decisions. This technique is best suitable for Bayers CFA (Color Filter Array) pattern and is tested against Lukae CFA pattern for the performance parameters. This interpolation method uses edge-based filtering technique and it also uses the high strength edge for the finding of missing values. This technique works according to the edge detection and the directional approach. It can produce good CPSNR (color peak signal-to-noise ratio) value.

Daisuke Kiku et al. (2018) proposed another work which is a minimized-laplacian residual interpolation (MLRI). This work is the same as residual interpolation but small modifications are done. In the MLRI, tentative estimates of R image are calculated by reducing the Laplacian energies of the residuals. This process will be done in both horizontal and vertical way. The upsampling of observed R and B images are called the tentative image. The upsampling will be done by guided filter (edge preserving filter). By this Laplacian energy minimization, bilinear interpolation technique gives the accurate result with small Laplacian energy. This method is combined into GBTF (Gradient-based Threshold-Free) algorithm. This method makes the process very accurate. This method also provides sharply interpolated color image without artifacts. The CPSNR value for this method is 38.35.

The main aim of this chapter is to develop a demosaicing method based on FPGA. Implementing image processing on hardware reduces the cost and make simpler for debugging and verification [

Demosaicing is the process of renovating the lost color channels of a pixel in the color-filtered image. Many demosaicing algorithms are developed based on image quality and computational complexity. Implementation of image demosaicing using interpolation methods suggestively performs the present image demosaicing methods in several computer vision algorithms.

A demosaicing (sometimes de-mosaicing, demosaicking, or debayering) algorithm is a digital image reconstruction technique that uses incomplete colour samples generated from an image sensor overlaid with a colour filter array to recreate a full colour image CFA. It’s also known as colour reconstruction or CFA interpolation. There are a variety of image demosaicing techniques available, but they can be divided into six categories as shown in

Though, Image demosaicing using interpolation is complex, a time-consuming process with high hardware cost. To solve these problems, a novel Gregory-Newton interpolation is proposed to reconstruct the image without any loss. The FPGA based Newton Gregory interpolation is described in this section. The main objective of proposed methodology is to reconstruct the image using combination of Red-Green-Blue Color Filter Array (RGBCFA) and Gregory-Newton interpolation method. Newton- Gregory interpolation technique produces the full resolution of green channel with red and blue channels. The proposed system consists of two parts such a sensor captures Bayer image using Color Filter Array (CFA), and demosaicing using Newton Gregory interpolation. The overall diagram of the proposed image demosaicing using RGBCFA Gregory-Newton interpolation is given in

The main step in the planned demosaicing approach is the interpolation of green channel. In this proposed work the interpolation of Green Channel is calculated using Gregory Newton interpolation method which is implemented on FPGA.

The block diagram representation of proposed Newton-Gregory interpolation-based FPGA design is given in

The block diagram of Newton-Gregory interpolation technique-based image demosaicing is shown in

In Newton-Gregory interpolation technique, the coordinates of original image are located in the position of (1, 1) in the memory and interpolated pixels are relocated based on the coordinates which are scaled at (1/2SR, 1/2SR) where SR denotes the scaling ratio of coordinates. Coordinates of an original image, interpolated coordinates of an image and its mapping technique is exposed in

The determination of next interpolated coordinates can be done by scaling the time interval which is 1/SR in any one of the directions x/y and these are computed in row wise. The position of a^{th} interpolated pixel in x direction and b^{th} interpolated pixel in y direction is as given in

After obtaining the position of an interpolated pixel (xa, yb), the row address (row_addr) is computed based on yb of the interpolated pixel. The row address of an original pixel (Ak, l) is computed based on

The neighbors of row_addrk such as row_addrk + 1, row_addrk + 2 and row_addrk + 3 are computed by adding C as a constant given in

The architecture of row/column address generator is shown in

The weighting factor updating is simplified by computing the distance in vertical direction (wv) and horizontal direction (wh) as given in

Vertical weighting coefficients are denoted by vwk, vwk + 1, vwk + 2, vwk + 3 for its corresponding pixels Ak, l, Ak + 1, l, Ak + 2, l, Ak + 3, l explained in

Similarly, the horizontal weighting coefficients are computed by the

After the weight updating is completed, forward interpolation is carried out. The forward interpolation is calculated using row wise. The forward interpolation process is given in

After the weight updating is completed, backward interpolation is carried out. The backward interpolation is calculated using column wise. The backward interpolation process is given in

After the process, finally the interpolated pixel is obtained. This process is repeated for all the pixels present in the input image captured by CFA. Finally, the interpolated green channel is obtained.

After calculating full resolution of green channel (G), calculate full resolution of blue and red channels for reconstructing the Bayer pattern into color image. To calculate full resolution of blue and red channels, utilize bilinear interpolation.

In bilinear interpolation the missing

For experimental analysis, two types of dataset are utilized such as Kodak and UPOL dataset. The description of both datasets is explained in this section.

The Kodak database contains 24 pictures, these are films caught and afterward digitized at the resolution of 512 × 768 pixels with 8-bit depth per color module. The ubiquity of Kodak picture database is chiefly because of the way that database covers normal genuine scenes and differs in unpredictability and shading attendances.

The UPOL data base contains iris picture. For experimentation, in the proposed work 10 pictures are used. The irises were examined by TOPCON TRC50IA (Tunnel Oxide Passivated Contact) optical gadget associated with SONY DXC-950P 3CCD (SONY Digital cross connect system) camera. The pictures are 24 bit, RGB (Red Green Blue), 576 × 768 pixels and it is PNG (Portable Network Graphics) record design.

Experimental results obtained from proposed image demosaicing are explained in this section. Initially the image is captured using RGB CFA sensor and then the image is given to the Newton Gregory interpolation method. In this chapter, Newton Gregory interpolation is implemented using FPGA.

Images | PSNR |
PSNR |
---|---|---|

37.14 | 36.23 | |

40.53 | 37.64 | |

36.36 | 35.81 | |

38.47 | 36.56 | |

39.42 | 38.15 | |

40.31 | 36.23 | |

39.3 | 37.71 | |

37.31 | 35.82 | |

36.3 | 36.3 |

The performance comparison of proposed image demosaicing algorithm with other demosaicing FPGA implementations is presented in

Demosaicing methods | Image size |
Hardware device | Used slices | Memory used (KB) | Frequency |
Time (ms) |
---|---|---|---|---|---|---|

Bilinear interpolation | 256 × 256 | Xilinx Vertex-E | 1530/4835 | 1125 | 165 | 0.65 |

Greenplane interpolation | 256 × 256 | Altera Stratix-II | 4553/71680 | 1325 | 264 | 0.57 |

Residual interpolation | 360 × 280 | Altera Cyclone | - | 985 | 240 | 0.48 |

Local directional interpolation | 512 × 512 | Xilinx Virtex-5 | 23904/37440 | 15264 | 208 | 0.37 |

Proposed | 512 × 512 | Xilinx Virtex-5 | 17928/28495 | 278 | 272 | 0.25 |

Interpolation methods | Multiplication (number) | Addition (number) |
---|---|---|

Bilinear interpolation | 8 | 45 |

Green plane interpolation | 19 | 21 |

Residual interpolation | 20 | 26 |

Local directional interpolation | 13 | 14 |

Proposed | 8 | 7 |

The Peak Signal to Noise Ratio (PSNR) values shown in

Color | PSNR (dB) | ||||||
---|---|---|---|---|---|---|---|

Image no | Bilinear interpolation | Green plane interpolation | Residual interpolation | local directional interpolation | Convolution interpolation | Proposed method | |

1 | RGB | 39.465 |
38.565 |
39.465 |
38.295 |
39.417 |
41.826 |

2 | RGB | 42.668 |
43.310 |
43.668 |
43.138 |
43.693 |
43.492 |

3 | RGB | 40.656 |
40.113 |
40.656 |
39.969 |
40.885 |
42.961 |

4 | RGB | 42.278 |
41.433 |
42.278 |
41.525 |
42.676 |
41.944 |

5 | RGB | 39.465 |
38.565 |
39.465 |
38.295 |
39.417 |
41.826 |

6 | RGB | 42.758 |
43.228 |
43.758 |
42.093 |
42.411 |
43.525 |

7 | RGB | 39.065 |
39.586 |
40.065 |
39.328 |
41.200 |
39.720 |

8 | RGB | 41.457 |
43.062 |
43.457 |
42.446 |
43.339 |
43.424 |

9 | RGB | 35.242 |
34.407 |
35.242 |
34.257 |
35.331 |
40.518 |

MSE | ||||||
---|---|---|---|---|---|---|

Image no | Bilinear interpolation | Green plane interpolation | Residual interpolation | Local directional interpolation | Convolution interpolation | PROPOSED |

1 | 12.404 | 19.981 | 19.407 | 23.915 | 18.844 | 10.917 |

2 | 4.733 | 3.401 | 3.733 | 5.314 | 4.116 | 2.940 |

3 | 6.828 | 7.697 | 6.828 | 8.864 | 6.172 | 5.542 |

4 | 3.456 | 3.051 | 3.456 | 3.969 | 2.982 | 3.753 |

5 | 6.748 | 6.204 | 6.748 | 8.087 | 5.607 | 5.207 |

6 | 4.771 | 5.152 | 5.771 | 7.497 | 5.712 | 3.719 |

7 | 6.563 | 9.378 | 9.563 | 12.309 | 7.525 | 7.288 |

8 | 8.554 | 8.321 | 8.554 | 10.854 | 9.890 | 5.654 |

9 | 11.880 | 3.051 | 21.880 | 24.015 | 16.087 | 8.330 |

10 | 3.479 | 2.496 | 2.479 | 3.758 | 3.001 | 2.237 |

11 | 8.542 | 7.002 | 7.542 | 8.848 | 5.212 | 7.178 |

12 | 5.735 | 3.035 | 3.735 | 4.519 | 3.764 | 3.504 |

An efficient image demosaicing using Newton Gregory interpolation in FPGA implementation is explained in this chapter. Implementing image processing on hardware reduces the cost and makes simpler for debugging and verification. The Newton interpolation with FPGA framework was implemented and the mathematical expressions are derived. To verify the efficiency of the proposed technique, the operation of planned system is equated with existing interpolation algorithms. The overall performance of FPGA and MATLAB implementation is analyzed. The FPGA implementation attains the average PSNR of performance of 38.8 dB and MATLAB implementation attains 36.71 dB. This technique was done for both natural and medical images and experimental results carried out for FPGA and MATLAB.

The authors would like to thank Anna University and also, we like to thank Anonymous reviewers for their so-called insights.