Deep Learning and SVM-Based Approach for Indian Licence Plate Character Recognition

: Every developing country relies on transportation, and there has been an exponential expansion in the development of various sorts of vehicles with various configurations, which is a major component strengthening the automobile sector. India is a developing country with increasing road traffic, which has resulted in challenges such as increased road accidents and traffic oversight issues. In the lack of a parametric technique for accurate vehicle recognition, which is a major worry in terms of reliability, high traffic density also leads to mayhem at checkpoints and toll plazas. A system that combines an intelligent domain approach with more sustainability indices is a better way to handle traffic density and transparency issues. The Automatic Licence Plate Recognition (ALPR) system is one of the components of the intelligent transportation system for traffic monitoring. This study


Introduction
In a developing country like India, traffic turmoil is the major nuisances because of massive population, limited roads, and infrastructure. The Intelligent Transportation System (ITS) has emerged as a potential solution to the aforementioned problem. ITS implementation is a solution to problems such as parking, traffic control, automatic toll collection, traffic safety, and enforcing traffic regulations, among others [1]. In a country like India, where over a thousand accidents occur every day, there is a pressing need to enforce regulations. Furthermore, toll plaza traffic congestion requires additional resources, such as time. To get rid of it, we need accurate vehicle recognition in a short amount of time. The difficulty can be solved with the use of an Automatic Licence Plate Recognition (ALPR) system. Despite the fact that there are several answers to the problem outlined above, there is still no complete solution. Looking into it more, there is a need to improve the current APLR systems.
Microwave, infrared, Radio Frequency Identification (RFID), and image recognition are all used in the current vehicle detection system. The installation of transponders on the vehicle is managed by first three strategies. However, there are other concerns, such as in the event of fast driving, when the precision rate of identifying and detecting number plates is lower with a transponder system. In most cases, the ALPR system can be leveraged to improve precision. In the meanwhile, there are various unexplained annoyances. The transponder may be illegal in some instances, and it may compromise user privacy [2]. The plate detection device mounted on the roadside faces the challenge of distance and vehicle speed. The law enforcement uses video cameras installed on major roadways to investigate vulnerable automobiles. This implanted detecting technology aids law enforcement by allowing traffic police officers to get information on vehicles from archived footage when needed in the future. The issue of stolen cars can be readily remedied with the ALPR system put on vehicles such as buses, taxis, or police cars. It will be resolved instantly by detecting the car by comparing the vehicle information to the stolen vehicle database [2].

Automatic Licence Plate Recognition (ALPR) System
ALPR is a group surveillance method that interprets vehicle licence plates using optical character detection and identification of imagery. Vehicles can be easily and precisely recognised by their licence plate number from an image or a video of a moving vehicle using this technology. ALPR systems play an important role in a variety of applications, including traffic control and management, vehicle tracking and location, car parking automation, and toll collection systems. It can be used for automatic toll collection on highways, parking in retail malls, airport parking, and hospitals, among other things. Fig. 1 [3] shows a typical ALPR system which involves majorly four steps in the process of recognition.

Character Segmentation
Pre-processing

Character detection and Recognition
Image of vehicle as Input Figure 1: A canonical ALPR system

Input Image of Vehicle
To begin, a camera is used to take a picture that includes the licence plate. The quality of the image obtained is crucial in the recognition of the licence plate. As a result, high-resolution cameras are required to capture the image.

Pre-processing
The preprocessing technique considers the captured image of the vehicle for evaluation, taking into account the total image with a set background for analysis. Firstly, the image is transformed into a grey image and then edges in the images are found using filtering. The filtering can be done with the number of operators such as Sobel, Canny, Gabor etc. [4][5][6]. After that dilation of the image is done with the square structuring element. The holes in the image are filled and the connected components are found and labelled. The histogram is used to find out a number of lines and characters in the region.

Character Segmentation
As the height and width of all the characters are equal so these features can be used for segmentation of characters. In literature, the segmentation is done by using a number of methods like connected component analysis, histogram analysis, window scanning [7] etc.

Character Recognition
Characters that are feature dominated and have relation with an acquired image are segmented and analysed after extraction. As vital aspects in character recognition on a licence plate, features play an important part in defining a procedure. In [5], fuzzy logic based character recognition is proposed. Other techniques proposed by researchers in literature are template matching, Artificial Neural Network (ANN), Support Vector Machine (SVM) etc. These classifiers perform extremely well for various applications such as diabetic diagnosis [8,9], handwritten character recognition [10,11], etc.
The aforementioned technology is gaining a name in traffic installations and security for a variety of applications, including toll collecting at toll plazas, red light jumping at the roadside, and any other traffic rule violation.

ALPR Systems Methodologies
There have been numerous studies conducted using various approaches for an ALPR system. The approaches used to accomplish the three key stages after image acquisition, namely, extraction of the licence plate from the input image, character segmentation from the extracted image, and identification of the segmented characters, are discussed.

Licence Plate Extraction
The drawing licence plate area retrieved from the captured image or video is crucial for the ALPR system since it determines the outcome. The collected vehicle image and a region of the vehicle image that most likely holds a licence plate serve as input and output to the first stage, respectively. Because the size and elements required to characterise a process in a measurable domain differ based on the licence plates employed in a system, inaccurate character judgement occurs. As a result, characteristics serve as a criterion for determining size and elements for defining a quantifiable process. Characters mentioned on a licence plate in a precise pattern are used to create the image's features. There is a number of licence plate features such as colour, rectangular shape, etc. [1]. The licence plate colour is usually distinct from the car colour; this difference is known as texture, and it can be taken into account while extracting the licence plate from the image. In addition, two or more picture attributes can be combined and used to extract a specific licence plate. Tab. 1 compares several licence plate extraction procedures based on technique, plate size, and success rate. A preset requirement, such as camera position, vehicle position, and lighting condition, constrains the execution of alternative strategies.

Character Segmentation
The retrieved licence plate is then processed for character segmentation. Threshold is used to transform the extracted licence plate into a binary image. The threshold must be determined to avoid character joining, which can result in complex character segmentation [27]. Due to a bad choice of binarization threshold, the frame of the licence plate might sometimes become stuck to the text. One solution to the problem is to increase the image quality before converting it to binary by doing prepossessing. In the literature, histogram equalisation, noise removal, and contrast enhancement are some of the approaches used to improve image quality. The various character segmentation techniques are compared on the basis of plate size and success rate are given in the Tab. 2.

Character Recognition
Recognition of the characters obtained after segmentation is the next and last stage. Due to camera magnification, irregular size, and thickness of segmented characters, character recognition in ALPR systems is complicated. The aforementioned issues are overcome by resizing the characters to a predetermined identical size before proceeding with the processing. Other difficulties include the font style of the characters, which varies by country, and the extracted characters being broken, noisy, or skewed. Character identification and segmentation are assessed based on the region covered, the area of interest, and the quantity of characters involved (not variable). The accuracy of recognition is determined by a number of criteria, including resolution and image quality, as shown in Tab. 3. Khan et al. [32] suggested a new ALPR technique based on template matching and NN for still photos and videos. Several character feature extraction and recognition techniques have been investigated, along with their benefits and drawbacks. In the literature, classifiers such as ANN [33], SVM [34], and HMM [35] are used to recognise characters using extracted characteristics. The classifiers can also be used in parallel [36], multistage [37]. Tab. 3 shows a comparison of the character identification rates of various approaches.

Soft Computing Techniques Based ALPR Systems
Soft computing is used to deploy approximate calculations to provide practical answers for complex computer problems, although it is imprecise. Problems that are difficult or impossible to solve with current hardware can be solved using this method. Soft computing techniques are similar to formal logical systems that focus largely on computer-aided numerical analysis and use biological processes as hallmarks rather than traditional techniques. The ALPR system's steps are shown in Fig. 2 and discussed in depth further below.

Input Stage
An ALPR system acquires images in any format immediately, as long as the licence plate's borders are sublimed as well. For picture acquisition, cameras are placed along the roadside. The image can be captured by the camera, or one of the video frames can be used for further processing. Camera quality, camera position, distance between vehicle and camera, angle of picture capture, camera zooming factor, lighting conditions, ambient circumstances, vehicle speed, and so on are all aspects that influence the entire process of area recognition. These elements are critical and cannot be overlooked because the input image is the most important factor in recognition. High-resolution cameras provide images with more pixels, which implies that all small and important details are taken into account, resulting in a greater identification rate. Fig. 3 shows an input car image with a licence plate.

Pre-processing Module
The pre-processing of the input image is an important module for recognition. The following are the steps involved in the pre-processing:

Binarization
The binarization begins with the resizing the input vehicle image. The vehicle image is resized to image with 400 rows while maintaining the same aspect ratio as of the original input image. This is done to have uniformity in the processing of different image taken by different cameras. This transformation from RGB to grayscale do not alter the luminance, however the Hue and Saturation are removed. Thereafter intensity image is transformed to binary image using a threshold using Otsu's method.

Filtering
The filtration is done using a 3 × 3 median filter to eliminate salt and pepper noise of the image. This filter minimises noise at first without affecting the edge information, which is crucial in this scenario. The masked values are ordered in increasing order for filtering purposes, and their median is determined.

Dilation
Dilation is mathematical morphology operator. Dilation is applied on binary images to progressively broaden the boundary regions of forefront pixels. The binary dilation of element A by structuring element B is denoted A⊕B, is defined as the set operation given by Eq. (1) given below. The structuring elements are used for dilation of an image to expand its shapes.

Erosion
Erosion is also mathematical morphology operator. Erosion is applied on binary images to progressively erode the boundaries regions of forefront pixels. The binary erosion of image A by structuring element B is denoted by A B, is defined as the set of pixels given by Eq. (2) given below.

Morphological Gradient
The eroded image is subtracted from the dilated image to obtain morphological gradient. The shrunk image obtained using erosion is subtracted out of the broadened image obtained using dilation and the results obtained are the edges present in the input image.

Brightening of Edges
The edge information is obtained from morphological gradient is further processed for more brightening. The brightened edges are obtained by doing 2D convolution of the image with edges obtained from the morphological gradient and [1 1, 1 1]. The convolution formula used for twodimensional variables A and B is given in Eq. (3) given below.
where, p and q run overall values that lead to legal subscripts of A(p, q) and B(j − p + 1, k − q + 1). The morphological gradient image is convolved with [1 1, 1 1] to brighten the edges.

Elimination of Unwanted Horizontal Lines
The image got after 2D convolution with kernel function [1 1, 1 1] is further searched for horizontal lines. In a licence plate, the possibilities of long horizontal lines due to the written characters are very low. So, to remove these unwanted horizontal lines in the image, firstly image is eroded with a structural element which is a line with 51 elements and zero degrees angle and the result obtained is subtracted from the convolved image.

Image Hole Filling
The image obtained from the previous step is having holes. For further processing, these holes need to be filled. Here, a hole is a group of pixels with value '0' and they are covered with pixels with value '1' and even if zeros from the edges are filled they cannot be reached out. The image obtained after horizontal lines is processed to fill the holes present in it.

Isolation of Characters
The characters in the image due to dilation may get connected to ensure the proper isolation of the characters the thinning operation is performed on the image. Further, the eroding is done using structural element [1; 1; 1].

Removal of Objects
Once the characters are isolated then all the objects in the image with pixels less than 300 are not likely to be the characters of the licence plates. Therefore all the objects in the image with pixels less than 300 are removed.

Identification of Connected Components in the Image
The image obtained for the last step needs to be identified. For their identification, these objects are labeled to differentiate from each other.

Measurement of the Labeled Connected Components
All the labeled connected components are measured for their property. The main idea behind using these measurements is to underline structured elements and lie along with borderline submerged images in order to provide sufficient rate for preprocessing. This structured analysis has been figured out and made available for analysis with eleven attributes so that annealing effect can be imposed and filter out antialiasing effect in which high frequency components superimpose low frequency components. This kind of processing also helps to add further processing.

Character Segmentation Module
The pre-processed connected components and their attributes are utilised to locate the characters in the licence plate. It aids in distinguishing the characters from the rest of the image. Because all characters are printed horizontally, their breadth along y will be same. Furthermore, if the object's upper left corner coordinates are carefully examined, the y-coordinates are likely to be the same or very similar to each other. This element aids in the identification of the characters on the licence plate. The ycoordinates and width along y of ten connected components are nearly the same. These ten connected components are likely to be the licence plate characters, hence separated out and their properties are given in the below Tab. 4. These upper left corners coordinate and width along x and width along y are used to crop the input image for the segmentation. The segmented characters obtained are as shown in Fig. 4 given below.

Character Recognition Module
The recognition is performed by trained models based on soft computing techniques i.e., RF, ANN, SVM, and CNN. These four soft computing techniques are, firstly used for the implementation of recognition module and later on the basis of their performance the hybrid technique is also implemented. The implementation of soft computing techniques RF, ANN, SVM, CNN and CNN-RF are described in [42]. Results motivate to implementation of CNN and SVM based hybrid ALPR system.

Hybrid Scheme Implementation for ALPR System
Using the training data of alphabets and numerals, various soft computing-based recognition modules are trained, and then the features of testing licence plates are extracted and recognised using the trained model. Arial, Lucida, Bookman Old Style, Calibri, Cambria, Consolas, Lucida Bright, Rockwell, Tahoma, Times New Roman, and Verdana are among the basic fonts used in different sets with pre inclined values [42]. The trained models were then put to the test for identifying Indian cars. Two hundred licence plates from five Indian states (Haryana, Punjab, Chandigarh, Uttrakhand, and Delhi) were used for testing and training in this work. Fig. 5 depicts a series of test photos of a car with a licence plate. Because each licence plate has four alphabets and six numerals, the modules below are used to test a total of 800 alphabets and 1200 numerals.

CNN and SVM Hybrid Recognition Module
For the recognition of the licence plate characters, the deployed technique uses a blend of CNN and SVM. To begin, the ALPR system is given the car image as input. To isolate the licence plate region from the rest of the image, the input image is pre-processed with binarization, median filtering, dilation, erosion, subtraction, linked component labelling, and the histogram. The properties of the item recognised in the input image are used for segmentation. For subsequent processing, the segmented characters are scaled to 28 × 28 images. The CNN network, which is a seven-layer network comprising of an input layer, convolutional layer, ReLU layer, max-pooling layer, fully connected layer, classification layer, and a softmax layer, is given the segmented region of the picture for feature extraction. The CNN training parameters are listed in Tab. 5. The input layer pre-processed a raw image with a size of 28 × 28 pixels, segmenting each letter and numeral with a structured frame. Different frames might be chosen depending on the image quality and feature mechanism. The CNN input layer functions as a pre-processor, keeping the segmented image organised. In a structured formation of frame, the second layer functions as a barrier for undesirable materials to penetrate and provides filters of quantized values of 9 × 9 with significant and relevant numbers. Filters decide the memory and weight values of a network and should be less in number in order to make a system valuable. Fixed Threshold function also termed as objective function is provided for linearization by third layer of a network. It can be also considered as minimum value of feature extraction limit without any loss of an image and minimizing time for feature extraction. Fourth layer is used to maximize the set attributes of pool value of 2 × 2 and minimize the number of elements keeping features intact. However; fifth layer is connected layer with intent to provide single vector with soft function terminology. Sixth Layer is the softmax layer which is used to calculates a probability for every possible class. Seventh layer deals with activation function for SVM module as it classifies and provide inputs for SVM trained network. The activation values of the classification layer are used as extracted features and given as input to the SVM module. The trained SVM module is used for the final recognition of testing licence plates dataset. The seven hundred eighty one alphabets are recognized out of eight hundred alphabets which results in 97.63% recognition rate for alphabets.
While one thousand one hundred eighty eight numerals out of twelve hundred are recognized, resulting in a recognition rate of 99%, while the overall licence plate recognition rate obtained is 98.45%.
Tab. 6 shows recognition rates obtained for various implemented techniques. It has been concluded from results that CNN based ALPR system performed better than RF, ANN, and SVM based ALPR systems. Furthermore, the results obtained are improved by using hybrid models for recognition purpose. The hybrid model of CNN and RF performed better by 1.75% than the CNN. Moreover, the hybrid model based on CNN with SVM performs better than the hybrid model based on CNN with RF by 0.65%. In the undertaken work, an ALPR system with high recognition rate has been implemented and tested. The Soft Computing Techniques, i.e., RF, ANN, SVM, and CNN are implemented for the recognition of the licence plate. The Soft Computing Techniques based created models are tested using 200 images of Indian licence plates and concluded that the implemented soft computing techniques i.e., RF, ANN, SVM, and CNN soft computing technique performs better than the conventional techniques. The obtained results have motivated for further hybridization of CNN with RF and CNN with SVM. Based on implemented soft computing techniques i.e., RF, ANN, SVM, and CNN, it is concluded that CNN performs better at 2.8%, 1.6% and 0.45% than RF, ANN and SVM based ALPR systems respectively. The implemented hybrid (CNN-RF and CNN-SVM) ALPR system performs better than the RF, ANN, SVM, and CNN based ALPR systems. The recognition rates obtained for implemented hybrid techniques, CNN-RF and CNN-SVM are 97.8% and 98.45%, which is better compared to the recognition rates of 93.25%, 94.45%, 95.6%, and 96.05% obtained with RF, ANN, SVM, and CNN, respectively. CNN-SVM hybrid technique promisingly performs better than CNN-RF. The recognition rate obtained for CNN-SVM is better at 0.65% than the CNN-RF based ALPR system. The research effort has been intended to come up with a new scientifically validated ALPR system. The work will have applications in different areas like traffic analysis, toll management, security, parking, law enforcement, and stolen vehicle identification and so on.
The implemented technique can also be improved and enhanced to develop fully automatic licence plate extraction and recognition system with full automation, right from the acquisition of the licence plate till the final recognized plates output as required by toll plaza system, parking management system, crime investigation system, traffic monitoring system, border control etc. The developed approach is trained to recognise Indian licence plates with English characters and numerals; however, the current research work can be extended to recognise licence plates from other countries or licence plates written in other scripts and languages such as Hindi, Punjabi, Tamil, and others. The future researcher can focus on the image with multiple licence plates, video-based recognition, processing of high definition plate images. The focus can be on pre-processing techniques to deal with noise, illumination problem for improving the recognition rate with reduced recognition time. The findings of this research work can be applied to other related areas of image processing and pattern recognition like handwriting recognition, signboard text detection, signature verification system etc. The recognition of the manufacturer and model of the vehicle along with the licence plate can also be extended in the ALPR system.