Recently, Advanced Driver Assistance System (ADAS) has been developing speedily such as Active Cruise Control, Parking Assistance, and Lane Departure Warning System. However, people should pay a lot of cost for ADAS options when they buy a new car. Auto parking system is one of these ADAS. Commercial auto parking system needs some extra sensors such as cameras, ultrasonic sensors, and so on. This auto parking system is no more than one of assist system in steer and brake for drivers which are not an active control. Autonomous vehicle uses GPS signal to identify their absolute position on ground which makes auto parking system possible. However, they use expensive GPS sensors and cannot utilize the signal in internal area. In this research, color mark based vehicle position; attitude estimation method and parking space determination method are introduced. Additionally, the method to use a surveillance camera for surround view system is suggested. Using this kind of approach to auto parking system, vehicles do not need extra expensive sensors but need a communication tool with a central computer in a parking lot. Auto parking system can be realized in any parking lot with a surveillance camera if the central computer has the control permission for vehicles. Drivers do not need to span their money for the auto parking system any longer and can experience the system in cheap [1].

For the Auto parking system suggested in this research, first stage is to estimate the vehicle position and attitude (rotation angle) in image coordinate system which is explained in detail in section 2. The next stage is parking space determination described in section 3. And third stage is to control a down scaled vehicle into a parking space mentioned in section 4. Additionally, surround view system is described in section 5. The entire system is developed for the scenario like the flow chart Figure 1.

Figure 1: Entire scenario of auto parking system.

In this research, a small children vehicle (HENES M7 Premium model), Fly Capture USB 3.0 camera of Point Grey assumed as a surveillance camera and a laptop as a central processing computer are utilized for a scaled down experiment. The camera is attached on 2 meters high on the side corner of the experimental parking lot like Figure 2 in consideration of the scale factor between a real car and a small children car. The small children car has the height of approximately 0.4 meters and a real car, for example, AVANTE MD series of Hyundai Motors has the height of approximately 1.5 meters and also most parking space in Korea has almost 5 meters height and at least 2.5 meters above in legal in Korea. The vehicle position and attitude estimation is performed in top view image of parking lot which is the coordinate system of auto parking system.

Figure 2: Experimental parking space.

The first step to estimate vehicle position and attitude exactly is to detect red and green color marks attached on the front top center and the rear top center like Figure 3 and find out the each center position of the color marks in image. Front and rear centers can be distinguishable because of difference of colors. The second step is to change the top centers of front and rear of the vehicle (a.k.a. top front and rear centers of the vehicle) into the real vehicle front and rear centers [2]. This happens because coordinate systems that camera is looking at are different depending of the height on the ground. The vehicle image is stretching into opposite side of camera viewing direction because the experimental vehicle has 3D shape. That is the characteristics of camera that is not changeable artificially. That is why the correction process should be needed into the real vehicle front and rear centers (a.k.a. real front and rear centers of the vehicle). And then, the vehicle centers and attitude are calculated using the real front and rear centers of the vehicle.

Figure 3: Down scaled experimental vehicle.

Front and rear top centers of the vehicle

Color marks are attached on vehicle front top center and rear top center respectively like Figure 4 for the entire vehicle position and attitude estimation. Starting from the first step, camera calibration process is implemented to remove image distortion around outside of image and the side view image of the parking lot is changed into bird eye view (top view) for convenience of coordinate system to control vehicle. And then, the format of bird eye view image which is originally RGB format is changed into HSV format. Then, two kinds of simple HSV filter is utilized to filter out red color mark and green color mark into each two image. Then, the two filtered images for red and green colors should be transformed into binary image and blob labeling method is utilized to pick up the red and green color marks’ pixel positions in each image which results in numbered matrix of each red and green color pixel positions. Then, the central pixel position of red and green color marks should be estimated. The important thing is that the processing time of estimating the central pixel position of red and green color marks is considerably dependent on how to calculate the center positions of color marks. The mean pixel position method (Equation (1)) of each numbered matrix is used to calculate the center position which needs relatively long processing time compared to the method to use the two corner pixel position.

Figure 4: Triangular method.

                           (1)

The superscript ‘k’ means red color mark’s center for 0 and green color mark’s center for 1.

However, the mean pixel position method gives relatively exact center positions of color marks because the two corners of each color marks are chattering due to multiple causes that is from camera resolution issue, and the brightness change of color marks’ surface when HSV filtering, and the performance of the blob detection algorithm [3]. The more pixel is far from the camera installation position, the more chattering is significant because the further from the camera position, the larger the size of one pixel is which results from the transformation process from the side view image to the top view image. The processing time is on trade-off relation with the accuracy to estimate the center position of marks.

Position correction of the front and rear top centers

As mentioned before, the centers of the color marks are not the real front and rear centers of the vehicle because the vehicle has 3D shape, that means the car cannot be positioned properly for auto parking system. Thus, color marks’ centers should be corrected into real front and rear centers of vehicle on ground using the triangular method like Figure 4. Color marks’ position, Camera position in pixel image, the installation height of the camera and the height attached color marks are all known. The equation is like below.

                                (2)

The subscript ‘corrected’ means the coordinate is corrected into the real centers of color marks and the superscript ‘k’ means front when k is 0 and rear when k is 1.

Vehicle center and attitute

Finally, the vehicle center is calculated from the half point of the compensated marks’ center position induced from section 2.2 using the equation (3).

            (3)

The superscript ‘0’ means the front center and ‘1’ means the rear center. And also, the vehicle attitude which means the rotated degrees of the vehicle should be calculated using the equation (4).

     (4)

The result for section 2 is like the Figure 5.

Figure 5: Result of center correction.

The experimental parking lot like Figure 2 was used for the entire realization of auto parking system. The assumption that all parking spaces are occupied except for one space which the vehicle will be parked is applied for simplified realization. Blue color mark is used for the parking space determination like Figure 6. The target position that the vehicle should be parked in is matched with each parking space in advance because the color mark is not attached to stick out to camera on the exact center of parking space which the car should be park in. The process to detect the blue color mark is the same with the process to detect the red and green color mark for the estimation of the vehicle center and attitude estimation. The top view image is changed into HSV format image and a HSV filter is used for filtering blue color mark’s pixels and the filtered image is transformed into binary image. Then, blob labelling method is applied to get numbered pixels of the color mark. And, a different thing with section 2.1 is that the two opposite corners of the color mark are used to calculate the color marks’ center position because high accuracy is less needed compared to when the vehicle center is calculated but fast processing time for the process is important for the entire auto parking system. In the entire parking system, the vehicle center and attitude detection, and the control of the vehicle are the most important processes. And, the central computer determines whether the blue color mark is detected or not using the equation (5).

Figure 6: Parking space determination using blue color mark.

       (5)

Then, the specific parking space attached the detected blue color mark is chosen for parking and finally the central computer decides target position of the parking space that the vehicle should be parked in.

Steering control

In this research, the simple proportional steering control logic is used like the Figure 7(a). But, Fuzzy logic to put a little hysteresis on the target encoder signal is added to the logic because the signal of the potentiometer encoder is chattering a lot. The steering control stops when the encoder signal is within hysteresis like the logic Figure 7(b).

Figure 7: Steering control.

Position and attitude initialization

For convenience, forward and backward direction control is applied to the vehicle control in low speed because most people drives vehicle in low speed during parking. And also, the communication method based on Bluetooth 2.0 is used between the vehicle and the central computer for vehicle parking closed loop control and the vehicle center position and attitude from image, and the target parking position from section. 3 are utilized. The Entire logic is like Figure 8.

Figure 8: Vehicle control algorithm.

Firstly, when assuming controlled vehicle is on any random position, there are so many paths into the target parking position. So, initialization process of vehicle position and attitude is needed to remove the complexity of path planning.

The forward and backward parking method is considered and it is assumed that the car will approach to the right side of the parking lot. The initialization position is different according to the forward and backward parking to use the rotation space maximally. For example, in Figure 2, in case of forward parking and the target position is in the number 1, 2, 3, 4, the initial position should be at the lower right part of the space between number 1, 2, 3, 4 spaces and 7, 8, 9, 10 spaces. Meanwhile, in case of backward parking, the initial position should be at the upper right part of the approach space. Some example of initialization method using Fuzzy logic is described in Figure 9.

Figure 9: Position initialization Fuzzy logic.

The equation (6) is used for index calculation. Steering input which is the target encoder value is decided from Fuzzy controller.

           (6)

The reason why there is only x term in the index equation is y term is only related to the steering control and x term is only related to the forward and backward on-off control. And, the Fuzzy logic to put some hysteresis is used based on proportional controller.

Path planning and parking control

Another Fuzzy logic that is designed in advance is applied to park from the initial point to the final target position. For example, assuming the vehicle should be parked at the target position for the number 4 parking space in Figure 10, the central computer select the control strategy which includes the path and when the car should start steering. For more detail, to see the Figure 10, in the case that the target position is in the number 4 parking space in forward method, the car should start steering on the black circle position labeled ‘Steering start position’. The steering control method in section 4.1 and Index method described in section 4.2 are continuously used during the parking process. The equation (7) is used for the determination whether the car arrives at the starting position of steering and the equation (8) for the car arrives at the target position. Then, the car is arranged in forward direction and the control is over.

Figure 10: Parking path when parking in number 4 space.

      (7)

          (8)

The superscript ‘steer’ means the steering start position and ‘target’ means the target position from section 3.

Surround view system is another application to use the surveillance camera helpfully. When the car is approaching to the parking lot, the central computer can send the surround view image to the car and drivers can check which spaces are empty and can be assisted in parking through the surround view image. In this research, two images are combined together. One is the top view image of the parking lot pictured in advance when the camera is installed and one is the real time pictured image in top view [4]. The method to make images to the top view is mentioned in section 2.1. The reason to combine two images is that camera cannot see the hidden space because of some barriers like columns of a building. Drivers can see parking lines at least through the combined image. The equation (9) is used to combine two images.

       (9)

K₁, K₂ is the scale factor of the combined image. And also, for the view image around the vehicle, the combined image is rotated as much as the vehicle rotated angle induced in section 2.3. Then, the rectangular surround view image is cut with the vehicle center as the center. The result is like Figure 11. However, there is limitation with this method because somethings exist in the hidden space because of barriers cannot be shown in image. This problem can be solved to combine the image with one more vehicle view image which can be obtained camera sensors on the vehicle or to install more cameras in the parking lot and combine all images taken from each camera with basic parking lot image taken in advance.

Figure 11: Surround view image around the vehicle.

In this research, vehicle center and attitude estimation, parking space detection method, and vehicle control method are described. And, the Entire system logic is processed in 5 Hz which is possible to be used in real application. However, the camera image is vulnerable to the light intensity and brightness. Thus, the solution for this is needed in view point of commercialization. And, there is a drawback that is color marks should be attached on vehicles. However, this can be solved through change of marks like pattern or QR code on the top of vehicles which is not ugly for appearance. And when the central computer controls a few vehicles, tracking method seems to be applied.

This works was supported by a National Research Foundation of Korea (NRF) grant funded by theKorean government (MSIP) (No. 2010-0028680).