In recent years, the use of Unmanned Aerial Vehicles (UAVs) has increased significantly as these vehicles provide cost-effective and safe alternatives to manned flights in several civil and military applications. These robotic aircraft can employ a variety of sensors, as well as multisensor data fusion algorithms, to provide autonomy to the platform in the accomplishment of mission- and safety-critical tasks. UAVs are characterized by higher manoeuvr ability, reduced cost, longer endurance and less risk to human life compared to manned systems. One of the most important concepts is to use a multi-sensor integrated system to cope with the requirements of long/medium range navigation and landing. This can reduce cost, weight/volume and support requirements and, with the appropriate sensors and integration architecture, give increased accuracy and integrity of the overall system. The best candidates for such integration are indeed satellite navigation receivers and inertial sensors. In recent years, computer vision and Vision-Based Navigation (VBN) systems have started to be applied to UAVs.VBN can provide a self-contained autonomous navigation solution and can be used as an alternative (or an addition) to the traditional sensors (GPS, INS and integrated GPS/INS). The required information to perform autonomous navigation can be obtained from cameras which are compact and lightweight sensors. This is particularly attractive in UAV platforms, where weight and volume are tightly constrained. Sinopoli et al. [1] used a model-based approach to develop a system which processes image sequences from visual sensors fused with readings from GPS/INS to update a coarse, inaccurate 3D model of the surroundings. Digital Elevation Maps (DEM) was used to build the 3D model of the environment. Occupancy Grid Mapping was used in this study, in which the maps were divided into cells. Each cell had a probability value of an obstacle being present associated with it. Using this ‘risk map’ and the images from the visual sensors, the UAV was able to update its stored virtual map. Shortest path optimization techniques based on the Djikstra algorithm and Dynamic Programming were then used to perform obstacle avoidance and online computation of the shortest trajectory to the destination. Se et al. [2] proposed a system which deals with vision-based SLAM using a trinocular stereo system. In this study, Scale-Invariant Feature Transform (SIFT) was used for tracking natural landmarks and to build the 3D maps. The algorithm built submaps from multiple frames which were then merged together. The SIFT features detected in the current frame were matched to the pre-built database map in order to obtain the location of the vehicle. Stereo vision-based navigation systems were proposed by various authors including Cui et al. [3] and Matsumoto et al. [4] proposed a representation of the visual route taken by robots in appearance-based navigation. This approach was called the View-Sequenced Route Representation (VSRR) and was a sequence of images memorized in the recording run along the required route. The visual route connected the initial position and destination via a set of images. This visual route was used for localization and guidance in the autonomous run. Pattern recognition was achieved by matching the features detected in the current view of the camera with the stored images. Criteria for image capture during the learning stage were given in the study. The visual route was learnt while the robot was manually guided along the required trajectory. A matching error between the previous stored image and current view was used to control the capture of the next key image. The current view was captured and saved as the next key image when a preset error threshold was exceeded. Localization was carried out at the start of the autonomous run by comparing the current view with the saved visual route images. The key image with the greatest similarity to the current view represented the start of the visual route. The location of the robot depended purely on the key image used and no assumption was made of its location in 3D space. During the autonomous run, the matching error between the current view and key images was monitored in order to identify which image should be used for guidance. The robot was controlled so as to move from one image location to another and finally reach its destination. This ‘teach-and-replay’ approach was adopted by Courbon et al. [5,6], Chen et al. [7] and Remazeilles et al. [8]. In Courbon et al. [5,6], a single camera and natural landmarks were used to navigate a quadrotor UAV along the visual route. The key images were considered as waypoints to be followed in sensor space. Zero Normalized Cross Correlation was used for feature matching between the current view and the key images. A control system using the dynamic model of the UAV was developed. Its main task was to reduce the position error between the current view and key image to zero and to stabilize and control the UAV. Vision algorithms to measure the attitude of a UAV using the horizon and runway were presented by Xinhua et al. [9] and Dusha et al. [10]. The horizon is used by human pilots to control the pitch and roll of the aircraft while operating under visual flying rules. A similar concept is used by computer vision to provide an intuitive means of determining the attitude of an aircraft. This process is called Horizon-Based Attitude Estimation (HBAE). In [9], grayscale images were used for image processing. The horizon was assumed to be a straight line and appeared as an edge in the image. Texture energy method was used to detect it and this was used to compute the bank and pitch angle of the UAV. The position of the UAV with respect to the runway was found by computing the angles of the runway boundary lines. A Canny Edge detector was applied to part of the image below the horizon. The gradient of the edges was computed using the pixel coordinates which gave a rough indication of where the UAV was situated with respect to the runway. Dusha et al. [10] used a similar approach to develop algorithms to compute the attitude and attitude rates. A Sobel edge detector was applied to each channel of the RGB image. The three channels were then combined and Hough transform was used to detect the horizon. In this research, it was assumed that the camera frame and the body frames were coincidental and equations were developed so as to calculate the pitch and roll angle. The angular rates of the UAV were derived using optical flow of the horizon. Optical flow gives us additional information of the states of the UAV and is dependent on the angular rates, velocity and the distance of the detected features. During this research, it was observed that the image processing frontend was susceptible to false detection of the horizon if any other strong edges were present in the image. Therefore, an Extended Kalman Filter (EKF) was implemented to filter out these incorrect results. The performance of the algorithms was tested via test flights with a small UAV and a Cessna 172. Results of the test flight with the UAV showed an error in the calculated pitch and roll with standard deviations of 0.42 and 0.71 degrees respectively. Moving forward from these results, in our research we designed and tested a new VBN sensor specifically tailored for approach/landing applications which, in addition to horizon detection and image-flow, also employed runway features extraction during the approach phase. Additionally, various candidates were considered for integration with the VBN sensor, including Global Navigation Satellite Systems (GNSS) and Micro Electro Mechanical Systems (MEMS) based Inertial Measurement Units (IMUs). MEMS-IMUs are low-cost and low-volume/weight sensors particularly well suited for small/medium size UAV applications. However, their integration represent a challenge, which need to be addressed either by finding improvements to the existing analytical methods or by developing novel algorithmic approaches that counterbalance the use of less accurate inertial sensors. In line with the above discussions, the main objective of our research is to develop a low-cost and low-weight/volume Navigation and Guidance System (NGS) based on VBN and other low-cost and low-weight/volume sensors, capable of providing the required level of performance in all flight phases of a small/medium size UAV, with a special focus on precision approach and landing (i.e., the most demanding and potentially safety-critical flight phase), where VBN techniques can be fully exploited in a multisensory integrated architecture. The NGS is composed by a Multisensor Integrated Navigation System (MINS) using an Extended Kalman Filter (EKF) and an existing controller that employs Fuzzy Logic and Proportional-Integral-Differential (PID) technology.

As discussed above, VBN techniques use optical sensors (visual or infrared cameras) to extract visual features from images which are then used for localization in the surrounding environment. Cameras have evolved as attractive sensors as they help design economically viable systems with simpler hardware and software components. Computer vision has played an important role in the development of UAVs [11]. Considerable work has been made over the past decade in the area of vision-based techniques for navigation and control [9]. UAV visionbased systems have been developed for various applications ranging from autonomous landing to obstacle avoidance. Other applications looked into the possible augmentation INS and GPS/INS by using VBN measurements [12]. As discussed above, several VBN sensors and techniques have been developed. However, the vast majority of VBN sensor schemes fall into one of the following two categories [13]: Modelbased Approach (MBA) and Appearance-based Approach (ABA). MBA uses feature tracking in images and create a 3D- model of the workspace in which robots or UAV operates [14]. The 3D maps are created in an offline process using a priori information of the environment. Localisation is carried out using feature matching between the current view of the camera and the stored 3D model. The orientation of the robot is found from 3D-2D correspondence. MBA has been extensively researched in the past and is the most common technique currently implemented for vision-based navigation. However, the accuracy of this method is dependent on the features used for tracking, robustness of the feature descriptors and the algorithms used for matching and reconstruction. The reconstruction in turn relies on proper camera calibration and sensor noise. Knowledge of surroundings so as to develop the 3D models is also required prior to implementation which may not be case in most situations.ABA algorithms eliminate the need for a metric model as they work directly in the sensor space. This approach utilizes the appearance of the whole scene in the image, contrary to model-based approach which uses distinct objects such as landmarks or edges [4]. The environment is represented in the form of key images taken at various locations using the visual sensors. This continuous set of images describes the path to be followed by the robot. The images are captured while manually guiding the robot through the workspace. In this approach, localisation is carried out by finding the key image with the most similarity to the current view. The robot is controlled by either coding the action required to move from one key image to another or by a more robust approach using visual servoing [8,15]. The ABA approach is relatively new and has gained active interest. The modelling of the surrounding using a set of key images is more straightforward to implement compared to 3D modelling. A major drawback of this method is its limited applicability. The robot assumes that the key image database for a particular workspace is already stored in its memory. Therefore, the key images need to be recaptured each time the robot moves to a new workspace. It is limited to work in the explored regions which have been visualised during the learning stage [16]. The ABA approach has a disadvantage in requiring a large amount of memory to store the images and is computationally more costly than MBA. However, due to improvements in computer technology, this technique has become a viable solution in many application areas. We selected the ABA approach for the design of our VBN sensor system.

Learning stage

The first step required for appearance based navigation is the learning stage. During this stage, a video is recorded using the onboard camera while guiding the aircraft manually during the landing phase. The recorded video is composed of a series of frames which form the visual route for landing. This series of frames is essentially a set of images connecting the initial and target location images. The key frames are first sampled and the selected images are stored in the memory to be used for guidance during autonomous landing of the aircraft. During the learning stage, the UAV is flown manually meeting the Required Navigation Performance (RNP) requirements of precision approach and landing. If available, Instrument Landing System (ILS) can also be used for guidance. It should be noted that the visual route captured while landing on a runway, can only be used for that particular runway. If the UAV needs to land at multiple runways according to its mission, the visual route for all the runways is required to be stored in the memory. The following two methods can be employed for image capture during the learning stage.

Method 1: Frames are captured from the video input at fixed time intervals. The key frames are selected manually in this case.

Method 2: Frames are captured using a matching difference threshold [4]. This matching difference threshold is defined in number of pixels and can be obtained by tracking the features in the current view and the previously stored key image. The key images can then be selected based on the threshold and stored in the memory.

The algorithm starts by taking an image at the starting point. Let this image be M_i captured at location i. As the aircraft moves forward, the difference between the current view (V) and the image M_i increases. This difference keeps increasing until it reaches the set threshold value (x). At this point, a new image M_i+1 is taken (replacing the previous image M_i) and the process are repeated until the aircraft reaches its destination. The learning stage process is summarized in figure 1a.

Figure 1: Image capture (a) and localisation (b).

Localisation

Localization is a process which determines the current location of the aircraft at the start of autonomous run. This process identifies the key image which is the closest match to the current view. The current view of the aircraft is compared with a certain number of images, preferably the ones at the start of the visual route. The key image with the least matching difference is considered to be the start of the visual route to be followed by the aircraft. At the start of the autonomous run, the aircraft is approximately at the starting position of the visual route. The current view, captured from the on-board camera is compared with a set of images (stored previously in the memory during the learning stage) in order to find the location of aircraft with respect to the visual route. The key image with the least matching difference with the current view is considered to be the location of the aircraft and marks the start of the visual route to be followed. The process is summarized in figure 1b.

Figure 2: Autonomous run (a) and matching difference/key-frame selection (b).

In this example, the number of images to be compared (X) is taken as 20. First, the algorithm loads the current view (V) and the first key frame (M_i). Then the difference between the current view and the current key frame is computed. The algorithm then loads the next key frame M_i+1 and again computes the difference with the current view. If this difference is less than the previous difference, M_i+1 replaces M_i, and the process is repeated again. Otherwise, M_i is considered as the current location of the aircraft.

Autonomous run

During the autonomous run phase, the aircraft follows the visual route (previously stored in memory during the learning stage) from the image identified as the current location of the aircraft during localization. The set of key images stored as the visual route can be considered as the target waypoints for the aircraft in sensor space. The current view is compared to the key images so as to perform visual servoing. The approach followed to identify the key image to be used for visual servoing, is describes as follows. Let M_j be the current key frame, i.e. image with the least matching difference with the current view. During the autonomous run, the current key image and the next key image (M_j+1) are loaded. The matching differences of the current view V with M_j and M_j+1 (which are D_Mj,V and D_Mj+1,V respectively) are tracked. When the matching difference D_Mj,V exceeds D_Mj+1,V, M_j+1 is taken as the current key image replacing M_j and the next key image is loaded as M_j+1. This same process keeps repeating until the aircraft reaches its destination, that is the final key frame. Figure 2a summarises the process of autonomous run in the form of a flow chart while the change in matching difference for different key frames during autonomous run is presented in figure 2b.

The proposed vision based navigation process is depicted in figure 3. The key frames represent the visual route the aircraft requires to follow.

Figure 3: VBN process.

The figure 4 shows that the key frame 2 is identified as the starting point of the visual route during the localization process. The onboard computer tracks the matching difference between current view and the second and third key frames until the difference for key frame 2 and the current view exceeds the difference of key frame 3 and the current view. At this stage, key image 3 is used to control the UAV and the matching differences between key frames 3, 4 and the current view are monitored. This process is repeated until the UAV reaches its destination. To capture the outside view, a monochrome Flea camera from Point Grey Research was used. The main specification of the camera and lenses are listed in table 1.

Sensor type	Sony ICX204AQ/AL 1/3" CCD sensor Scan type : Progressive
Resolution	1024x768 BW
Format	8-bit or 16-bit, 12-bit AtoD
Pixel Size	4.65μm x 4.65μm
Frame Rates	1.875, 3.75, 7.5, 15, 30fps
Video output signal	8 bits per pixel / 12 bits per pixel digital data
Interfaces	6-pin IEEE-1394 for camera control and video data transmission 4 general purpose digital input/output pins
Voltage Requirements	8-32V
Power Consumption	< 3W
Gain	Automatic/Manual modes at 0.035dB resolution (0 to 24dB)
Shutter	Automatic/Manual/Extended Shutter modes (20μs to 66ms @ 15Hz)
Trigger Modes	DCAM v1.31 Trigger Modes 0, 1 and 3
SNR	50dB or better at minimum gain
Camera Dimensions (no lenses)	30mm x 31mm x 29mm
Mass	60g without optics
Operating Temperature	0° to 45°C
Focal length	3.5- 8.0 mm
Max CCD Format	1/3"
Aperture	F1.4 – 16 (closed) - Manual control
Maximum Field of View (FOV)	Horizontal: 77.6°/Vertical: 57.6°
Min working distance	0.4m
Lenses Dimensions (mm)	34.0 diameter x 43.5 length

Table 1: Point Grey Flea and Lenses Specifications.

Figure 4: Functional Architecture of the IPM.

This camera was also used in a previous study on stereo vision [17] and was selected for this project. The Flea camera is a pinhole Charged Coupled Device (CCD) camera with a maximum image resolution of 1024×768 pixels. It is capable of recording videos at a maximum rate of 30 fps. An IEEE 1394 connection was used to interface the camera and computer with a data transfer speed of 400 Mbps. A fully manual lens which allows manual focus and zoom was fitted to the camera. A UV filter was employed to protect the lens and to prevent haziness due to ultraviolet light.

Image processing module

The Image Processing Module (IPM) of the VBN system detects horizon and runway centreline from the images and computes the aircraft attitude, body rates and deviation from the runway centreline. Figure 5 shows the functional architecture of the IPM.

Figure 5: FImage processing module flowchart.

The detailed processing performed by the IPM isillustrated in figure 6. As a first step, the size of the image is reduced from 1024×768 pixels to 512×384 pixels. After some trials, it was found that this size reduction speeds up the processing without significantly affecting the features detection process. The features such as the horizon and the runway centreline are extracted from the images for attitude computation. The horizon is detected in the image by using canny edge detector while the runway centreline is identified with the help of Hough Transform. The features are extracted from both, the current view (image received from the on-board camera) and the current key frame. The roll and pitch are computed from the detected horizon while the runway centreline in used to compute the deviation of aircraft from the runway centreline. Then the roll and pitch difference are computed between the current view and the current key frame. Optical flow is determined for all the points on the detected horizon line in the images. The aircraft body rates are then computed based on the optical flow values. The image processing module provides the aircraft attitude, body rates, pitch and roll differences between current view and key frame, and deviation from the runway centreline. The attitude of the aircraft is computed based on the detected horizon and the runway. The algorithm calculates the pitch and roll of the aircraft using the horizon information while aircraft deviation from the runway centreline is computed using the location of runway centreline in the current image.

Figure 6: Coordinate system.

Figure 6 shows the relationship between the body (aircraft) frame (O_b,X_b,Y_b, Z_b), camera frame (O_c, X_c,Y_c, Z_c) and Earth frame coordinates (O_w, X_w, Y_w, Z_w).

The position of a 3D space point P in Earth coordinates is represented by a vector X_p^w with components x_p, y_p and z_p in the Earth frame. The position of aircraft centre with respect to the Earth coordinates is represented by the vector X_b^w components x_b, y_b and z_b in the Earth frame. The vector X_p^c represents the position of the point P with respect to the camera frame with components x_cp, y_cp and z_cp in the camera frame. The position of centre of camera lens with respect to the body frame is represented by the vector X_c^b. The vector X_c^w represents the position of lens centre with respect to the ground frame with components x_c, y_c and z_c in the ground frame. The position of point P with respect to body frame with components in the Earth frame can be computed as .The transformation matrix from Earth frame to body frame C_w^b can be obtained in terms of the yaw ψ, pitch θ, and roll angle ф as:

(1)

The position of point P with respect to the aircraft’s body with components in the body frame can be obtained as:

(2)

The position of point P with respect to the camera frame can also be found in a similar way as:

(3)

Where Cbc is the constant transformation matrix from the body frame to the camera frame. With the assumption that the camera is fixed with respect to the body and the angle from the camera optical axis to the longitudinal axis of the aircraft is fixed value, is a known constant vector with components k_x, k_y and k_z in the camera frame. In this case, the velocity and rotation rate of aircraft are the same as those of the camera. Thus, the position and attitude of the aircraft can be easily computed according to those of the camera as:

(4)

(5)

(6)

(7)

Where ф_c is the roll, θ_c is the pitch, ψ_c is the yaw and θ₀ is the angle of incidence of the camera. The transformation matrix from camera frame to the ground frame, represented by C_c^w, is obtained from:

(8)

(9)

Eq. (9) represents the transformation matrix from the Earth frame to the camera frame coordinates [2]. From now onwards, only the state estimates of the camera are considered. The position of 3D point P with respect to camera frame is given by:

(10)

With components in the camera frame given by:

(11)

Then, the coordinates (u,v) of P in the image plane can be obtained from:

(12)

(13)

Using the coordinate previously defined, the point P is assumed to be located on the detected horizon line. As the Earth’s surface is approximated by a plane [3,9], a normal vector to the plane, n_w can be described as:

(14)

If the horizon line is described by a point X_p^w and a direction vector l_w tangential to the line of horizon visible to the image plane, then:

(15)

(16)

Where x is an arbitrary point along x-axis and d is the distance to horizon along z-axis. If the camera is assumed to be placed directly above the origin of the ground frame, the position of camera can be described as:

(17)

Then, a point on horizon may be expressed as:

(18)

The horizon projection on the image plane can be described by the point p and a direction vector m as:

(19)

(20)

Where gives the gradient of the horizon line. As the position of the horizon lies on the surface of the ground plane, therefore:

(21)

Substituting gives:

(22)

The direction vector of the horizon line l_w lies on the plane and is therefore orthogonal to the normal vector. Therefore:

(23)

Equations (22) and (23) are in form known as line plane correspondence problem as proposed by [18]. Recalling the equations for a projective perspective camera:

(24)

Substituting Eq. (24) into Eq. (22), roll angle can be derived as [6]:

(25)

Which is an intuitive result that roll angle is dependent on the gradient of the horizon line on the image plane. Similarly, it can be shown that substituting Eq. (20) into Eq. (22), the pitch angle can be derived as [6]:

(26)

If the distance to the horizon is much greater than the height of the aircraft (i.e., d >> h , the expression for pitch reduces to the following:

(27)

Which shows that pitch is dependent on roll angle, focal length and the position of horizon in the image plane. Optical flow depends on the velocity of the platform, angular rates of the platform and the distance to any features observed [6]. Differentiating Eq. (24) we obtain:

(28)

(29)

Substituting Eq. (28) into the time derivative of Eq. (18) yields the classical optical flow equations [6]:

(30)

If the observed point lies on the horizon, then Z will be large and the translational component will be negligible. In this case, Eq. (30) reduces to:

(31)

To minimize the effect of errors, a Kalman filter is employed. The state vector consists of the roll angle, pitch angle and body rates of the aircraft. It is assumed that the motion model of the aircraft is disturbed by uncorrelated zero-mean Gaussian noise.

(32)

(33)

Where x (k ) is the state vector of the aircraft while η (k ) is the uncorrelated zero-mean Gaussian random vector with diagonal covariance matrix Q(k ). The measurement vector at time k is represented by z(k) and w(k) is the zero-mean Gaussian noise vector with a diagonal matrix R(k). If the body rates are assumed to be approximately constant during the sampling interval Δt and first order integration is applied, then the state transition equations are as follows:

(34)

Where

(35)

(36)

The measurement equations are comprised of direct observations of the pitch and the roll from the horizon and i optical flow observations on the detected horizon line. Therefore, the length of the measurement vector z(k) is 2(i+1). The relation of measurement vector and the states is represented by following linear equations:

(37)

VBN sensor performance

Based on various laboratories, ground and flight test activities with small aircraft and UAV platforms, the performance of the VBN sensor were evaluated. Figure 7 shows a sample image used for testing the VBN sensor algorithms and the results of the corresponding horizon detection process for attitude estimation purposes.

Figure 7: Horizon detected from the test image (landing phase).

The algorithm detects the horizon and the runway centreline from the images. The horizon is detected in the image by using canny edge detector with a threshold of 0.9 and standard deviation of 50. In this experiment, the values of the threshold and the standard deviation were selected by hit-and-trial method. The resulting image after applying the canny edge detector is a binary image. The algorithm assigns value ‘1’ to the pixels detected as horizon while the rest of the pixels in the image are assigned value ‘0’. From this test image, the computed roll angle is 1.26° and the pitch angle is -10.17°. To detect the runway in the image, kernel filter and Hough Transform are employed. The runway detected from the same test image is shown in figure 8. For this image, the location of the runway centreline was computed in pixels as 261. The features were extracted from both the current view (image received from the on-board camera) and the current key frame. After the pitch, roll and centreline values were determined, the roll/pitch differences and the deviation from centreline are computed between the current view and the current key frame.

Figure 8: Runway detected in the test image.

The algorithm also computes the optical flow for all the points on the detected horizon line in the images. The optical flow is determined based on the displacement of points in two consecutive frames of a video. The algorithm takes two consecutive frames at a time and determines the motion for each point on the horizon. These optical flow values are used to compute the body rates of the aircraft. An example of the optical flow calculation is shown in Fig. 8, where the original image (from the camera) is shown on the left and the image on the right shows the optical flow vectors (in red) computed for the detected horizon line. The vectors are magnified by a factor of 20. Since the vectors on the right half of the horizon line are pointing upwards and the vectors on the left halfare pointing downwards, the aircraft is performing roll motion (clockwise direction).

The real-time performance of the IPM algorithms were evaluated using a combination of experimental data(from the VBN camera) collected in flight and IPM simulation/data analysis performed on the ground using Matlab. The algorithm processed the video frame by frame and extracted horizon and the runway from each frame. The roll and pitch of the aircraft were computed based on the horizon detected in each frame. The algorithm also identified the location of runway centreline in each frame which was further used to calculate the deviation of the aircraft from the runway centreline. Kalman filter was employed to reduce the effect of errors in the measurements. The roll and roll-rate results obtained for 800 frames are shown in figure 9. Similarly, figure 10 depicts the results for pitch and pitch-rate. The computed location of centreline (pixels) and the centreline drift rate (pixels per second) are shown in figure 11.

Figure 9: Received image from camera (left) and optical flow computed for the detected horizon in the image (right).

Figure 10: Roll and roll-rate computed from the test video.

Figure 11: Roll and roll-rate computed from the test video.

Although the test activities were carried out in a limited portion of the aircraft/UAV operational flight envelopes, some preliminary error analysis was performed comparing the performance of the VBN sensor an INS. The mean and standard deviation of the VBN attitude and attitude-rate measurements are listed in table 2.

Table 2: VBS Attitude and Angular Rates Errors Parameters.

The performance of the VBN sensor is strongly dependent on the characteristics of the employed camera. The developed algorithms are unable to determine the attitude of the aircraft in case of absence of horizon in the image. Similarly, the deviation of the aircraft from the runway centreline cannot be computed in the absence of runway in the image. The most severe physical constrain is imposed by the FOV of the camera. The maximum vertical and horizontal FOVs of the Flea Camera are 57.6° and 77.6° respectively. Due to this limitation, the VBN sensor can compute a minimum pitch angle of -28.8° and a maximum of +28.8°. Additionally, environmental factors such as fog, night/lowlight conditions or rain also affect the horizon/runway visibility and degrade the performance of the VBN system.

There are a number of limitations and challenges associated to the employment VBN sensors in UAV platforms. VBN is best exploited at low altitudes, where sufficient features can be extracted from the surrounding. The FOV of the camera is limited and, due to payload limitations, it is often impractical to install multiple cameras. When multiple cameras are installed, additional processing is required for data exploitation. In this case, also stereo vision techniques can be implemented. Wind and turbulence disturbances must be modelled and accounted for in the VBN processing. Additionally the performance of VBN can be very poor in low-visibility conditions (performance enhancement can be achieved employing infrared sensors as well). However, despite these limitations and challenges, VBN is a promising technology for small-to-medium size UAV navigation and guidance applications, especially when integrated with other low-cost and low-weight/volume sensors currently available. In our research, we developed an integrated NGS approachemployingtwo state-of-the-art physical sensor: MEMS-based INS and GPS, as well as augmentation from Aircraft Dynamic Models (Virtual Sensor) in specific flight phases.

GNSS and MEMS-INS sensors characteristics

GNSS can provide high-accuracy position and velocity data using pseudorange, carrier phase, Doppler observables or various combinations of these measurements. Additionally, using multiple antennae suitably positioned in the aircraft, GNSS can also provide attitude data. In this research, we considered GPS Standards Positioning Service (SPS) pseudorange measurements for position and velocity computations. Additional research is currently being conducted on GPS/GNSS Carrier Phase Measurements (CFM) for attitude estimation. Table 3 lists the position and velocity of state-of-the-art SPS GPS receivers. Position error parameters are from [18] and velocity error parameters are from [19], in which an improved time differencing carrier phase velocity estimation method was adopted. Typically, GPS position and velocity measurements are provided at a rate of 1 Hz.

Table 3: GPS position and velocity errors.

An Inertial Navigation System (INS) can determine position, velocity and attitude of a UAV based on the input provided by various kinds of Inertial Measurement Units (IMUs). These units include 3-axis gyroscopes, measuring the roll, pitch and yaw rates of the aircraft around the body-axis. They also comprise 3-axis accelerometers determining the specific forcesin the inertial reference frame. In our research, we considered a strap-down INS employing low-cost MEMS Inertial Measurement Units (IMUs). MEMS-based IMUs are low-cost and low-weight/volume devices that represent an attractive alternative to high-cost traditional INS sensors, especially for general aviation or small UAVs applications. Additionally, MEMS sensors do not necessitate high power and the level of maintenance required is far lower than for high-end INS sensors [20]. The main drawback of these sensors is the relatively poor level of accuracy of the measurements that they provide. In our research, INS-MEMS errors are modeled as White Noise (WN) or as Gauss-Markov (GM) processes [21,22]. Table 4 lists the MEMS-INS error parameters considered in our research.

Table 4: MEMS-INS error parameters.

ADM virtual sensor characteristics

The ADM Virtual Sensor is essentially a knowledge-based module used to augment the navigation state vector by predicting the UAV flight dynamics (aircraft trajectory and attitude motion). The ADM can employ either a 6-Degree of Freedom (6-DOF) or a 3-DOF variable mass model with suitable constraints applied in the different phases of the UAV flight. The input data required to run these models are made available from aircraft physical sensors (i.e., aircraft data network stream) and form ad-hoc databases. Additionally, for the 3-DOF case, an automatic manoeuvre recognition module is implemented to model the transitions between the different UAV flight phases. Typical ADM error parameters are listed in table 5 [21,22]. Table 6 lists the associated error statistics obtained in a wide range of dynamics conditions for 20 seconds runtime.

Table 5: ADM error parameters.

Table 6: ADM error statistics.

The data provided by all sensors are blended using suitable data fusion algorithms. Due to the non-linearity of the sensor models, an EKF was developed to fuse the information provided by the different MINS sensors and to provide estimates of position, velocity and attitude of the platform in real-time. Two different integrated navigation system architectures were defined, including VBN/IMU/GPS (VIG) and VIG/ ADM (VIGA). The VIG architecture uses VBN at 20 Hz and GPS at 1 Hz to augment the MEMS-IMU running at 100 Hz. The VIGA architecture includes the ADM (computations performed at 100 Hz) to provide attitude channel augmentation. The corresponding VIG and VIGA integrated navigation modes were simulated using MATLAB™ covering all relevant flight phases of the AEROSONDE UAV (straight climb, straight-and-level flight, straight turning, turning descend/ climb, straight descent, etc.). The navigation system outputs were fed to a hybrid Fuzzy-logic/PID controller designed at Cranfield University for the AEROSONDE UAV and capable of operating with stand-alone VBN, as well as with VIG/VIGA and other sensors data [23,24].

VIG and VIGA architectures

The VIG architecture is illustrated in figure 12. The INS position and velocity provided by the navigation processor are compared to the GPS position and velocity to form the measurement input of the data fusion block containing the EKF. A similar process is also applied to the INS and VBN attitude angles, whose differences are incorporated in the EKF measurement vector. The EKF provides estimates of the Position, Velocity and Attitude (PVA) errors, which are then removed from the sensor measurements to obtain the corrected PVA states. The corrected PVA and estimates of accelerometer and gyroscope biases are also used to update the INS raw measurements.

Figure 12: Computed location and rate of change of centreline location from the test video.

The VIGA architecture is illustrated in figure 13. As before, the INS position and velocity provided by the navigation processor are compared to the GPS data to form the measurement input of EKF. Additionally, in this case, the attitude data provided by the ADM and the INS are compared to feed the EKF at 100 Hz, and the attitude data provided by the VBS and INS are compared at 20 Hz and input to the EKF. Like before, the EKF provides estimations of PVA errors, which are removed from the INS measurements to obtain the corrected PVA states. Again, the corrected PVA and estimates of accelerometer and gyroscope biases are used to update INS raw measurements.

Figure 13: VIG architecture.

Both the VIG and VIGA multi-sensor architectures were tested by simulation in an appropriate sequence of flight manoeuvres representative of the AEROSONDE UAV operational flight envelope. An FLC/PID controller was used for simulation. The duration of the simulation is 1150 seconds (approximately 19 minutes). The horizontal and vertical flight profiles are shown in figure 14.

Figure 14: VIGA architecture.

The list of the different simulated flight manoeuvres and associated control inputs is provided in table 7. The numbered waypoints are the same shown in figure 15.

Figure 15: Horizontal and vertical flight profiles.

Table 7: Flight manoeuvres and control inputs.

The VIG position error time histories (east, north and down) are shown in figure 16. For comparison, also the GPS position errors (unfiltered)are shown. Table 8 presents the position error statistics associated to each flight phase.

Table 8: VIG position error statistics.

Figure 16: VIG position error time histories.

The VIG velocity error time histories are shown in figure 17. For comparison, the GPS velocity error time histories are also shown. GPS is the dominating sensor for velocity computations but a significant improvement is with the VIG system on the accuracy of the vertical data. Table 9 shows the velocity error statistics associated to each flight phase. The velocity error is determined by comparing the velocities calculated in presence of modeled errors, with the nominal velocity values.

Table 9: VIG velocity error statistics.

Figure 17: VIG velocity error time histories.

The attitude error time histories of the VIG system are shown in figure 18. Table 10 presents the associated attitude error statistics.

Table 10: VIG attitude error statistics.

Figure 18: VAttitude error time histories.

As discussed above, the ADM data were used in the VIGA architecture to update the attitude channel (the position and velocity channels are derived from the VIG system). Therefore, only the attitude error statistics of the VIGA system are presented here. The time histories of the VIGA attitude errors are shown in figure 19 and compared with the corresponding VIG attitude errors. Table 11 presents the VIGA attitude error statistics.

Figure 19: Comparison of VIGA and VIG attitude errors.

Table 11: VIGA attitude error statistics.

During the initial VIGA simulation runs it was evidenced that the ADM data cannot be used without being reinitialized regularly. For the AEROSONDE UAV manoeuvres listed in table 12, it was found that the optimal period between ADM reinitialisation was in the order of 20 seconds. Converting the data in Table 11 and 12 to the corresponding RMS (95%) values, we obtain the error data in table 13 and 14. Comparing the two tables, it is evident that the ADM virtual sensor contributes to a moderate reduction of the overall attitude error budget in all relevant flight phases.

Table 12: VIG attitude RMS-95% errors.

Table 13: VIGA attitude RMS-95% errors.

To conclude the simulation data analysis, table 14 shows a comparison of the VIG/VIGA horizontal and vertical accuracy (RMS- 95%) with the required accuracy levels for precision approach [25,26]. The VIG/VIGA performances are in line with CAT II precision approach requirements. Future research will address the possible synergies of the VIG/VIGA architectures with GPS/GNSS space, ground and aircraftbased augmentation systems.

Table 14: VIGA attitude RMS-95% errors.

In this paper we have described the research activities performed to design a low-cost and low-weight/volume integrated NGS system suitable for small/medium size UAV applications. As a first step, we designed and tested a VBN sensor employing appearance-based techniques and specifically tailored for UAV low-level flight, including precision approach and landing operations. In addition to horizon detection and image-flow, the VBN sensor also employed runway features extraction during the approach phase. Various candidates were considered for integration with the VBN sensor and, as a result, GPS and MEMS-IMUs (with possible augmentation from ADM) were finally selected. The multisensory integration was accomplished with an EKF. The attitude/attitude-rate accuracies obtained with the VBN sensor were evaluated by a combination of laboratory, ground and flight test activities. The results were satisfactory in low-level flight and during the approach and landing phase of the UAV flight. However, the VBN sensor performance was strongly dependent on the characteristics of the employed camera. The algorithms developed are unable to determine the attitude of the aircraft in case of absence of horizon in the image. Similarly, the deviation of the aircraft from the runway centreline cannot be computed in the absence of runway in the image. The most severe physical constrain is imposed by the angular FOV of the camera. The maximum vertical and horizontal FOVs of the employed camera are 57.6° and 77.6° respectively. Due to this limitation, the VBN sensor can compute a minimum pitch angle of -28.8° and a maximum of +28.8°. Current research shows that for wind speeds greater than 20 m/s, the VBN/IPM algorithms are marginally usable for navigation purposes as well as for guidance. Environmental factors such as fog, night/low-light conditions or rain also affect the horizon/runway visibility and degrade the performance of the VBN system. To cope with these limitations, current research is investigating the potential synergies obtained by integrating daylight camera vision sensors with Infrared and Night Vision Imaging Sensors (IR/NVIS). Simulation of the VIG integrated navigation mode showed that this integration scheme can achieve horizontal/vertical position accuracies in line with CAT-II precision approach requirements, with a significant improvement compared to stand-alone SPS GPS. An improvement was also observed in the accuracy of the vertical velocity data. Additionally, simulation of the VIGA navigation mode showed promising results since, in most cases, the attitude accuracy is higher using the ADM/ VBS/IMU rather than using VBS/IMU only. However, due to rapid divergence of the ADM virtual sensor, there is a need for a frequent re-initialization of the ADM data module, which is strongly dependent on the UAV flight dynamics and the specific manoeuvres/flightphase transitions performed. In the considered portion of the UAV operational flight envelope, the required re-initialization interval was approximately 20 seconds. To cope with this issue, the original ADM is being modified to take into account specific manoeuvre constraints and the transition states between various manoeuvres are being carefully modeled. Additionally, an automatic manoeuvre recognition algorithm is being developed for updating the ADM in real-time and providing direct feedback to the VBN sensor. It is expected that, adopting this approach, the performances of the multisensory integrated NGS will be significantly enhanced both in terms of data accuracy and data continuity.