The performance of a stereo vision system depends on several processing stages, from stereo system calibration to 3D data extraction, including feature detection and matching cost functions. Although a large amount of work can be found in the stereo vision literature, when sensors with similar characteristics are used, the multimodal case needs some adaptations or the redefinition of classical stereo algorithms. For instance, in the multimodal case similarity functions such as: SAD (sum of absolute differences), NCC (normalized cross correlation), SSD (sum of squared differences) or Census transform cannot be used since a linear correlation between the data cannot be assumed.
In the SiMeVé project different non linear similarity functions will be studied to establish the relationship between multispectral images. In other words, different strategies to associate information content between IR (both including LWIR and NIR) and VS images will be explored in order to obtain sparse and dense disparity maps. Multimodal matching has been widely studied in registration and fusion problems, specially in medical imaging. However, there are few research related with the correspondence problem when infrared and color images are considered.
The pipeline generally studied in the classical stereo vision domain will be reviewed and adapted in the SiMeVé project, in particular some of the taks to be tackled are as follow:
-
Multimodal stereo calibration
-
Multimodal feature detection & description
-
Multimodal matching cost function
-
Multimodal sparse/dense disparity map generation