(45-4) 14 * << * >> * Russian * English * Content * All Issues

Strategies for generating panoramic video images without information about scene correspondences for multispectral distributed aperture systems
I.A. Kudinov 1, M.B. Nikiforov 1, I.S. Kholopov 1

1 Ryazan State Radio Engineering University named after V.F. Utkin,
390005, Ryazan, Russia, Gagarina 59/1

 PDF, 1800 kB

DOI: 10.18287/2412-6179-CO-846

Pages: 589-599.

Full text of article: Russian language.

Abstract:
We derive analytical expressions for calculating the number of elementary computational operations required to generate several personal regions of interest in a panoramic computer-vision distributed-aperture system using two alternative strategies: strategy 1 involves acquisition of a complete panoramic frame, followed by the selection of personal regions of interest, while with strategy 2 the region of interest is directly formed for each user. The parameters of analytical expressions include the number of cameras in the distributed system, the number of users, and the resolution of panorama and user frames. The formulas obtained for the given parameters make it possible to determine a strategy that would be optimal in terms of a criterion of the minimum number of elementary computational operations for generating multiple personal regions of interest. The region of interest is generated using only a priori information about the internal and external camera parameters, obtained as a result of their photogrammetric calibration with a universal test object, and does not take into account information about scene correspondences at the boundaries of intersecting fields of view.

Keywords:
panoramic image, camera calibration, quaternions.

Citation:
Kudinov IA, Nikiforov MB, Kholopov IS. Strategies for generating panoramic video images without information about scene correspondences for multispectral distributed aperture systems. Computer Optics 2021; 45(4): 589-599. DOI: 10.18287/2412-6179-CO-846.

References:
  1. Lazarenko VP, Dzhamiykov TS, Korotaev VV, Yarishev SN. Method for creation of spherical panoramas from images obtained by omnidirectional optoelectronic systems [In Russian]. Nauchno-tehnicheskij vestnik informatsionnyh tehnologij, mehaniki i optiki 2016; 16(1): 46-53.
  2. Prudnikov NV, Shlishevsky VB. All-round looking optoelectronic surveillance systems [In Russian]. Vestnik SGUGiT 2016; 33(1): 148-161.
  3. Pavlov OV. Determination of helmet spatial position in helmet-mounted information and control system [In Russian]. Izvestiya TulGU. Technicheskie nauki 2018; 9: 126-136.
  4. Rash ET, Russo MB, Letowski TR, Schmeisser ET, eds. Helmet-mounted displays: sensation, perception, and cognition issues. 1st ed. Fort Rucker: US Army Aeromedical Research Laboratory; 2009.
  5. Eremeev SV, Andrianov DE, Titov VS. An algorithm for matching spatial objects of different-scale maps based on topological data analysis. Computer Optics 2019; 43(6): 1021-1029. DOI: 10.18287/2412-6179-2019-43-6-1021-1029.
  6. Vizilter YuV, Gorbatsevich VS, Rubis AYu, Vygolov OV. Image shape matching using diffusion morphology and diffusion correlation. Computer Optics 2015; 39(2): 265-274. DOI: 10.18287/0134-2452-2015-39-2-265-274.
  7. Araújo SA, Kim HY. Grayscale template-matching invariant to rotation, scale, translation, brightness and contrast. IEEE Pacific-Rim Symposium on Image and Video Technology 2007; 4872: 100-113.
  8. Bessmeltsev VP, Bulushev ED. Fast image registration algorithm for automated inspection of laser micromachining. Computer Optics 2014; 38(2): 343-350. DOI: 10.18287/0134-2452-2014-38-2-343-350.
  9. Goshin YeV, Kotov AP. Method for camera motion parameters estimation from a small number of corresponding points using quaternions. Computer Optics 2020; 44(3): 446-453. DOI: 10.18287/2412-6179-CO-683.
  10. Brown M, Lowe D. Automatic panoramic image stitching using invariant features. Int J Comput Vis 2007; 74(1): 59-73.
  11. Goshin YeV, Kotov AP, Fursov VA. Two-stage formation of a spatial transformation for image matching. Computer Optics 2014; 38(4): 886-891. DOI: 10.18287/0134-2452-2014-38-4-886-891
  12. Efimov AI, Novikov AI. An algorithm for multistage projective transformation adjustment for image superimposition. Computer Optics 2016; 40(2): 258-265. DOI: 10.18287/2412-6179-2016-40-2-258-265.
  13. Donon Y, Paringer R, Kupriyanov A, Goshin Y. Blur-robust image registration and stitching. J Phys Conf Ser 2019; 1368(5): 052043. DOI: 10.1088/1742-6596/1368/5/052043.
  14. Kudinov IA, Pavlov OV, Kholopov IS, Khramov MYu. The algorithm for a video panorama construction and its software implementation using CUDA technology. CEUR Workshop Proc. 2017; 1902: 37-42.
  15. Kudinov IA, Pavlov OV, Kholopov IS, Khramov MYu. Real-time multispectral video panorama construction. CEUR Workshop Proc. 2018; 2210: 1-5.
  16. Kudinov IA, Nikiforov MB, Kholopov IS. Camera and auxiliary sensor calibration for a multispectral panoramic vision system with a distributed aperture. J Phys Conf Ser 2019; 1368(3): 032009. DOI: 10.1088/1742-6596/1368/3/032009.
  17. Kostyashkin LN, Nikiforov MB, eds. Image superimposition in correlation-extreme navigation systems [In Russian]. Moscow: "Radiotehnika" publisher; 2015.
  18. Kostyashkin LN, Nikiforov MB, eds. Image processing in avionic vision systems [In Russian]. Moscow: "Fizmatlit" Publisher; 2016.
  19. Brown M, Susstrunk S. Multi-spectral SIFT for scene category recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition 2011: 177-184. DOI: 10.1109/CVPR.2011.5995637.
  20. Efimov AI, Novikov AI, Sablina VA. Image superimposition technique in computer vision systems using contour analysis methods. Proc 5th Mediterranean Conf on Embedded Computing 2016: 132-137. DOI: 10.1109/MECO.2016.7525721
  21. Efimov AI, Kudinov IA, Melnik OV, Nikiforov MB, Kholopov IS. Digital multispectral images superimposition based on preliminary calibration or contour analysis: advantages and disadvantages. Proc ELEKTRO 2020 Conf 2020: 1-6. DOI: 10.1109/ELEKTRO49696.2020.9130339.
  22. Szeliski R. Image alignment and stitching: a tutorial. Found Trends Comput Graph Vis 2006; 2(1): 1-104.
  23. Ikeda S, Sato T, Yokoya N. High-resolution panoramic movie generation from video streams acquired by an omnidirectional multi-camera system. Proc IEEE Int Conf on Multisensor Fusion and Integration for Intelligent Systems (MFI’2003) 2003: 155-160. DOI: 10.1109/MFI-2003.2003.1232649.
  24. Rehder J, Siegwart R. Camera/IMU calibration revisited. IEEE Sens J 2017; 17(11): 3257-3268.
  25. Xiao Y, Ruan X, Chai J, Zhang X, Zhu X. Online IMU self-calibration for visual-inertial systems. Sensors 2019; 19(1624): 1-26.
  26. Hartley R, Zisserman A. Multiple view geometry in computer vision. 2nd ed. Cambridge: Cambridge University Press; 2003.
  27. Li J, Su J, Zeng X. A solution method for image distortion correction model based on bilinear interpolation. Computer Optics 2019; 43(1): 99-104. DOI: 10.18287/2412-6179-2019-43-1-99-104.
  28. Chelnokov YuN. Quaternion and biquaternion models and methods of mechanics of solids and their applications [In Russian]. Moscow: “Fizmatlit” Publisher; 2006.
  29. Kuipers JB. Quaternions and rotation sequences. 5th ed. Princeton, Oxford: Princeton University; 2002.
  30. Ming D, Zhang W, Zhong H, Qi H, Di S, Yongfei C, Xudong J, Guojun C, Fengyang L, Ting H, Hui W. Device employing checkerboard for calibrating visible light camera and thermal infrared imager camera [In Chinese]. Pat CN of Invent N 204287725 of April 22, 2015.
  31. Zhang JX, Zhong H, Qi H, Di S, Yongfei C, Ming D, Guojun C, Fengyang L, Jian W, Ting H, Hui W. Novel parameter combined calibration board for visible light camera and infrared thermal imager camera [In Chinese]. Pat CN of Invent N 204301863 of April 29, 2015.
  32. Vidas S, Lakemond R, Denman S, Fookes C, Sridharan S, Wark T. A mask-based approach for the geometric calibration of thermal-infrared cameras. IEEE Trans Instrum Meas 2012; 61(6): 1625-1635. DOI: 10.1109/TIM.2012.2182851.
  33. Ramegowda D, Mohideen MI, Boregowda LR, Sai B, Venkoparao VG. Thermal camera calibration. Pat US of Invent N 9621823 of April 11, 2017.
  34. Choinowski A, Dahlke D, Ernst I, Pless S. Automatic calibration and co-registration for a stereo camera system and a thermal imaging sensor using a chessboard. In: Proc Int Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2019; XLII-2/W13: 1631-1635. DOI: 10.5194/isprs-archives-XLII-2-W13-1631-2019.
  35. Alekseev AP, Goshin EV, Davydov NS, Ivliev NA, Nikonorov AV. Visual-inertial odometry algorithms on the base of thermal camera. CEUR Workshop Proc 2019; 2416: 183-188. DOI: 10.18287/1613-0073-2019-2416-183-188.
  36. St-Laurent L, Mikhnevich M, Bubel A, Prevost D. Passive calibration board for alignment of VIS-NIR, SWIR and LWIR images. Quant Infrared Thermogr J 2017; 14(2): 193-205. DOI: 10.1080/17686733.2017.1319529.
  37. Kudinov IA, Pavlov OV, Kholopov IS. Test pattern for calibration of video sensors of multispectral vision system [In Russian]. Pat RF of Invent N 2672466 of November 14, 2018, Russian Bull of Inventions N32, 2018.
  38. Chen Y. Huang F-y, Shi F-m, Liu B-q. Plane chessboard-based calibration method for a LWIR ultra-wide-angle camera. Appl Opt 2019; 58(4): 744-751. DOI: 10.1364/AO.58.000744.
  39. Zoetgnande YWK, Fougeres A-J, Cormier G, Dillenseger JL. Robust low-resolution thermal stereo camera calibration. Proc SPIE 2018; 11041: 110411D.  DOI: 10.1117/12.2523440.
  40. Wu H, Zhang J, Zhao Y, Yu X, He B. Camera calibration template [In Chinese]. Pat CN of Invent N 204695399 of October 07, 2015.
  41. BrightniteTM. Source: <https://elbitsystems.com/media/Brightnite_2016.pdf>.
  42. Layout of video panorama from 5 cameras and thermal imager [In Russian]. Source: <https://www.youtube.com/watch?v=HHdj0JcKpI8>.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20