(49-1) 14 * << * >> * Russian * English * Content * All Issues

Towards using smartphones as hyperspectral cameras
D. Reutsky 1,2, A. Pogadaev 1,2, D. Vladimirov 1,2, E. Ershov 1,2

1 Institute for Information Transmition Problems (Kharkevich Institute),
127051, Moscow, Bolshoy Karetny per. 19, build. 1;
2 Moscow Institute of Physics and Technology (National Research University),
141701, Russia, Dolgoprudny, Moscow Region, Institutskiy per. 9

 PDF, 6673 kB

DOI: 10.18287/2412-6179-CO-1315

Pages: 112-120.

Full text of article: Russian language.

Abstract:
Spectral reconstruction (SR, recovering spectra from RGB measurements) is a vital problem of computational photography. However, even recent algorithms do not provide acceptable accuracy. As a matter of curiosity, modern mobile devices open a new opportunity to improve the quality of SR by utilizing images from several cameras at once. This leads to the idea of creating a mobile hyperspectral camera for the general public. In this paper we investigate the achievable accuracy when using several identical cameras simultaneously in combination with different spectral filters. To find optimal filters, two algorithms are proposed: one learns linear regression for SR and the other learns fully connected neural network. As a result of numerical experiments, it is found that in the case of 4 cameras and 4 filters, the SR accuracy is three times higher than in the case of SR from RGB images.

Keywords:
spectrum, hyperspectral imaging, hyperspectral reconstruction.

Citation:
Reutsky DA, Pogadaev AV, Vladimirov DM, Ershov EI. Towards using smartphones as hyperspectral cameras. Computer Optics 2025; 49(1): 112-120. DOI: 10.18287/2412-6179-CO-1315.

References:
  1. Leon R, Martinez-Vega B, Fabelo H, et al. Non-invasive skin cancer diagnosis using hyperspectral imaging for in-situ clinical support. J Clin Med 2020; 9(6): 1662. DOI: 10.3390/jcm9061662.
  2. Fabelo H, Halicek M, Ortega S, et al. Deep learning-based framework for in vivo identification of glioblastoma tumor using hyperspectral images of human brain. Sensors 2019; 19(4): 920. DOI: 10.3390/s19040920.
  3. Underwood E, Ustin S, DiPietro D. Mapping nonnative plants using hyperspectral imagery. Remote Sens Environ 2003; 86(2): 150-161. DOI: https://doi.org/10.1016/S0034-4257(03)00096-8.
  4. Chu X, Miao P, Zhang K, Wei H, Fu H, Liu H, Jiang H, Ma Z. Green banana maturity classification and quality evaluation using hyperspectral imaging. Agriculture 2022; 12(4): 530. DOI: 10.3390/agriculture12040530.
  5. Specim, Spectral Imaging Ltd. Case study: Picvisa – machine vision solutions for waste treatment and recycling. 2020. Source: <https://www.specim.com/case-study-picvisa-machine-vision-solutions-for-waste-treatment-recycling>.
  6. Specim, Spectral Imaging Ltd. Spectral imaging in machine vision solutions. 2022. Source: <https://www.specim.com/industry>.
  7. Arad B, Timofte R, Ben-Shahar O, et al. NTIRE 2020 challenge on spectral reconstruction from an RGB image. 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops (CVPRW) 2020: 1806-1822. DOI: 10.1109/CVPRW50498.2020.00231.
  8. Aeschbacher J, Wu J, Timofte R. In defense of shallow learned spectral reconstruction from RGB images. 2017 IEEE Int Conf on Computer Vision Workshops (ICCVW) 2017: 471-479. DOI: 10.1109/ICCVW.2017.63.
  9. Shi Z, Chen C, Xiong Z, Liu D, Wu F. HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops (CVPRW) 2018: 939-947. DOI: 10.1109/CVPRW.2018.00139.
  10. Cai Y, Lin J, Lin Z, Wang H, Zhang Y, Pfister H, Timofte R, Van Gool L. MST++: Multi-stage spectral-wise transformer for efficient spectral reconstruction. 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops (CVPRW) 2022: 745-755. DOI: 10.1109/CVPRW56347.2022.00090.
  11. Helling S, Seidel E, Biehlig W. Algorithms for spectral color stimulus reconstruction with a seven-channel multispectral camera. CGIV 2004: The Second European Conf on Colour Graphics, Imaging and Vision 2004; 2: art00051. DOI: 10.2352/CGIV.2004.2.1.art00051.
  12. Valero EM, Nieves JL, Nascimento SMC, Amano K, Foster DH. Recovering spectral data from natural scenes with an RGB digital camera and colored filters. Color Res Appl 2007; 32(5): 352-360. DOI: 10.1002/col.20339.
  13. Finlayson GD, Zhu Y. Designing color filters that make cameras more colorimetric. IEEE Trans Image Process 2020; 30: 853-867. DOI: 10.1109/TIP.2020.3038523.
  14. Ng D-Y, Allebach JP. A subspace matching color filter design methodology for a multispectral imaging system. IEEE Trans Image Process 2006; 15(9): 2631-2643. DOI: 10.1109/TIP.2006.877384.
  15. Nie S, Gu L, Zheng Y, Lam A, Ono N, Sato I. Deeply learned filter response functions for hyperspectral reconstruction. 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition 2018: 4767-4776. DOI: 10.1109/CVPR.2018.00501.
  16. OMDIA. Smartphone oems shifting to higher camera resolution configurations with fewer lenses in response to continued semiconductor shortage, finds new Omdia research. 2021. Source: <https://omdia.tech.informa.com/pr/2021-dec/smartphone-oems-shifting-to-higher-camera-resolution-configurations-with-fewer-lenses-in-response-to-continued-semiconductor-shortage-finds-new-omdia-research>.
  17. Foster DH, Amano K, Nascimento SMC, Foster MJ. Frequency of metamerism in natural scenes. J Opt Soc Am A 2006; 23(10): 2359-2372. DOI: 10.1364/JOSAA.23.002359.
  18. Lin Y-T, Finlayson G. Physically plausible spectral reconstruction. Sensors 2020; 20(21): 6399. DOI: 10.3390/s20216399.
  19. Li Y, Fu Q, Heidrich W. Multispectral illumination estimation using deep unrolling network. 2021 IEEE/CVF Int Conf on Computer Vision (ICCV) 2021: 1-8. DOI: 10.1109/ICCV48922.2021.00267.
  20. Colorimetric illuminants. 2024. Source: <https://cie.co.at/publications/colorimetric-illuminants>.
  21. Rochester Institute of Technology. Camera spectral sensitivity database. 2013. Source: <http://www.gujinwei.org/research/camspec/db.html>.
  22. Shepelev DA, Bozhkova VP, Ershov EI, Nikolaev DP. Simulating shot noise of color underwater images. Computer Optics 2020; 44(4): 671-679. DOI: 10.18287/2412-6179-CO-754.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20