1.1.

Obstacles in Pathology

There is an inherent compromise in skin cancer diagnosis, to both not miss cases in the early stage and to avoid unnecessary excisions. The gold standard is a histopathological biopsy, which requires several days to produce results. Although the clinical setting differs between primary and secondary care, the compromise persists. Grossing is performed by an experienced anatomic pathologist, who evaluates the excised tissue specimen before the microscope evaluation. The purpose of anatomic pathology is to accurately determine tumor margins. Early detection in situ, namely before grossing, improves considerably the disease prognosis, especially for melanoma cancers. The current state-of-the-art in clinical practice is dermoscopy, which can improve the diagnosis of skin cancer, but still suffers in terms of specificity.⁶ Gross pathology lacks automation and cross-laboratory protocol standardization, which introduces discrepancies in diagnosis. In addition, it results in over-reliance on the skill of the pathologist. Therefore, patient and medical staff must wait for the biopsy, which apart from inducing costs due to the delay increases the probability of resection due to the unavailability of tumor margins pre- and intraoperatively. The discovery of compromised cancer margins results in increased overall treatment costs, treatment duration, and patient discomfort increase. Along with the increased prevalence of skin cancer^1,2 and the shortage of experienced pathologists, a substantial workload is assigned to the pathology lab. Furthermore, amid the global Covid-19 pandemic, enhancing the digital flow and offering opportunities for remote education and diagnosis are quickly becoming the turning point of medical research.

1.2.

Optical Properties of the Skin

Optical properties of the tissue are commonly used in diagnostic systems. Considering the reflectance model, an incident illumination ray is partially refracted inside the target and partially reflected. Reflectance is a function of wavelength, modified by the target tissue in accordance to Beer–Lambert Law of absorbance.⁷ Light rays scatter heavily inside human skin, which is structured in layers.⁸ Carcinogenesis and associated metabolic changes, known as Warbung effect,⁹ modify the molecular structure of the affected tissue, altering the distribution of chromophores. These heterogeneous changes affect the ratio of absorbed and reflected light. Thus, the reflectance spectrum can be regarded as a descriptive signature of the chromophore contents of the tissue, analogous to the human fingerprint. Consequently, discordant signatures can describe atypical concentrations of skin chromophores and be an indicator of abnormal growths.¹⁰ HSI or MSI systems are one of the means to record such signatures.

1.3.

Spectral Imaging

A range of spectral imaging technologies have been developed, owing to its noninvasive nature.¹¹ Raman spectroscopy (RS) uses fiber optic probes to capture information about the molecular fingerprint of a tissue. Although applications of RS in macrolevel diagnosis have been attempted, the complexity of the imaging system design and the need for guidance remains an obstacle.¹² Another approach on ex vivo gross samples is multispectral Mueller polarimetry imaging,¹³ a complex laser-based technique. When depth-imaging is the focus, multispectral optoacoustic tomography can be employed. This label-free technique uses fast laser pulses that excite the tissue, producing waves that can be reconstructed using backpropagation.¹⁴ The main drawback of those methods in a gross-level application is the increased component complexity, need for customization, limited field of view, and bulky equipment. In addition, the aforementioned methods are point-based, which impedes wide-area snapshots and imaging duration. Alternatively, spectral imaging equipment can be used in combination with fluorescent agents, to provide labeled images of the tissue. Despite the labeling advantage, this approach suffers from poor spatial resolution and poor tissue mapping due to movement.

1.4.

Hyper/Multispectral Imaging

HSI was originally developed for remote sensing and space applications. However, HSI and MSI are emerging imaging modalities for medical applications, as they can capture the tissue’s spectral signature. RGB cameras mimic the behavior of cone cells in the human eye, showing three distinct wide-band responses to visible light. S-cells integrate radiation information in the range 420 to 440 nm, M-cells in 535 to 545 nm, and L-cells in 564 to 580 nm. RGB cameras have similar wide-band integration filters, therefore, are susceptible to metamerism, i.e., the inability to recognize different colors under a certain illumination.¹⁵ In contrast, HSI and MSI use narrow-band filters with a width of a few nm. The spectral range can be either at the visible wavelengths (VIS, 380 to 780 nm) or the near-infrared range (NIR, 780 to 2500 nm). Longer wavelengths offer the additional ability to penetrate through the deeper layers of the skin. Depending on the imaging equipment, it is possible to capture a two-dimensional (2D) surface instantly, with good spatial accuracy. HSI and MSI differ in terms of the number of channels. MSI systems are usually customized to the absorbance features of the target tissue. As a result, an MSI pixel describes essentially a feature vector, in contrast to the smooth HSI reflectance curves. In this study, we will consider images with $\le 20$ channels as MSI, while the rest will be considered HSI.¹⁶ We refer to both techniques jointly as hyper/multispectral imaging (HMSI).

1.5.

Potential in Diagnosis

The application of HMSI in pathology shows significant potential.¹⁷ First, it is a noncontact, noninvasive, nonionizing imaging method. HMSI does not modify the physical (cellular- and tissue-level) properties of the tissue while preserving the spatial dimension of the distribution of tissue chromophores. Second, HMSI is fast compared with histopathology. An image can be acquired and processed in a few seconds or minutes instead of days. Coupled with a semi- or fully automated processing tool, HMSI-based tumor segmentation can be implemented easily and with minimal training of the medical personnel. In turn, associated costs can be reduced, and resources can be reserved for the diagnosis of more difficult cases. Skin tissue, which is characterized of an increased presence of chromophores and surface inconsistencies, is an ideal target for colorimetric and texture analysis. Technological advances in hardware, GPU programming, and machine learning libraries facilitate the processing of HMSI data, which often require a few GB of memory per image. HMSI has been investigated for applications in tumor detection, dermoscopy,¹⁸ and temporal monitoring.¹⁹ Previous studies on commercial MSI-based diagnostic tools showed a considerable increase in sensitivity and specificity for histology guidance by dermatologists and non-dermatologist clinicians.^20,21 Several studies report trends in applications²² and classifications schemes for HMSI.²³ Specifically for medical applications, previous reviews focused on noninvasive methods,²⁴ comparison of HMSI to RGB images and dermoscopy,²⁵ and commercial systems^26,27 for in situ melanoma detection, with mixed findings. However, as of writing this report, we are not aware of any systematic review that investigates HMSI for gross pathology for different types of skin lesions, including nonmelanoma cancers.

1.6.

Objectives

HMSI has the potential to standardize, accelerate, and facilitate diagnosis, by (a) producing tumor segmentation in the form of optical biopsies (intraoperatively) or by (b) clarifying cancer margins at the clinical stage (preoperatively). An HMSI imaging system combined with a powerful processing algorithm would reduce diagnostic costs, due to the lack of moving parts, fast acquisition, robustness, and application on multiple pathologies. Through this systematic review, we expect to identify current trends in data processing for HMSI-based segmentation schemes for skin lesions at the macroscopic level. In addition, we summarize HMSI acquisition systems and preprocessing techniques, as well as study effects and limitations.

This study is structured as follows. In Sec. 2, we describe the methodology for this systematic review. In Secs. 34–5, we explain in detail proposed schemes for data acquisition, preprocessing, and classification/segmentation, respectively, show cumulative results from the review and explain subcategories. In Sec. 6, we discuss effects, drawbacks, and common limitations of HMSI processing for skin lesions. The final conclusions are summarized in the last section.

2.1.

Characteristics of the Skin

The optical signature of a tissue is characterized by the concentrations of chromophores inside the tissue. The main chromophores of the skin are melanin (eumelanin and pheomelanin), oxygenated hemoglobin (${\mathrm{HbO}}_2$), and deoxygenated hemoglobin (Hb), with strong absorbance in the VIS range. Lipids and water have strong absorbance in the NIR. Absorbance is proportional to the extinction coefficient for fixed concentration and optical path, according to the Beer–Lambert Law. The extinction coefficient spectra of skin chromophores are shown in Fig. 1. Response to characteristic peaks and valleys of these curves is routinely used in the development of HMSI systems.

Fig. 1

Spectra of the extinction coefficient for the three main chromophores of skin tissue. Reconstruction using data from Ref. 28.

Human skin is an organ comprised of tissue layers consisting of three different types of cells, namely squamous, basal, and melanocytic cells. Malignancies appear when skin cells start to multiply uncontrollably. Although each skin cell category is responsible for different types of skin cancer, irregular growth of melanocytes is the cause of melanoma. Melanoma is the most aggressive skin cancer and is associated with most skin cancer-related deaths. Occasionally, irregular cell growth can be benign, with the example of nevi and Spitz nevi, which are nonthreatening melanocytic tumors. Therefore, discrimination between benign and malignant melanocytic lesions is difficult due to the similar cellular substrate. A common category for HMSI-related research is pigmented skin lesions (PSL), because of the enhanced and heterogeneous presence of skin chromophores. Pigmented skin lesions include MM, basal cell carcinoma (BCC), and squamous cell carcinoma. Another category rich in color information is injury lesions, bruises, and burns.

Knowledge about spectral properties of skin tissue can assist the development of diagnostic models. A high rate of vascularization combined with a high amount of melanocytic cells, high hemoglobin, and melanin concentrations, respectively, can indicate the presence of malignancy. ${\mathrm{HbO}}_2$ and Hb show absorbance peaks around 430 nm, while ${\mathrm{HbO}}_2$ has a local minimum of absorbance at 470 nm. ${\mathrm{HbO}}_2$ shows twin peak absorbance at 550 and 570 nm and Hb a local peak at the same area. At 660 nm absorbance of ${\mathrm{HbO}}_2$ drops suddenly. After 730 nm, the absorbance of melanin starts to become prominent and ${\mathrm{HbO}}_2$ can be discriminated from Hb. Spectral bands at 950 nm provide information from the deeper layers of the skin. Borisova et al.²⁹ and Zherdeva et al.³⁰ showed that the reflectance spectra of pigmented MM above 600 nm are consistently lower than that of nevi or other pigmented lesions. Healthy tissue has consistently higher reflectance compared with lesion tissue. Healthy skin and pigmented lesions rich in blood content show a sudden reflectance hike at 570 nm, although reaching a different maximum reflectance value.

2.2.

HMSI Imaging

HMSI describes jointly HMSI systems. Some studies do not discriminate between HSI and MSI due to their similar nature. However, some notable differences are present. Due to obstacles in technology, initial remote sensing systems were capable only of MSI.¹⁶ MSI systems sample only specific wavelengths or wavelength bands. Therefore, one can select individual wavelengths with classification capabilities and record only their specific reflectance values, resulting in a small number of channels. As technology advanced, HSI systems became feasible. HSI systems sample the entire spectrum with a fixed step and provide continuous spectral signatures. In problems where target wavelengths are unknown, the continuous spectral signature might reveal a spectral pattern that is concealed in the MSI. On the other hand, the increase in the number of channels adds additional complexity in data storage and processing, as well as increases processing time.

HMSI for pathology-related tasks is usually performed in the VIS range, where chromophore absorbance is highest. The NIR range is useful when detailed information for depth structures is required because longer wavelengths can penetrate deeper. However, the spatial quality of NIR tends to be reduced, due to increased scattering and attenuation phenomena. The quality of HMSI differs between in situ and ex vivo imaging, due to the influence of breathing and patient’s movement.

2.3.

Review Methodology

In this systematic review, we followed the methodology proposed in the updated 2020 Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement.³¹

2.3.12.

Study selection

The flowchart of publication search and selection is described in Fig. 2. The search returned 744 entries from Scopus and 714 entries from PubMed. Afterward, we screened for eligibility (a) reports that cited to and (b) reports that were cited by these initial reports. A total of 37 reports fulfilled the eligibility criteria. A list of all eligible studies and additional information (sample size, channel number, spectral range, validation labels, and tissue type) is provided in Table 1.

Fig. 2

Flowchart of publication selection for this systematic review.

Table 1

List of eligible studies.

Ref.	Range (nm)	Channels	Dataset size	Validation	Tissue
33	400 to 1000	162	23 subjects	hist.	In situ
34	405 to 970	19	96 lesions	derm.	In situ
35	VIS-NIR	N/A	1211 lesions	mult.	In situ
36	N/A	10	1632 lesions	derm.	In situ
37	450 to 950	50	81 lesions	hist.	In situ
38	450 to 950	50	82 lesions	derm.	In situ
39	380 to 780	124	24 subjects	hist.	In situ
40	380 to 780	124	27 subjects	hist.	In situ
41	400 to 1100	71	26 lesions	hist.	In situ
42	380 to 780	124	20 subjects	hist.	In situ
43	540 to 950	3	3 lesions	derm.	In situ
44	1000 to 2400	140	1 subject	other	In situ
45	360 to 1000	1127	64 subjects	mult.	In situ
46	500 to 850	71	52 lesions	derm.	In situ
47	450 to 990	55	32 lesions	hist.	In situ
48	380 to 780	124	80 subjects	hist.	In situ
49	500 to 850	71	19 subjects	derm.	In situ
50	400 to 982	8	18 lesions	hist.	In situ
20	430 to 950	10	6 lesions	derm.	In situ
51	500 to 885	76	8 lesions	hist.	In situ
52	450 to 750	61	4 lesions	N/A	Both
30	450 to 750	61	45 subjects	hist.	In situ
53	540 to 950	51	126 lesions	derm.	In situ
54	500 to 850	71	32 lesions	hist.	In situ
55	400 to 720	33	26 subjects	hist.	In situ
56	N/A	10	360 lesions	hist.	In situ
57	N/A	N/A	N/A	N/A	N/A
58	414 to 995	8	429 lesions	derm.	In situ
59	450 to 950	50	124 lesions	N/A	In situ
29	345 to 1040	N/A	N/A	N/A	Ex vivo
60	500 to 850	71	16 lesions	hist.	In situ
61	450 to 630	7	10 subjects	derm.	Ex vivo
62	450 to 750	125	49 lesions	derm.	In situ
63	450 to 780	34	4 subjects	derm.	In situ
64	450 to 750	125	76 lesions	derm.	In situ
65	398 to 757	84	619 lesions	hist.	In situ
66	398 to 757	84	619 lesions	hist.	In situ

Note: N/A, not available; derm., dermatology,; hist., histology,; mult., multidisciplinary. Items are ordered according to publication year.

3.1.

Commercial Systems

A relatively fast approach for studies that investigate proof-of-concept is the use of commercial HSI systems. Four studies used commercially available HSI³⁵ and MSI systems^36,20,56 specifically designed for skin diagnosis. MelaFind, a commercial tool for melanoma detection, which uses 10 MSI channels (430, 470, 500, 550, 600, 650, 700, 770, 880, and 950 nm), showed an increase in specificity and sensitivity compared with dermoscopic/clinical diagnosis.³⁶ SIAscope, another commercial solution, captures spectral information in both the VIS and NIR range.³⁵ Several reports used the Nuance EX system coupled with additional halogen lamps and a polarizing film on the lens^67,38,59 or with a single halogen lamp at 45 deg/0 deg geometry.⁴⁴ Nishino et al.⁴⁴ used the VIS HSI only for visualization purposes, while simultaneously using an NIR line-scan camera for data collection. Randeberg et al.³³ used the VNIR camera and illuminated the scene with two halogen lamps, with sandblasting to reduce specular reflectance. Liu et al.⁴⁵ combined the HSI camera with a halogen source and a reflection fiber optics probe, adjusted to achieve illumination at 45 deg.

3.2.

Hyperspectral Prototype Systems

Some of the reports made use of immobile imaging structures, created specifically for their use case. Nagaoka et al.³⁹ used a prototype HSI system that combined an imaging spectrograph with an electron multiplying charge-coupled device camera. The system included a halogen lamp with optic fiber, a cylindrical lens, and two polarizers. They used a frame rate of $30\text{frames}/\mathrm{s}$ to time line-scanning. Kim et al.⁵⁵ created a prototype HSI system using a monochromatic charge-coupled device (CCD) camera and a liquid crystal tunable filter. A telecentric lens with $0.3\times$ magnification was mounted on the camera. Diffuse illumination was provided through a ring light. Suárez et al.⁴¹ used two different monochromatic CCD cameras and tunable filters to capture spectra in the VIS and NIR ranges. The cameras had different resolutions, so they were registered by mutual information maximization. Carmona Jaramillo et al.⁶³ analyzed images from research prototype IMPULSO, but did not provide additional information regarding system design. Zherdeva et al.³⁰ used an experimental setup comprised of a tunable monochromator and broadband LED. The filtering at each wavelength was achieved using a controller-operated acoustic wave generator combined with two amplifiers. Two polarizers are also included in the design.

Apart from immobile systems, a few handheld HSI devices have been proposed. Neittaanmäki-Perttu et al.⁴⁶ used a prototype handheld HSI camera, based on a Fapry–Perot interferometer (FPI), which can acquire data in a snapshot-like manner. The device was positioned at a short distance from the skin and illumination was provided by a halogen lamp with fiber optic ring. Fabelo et al.⁶² also used a handheld prototype, with a snapshot HSI camera at its core. The system captures a $12\times 12{\mathrm{mm}}^2$ area with a small spatial resolution ($50\times 50\text{pixels}$) in less than a second. They used a similar halogen illumination system with a fiber optic ring guide. The device is attached to the skin surface by a dermoscopic contact structure. Reports by Prigent et al.³⁴ and Borisova et al.²⁹ did not provide any information about imaging design, spatial resolution, or captured area.

3.3.

Multispectral Prototype Systems

Most MSI systems consisted either of a monochrome sensor and multicolor LED lights or of a tunable filter and a single light source. Li et al.⁵⁰ used a 12-bit monochromatic camera, together with an infrared enhanced lens, halogen illumination, and a bandpass filter for wavelength selection. The filter targeted eight center wavelengths in the range 400 to 1000 nm (420, 542, 581, 601, 726, 800, 860, and 972 nm) with full-width at half-maximum 10 to 41 nm. Delpueyo et al.⁵⁸ combined a monochromatic CCD sensor with 32 LEDs (of eight wavelengths in range 400 to 1000 nm) in a ring formation instead of a filter. The LED wavelengths (414, 447, 477, 524, 671, 735, 890, and 995 nm) were selected to match the absorbance characteristics of skin chromophores. Aloupogianni et al.⁶¹ used a similar approach, but with an RGB camera and seven LEDs in the range 400 to 700 nm (450, 465, 505, 525, 575, 605, and 630 nm), instead. They combined the response of each RGB channel under a specific LED light to construct nine-channel MSI. Li et al.⁵⁰ used a diffuser to scatter the light, while Delpueyo et al.⁵⁸ and Aloupogianni et al.⁶¹ installed crossed polarizers between the illumination source and the sensor.

3.4.

Takeaways

Since HMSI is still a developing technology, there is great variability in the imaging equipment used in each eligible study, as shown in Tables 2 and 3. The bulk of the reports used either general HMSI cameras or research prototypes. All acquisition systems captured information in at least the VIS range. Some systems acquired information up to the NIR range. The maximum spectral resolution for HSI was $\sim 0.6\mathrm{nm}$ and the maximum number of channels was 10 for MSI and 1127 for HSI. For applications in gross pathology of the skin, experimentation started with MSI prototypes and gradually expanded to HSI. The initial trend for melanoma versus nevus classification was to target specific wavelengths that coincide with critical points of the absorbance curves in Fig. 1. However, for a larger capture area or a variety of pathologies using the entire HSI spectrum is more appropriate.

Most studies used immobile HMSI systems. Only a few studies reported on the capture area, spatial resolution, or imaging speed. Capture duration ranged from seconds to minutes. Regarding applications on in vivo tissue, breathing or unconscious movement should be taken considered. An immobile system with snapshot scanning is more appropriate to acquire accurate spectral signatures. While it is noted that a light-weight handheld device is easier to use in a clinical setting, this should be limited to small ($\le 10\mathrm{mm}$) capture areas. In addition, fast capture time is preferable to avoid data noise due to movement. Systems that try to emulate the function of a dermoscope are attached to the skin surface or positioned a few mm away. However, most systems were positioned at a distance of a few cm away from the target. This affects the spatial resolution and the maximum area that can be imaged. In this regard, a larger distance is preferable, but a macroscopic lens can be used to improve spatial resolution. Another parameter is the illumination condition. Studies were split in half between those that used LED lamps and those that used halogen lamps. Fiber-optic lighting is also an option. The use of polarizers helps to reduce noise and saturation, therefore is suggested. In addition, a dark box for measurements should be considered, to avoid the influence of ambient light.

Based on the above considerations, the following HMSI systems show the most promise. The system proposed by Suárez et al.⁴¹ is suitable for capturing a large area, which can facilitate the diagnosis of multiple types of lesions. It is capable of depth measurements, due to the high penetration of wavelengths in the NIR range. For lesions small in size and a dermoscope-like function, one can build a system similar to the one proposed by Fabelo et al.⁶² or Neittaanmäki-Perttu et al.⁴⁶ Portable handheld devices can be easily incorporated in clinical practice. The fast acquisition time in a snapshot manner minimizes noise due to patient movement.

4.1.

Normalization

Normalization of spectra can refer to minimizing bias from nonuniform spatial illumination, different tissue type, or different patient. Measurement normalization greatly affects the performance, especially when machine learning is used. Moreover, it is necessary in order to convert HMSI measurements into reflectance ratios. The necessity of interpatient normalization depends on the task. For example, it is required for training an on-the-fly system of intraoperative margin detection, using previous samples of the same patient. However, it might not be necessary for a coarse classifier used in broader applications. For a large capture area, the spatial distribution of illumination intensity becomes inhomogeneous and nontrivial. Since the optical properties of tissue are affected by water content and temperature,⁷⁰ additional corrections are required. For this reason, it is essential to control experimental conditions and calibrate the captured signals.

The most popular approach for biological spectra is min-max scaling, using a dark current image and a white reference image.^{39,45,52,55,62} The reflectance spectrum $r$ then becomes

Fig. 6

Ex vivo spectra for PSL, adapted from unpublished work by Ref. 71. The blue line represents the average. Depending on the tissue type, spectral signatures of each pixel differ in shape and value. Spectral signatures for mucinous carcinoma and Bowen’s disease are smoothly rising lines, whereas for BCC there is a sudden slope around 570 nm, consistent with the local maximum of hemoglobin absorption. Reproduced with permission.

4.2.

Quality Enhancement

Noise is omnipresent in all measurements, but is exacerbated in HSI due to noise induced by patient movement (during in situ measurement), nonuniformity of illumination and illumination fluctuation in the long period that is required for capture. There are two options for denoising, in the form of smoothing in the spatial dimensions (used in Ref. 33) or smoothing in the spectral dimension (used in Refs. 62, 64). Errors in normalized values can be fixed by limiting them to range [0, 1]. Fabelo et al.⁶² and Leon et al.⁶⁴ used a calibration scheme based on min-max scaling, followed by band-pass filtering of extreme bands, noise removal by smoothing the spectrum and another rescaling step to scale spectra in range [0, 1]. Commercial software, such as ImageJ, can be used to stabilize artifacts induced by patient motion.³⁷

In cases of noise with great variability, more complex schemes can be employed. Li et al.⁵⁰ developed an algorithm for the removal of outlier spectra, based on variance restriction. Outliers were present regardless of tissue type and performance increased 20% from the baseline model in terms of accuracy. However, it should be noted that both baseline and improved performance was low ($<80\%$) and hyperparameters are empiric, requiring adjustments depending on the dataset and application.

4.3.

Spectra Reconstruction

In cases of RGB or MSI systems, it can be hypothesized that useful information is missing due to the small number or selection of bands. For this reason, some studies attempted to reconstruct HSI spectral curves from a few RGB/MSI measurements. Assuming that MSI measurements $g$ are

4.4.

Feature Selection

Feature selection maintains the original values of features, hence is necessary when model interpretability is important. During development stages for HMSI systems, spectral bands of interest are selected empirically. Quinzán et al.⁴⁷ and Liu et al.⁴⁵ aimed to create an appropriate filter bank for a targeted pathology, therefore, used supervised sequential floating forward selection, with a distance metric. Liu et al.⁴⁵ performed a stability analysis that showed that modifications in the center wavelength of selected filters, affected performance considerably, and a shift should be limited to 4 nm. Kato et al.⁶⁶ proposed two manually selected channel sets, namely [530, 540, 590] nm and [500, 620, 740] nm, with the former performing better.

4.5.

Feature Extraction

Feature extraction is used to transform the dataset into a subspace that is more appropriate for classification and segmentation tasks. The number of transformed components can be limited, effectively reducing the dimensions of transformed samples. The most common unsupervised method for dimension reduction is principal component analysis (PCA) and similar variants, such as singular value decomposition (SVD) and Karhunen–Loeve (KL) transform. The goal of PCA is to transform the dataset in a new subspace so sample variance is maximized, under constraints of orthogonality. At the same time, PCA denoises the dataset, under the assumption that noise has low variance. Pardo et al.⁵⁹ used KL in the form of Sequential KL (SKL) to transform the dataset while preserving information fidelity. SKL calculates SVD efficiently and with a dynamic threshold for dimension reduction. The training was done using manually selected, square regions of interest (ROI), that do not include border or fringe regions. Randeberg et al.³³ used a variation of PCA called minimum noise fraction transform (MNFT), which performs well on signal-dependent noise.⁷⁴ Forward and inverse MNFT was used for denoising, and then MNFT was applied again for feature extraction purposes. Prigent et al.³⁴ proposed the use of projection pursuit (PP) to reduce dimensions before classification. PP searches for non-Gaussian projections in a lower dimension but is computationally expensive. Independent component analysis (ICA) is an alternative unsupervised technique that assumes independent components with non-Gaussian distributions, instead of principal components. ICA can be implemented with FastICA³³ or JADE³⁴ for a small number of components. Neittaanmäki-Perttu et al.^46,49,54,60 used vertex component analysis (VCA),⁷⁵ an unsupervised linear unmixing technique that detects a predefined number of pure components from the spectral signature. The coefficients of pure components were presented in the form of ”abundance” or concentration maps, in which patterns are unmixed further using filter vector algorithm (FVA). These maps served as input to classification.

The techniques above try to solve a problem of blind source separation because they assume no or minimal knowledge about the components that comprise the HMSI signal. However, other techniques can be trained on labeled data, if available. In many instances, linear discriminant analysis (LDA) is more powerful than PCA, because it maximizes interclass variance while minimizing intraclass variance. However, LDA is supervised and required prior class labels. Liu et al.⁴⁵ used PCA to reduce HSI data to a 28-point space, before additional reduction with multiclass LDA. Nishino et al.⁴⁴ employed a flavor of multiclass LDA that is based on canonical coordinates and canonical discriminant analysis (CDA) for NIR spectra.

Apart from spectral features, HMSI systems provide also texture and color features. To make use of spectrospatial information, Aloupogianni et al.⁶¹ applied a multispectral multiscale local binary pattern operator that extracts texture characteristics from slices of the HSI data cube. Incorporation of texture information in the classification showed better performance compared with the case of using standalone spectral information. Delpueyo et al.⁵⁸ used as features the color coordinates in CIELAB space and color differences $\mathrm{\Delta }E$. On top of the extracted feature vector, they calculated texture features in an image segment, based on first- and second-order statistics, including mean, variance, entropy, energy, and third central moment. Lorencs et al.⁵³ also extracted measurement statistics as features. Biological indexes that describe chromophore concentrations have also been proposed, such as Erythema index $E=I(660)/I(545)$^58,69 and Bilirubin index $B=I(450)/I(660)$.⁵⁸ In an unconventional approach, Zheludev et al.⁵¹ first increased the dimension of input vectors via Framelet transform, selected some dozens of features, and then reduced dimension with diffusion maps.

4.6.

Takeaways

Various preprocessing steps can be used sequentially for HMSI analysis. Dataset-wide normalization using a fully reflective and a dark reference is an essential step, especially if complex models are used later. Quality enhancement with average filtering should be used (if used at all) with caution, because it may erase spectral features in HMSI data with low spatial resolution. Dimension reduction is optional, while it depends on the size of the dataset. Feature extraction or selection can alleviate ill-posed problems with a large input vector compared with the available number of data samples. However, it is an optional step and if used, methods with different assumptions about the data should be compared. PCA and VCA are suitable as a base technique. In addition, feature extraction is not recommended when a complex decision model is used, to avoid overfitting. In studies where texture information was incorporated in segmentation, it assisted performance.

5.1.

Overview of Proposed Methods

Table 5

Summary of proposed discrimination indices.

Despite the variety of proposed methods, they are defined by different assumptions, thus producing sometimes different results. SVM and RF are classifiers commonly used in medical applications with good classification results. SVM is effective in high-dimensional spaces and memory efficient, but may fail with a large dataset or noisy data. On the other hand, RF is robust to overfitting and with a few hyperparameters, but is slow in producing predictions real-time and does not focus on data description, making it hard to interpret. MAP produces fixed point estimates for predictions, which can be useful to denote confidence in the results. LDC produces impressive results assuming linear separation, but might underperform on uncommon data that do not follow normal distribution. KNN is a lazy model, which is easily affected by outliers and cannot be scaled effectively. ANN and pretrained models are preferred for segmentation problems, where the result is a 2D mask of class labels. However, they require large amounts of data samples. KC and SAM can be combined for segmentation problems and be applied on standalone image data.

5.2.

High-Performing Systems

5.2.1.

Melanoma versus nevus

A straightforward approach toward discriminating between melanoma and benign nevus is to calculate an HSI-based index or map, evaluate it visually, and apply a cutoff threshold. Nagaoka et al.^39,40,42,48 took into account the unstructured nature of melanoma and calculated a discriminator index based on the entropy of a probability density function (PDF). An example of SAM angles for MM and benign dermatofibroma is shown in Fig. 8. They used SAM as a building block for the probability function. Afterward, they applied a threshold for discrimination between melanoma and other PSL. They modified the index by adding an upper limit to SAM frequencies, so melanoma false negatives are reduced. They found a possible correlation between this index and the existence of melanoma cell at the dermal/epidermal junction, which can be useful for cancer staging. Alternatively, another option is to calculate multiclass conditional probabilities of the preprocessed spectral signatures. Pardo et al.⁵⁹ calculated probabilities using multivariate kernel density estimation (KDE) with Silverman’s rule of thumb estimator. Classification of the reduced spectrum is done with MAP, given dynamically adjusted class weights. The three trained feature bases did not extract specific absorbance properties, but instead a cumulative response, focusing at the 550 nm (hemoglobin absorbance peak) and the red and near infrared region (melanin absorbance slope). Their proposed method achieves fast execution and good results under various cross-validation schemes, includes only one degree of freedom and can be optimized to eliminate false negatives.

Fig. 8

Example of initial processing for an MM. Spectral signatures from HSI subimages in (a) are used to calculate SAM values in (b). Pseudo-RGB images and SAM angles for MM (left) and benign dermatofibroma (right) are presented in (b). Spectral signatures of the highlighted red points in (b) are shown in comparison to the reference signature of healthy skin in (c). The angles of melanoma signatures are larger than those of dermatofibroma. The blue marks surrounding the melanoma tissue are pathology stains. Adapted from unpublished work by Ref. 71. Reproduced with permission.

5.3.

Takeaways

Many of the eligible studies emphasized feature extraction, resulting in simple visual evaluation of index images or semiautomatic thresholding. There were a few studies that used traditional machine learning classifiers and only five that used deep learning. For clearly identified tasks such as melanoma versus nevus classification, the use of thresholding on a discrimination index seems sufficient.⁴⁸ In this regard, an MSI with a few channels is enough. However, for a multiclass problem, a more complex approach is necessary. Staple classifiers such as SVM can provide good results.^59,64 It should be noted that synthetic minority oversampling technique (SMOTE)^41,47 can alleviate the problem of unbalanced training classes for an SVM model. Integration of concepts prepared for remote sensing HSI, such as SAM and endmembers, can improve performance. On the other hand, ANN systems generally perform poorly compared with simple methods.^65,66 This could be attributed both to the relatively small size of the training dataset, the large number of training parameters, and the unsuccessful learning of rare features.

6.1.

Data Acquisition

All but one, the systems in this study focused on the VIS range of wavelengths. In comparison to the extinction coefficient graph in Fig. 1, indeed characteristics of the curves are included in the 400 to 800 nm range. With the advancement of HMSI cameras, snapshot cameras can replace line-scan cameras. The capture of a small ROI can last as little as a few seconds, minimizing discomfort to the patient. The size of captured images varied widely, from $50\times 50$ to $1200\times 1400\text{pixels}$. Because of the variety in sensor equipment, spectral resolution varied as well. Therefore, any preprocessing and classification scheme should be adjusted to the specifics of the HMSI system that was used for acquisition.

6.2.

Preprocessing Schemes

Normalization greatly affects the performance, especially when machine learning is used. OD as an expression of absorbance is becoming obsolete. Min-max scaled reflectance information is preferred instead. In some cases, noise filtering was applied. However, depending on the resolution, this is not generally recommended, since such filters might also reduce peaks in the spectrum that do not represent noise. In addition, in the case of biological spectra, noise affects the location of a peak instead of the height of the peak. Therefore, that peak shift might disappear with spectral filtering. However, it might be beneficial to reduce artifacts around hair follicles,³³ which contain high melanin content.

Relevant studies included a variety of feature extraction and feature selection methods. During the review process, it was evident that preprocessing techniques were selected “as is,” with empirical hyperparameter selection and no comparison to alternative schemes. For example, PCA and ICA have different assumptions about a component’s contribution to the total variance or intercomponent independence. Furthermore, apart from general methods, there are available dimension reduction techniques specifically for the classification of biological HSI spectra.⁷⁸ There is a need for comparative evaluation of feature extraction methods to identify which is more appropriate according to lesion type and the task at hand. Further research is needed on whether preprocessing induces overfitting of the result or discards valuable components. In addition, there is a concern on whether it is more appropriate for dimension reduction to be trained on a patient’s reference data (interpatient) or a database of patient data (intrapatient).

6.3.

Classification and Segmentation

HMSI methods in this review performed well compared with dermoscopy, with many studies reporting sensitivity and specificity $<80\%$. A study on 463 lesions (of which 30% were malignant) reported 80% sensitivity and 82.6% specificity when using dermoscopy.⁷⁹ A meta-analysis reported a cumulative 90.1% sensitivity and 74.6% specificity of dermoscopy-based systems.⁵⁷ Dermoscopy and histological biopsy, despite both being gold standards at different stages of diagnosis, evaluate different parts of the tissue.⁸⁰ Dermoscopy evaluates the entire tissue area, including color and patterns. Histopathological biopsy evaluates vertical sections and the cell structure in depth but evaluates only a small percentage of the tumor. In this context, depending on the acquisition range, HMSI can combine the two, provide detailed color and texture information together along with some depth information.

However, the studies with exceptionally high specificity and sensitivity in this review were evaluated on small datasets with carefully selected target lesions, which might positively skew performance. Larger datasets that contained a variety of lesions showed worse performance. Moreover, the classification systems should be able to be updated and retrained on a larger dataset to incorporate newly accumulated knowledge. Some classification models were dependent on biological absorbance properties, while others were data-driven. There is a lack of studies that evaluate alternative methods simultaneously. To minimize the need for participation of the medical staff in the analysis process, it is essential to develop fully automated classification methodologies.

A point of concern in classification is robustness and interpretability. Liu et al.⁴⁵ assessed the stability of the classification system using Monte Carlo analysis. Other studies did not perform robustness checks. The explainability of the results is also contentious. Some results associated important wavelength for classifications with characteristic wavelengths for skin chromophores. However, when applying classification on an HMSI patch, there is a need to investigate ROI that influenced the classification decision, apart from the spectral dimension. Methods such as Grad-CAM⁸¹ and LIME⁸² have been proposed to explain the results of machine learning models for RGB images. Using explainable visualizations of the classification system, it is easier to develop a robust system, as well as to convince medical staff of the system’s fidelity. None of the eligible studies reported concerns regarding the patients’ safety. In some instances, opinions among medical staff are divergent. Most studies did not mention the number of doctors that performed manual classification and labeling. Labeling, training, and performance are subject to change when the majority vote of multiple doctors is included during the data collection and evaluation stages.

6.4.

Limitations and Obstacles

A crucial issue in the development of CAD systems is validation with real data. Due to the high workload of pathologists and the discrepancies between clinical protocol and CAD system development requirements, it is difficult to obtain fully labeled datasets. In addition, to properly validate a system, a balanced dataset of both healthy and malignant samples is preferred. However, it is not ethical to perform histological biopsies on every single patient. This deficit of healthy labeled data affects the training of the CAD system and might increase false negatives. SVM is claimed to perform well with unbalanced datasets and indeed performed best in studies in this review. Furthermore, in cases where histology slides are available, they need to be registered to the HMSI cube. Movement and deformation of the tissue complicate image registration. In addition, histology slides are prepared from cross-sections of the tissue, while gross-level HMSI captures the surface of the tissue.

To train the segmentation of tumor effectively, a large number of samples is required. This makes the use of each pixel as one data point the obvious choice. Training on pixelwise labels ignores spatial information but provides more samples for training, improving accuracy. However, pixelwise samples will contain inherent correlations, due to being extracted from the same lesion/patient. On the other hand, training on patch-wide labels utilizes both spatial and spectral information. Therefore, an even larger number of tissue samples is required for adequate training and validation of the diagnostic model. This limitation in acquiring large, labeled datasets is what is hindering the application of deep learning algorithms on HMSI of skin lesions. Recently, Halicek et al.⁸³ applied deep learning for tumor margin detection of nonskin head and neck tissue samples with promising results. Active learning can be used to speed-up development amid a lack of labeled HMSI.^84,85

While HMSI-based classifiers show potential for automatic detection of cancer margins, there is a long way to go until they are incorporated in the clinical practice. Fink et al.²⁴ compared a variety of noninvasive imaging approaches for melanoma detection and found none was able to provide a definite and final diagnostic result. Indeed, in eligible reports, there were cases with unexpected false positives and false negatives. To be useful in practice, HMSI-based CAD systems need to include fidelity estimates and display deciding factors regarding the segmentation result they produced.

Based on the results of this review, in our future work, we will aim to compare different preprocessing and segmentation schemes with each other and a common baseline. We will consider data augmentation and transfer learning methods to investigate the application of deep learning. In addition, we aim to investigate analysis techniques that are specifically developed for HMSI images, instead of traditional learning techniques. Finally, we will try to adjust the training process and performance toward good interpretability and explainability of the segmentation model.