Spectral machine vision enables the identification of objects and materials through their wavelength-dependent optical signatures, with broad applications including, e.g., agriculture, chemical sensing and autonomous systems. However, conventional approaches collect dense spectral-spatial data, which is orders of magnitude larger in volume compared to normal camera videos, and digitally process it into scene recognition. When deploying spectral machine vision on satellites, drones, cell phones, and robots, the data bottleneck usually causes slow speed, poor resolution, and short operation time with batteries. 

Rationale:

To address these bottlenecks, we introduce spectral kernel machines (SKMs), an optoelectronic device architecture that directly compresses spectral analysis inside the sensor. By co-designing electrically tunable bipolar photodetectors and a real-time training algorithm, the SKM framework allows for intelligent spectral inference via the analog photocurrent output, eliminating the need for post-processing and significantly reducing power consumption and latency.

Results:

We experimentally demonstrated different SKMs to perform machine learning analysis over complex, visible to mid-infrared (MIR) incident spectra, with the readout photocurrent directly providing the final inference. This process mathematically resembles the kernel machine algorithms typically used for digital machine learning, making each photodetector a spectral kernel machine. It can ‘sniff-and-seek’, conceptually inspired by retriever dogs, learning from examples to recognize spectral features within a complex scene. We experimentally demonstrated different SKMs to perform diverse tasks, including image segmentation, nano-scale thickness metrology of wafer oxide layers, discrimination of natural/artificial leaves, and identification of leaf hydration levels, achieving high accuracy under blind testing. We further innovated SKM devices in the MIR band using two-dimensional nanomaterials, black phosphorus and MoS2. The device was operated at room temperature to identify chemicals and quantify mixture concentrations indistinguishable in the visible bands. We also modeled the SKM performance using benchmark hyperspectral real landscape datasets. The results, published on Science (Science 390.6776: eady6571, (2025)), demonstrated that SKMs can deliver comparable accuracy to digital methods, while offering over two orders of magnitude improvements in speed and energy efficiency.

Conclusion:

The SKM framework offers orders of magnitude higher power efficiency and speed, and large training flexibility to resolve, recognize, and compress high-dimensional spectral-spatial information. The findings will inspire new intelligent sensors with potential applications in e.g., precision agriculture, autonomous driving, chemical sensing, metrology, and providing new products in the original equipment manufacturer (OEM) industry.