Nail Polish Try-On: Realtime Semantic Segmentation of Small Objects for Native and Browser Smartphone AR Applications
This addresses the need for efficient, client-side AR experiences in beauty and retail applications, though it is incremental as it optimizes existing segmentation methods for a specific use case.
The paper tackles the problem of enabling real-time nail polish try-on AR applications on mobile devices by developing a system for semantic segmentation of small objects, achieving 94.5 mIoU at 29.8ms runtime on an iPad Pro.
We provide a system for semantic segmentation of small objects that enables nail polish try-on AR applications to run client-side in realtime in native and web mobile applications. By adjusting input resolution and neural network depth, our model design enables a smooth trade-off of performance and runtime, with the highest performance setting achieving~\num{94.5} mIoU at 29.8ms runtime in native applications on an iPad Pro. We also provide a postprocessing and rendering algorithm for nail polish try-on, which integrates with our semantic segmentation and fingernail base-tip direction predictions.