Classification of Facial Skin Problems Using Convolutional Neural Networks (CNN) and Content-Based Filtering for Skincare Product Recommendation

Abstract

Many consumers find it difficult to independently identify their facial skin problems due to limited access to dermatological services. This study aims to classify seven facial skin problems using a CNN with ResNet50V2 and generate skincare product recommendations using Content-Based Filtering with TF-IDF and Cosine Similarity. The dataset consists of 2,450 facial skin images, with 350 images for each class. The model was trained through three progressive fine-tuning phases using a Dual Pooling classification head and Test-Time Augmentation, complemented by input validation through face detection and a confidence-score threshold. The results show that the model achieved a test accuracy of 82.53% and a macro F1-score of 0.82. The acne class achieved the highest performance with an F1-score of 0.89, while the dull skin/dark spots class obtained the lowest F1-score of 0.78, which was associated with visual similarities to signs of aging. The recommendation system generated product recommendations with similarity scores ranging from 0.278 to 0.321 based on the relevance of product attributes and active ingredients. Black-box testing showed that all tested system scenarios were valid.