In recent years, the application of machine learning in education has gained significant attention for its potential to uncover hidden patterns and improve student learning outcomes. This study employs a hybrid unsupervised-supervised machine learning approach to achieve two primary objectives: first, to uncover hidden learning profiles of Cambodian high school students, and second, to identify and validate the most significant factors contributing to their mathematics achievement. Using a dataset of 7,188 students, a two-phase analytical pipeline integrating unsupervised and supervised learning was developed. In the first phase, k-modes and k-prototypes clustering were applied with multiple internal validation and stability metrics to analyze latent student profiles. This exploratory phase informed a the second phase using CatBoost, Decision Tree, and Random Forest classifiers, with feature significance determined through a consensus of Mean Decrease in Impurity (MDI), Permutation Feature Importance (PFI), and Recursive Feature Elimination (RFE) metrics. Results indicate that k-prototypes produced a superior two-cluster solution (stability score: 0.9941) compared to k-modes (0.1181), segmenting students primarily by math anxiety, grade level. Supervised modeling identified previous semester math achievement, anxiety, and grade level as the most stable predictive features, with the core feature set achieving a cross-validation accuracy of 0.7068. The integrated analytical framework successfully translated computational metrics into actionable educational insights, providing a validated, data-driven foundation for targeted intervention strategies to support mathematics learning in Cambodian high schools.