Payment card fraud losses worldwide reached $33.83 billion in 2023 (Nilson Report, 2025). Alarmingly, Deputy Prime Minister and Minister of Interior Sar Sokha (2024) stated that in the first semester of 2024, Cambodians lost nearly $40 million to online and digital fraud. To prevent these significant financial losses, it's crucial to identify the predictive models that can more accurately detect the anomalies in transactions. This study investigates which predictive models work best in predicting the anomalies in the payment transaction. The dataset contains types of online transactions, the amount of the transactions, names of the sender and receiver, the sender’s balance of account balance before and after the transaction, and the receiver’s account balance before and after receiving money. Autoencoder, LightGBM, Neural Network, Logistic Regression, Random Forest, and CatBoost were built as prediction models within this research. Each of these algorithms is employed to create prediction models, which are meticulously fine-tuned to yield the most accurate prediction of the fraud payment in the system involving optimizing hyperparameters and selecting the best features to enhance the models’ prediction power. Common performance metrics such as Precision, Recall, F1-Score, and AUC-ROC were used to test each model's performance. Extensive experimentation data shows the best performance of CatBoost (AUC-ROC of 0.895 for Oversampled and 0.999 for other kinds of datasets) and Random Forest model (AUC-ROC of 0.99), which consistently outperforms other machine learning methods in accurately predicting payment fraud, whether training with an imbalanced or balanced dataset. These results highlight the model's ability to handle complex, non-linear relationships within the data and its effectiveness in generalizing across different scenarios and conditions. In conclusion, the findings of this study will be beneficial mostly to the banking system as they can apply the models we found in their system to prevent any fraudulent activities. Spotting the fraudulent activities in the early stage plays an immense role in preventing the loss.