AI-Driven Transaction Monitoring in Fintech Environments: Comparative False-Positive Reduction Across Rules-Based and Machine-Learning Detection Models

Abstract

This review evaluates the comparative effectiveness of rules-based and machine-learning approaches to transaction monitoring in fintech environments, with particular emphasis on reducing false-positive alerts without weakening financial-crime detection. The study adopts a structured review approach, synthesizing scholarly evidence on monitoring evolution, detection architectures, model performance, data quality, feature engineering, operational efficiency, explainability, governance, regulatory compliance, implementation barriers, and emerging analytical techniques. The findings show that rules-based systems remain important because they provide transparent, auditable, and regulator-aligned controls for recognized typologies, yet their reliance on static thresholds often produces excessive alerts and substantial investigative burden. Machine-learning models offer stronger contextual discrimination, behavioural pattern recognition, and adaptive risk scoring, improving alert precision when supported by representative data, robust feature construction, appropriate class-imbalance treatment, and continuous validation. However, their deployment introduces challenges related to explainability, concept drift, privacy, model risk, workforce capability, and regulatory accountability. The review further identifies hybrid architectures as a particularly effective direction because they combine deterministic safeguards with predictive analytics, dynamic prioritisation, and human oversight. It concludes that sustainable false-positive reduction depends on balancing precision with recall, false-negative risk, investigator workload, model stability, customer impact, and auditability. Fintech institutions are therefore advised to strengthen data governance, recalibrate models and thresholds continuously, maintain transparent model documentation, embed investigators within feedback loops, and adopt phased implementation strategies. Future research should prioritise graph-based analytics, federated learning, synthetic data, adaptive retraining, concept-drift management, and reproducible benchmarking across realistic transaction datasets and jurisdictions. These priorities can improve scalability, cross-institutional learning, and responsiveness while preserving proportionate controls and defensible oversight across complex digital payment ecosystems.