I’ve spent the last 9 years building and shipping ML products end-to-end.
What I love most is turning ML into something reliable and useful. Today at Payrails, that means one clear mission:
I build reliable ML systems to quickly detect issues and optimize payments at real-time scale.
After years of working on flagship AI initiatives across London and Berlin, joining Payrails felt like the perfect next step: high impact, real-world complexity, and the opportunity to build ML capabilities from the ground up in one of the most demanding domains there is.
Why Payrails?
Payments are deceptively hard. Merchants operate in a landscape where small failures can have massive business impact.
Payrails exists to make payments and other interconnected financial operations smarter and simpler, helping merchants optimize their payment strategies without needing to manually manage a complex web of providers, rules, and performance metrics.
When it comes to machine learning, the world of payments presents two exciting challenges:
- Real-time decision-making: payment performance changes quickly and needs immediate reaction
- Scale: working with millions of transactions where even a small shift matters
When I joined Payrails a few months ago and first discussed why ML initiatives to prioritize with our Head of Data, we immediately aligned not only on what we should build first, but how we should build it:
- Choose a high-impact problem
- Build a POC fast
- Test it with real-world data and internal stakeholders
- Iterate quickly
- Invest in production-ready infrastructure once value is proven
What I did not expect was how quickly we would see meaningful results.
The first use case: Intelligent anomaly detection to monitor payment authorization success rates
The first machine learning use case we tackled was monitoring payment authorization success rates.
When it comes to authorization rates, merchants want to know immediately when something unusual happens, especially when success rates drop. A sudden dip might indicate:
- Payment provider instability
- Routing issues
- Upstream technical failures
- Unexpected behavior in a specific market or segment
We chose this as our first use-case because it’s a well defined problem statement and allowed us to build out key infrastructure pieces for ML. A missed anomaly can lead to lost revenue, but it can also cause something even harder to recover: loss of merchant trust.
One of the key differentiators of Payrails is our hands-on approach to working with enterprises. We act as an extension of your team, diving deep into your payment data to uncover opportunities and solve problems together. It's a collaborative relationship based on a shared commitment to unlocking better performance.
In order for this relationship to be successful, it depends on trust. That trust is built through reliability, transparency, and efforts to continuously improve: all of which made intelligent anomaly detection a perfect use case for ML.
Why manual anomaly detection doesn’t scale
Many merchants today rely on static thresholds.
For example: “Alert me if acceptance drops below 90%.”
But thresholds have serious limitations.
For one, manual setup takes time. For every payment provider, market, and transaction type, someone needs to manually configure and test thresholds, delaying time-to-value.
Second, static rules don't adapt to seasonal shifts. What looks normal during Black Friday will trigger false alarms in January, forcing teams to constantly recalibrate.
Beyond the time investment, these threshold-based systems simply don't scale as payment operations grow more complex. When you're managing multiple workspaces, each with its own payment providers, routing logic, and transaction patterns, maintaining separate threshold configurations for every combination becomes impractical.
And perhaps most frustrating: even perfectly configured thresholds produce high rates of false positives and missed anomalies. Teams waste time investigating non-issues while real problems slip through undetected.
At Payrails, the complexity compounds: each merchant can have multiple workspaces, and each workspace can have multiple payment providers configured. Monitoring acceptance rate manually across all those combinations becomes unmanageable.
So we set out to build something intelligent.
The solution: ML-powered anomaly detection
We built an anomaly detection system that learns what “normal” looks like and flags statistically meaningful deviations automatically.
The goal was to detect sudden drops in authorization success rate quickly and reliably, with context.
To make the system truly useful, we needed it to detect anomalies at the right level of granularity.
For the POC, we trained multiple models specifically for each merchant and provider combination.
That resulted in hundreds of detection models running in parallel.
We could go even more granular (e.g., by country, payment method, currency), but we intentionally didn’t, for two main reasons:
- Technical limitations: Forecasting models struggled with the sheer data size and complexity
- Signal quality: More granular segments were too noisy, leading to too many false positives
Choosing the right granularity became one of the most important technical decisions of the project.
How the system works (high-level overview)
At a high level, the anomaly detection system we designed is intended to continuously learn what “normal” payment behavior looks like and then surface deviations that are likely to indicate real issue, rather than noise.
Instead of relying on static rules or manually configured thresholds, the system builds an understanding of expected authorization success rates over time and compares real-world behavior against that baseline.
Here are a few examples:
Learning normal payment behavior
Payment traffic naturally follows patterns driven by time, user behavior, and transaction dynamics. Our approach captures these recurring structures and adapts to gradual changes, allowing the system to distinguish between expected fluctuations and genuinely unusual behavior.
By continuously updating its expectations, the system remains effective even as merchants grow, traffic shifts, or usage patterns evolve.
Detecting meaningful deviations
Once an expected range of behavior is established, the system monitors live data and flags deviations that are statistically significant. This makes it possible to detect sudden drops or unexpected changes in authorization success rate without overwhelming teams with false alarms.
Crucially, anomalies are identified based on how unusual they are relative to historical behavior, not against a one-size-fits-all threshold.
Providing context for faster decisions
Alerts are designed to be actionable. Alongside anomaly signals, the system provides contextual information that helps teams quickly assess severity and relevance, such as whether an issue is backed by sufficient transaction volume to warrant immediate attention.
This balance between sensitivity and context is especially important in payments, where over-alerting can be just as harmful as missing real incidents.
Built for evolution
The current implementation prioritizes speed of learning and practical usefulness, validating the approach with real payment data and real incidents. As we move toward a production-grade solution, this foundation allows us to further improve robustness, scalability, and precision, without changing the core principles behind the system.
From concept to reality: Shipping a POC in one month
We built and deployed the first internal version of the anomaly detection system within one month. That speed was only possible because of the approach we agreed on early: define the problem clearly, ship quickly, validate with stakeholders, and iterate with real incidents.
The POC is currently used internally by our Payment Experts team to better understand merchant health and by infrastructure engineers to spot system issues early. And most importantly, during the POC phase, it successfully caught real-time incidents as they occurred.
One of the hardest parts of bringing ML to a new context is balancing the need to deliver value fast today while simultaneously building infrastructure that supports ML tomorrow, all while educating people on which problems are solvable by existing systems versus real AI use cases.
In payments, "production-ready" doesn't just mean "works on a dataset." It means the system has reliability, monitoring, auditability, explainability, security and compliance controls, graceful degradation, and strong operational support. These requirements are fundamental to earning and maintaining merchant trust in a domain where every transaction matters.
What's next: Building production-grade ML at Payrails
Before scaling anomaly detection to merchants, we’re building the foundation needed to support robust ML systems in production:
- training and retraining pipelines
- experiment tracking
- model performance monitoring
- observability across pipelines and inference
- security and compliance controls for ML workflows
Once the infrastructure is in place, we’ll build a more mature anomaly detection system on top of it: with deeper robustness, smarter handling of sparse data, and improved granularity where it makes business sense.
The potential for machine learning in payments extends far beyond monitoring. What excites me most is working on problems where reliability and speed genuinely matter.
Payments are real-time, high impact, and unforgiving of failure. Building ML systems in this environment means building systems that merchants can depend on to protect revenue, prevent issues, and drive better outcomes.
And we're just getting started.
Jacek Filipczuk is a Senior Machine Learning Engineer at Payrails, where he builds ML systems to detect payment issues early and help merchants optimize financial operations at scale.
I am the text that will be copied.
