Models that survive the model-risk review

The latency budget on card-not-present traffic is set by the authorisation network, and the architecture is built to fit inside it end-to-end (ingress → score → response). Stack: pre-computed user/merchant/device features in Redis with a 50ms refresh budget, a quantised gradient-boosted model served from C++ via Triton, and graph features (mule-network proximity) computed offline and joined at read. We keep the hot path under 200 features; the heavy graph and behavioural signals run on a 5-minute streaming window and are checkpointed. We have also evaluated edge inference - co-locating the model at the issuer-processor boundary - which cuts network time at the cost of feature staleness.

We treat SR 11-7 as the development methodology, not a documentation tax. Each model gets: (1) a conceptual soundness write-up - why this technique, why these features, references to literature; (2) an outcomes analysis with backtesting, benchmarking against the existing model, and stability testing; (3) an ongoing monitoring plan with PSI on features, KS on scores, and outcome-based drift; (4) a challenger model maintained in shadow. The artefact pack feeds straight into the bank's MRM team. We have second-line validation review built into our release gate - a model does not promote to production without sign-off from the bank's independent validators.

Yes - that constraint shapes the architecture. We never write back to the core synchronously on the customer path; reads use the core's standard read replicas or a CDC stream into our analytical store. For Temenos T24 we use T24 Browser API plus the TAFC log stream; for FIS Profile, the IBS adapter; for Fiserv DNA, the OpenBank API; for Finacle, FI-direct and connect24. End-of-day batch windows are sacred - our pipelines respect cutover times and resume from CDC checkpoints once the batch is clean. The integration is designed so the batch window is never in the blast radius.

Three layers. First, in data: we do not feed protected-class variables, and we audit proxies (e.g. ZIP code is a known race proxy in the US - we use it with caution and bias-aware transforms). Second, in training: adversarial debiasing or constrained optimisation so the model cannot improve AUC by exploiting protected-class signal. Third, in validation: standardised disparate-impact analysis (80% rule), AUC parity, calibration parity, and an adverse-action consistency check. We publish a fair-lending technical report with every deployment, including statistical tests and the boundary cases the model gets wrong. CFPB Circular 2022-03 made explainability a regulatory requirement - our reason codes are designed to clear that bar.

We ship a self-hosted MLOps bundle: MLflow model registry, Argo Workflows for orchestration, Prometheus + Grafana for monitoring, a private container registry (Harbor), and a pip/conda mirror seeded from an approved package list. Updates come in over a one-way diode or as signed offline bundles, scanned at the air-gap boundary. Training data never leaves the enclave; model weights are versioned in-bank. The trade-off is slower iteration: a new dependency takes a week to clear, not five minutes.

Yes - and we strongly recommend it for any system that touches PAN, CVV, or full track data. Our reference deploy on AWS uses a dedicated VPC with PrivateLink endpoints (no public internet), KMS with customer-managed keys, VPC Flow Logs to your SIEM, and Macie for PII discovery. On Azure: Private Link + Key Vault HSM. On GCP: VPC-SC perimeters with CMEK. We tokenise PANs at the perimeter wherever possible so analytics, ML training, and inference operate on tokens - the cardholder data environment stays small and the PCI-DSS audit scope stays bounded. We can support FIPS 140-2 Level 3 HSMs where required.

We do not pretend the trade-off does not exist - we manage it. For low-stakes, high-volume decisions (e.g. routing a fraud alert to queue A or B), pure gradient-boosted or even neural models are fine and SHAP is sufficient post-hoc. For high-stakes regulated decisions (credit underwriting, AML SAR filing recommendations, robo-advisor allocations), we use intrinsically interpretable models: monotonic scorecards, GAMs (EBMs), or constrained GBMs where every feature has a known sign on the outcome. There is an AUC cost versus a free GBM - we measure it on your own portfolio rather than quoting an average - and it is worth paying when the regulator asks why a customer was declined.