Everyone is talking about xAI; the brand chosen by one of Silicon Valley's most prominent AI ventures for its latest generation large-language-model chatbot. In the water sector we could use a similar kind of explainable XAI: one that makes AI tools and simulation models genuinely understandable, trustworthy, and actionable for engineers, operators, and managers.
The black box problem
Imagine a process engineer at a large wastewater treatment plant. A machine-learning algorithm flags an alert on the control screen: "Increase methanol dosing to the post-denitrification stage by 18%." Would the operator act on it immediately? Probably not and rightly so.
This is the central challenge facing AI adoption in the water sector. Many data-driven tools that are available to operators in a control room function as black boxes: they ingest sensor data and SCADA streams, perform opaque statistical operations, and return a number for an indicator variable or raise an alert with no accompanying explanation. The prediction may be accurate, but without knowing the underlying reasoning — which process variables drove it, how confident the model is, what would change the recommendation — human trust is difficult to build. A conversational AI agent on the SCADA interface can assist the operator in querying the system in plain language, and that is genuinely useful. But a chatbot that translates a black box's output into readable sentences is not the same as a model whose reasoning is inherently transparent. One translates the black box. The other dismantles it. The human dimension of this challenge has been explored in this previous blog.
White boxes and black boxes
Process engineers in the water sector are not new to interpretable models. For decades, process simulation has relied on deterministic, mechanistic physics-based frameworks: the Activated Sludge Model family (ASM1, ASM2d, ASM3) to simulate wastewater treatment by the activated sludge process with biological nitrogen and phosphorus removal, the Anaerobic Digestion Model No.1 (ADM1) for sludge digestion, and hydraulic models rooted in mass-balances and fluid dynamics equations. These are the true "white box" models; every model parameter has a physical meaning, every equation can be questioned and is interpretable.
The limitations of white-box models are also well known; they are rigid, require intensive expert calibration and model validation, and struggle with the nonlinear, nonstationary dynamics of real plants operating under variable loads and ageing infrastructure. Pure ML models sit at the opposite extreme; they are powerful and adaptive, but structurally opaque. Neither extreme alone is sufficient in many cases.
Physics-Informed AI: grey-boxes with explainability by architecture
The most promising path forward in the modelling world lies – as often – in the middle: hybrid, or "grey box," models that embed deterministic process knowledge within a data-driven framework, what the broader engineering community is increasingly calling Physics-Informed AI. In a parallel hybrid architecture, a mechanistic model (e.g. a modified ASM describing biological and physical-chemical processes in a biofilm reactor) provides the structural backbone and mass-balance coherence, while an ML component learns the residuals — capturing variability, sensor drift, and process dynamics that equations alone cannot represent. A clear outcome of this type of approach is that hybrid modelling can improve wastewater treatment model accuracy significantly, reducing prediction errors under challenging validation conditions and increasing confidence for use as a decision-making tool for managers, operators, and process engineers. See a recent research article on hybrid modelling in wastewater treatment Serrao et al., 2024.
Explainable AI (XAI) is a set of processes and methods that allows users to understand and trust the results and output created by AI's machine learning (ML) algorithms. XAI provides the explanations accompanying AI/ML output. A hybrid grey-box model architecture is an explainable artificial intelligence by design. The mechanistic layer gives operators something they can reason about — the biological and chemical knowledge is preserved and visible — while the ML layer acts as an adaptive corrective lens rather than a mysterious predictor. When such a Physics-Informed AI model is embedded in a digital twin with real-time data connectivity, the result is a decision-support system where a recommendation like "increase aeration" can be traced back to specific process signals and model states. Physical grounding is what makes the model inherently explainable: a synergy that combines the predictive power of AI with the explanatory capacity of physics-based modelling.
Making predictions traceable: SHAP values and sensitivity analysis
Embedding physics into model architecture is the foundation. Two complementary tools make model outputs fully traceable in practice.
Sensitivity analysis. This is a classical technique long familiar to process engineers from ASM calibration. It asks a simple but powerful question: if input variable X changes by a small amount, how does model output Y respond? Applied to the hybrid model in our example case, it reveals which biological rate constants, hydraulic retention times, or influent load parameters most strongly drive predictions, focusing operator attention precisely where it matters most. It speaks the native language of process engineers: partial derivatives, parameter rankings, uncertainty ranges.
SHAP (SHapley Additive exPlanations) values, drawn from cooperative game theory, go a step further: for any individual prediction, they quantify the contribution of each input variable, such as sensor readings, water temperature, dissolved oxygen, nitrate concentration, to that specific model output. Instead of seeing "increase methanol dosing by 18%", the operator sees for example: "Nitrate spike detected at reactor inlet: +12%, drop in water temperature: +4%, low dissolved oxygen: +2%. Recommended to increase methanol dosing by 18%". The recommendation is the same; however, the reasoning is now auditable.
Applied together to a Physics-Informed hybrid model, sensitivity analysis interrogates the deterministic backbone while SHAP illuminates the adaptive ML layer. The combination delivers a full audit trail from raw process data to actionable recommendations, which is precisely what operators and managers need to move from skepticism to confidence.
Not every problem needs a hammer
A critical warning is in order. The enthusiasm around AI tools sometimes leads to their indiscriminate application, irrespective of whether a problem actually requires them. As argued in a recent critical review co-authored by this author: not every water-sector challenge is a nail that needs the ML hammer (Vanrolleghem et al., 2025).
For well-understood processes with stable dynamics and sufficient mechanistic knowledge, a calibrated deterministic model remains more appropriate — more interpretable, more robust to data scarcity, and more defensible to regulators. ML should be deployed where it adds genuine value: handling nonlinearities that physics-based models cannot capture, filling sensor gaps and creating virtual sensors, or accelerating real-time predictions at scale. Choosing the right tool requires intellectual honesty about what the data can and cannot support. This question of tool selection in AI-assisted water engineering has been addressed from a design and CAPEX perspective in an earlier blog.
From confidence to adoption
For utility managers, the business case is direct. Explainable model outputs reduce the frequency with which valid AI recommendations are overridden by skeptical operators; which is a well-documented barrier to digital-tool adoption. XAI features, whether post-hoc explanation methods such as SHAP and sensitivity analysis applied on top of ML layers, or structural interpretability built into Physics-Informed AI architectures from the start, should become standard requirements in procurement specifications for digital water management tools.
Different technology companies now offer large language models as conversational interfaces for industrial operations, and the water sector will rightly embrace them as a useful layer of support. But a chatbot that explains an opaque model's output in plain language is not the same as a model that is inherently explainable. Physics-Informed AI is not a product, it is a design philosophy that builds transparency in from the start, equation by equation, decision by decision.
