Anomaly Detection on Distribution Boards

Overview

What it is

A condition-monitoring tool that learns an electrical distribution board's normal daily rhythm from its own logs, flags departures from it, and records each investigation as a permanent, exportable case file.

It runs locally. It adds no sensors, no cloud service and no vendor dependency — it reads telemetry that was already being logged, every day, and read by nobody.

What it answers

One question: did something depart from normal today?

That is deliberately narrower than "is this board faulty". Fault detection already existed on site and worked well. What nothing covered was the space underneath it — the departures that are not yet faults, the ones nobody is looking for because no alarm is configured to look for them.

What it does

• Builds a per-minute-of-day baseline for each channel from the board's own clean days.
• Runs two detectors over that baseline — one for sustained shifts in the daily shape, one for sharp transients — designed so they cannot double-count each other.
• Ranks what it finds by severity and writes each investigation to a case file.
• Says "insufficient data" when it has too little clean history, instead of manufacturing a baseline.

Every output is deterministic arithmetic. No machine learning sits anywhere in the analysis path, which means every number in a report can be defended line by line to the electrician it is sent to.

What it doesn't claim

It learns normal from the board's own recent history. A fault that has always been present reads as baseline. This tool finds departures from habit — not absolute correctness.

It has been validated against one independently confirmed fault event, reconstructed retrospectively from upstream protection logs. One event is a demonstration, not a catch rate.

Where it led

This became the foundation for a second piece of work: drift prediction on a critical fan motor, built on the same ingestion pipeline, the same database, and the same rule that only measured values are ever read. This half asks did something depart from normal today? That half asks is this fan slowly changing across weeks?

Read Part 2 — Drift Prediction on a Critical Fan Motor →

Journey

It started with the data, not with a failure

There was no incident that prompted this. I was sitting in a contractor training session and noticed rows of telemetry being logged daily — per-phase voltages, currents, relay data, protection settings — accumulating continuously. I recognised it as rich and completely unused, and downloaded a dataset to look at properly.

I did not, at that point, know what half the fields meant. I could not have told you what THD was.

What I did know was that the system logging all of this existed for outage tracking. That told me something useful before I understood a single column: the organisation already treated this data as mission-critical. It was being collected carefully, retained, and relied on when something went dark. It just wasn't being read on the days when nothing went wrong.

That gap — mission-critical collection, zero routine interrogation — was the opening.

The strategic problem, and inverting it

Predictive maintenance in building services is dominated by established M&E vendors. Their model is consistent: they install their own sensors, under their own standard operating procedure, and sell the monitoring that sits on top.

Against that I had no money, no domain credibility, and no existing trust with the people who would have to approve anything I built.

Competing on their terms was not available to me, so I inverted the constraint. Rather than treating the absence of a sensor budget as the thing blocking the work, I treated the existing instrumentation as the thing that made the work possible. Every distribution board on site was already metered. Every protection device was already logging. The infrastructure had been paid for years ago.

That became the wedge: no hardware, no installation, no vendor lock-in. Not a compromise version of the vendor offering — a different proposition, available in places where the vendor proposition never gets approved.

The data was not what the documentation implied

The first real work was discovering how incomplete the datasets were.

Voltages missing on some boards. Relay data absent on others. Upstream protection coverage with no matching downstream component, so I could see that something had happened without being able to see where. The shape of what was logged varied board to board in ways that no schema described.

I could not resolve this from the files alone, so I stopped trying and went to the people who maintain the equipment. Those conversations established the distinction that mattered most: what is actually tracked versus what people believe is tracked. Those are not the same list, and building on the second one would have produced a tool that worked beautifully in development and failed on contact with a real export.

From there the task was concrete: build a platform that extracts the information relevant to predictive maintenance from the sets that genuinely exist.

The scope correction

My first build was fault detection.

It was wrong, for two reasons I found in sequence. First, the Building Management System already did fault detection, and did it well — I was rebuilding something that worked. Second, and more fundamentally, without a meter at the point of interest I could not capture the exact moment of a fault anyway. The temporal resolution required to characterise a fault event was not in the data I had.

The obvious move was to request hardware. I did not, and I think that was the right call — asking for a sensor budget on day one would have collapsed the entire premise of the project, which was that useful work could be done without one.

So I scaled back to the thing nothing currently covered: anomaly investigation. Not "has this board tripped" — that was handled — but "has this board's behaviour departed from its own established pattern, in a way nobody has been alerted to". A smoke alarm rather than a fire report.

The narrower scope turned out to be the more defensible one. It sits in genuine white space, it requires no new instrumentation, and it does not compete with a system that already works.

The one real fault

A genuine fault did occur at the facility during the period covered by the logs, and I reconstructed it retrospectively from upstream protection records.

The algorithm flagged it: a significant drop in fault current across all three phases.

The more interesting finding was how it had to be flagged. The target signal was masked by a strong daily load cycle — peak magnitude alone could not distinguish a fault day from a normal one, because the daily peak on a normal day was comparable. What discriminated the event was not the excursion itself but what happened after it: whether the baseline recovered to its previous level or stayed permanently depressed.

That reframing — from "how big was the departure" to "did normal come back" — is the single most useful thing this half of the project taught me.

Lessons

Look at what is already being collected before asking for anything new. The most valuable dataset available to me was one that already existed, was already trusted enough to be mission-critical, and was already being ignored.

A constraint you cannot remove is a position you can occupy. No sensor budget was not the obstacle to the project. It was the project's differentiation.

Ask what is tracked, not what should be tracked. Domain experts answer the second question by default. The first one takes deliberate asking, and it is the one that determines whether your code runs.

Not rebuilding something that works is a design decision worth defending. The BMS did fault detection well. Scaling back to the uncovered space was a reduction in scope and an increase in value.

The masking signal is often more informative than the signal. The daily load cycle hid the fault. Understanding the cycle well enough to see through it produced a better detector than trying to make the fault louder.

Technical

Architecture

A Python/Flask application running on localhost. SQLite for persistence. pandas and NumPy for analysis. Plots rendered as inline SVG rather than through a JavaScript charting library.

Single user, local only, no cloud service, no external calls. These were constraints I set deliberately rather than limitations I worked around: the data describes the electrical infrastructure of a live public facility, and the smallest defensible attack surface was one with no network in it.

Learning the baseline

The tool does not hold assumptions about what a distribution board should look like. It learns each board's normal from that board's own history.

Construction. For each channel, a per-minute-of-day profile is built from the board's clean days only. A minimum of three clean days is required; the window rolls over the most recent fourteen.

Centre and envelope. The centre of the baseline is a robust median. The normal envelope is ± k robust sigma, derived from the median absolute deviation rather than the standard deviation, so that a single bad day cannot inflate the envelope and blind the detector for the following two weeks.

The property that matters. Because the profile is per-minute-of-day, the board's scheduled behaviour — the morning ramp-up, the evening shutdown — sits directly on the baseline rather than appearing as a twice-daily excursion. Routine daily behaviour is never flagged. Normal is learned, not assumed, and that includes normal being different at 06:00 than at 18:00.

The two-detector design

A single detector cannot cover both failure modes. A spike detector goes blind to gradual change; a trend comparison averages spikes away. So the signal is split into two orthogonal components and each gets its own detector.

Detector 1 — daily-shape shifts. Reads the smoothed curve, a 15-minute rolling median. Flags sustained departures outside the normal envelope, minimum duration 30 minutes.

Detector 2 — transients. Reads the residual that the smoothing discards. Flags sharp excursions at or beyond 6σ of that day's own noise, maximum duration 10 minutes.

The important property is structural: they cannot double-count by construction. A slow shift leaves no residual, so Detector 2 stays silent. A sharp spike cannot move a 15-minute median, so Detector 1 stays silent. When both genuinely fire, the findings are linked in the report and presented as two observations of one event. They are never blended into a single fabricated risk score.

Design rules on honesty

These constraints do more to make the tool trustworthy than any of the detection logic does.

Measured values only. The logs contain status columns — trip status, last trip type — that are known to be unreliable. The tool never reads them. Not "reads them with a caveat": never reads them. A derived or self-reported channel that is wrong in an unknown direction is worse than an absent one, because it looks like evidence.

Known trips are validation, not input. Where a user supplies a known trip time, it is drawn on the timeline and cross-checked against what the detectors flagged. It is structurally unable to reach the detection path. The detectors cannot see it, so it cannot contaminate them, so the cross-check means something.

Insufficient data is an output. Fewer than three clean days produces "insufficient data", not a baseline assembled from whatever is available. A fabricated baseline in a maintenance report is worse than no report.

Nothing is silently dropped. Flat or idle days are set aside from baseline construction — and named in the output. If a channel was excluded, the report says which one and why.

Case files

Each investigation is recorded as a permanent case file, exportable to Word or CSV. This is less a technical feature than an organisational one: an anomaly that is looked at once and forgotten produces no institutional memory, and the value of a baseline-departure tool compounds only if the departures accumulate into a record someone can look back through.

Why no machine learning

Three reasons, in order of weight.

No training data existed. No monitored board had failed. There was nothing to learn from, and a model trained on synthetic faults learns the synthesis.

The output directs a person. These findings go to electricians and to management. A deterministic rule fires identically every time and can be argued from physics. A model's output can be reported but not defended, and the first time it is wrong in front of someone who knows the equipment better than I do, the tool is finished.

It would have been resume-driven engineering. The problem did not call for it. Adding it would have been for my benefit, not the facility's.

Testing

Every component shipped with a self-test. 234 automated assertions on this half of the tool.

The fixtures are not synthetic in the usual sense — they deliberately reproduce the real faults found in real exports, so that each bug discovered in production data became a regression test that would catch it again.

What's validated

Validated

The engineering. 234 automated assertions on this half of the tool, all passing, with fixtures modelled on faults found in real exports rather than invented failure modes.

One fault event. A genuine facility fault was reconstructed retrospectively from upstream protection logs, and the method correctly identified the departure — a significant drop in fault current across all three phases, with a baseline that did not recover afterwards.

That is a demonstration that the method works on a real event. It is not a catch rate. One confirmed event establishes that the detector can fire correctly; it says nothing about how many events it would miss.

Not validated

Detection sensitivity. Establishing what proportion of real anomalies this tool catches requires observing many more of them. That is a matter of elapsed time and accumulated incidents, not additional analysis.

False negative rate. Unknown, and not estimable from the data available.

Structural limitations

These are properties of the approach rather than gaps in the implementation. They do not go away with more data.

It finds departures from habit, not absolute correctness. The baseline is learned from the board's own recent history. A fault that has always been present is, to this tool, normal. If a board has been running with a defect since commissioning, it will read as healthy for as long as the defect stays stable. Detecting that class of problem requires a reference outside the board's own history — a sibling comparison, a design specification, or a physical measurement.

The duration dead zone. Detector 2 accepts events up to 10 minutes. Detector 1 requires 30 minutes or more. An isolated event lasting roughly 20 minutes can fall between them.

This is a real gap and it is documented rather than hidden. The two thresholds are not arbitrary — they follow from the 15-minute smoothing window, which sets both what the smoothed curve can resolve and what survives into the residual. Closing the gap means either a second smoothing scale or accepting more false positives at the boundary, and I have not yet established which trade is better.

No fault-moment resolution. Without a meter at the point of interest, the exact instant of a fault cannot be captured. This is why the scope is anomaly investigation rather than fault characterisation, and it is a hardware limitation rather than an analytical one.

Masking by load cycle. The strong daily load cycle can hide a genuine excursion when its magnitude is comparable to a normal daily peak. The recovery-based reasoning above works around this for one class of event. It is not a general solution.

On statistical rigour

Worth stating plainly. Decisions about baseline window length, envelope width, and what constitutes a statistically meaningful departure were made pragmatically — from what behaved sensibly against real exports — rather than from a grounded understanding of the underlying statistics.

The choices are defensible and the reasoning behind each is recorded. They are not derived. Formal training in analytical methods would let the same problems be approached with more rigour, and would extend what can responsibly be claimed from the results. I would rather say that than imply a foundation that isn't there.

Future work

Close the duration dead zone

The gap between the two detectors — events around 20 minutes long, too short for the sustained-shift detector and too long for the transient detector — is the clearest known defect in the design.

Two candidate fixes. A second smoothing scale would give a third detector covering the middle band, at the cost of a more complex report and a harder argument about whether three detectors are still orthogonal. Alternatively, relaxing the boundaries on the existing two would close the gap directly and increase false positives near the edges.

I have not decided between them, and I would want to see a real event land in the dead zone before committing — fixing a theoretical gap in a way that degrades real performance is a bad trade.

More assets, and sibling comparison

The most useful reference for a board with no failure history is an identical board next to it. Absolute thresholds require knowing what the number should be; sibling comparison only requires two units that ought to agree.

That pattern strengthens with every asset onboarded. It is already the strongest tool available on the fan work, where an order-of-magnitude difference between two identical units produced the most actionable finding of the project.

Onboarding is deliberately cheap — everything is learned per asset, so adding a board is a file upload rather than a configuration exercise.

Getting the mission-critical list named

The realistic next move does not come from me. It comes from the organisation naming the systems it considers mission-critical and wants tracked.

The model and its workings have been shown internally. There is appetite to explore further. There is not yet an owner, a date, or a budget, and I would rather say that than dress an internal demonstration up as a sanctioned pilot.

Keeping the fault list

In the meantime I maintain my own record of what actually fails on site.

This is the raw material for the eventual case. The argument for condition monitoring is not won with a methodology document; it is won with a list of things that broke, which of them gave warning in data that was already being collected, and what that warning looked like before anyone noticed.

Being on the ground and seeing what genuinely fails is the leverage I have that a vendor proposal does not.