Four Things Every RCA Taught Us to Check at Recommissioning

Time, grounding, DC supplies and the UPS. Unimportant to many, Important to run system reliably.

After a trip, everybody wants the same answer: what happened first?

Over many investigations in power plants I have looked for that answer. Different plants, different control systems, different countries. The root cause is different every time. But the reasons the investigation is slow, painful and sometimes inconclusive are almost always the same.

These four items keep coming back. None of them is exotic. None of them needs new technology. All of them are cheap to check during recommissioning and very expensive to reconstruct after a trip.

This article is about those four items, and about the tool we built years ago so that an RCA does not have to start with a spreadsheet of guesses.

1. Time and logs: the clocks nobody looks at

The first step of any root cause analysis is putting every log on one timeline. Control system events, protection relay records, breaker operations, inverter alarms. Only when they share one time base can you see the order of events, and the order of events is the investigation.

On one recent site we did exactly that, by hand. We merged all the logs into a single, very complicated spreadsheet and corrected each clock separately. What we found:

• generator protection relays several minutes off,
• other relays up to 35 minutes off,
• bus transfer relays still showing a date years in the past,
• inverters around half an hour out.

It was a lot of work, and the result was never precise. When two events are seconds apart and the clocks are minutes apart, you are no longer reading the sequence. You are estimating it.

Everybody checks the DCS and HMI time synchronisation. That is the part of the plant people look at every day. The problem sits in the devices nobody looks at: protection relays, bus transfer relays, inverters, and any intelligent device added later with its own internal clock.

The second half of the problem is retrieval. Logs that exist but take days to extract, or that are overwritten before anyone asks for them, are almost as bad as logs that do not exist.

What to check at recommissioning:

• Every device that writes a timestamp is on the plant time source, not only the DCS.
• Record the actual offset of each device, not just "sync enabled".
• Know where every log is stored, how long it is kept, and how fast it can be retrieved.

2. Grounding: measure, do not assume

Grounding is usually treated as a drawing. There is an earthing scheme, it was approved, and everybody assumes the plant still matches it.

In practice plants change. Cabinets are added, cables are rerouted, equipment is replaced. A tidy drawing is not a measurement.

The measurement is simple. Voltage between neutral, PEN and protective earth. Current on the earth conductor. Measured at many points, with defined limits, and recorded at recommissioning. Then you have a reference. If something changes later, you can compare against a known good state instead of arguing about what the plant was supposed to look like.

What to check at recommissioning:

• A written procedure for grounding measurements with acceptance limits.
• Measurements at enough points to cover each cabinet group and each power distribution path.
• Results stored with the commissioning records, so the next investigation can use them.

3. DC power supplies: one fault hides, two faults trip

In some designs the same DC supply feeds very different things. Field devices on the turbine, such as valve actuators. The transformer and generator protection. The protective relays. The trip coil of the main breaker.

That is efficient on paper. In the field it means a fault in the hottest, most vibrating part of the plant shares a supply with the devices that are supposed to protect it.

A ground fault on top of the turbine shifts the reference of the whole DC system. Often nothing happens. The plant keeps running, and the fault stays hidden even when alarm of Earth Fault is shown on the screen. Add a second ground fault somewhere else and you get the behaviour that makes RCAs so difficult: unexpected trips, or a breaker that does not open when it should.

These faults can also be intermittent. A cable that touches a hot surface only when a valve moves will look healthy on a quiet day.

What to check at recommissioning:

• Measure DC voltages and ground faults on every DC system.
• Do it while field devices are moving, not only at standstill.
• Know which loads share which DC supply, and treat shared supplies to protection and trip circuits as a risk to be reviewed.

4. Added UPS units: the earth path nobody seen comming

Somebody adds a small UPS "just to be safe". For a server, a switch, a monitoring panel. It is a sensible decision, and it is rarely part of the original design.

But a UPS has its own wiring and its own earthing. Earthing the protective earth at the UPS can create a second, unintended grounding path, and quietly undo the grounding scheme from point 2. The drawing still looks correct. The plant no longer is.

What to check at recommissioning:

• List every UPS in the plant, including the small ones added later.
• Include each one in the grounding measurements.
• Check how its neutral and protective earth are connected.

5. Why this belongs in recommissioning

All four items have something in common. They are easy to check when the plant is being brought back into service, and almost impossible to reconstruct after a trip.

During recommissioning the teams are on site, the cabinets are open, and checking things is the job. After a trip, the clocks have kept drifting, the faults may have cleared, and the UPS that caused the problem looks like it has always been there.

So the message is simple. These checks belong inside the recommissioning procedure, not after it. Done there, they prevent faults. And if a trip still happens, the next RCA starts with a common time base, retrievable logs and grounding records, instead of a spreadsheet of guesses.

6. Instead of a spreadsheet: one timeline

Merging logs in a spreadsheet is what you do when there is no better tool. We built that tool in 2014. It is called SOE Explorer, and it ran on a tablet.

The idea was simple: every event from the plant, on one timeline, readable by an engineer standing next to the machine.

The home screen shows years of plant events at a glance, from 2006 onwards, coloured by type. Below it are saved reports: each investigated trip kept as a study that can be reopened, with a note on what was found.The timeline shows every event of a trip in sequence, second by second. Alarms, warnings and faults line up against each other, so the order of events is visible instead of reconstructed.

The timeline shows every event of a trip in sequence, second by second. Alarms, warnings and faults line up against each other, so the order of events is visible instead of reconstructed.

For each signal you can open its history: how often it has fired over the years. A signal that fires once during a trip is different from one that has been firing for years and was simply ignored.

And each signal carries its meaning, not just its tag. The alarm text from the operating manual is right there: what the signal is, its significance, the consequences and what the operator should do. An investigation needs that context, and it should not depend on finding the right page of the right manual.

That was more than ten years ago. The principle has not changed. One time base. One timeline. Then the RCA.

A short checklist

For the next recommissioning:

1. Time: every timestamping device on the plant time source, actual offsets recorded, logs retrievable.
2. Grounding: measured at many points against defined limits, results recorded.
3. DC supplies: voltages and ground faults measured, including with field devices moving, shared loads known.
4. UPS units: every unit listed, included in the grounding check.

Point five?

These four come from real investigations. I am sure they are not the only ones.

What would you add as point five?

Petr Roupec

Managing Director, BM Global