user data failure suddenly occurs on working PLC

I have CPU 1518F-4 PN/DP with firmware v2.1.0 and TIA Portal V14 SP1 Upd 7.

The PLC has been installed and working for almost 2 years.  The local rack consists of (in order):

25W system PS

CPU (with connection to supply voltage L+)

Two CP 1543-1

Three TM Count2x24V

One DI 32x24VDC

One DQ 16x24VDC/0.5A HF

Suddenly the PLC had "User data failure of hardware component", event ID 16# 02:3958, followed by "Error: Hardware component removed or missing", event ID 16# 02:3951, on all of the TM Count, DI, and DQ modules.   According to the diagnostic buffer, they all failed and came back in a  couple of milliseconds.  This caused the CPU to go into STOP mode.  After putting CPU back to run, there have not been any further problems (as of 6 hours).

I'm wondering has anyone encountered something like this before?  Or have any ideas of the cause?

That is one possibility which I'm looking into.   The fault only lasted about 4 msec and has not happened again for over two days now.

The 24V that feeds into the system power supply and CPU also powers two other S7-1500 PLCs which did not have any problems.  So at the moment my suspects are the system power supply module, the CPU module, or the bus connectors.   I just don't see any way that the bus connector would fail.  And I'm unclear as to how either the PS or CPU could fault one time only like this.

Are there any large inductive loads, or their power wiring *near* the plc or its modules?

A bonding problem _might_ cause a module to temporarily get scrambled if a large voltage spike hit the ground plane, like if a large contactor was turned off right about the same time as the problem happened.

NSSEA Steve

I have CPU 1518F-4 PN/DP with firmware v2.1.0 and TIA Portal V14 SP1 Upd 7.

The PLC has been installed and working for almost 2 years.  The local rack consists of (in order):

25W system PS

CPU (with connection to supply voltage L+)

Two CP 1543-1

Three TM Count2x24V

One DI 32x24VDC

One DQ 16x24VDC/0.5A HF

Suddenly the PLC had "User data failure of hardware component", event ID 16# 02:3958, followed by "Error: Hardware component removed or missing", event ID 16# 02:3951, on all of the TM Count, DI, and DQ modules.   According to the diagnostic buffer, they all failed and came back in a  couple of milliseconds.  This caused the CPU to go into STOP mode.  After putting CPU back to run, there have not been any further problems (as of 6 hours).

I'm wondering has anyone encountered something like this before?  Or have any ideas of the cause?

There are three PLCs feed from a pair of Phoenix Contact 24V power supplies with paralleled outputs for redundancy.  Each power supply is capable of running everything by itself.  These are dedicated to the PLCs.  No other loads.  This PLC is the only one that even has local I/O.  All of this is in dedicated cabinet in office environment.

The DI and DQ modules are wired entirely locally.   Encoder cables do run out of the cabinet.  We ran shielded cables and try to keep good separation from other cables.

The DQ does drive some small 24VDC relays, only one at a time and they only switch at a specific event (Production run change), which most certainly was not at the time of the fault. 

Try running the power supplies non-redundant if the problem happens often enough that you could tell it was solved within a short time.

If the problem does not happen all that often, maybe use the redundant / parallel supply's fault contacts as fault inputs to the plc, or to some physical counter that counts the number of fault events the power supplies have.  When I say physical counter, I mean something that requires no power to see the count value, like you might see on a 1960's era control panel.

As to the DQ's running relays, I assume with all the effort described, all the inductive loads (relays included) have proper suppression to protect the PLC and power supplies from the inductive kicks that happen when the relays turn off.

If you have a voltmeter you can plug into a convenience outlet near the control panel, you could use it for min/max detection to see if there are voltage spikes.

You could look at the incoming power and the output voltage from the redundant power supply setup and check it each day, noting normal ranges.

When it happens again, see if it was an incoming power brown-out (in which case a UPS solves that), or something with the 24V supply setup, in which case a 24V UPS would maybe help.

NSSEA Steve

There are three PLCs feed from a pair of Phoenix Contact 24V power supplies with paralleled outputs for redundancy.  Each power supply is capable of running everything by itself.  These are dedicated to the PLCs.  No other loads.  This PLC is the only one that even has local I/O.  All of this is in dedicated cabinet in office environment.

The DI and DQ modules are wired entirely locally.   Encoder cables do run out of the cabinet.  We ran shielded cables and try to keep good separation from other cables.

The DQ does drive some small 24VDC relays, only one at a time and they only switch at a specific event (Production run change), which most certainly was not at the time of the fault. 

Hopefully unrelated, but.

NSSEA Steve

There are three PLCs feed from a pair of Phoenix Contact 24V power supplies with paralleled outputs for redundancy.  

The old teaching was to never connect switch-mode power supply (PSU) together.

Reason: The PSU measures its own output to regulate and also protect against short-circuit. The switching frequency generated by a "foreign" partner PSU can interfere with the measure process. Occasionally the PSUs can fight each other for dominance pushing the voltage too low.

Probably newer PSUs can handle it, though it will be nice if some specification sheet would say that parallel connection is possible. No all device ranges (even in the same brand) is created equal.

Additionally the following link suggests how redundant connection should be done - https://www.deltapsu.com/technical-articles/how-to-operate-parallel-and-series-connection

Other thoughts -
If a short-circuit happens, both PSUs will react to the occurrence. A brown-out to the supply power is one other enemy if there is no UPS type PSU.

Check for badly fastened wires or wires with bad connectors. (remembering experience of loose connector that gave problems 4 months after commissioning, which was not operator sabotage)

Are these 24V supplies by P.C. designed to operate in a redundant mode?  If not, please stop paralleling them and get power supplies designed to operate in parallel.

NSSEA Steve

There are three PLCs feed from a pair of Phoenix Contact 24V power supplies with paralleled outputs for redundancy.  Each power supply is capable of running everything by itself.  These are dedicated to the PLCs.  No other loads.  This PLC is the only one that even has local I/O.  All of this is in dedicated cabinet in office environment.

The DI and DQ modules are wired entirely locally.   Encoder cables do run out of the cabinet.  We ran shielded cables and try to keep good separation from other cables.

The DQ does drive some small 24VDC relays, only one at a time and they only switch at a specific event (Production run change), which most certainly was not at the time of the fault. 

Yes, they are.  I checked the tech specs specifically for that before buying them.  Phoenix Contact #2904601.   Each one is fed from a separate UPS.  Both UPSs are double-conversion on-line type, so they regulate output voltage and have no transfer delay.

I doubt that the relays driven by the DQs are a problem.  They only change state when we change product runs.  The PLC fault happened in middle of a run.  But I will check if there are flyback suppression diodes.

I've also verified that all the modules are tightly bolted to the base plate.

So far, the problem has not happened again after 3 1/2 days.   Once in 20 months isn't exactly often, but people crunching production data kinda like to have answers about the cause and some assurance that it won't happen again... especially when I'm asking them to approve a vacation request.  :-)

AutoGen_1801304

Probably newer PSUs can handle it, though it will be nice if some specification sheet would say that parallel connection is possible. No all device ranges (even in the same brand) is created equal.

Additionally the following link suggests how redundant connection should be done - https://www.deltapsu.com/technical-articles/how-to-operate-parallel-and-series-connection

Other thoughts -
If a short-circuit happens, both PSUs will react to the occurrence. A brown-out to the supply power is one other enemy if there is no UPS type PSU.

Check for badly fastened wires or wires with bad connectors. (remembering experience of loose connector that gave problems 4 months after commissioning, which was not operator sabotage)

 I did check the tech specs before selecting this model.  It does explicitly state that these may be paralleled.

Very informative link.  

We're also using double-conversion UPSs.  They have the inverter always online so that there is no switch-over delay and it actively regulates the output.  They also have a network management card which reports events to a central syslog server, as well as having internal web page with event log.