V90 - Spontaneous occurence of 421 technology alarm

• TIA Portal V18
• PLC CPU 1514SP F-2 PN
• 2x V90 6SL3 210-5FB10-4UF1
• The PLC controls two V90 drives via TO control
• Hardware added to project via the ["Drives and Starters" -> "SINAMICS Drives" -> "SINAMICS V90 PN"] route
• The V90 are daisy-chained and connected directly to port 2 of the PLC. This is also defined in the topology to satisfy IRT requirements. Their is not other network traffic on this port
• Profinet cables are ready-made straight from the manufacturer - we did not install our own connectors
• Telegram 3 is used for data exchange with the PLC

• The frequency varies, but this occurs 4-12 times a day
• Both drives go into fault at the same time
• There is no specific pattern to the fault occurrence: It can occur during standby, it can occur during servo motion.
• The only thing which goes into fault is the Technology Object. There are no corresponding faults or alarms on the V90 drives, or in fault/alarm history.
• The Error Detail in the TO data block is 421. I've attached the Help information for this alarm, although it did not as of yet help me identify the root problem

In your S7-1514 PLC's Diagnostics Buffer, what entries relate to the faulting of the V90 PLCs?

Are there clues provided?

Hopefully you set your PLC time to current time-of-day (TOD) so the diag buffer entries are TOD related.  I know it really helps when someone says at 2:12 PM today something weird happened.  Then you know when to look for odd problems showing up in the PLC diag buffer.

How about the V90 alarm / warning history?  What clues are found there?

Zivilin

So this is a bit of a mystery: I have to two identical machines, here is some information about the setup:

• TIA Portal V18
• PLC CPU 1514SP F-2 PN
• 2x V90 6SL3 210-5FB10-4UF1
• The PLC controls two V90 drives via TO control
• Hardware added to project via the ["Drives and Starters" -> "SINAMICS Drives" -> "SINAMICS V90 PN"] route
• The V90 are daisy-chained and connected directly to port 2 of the PLC. This is also defined in the topology to satisfy IRT requirements. Their is not other network traffic on this port
• Profinet cables are ready-made straight from the manufacturer - we did not install our own connectors
• Telegram 3 is used for data exchange with the PLC

The problem is that one machine (Machine A) works fine, while in the other (Machine B) the drive technology objects spontaneously go into fault. This is, naturally, very frustrating for the client. Here is some facts about the problem:

• The frequency varies, but this occurs 4-12 times a day
• Both drives go into fault at the same time
• There is no specific pattern to the fault occurrence: It can occur during standby, it can occur during servo motion.
• The only thing which goes into fault is the Technology Object. There are no corresponding faults or alarms on the V90 drives, or in fault/alarm history.
• The Error Detail in the TO data block is 421. I've attached the Help information for this alarm, although it did not as of yet help me identify the root problem

I've tried swapping the Profinet cables and increasing IRT Sync-Domain cycle time, but to no avail. The safety system is just a simple terminal STO, so I don't believe it is generating the problem. I've resorted to a dirty workaround of automatically resetting the servo fault if it occurs during motion but this is not a solution I would want to keep around long term.

I'd like to know whether anyone had something like this happen? What was their solution?

Alternatively, are there any further diagnostic steps I can take? After finding the drive fault/alarm history empty and checking the cables, I'm sort of stuck as to how to proceed.

The PLC diagnostic buffer simply reiterates what I already know: that the drives were shut down due to TO alarm 421. I would very much like to find out exactly what causes TO alarm 421, it from the description it seems to be safety related? I've attached an image of the diagnostic buffer.

We have an NTP server set up, so all PLCs are synced with the same Date/Time, which does indeed help with troubleshooting.

Concerning alarms on the drives themselves, I already mentioned this in the original post, but when I last checked there weren't any eye-catching alarms, at least not at the frequency at which the actual problem was occurring. That is, the day I had around 10 of these TO 421 faults, I would expect to have a similar number of faults in the Drive fault history, but all I had is a few 8501 Profidrive setpoint timeouts and 1912 Profidrive clock cycle synchronous operation sign-of-life failures scattered across a dozen days, probably corresponding to the times I or my colleague were downloading modifications to the safety system, causing the PLC to go into STOP.

Do you have profidrive diagnostics enabled?  Try running with it disabled and instead use the status word from the V90's telegram to discover a drive fault and handle it - hopefully more gracefully than you can with Profidrive telegram level diag enabled.

What is the IRT update rate?  Is it at the extreme lowest value you can use for the V90?

How are you handling the telegram updates for the IRT V90's?  Using OB1 (cyclic), or OB3x cyclic interrupt?  Hopefully OB3x.

Having _any_ sign-of-life failures or setpoint timeouts is a problem with the profinet configuration, either in the subnet, or IRT or not having a powerful enough S7-1500 PLC.

I know there is another machine not shutting down.  What diag buffer entries does it have?  Hopefully it just says it moved from stop to run and no other entries are found since it was last started.  That's what you _should_ see on both systems -- except when something is wrong - broken cable, drive component failure, load changes beyond original design criteria either due to wear and tear or trying to push past the boundaries without buying a new more capable machine.

Zivilin

The PLC diagnostic buffer simply reiterates what I already know: that the drives were shut down due to TO alarm 421. I would very much like to find out exactly what causes TO alarm 421, it from the description it seems to be safety related? I've attached an image of the diagnostic buffer.

We have an NTP server set up, so all PLCs are synced with the same Date/Time, which does indeed help with troubleshooting.

Concerning alarms on the drives themselves, I already mentioned this in the original post, but when I last checked there weren't any eye-catching alarms, at least not at the frequency at which the actual problem was occurring. That is, the day I had around 10 of these TO 421 faults, I would expect to have a similar number of faults in the Drive fault history, but all I had is a few 8501 Profidrive setpoint timeouts and 1912 Profidrive clock cycle synchronous operation sign-of-life failures scattered across a dozen days, probably corresponding to the times I or my colleague were downloading modifications to the safety system, causing the PLC to go into STOP.

0_Einstein

Do you have profidrive diagnostics enabled?  Try running with it disabled and instead use the status word from the V90's telegram to discover a drive fault and handle it - hopefully more gracefully than you can with Profidrive telegram level diag enabled.

I'm not sure what is meant by "enabling profidrive diagnostics". I check for historical faults and alarms directly from TIA Portal, using the Online & Diagnostics window (see attached image - please disregard the dates, the drives did not have their dates set, unlike the PLCs). In the PLC itself, I use a SINA_PARA_S to read out, I believe it is parameter r945, to read the currently active alarm and display it to the operator. Neither of these is something I could disable... well, I could NOT trigger SINA_PARA_S I suppose, but I don't think this is what you were referring to. If you could link me to some materials about the profidrive diagnostics you have in mind, I'll read up on them.

0_Einstein

What is the IRT update rate?  Is it at the extreme lowest value you can use for the V90?

I started out at 2ms, then bumped it up to 4ms. That helped my PLC cycle time a bit, but didn't help at all with the spontaneous error 421 thing.

0_Einstein

How are you handling the telegram updates for the IRT V90's?  Using OB1 (cyclic), or OB3x cyclic interrupt?  Hopefully OB3x.

This is probably another knowledge gap on my part... I thought telegram updates were handled by MC-Servo (OB91), which is generated automatically when You add a TO position axis? If you mean where all my MC_POWER, MC_MOVEABSOLUTE etc. blocks are located, then they are indeed all in OB1. Is putting them into OB3X standard practice? The only case I can imagine where that is necessary is if I need to change a position setpoint 'on the fly', faster then OB1 can update.Otherwise, wouldn;t it eat up even more of my PLC cycle time?

In any case, good engineering practices aside, I don't believe this is the root of the problem, since the other machine has the same PLC code and does not suffer from the dreaded 421 error.

0_Einstein

Having _any_ sign-of-life failures or setpoint timeouts is a problem with the profinet configuration, either in the subnet, or IRT or not having a powerful enough S7-1500 PLC.

As I mentioned, this is quite normal if these errors correspond to me or my colleague setting the PLC into STOP mode - a normal occurrence during initlal commissioning when The HW config and safety program are regularly being changed.

0_Einstein

I know there is another machine not shutting down.  What diag buffer entries does it have?  Hopefully it just says it moved from stop to run and no other entries are found since it was last started.  That's what you _should_ see on both systems -- except when something is wrong - broken cable, drive component failure, load changes beyond original design criteria either due to wear and tear or trying to push past the boundaries without buying a new more capable machine

Here's a an update, since it seems I have finally stumbled upon the solution to this phenomenon.

I had all but lost hope after communicating with local Siemens support last year, because their advice didn't go beyond the standard "Did you try checking the cables/connection?". The client got used to some minor inconveniences related to the workarounds I implemented and we left it at that.

However, not long ago, while searching through the forums for clues regarding a different technology alarm, I came across the following forum post:

Sinamics V90 Motion control fault MC 421 on S7-1500

A post there suggested the problem may be related to the dark test functionality used in safety output modules, where the safety output "goes dark" for a short period of time (1ms by default, I think?) for safety diagnostic purposes. The solution used by another user on that thread was to buffer the safety output through a relay, so that the relay's slower reaction speed "absorbed" the 1ms dark test. This left me somewhat doubtful, since I already had the same setup with a relay between the safety output and Drive STOs, so the machine should have already been immune to the problem. However, just to be sure, I disabled the test entirely in the FDQ module HW configuration and lo and behold, the mysterious 421 alarms miraculously stopped popping up.

A few ending thoughts:

• It seems that V90s don't really have any active alarms regarding the safety system having been triggered, which made it difficult to diagnose the problem - since I never found any direct alarm message stating that the drive was stopped due to the safety system being triggered.
• Disabling the dark test drops the maximum achievable SIL of the safety system, but fortunately this wasn't a SIL3 safety function.
• I'm still scratching my head as to why the relay didn't do the trick - the documentation states it had a reaction time of around 40ms, so a 1ms voltage drop shouldn't've had any effect on the downline STO inputs? At times like these I wish I had an oscilloscope handy.
• I have no idea why this problem affected one machine, but not the other