I/O access error from a non-existent device, followed by removal of hardware

Hello dear colleagues, 

We have been reported by one of our clients that a PLC on their system has switched to STOP mode and could only be set to RUN mode manually by switching the power OFF and ON. We acquired the diagnostics history and noticed a strange series of events. I wanted to ask your opinions before we take any actions. 

First I want to give you a brief explanation of the system to answer some of the questions you might have when you take a look at the diagnostics file:

• We are using configuration control to regonfigure the combination of IO modules according to the scale of the system that the PLC is used. Basically, there are two DI/DO modules with 8 and 16 IO and either none of them, one of each or both of them are used according to the selection of configuration.
• We have an optional PNBacnet gateway device in the configuration but it is not used on that particular system so it is deactivated after every startup but not before initiating a IO device failure. That is not relevant to our topic, at least we think so, but I still needed to mention it to clear some questions that might occur.

Now I will summarize the chain of events that is important to us, in chronological order, from the diagnostics poinf of view:

1. Prior to error, there are no changes for more than 3 days and everyhing is working nice.
2. (Event 25 of 50) We have a New IO access error from IW110, which is addressed on a AI1 signal board, but the diagnostics shows the device of the access error as if its SM 1223 DI16/DQ16, which is not even on the system according to the selected configuration. 
3. (Event 24 of 50) We have a Hardware configuration error which says that the module on the previous event has been removed and that caused the PLC to switch to STOP mode. Again this module should not be on the system according to the selected configuration.

We suspected that maybe the client changed the selection of configuration during runtime and did not want to tell us they did that but it does not also make sense, because even if they change the configuration a "Startup OB" is required for the new configuration to be applied, but there is no sign of a startup of PLC before those errors occured.

The address (IW100) that triggered the error is connected to a pressure sensor on the field, so it is possible that it might malfuntion or cause malfunction on AI1 signal board, so that is something that we can explain to our customer. We will also try "rebuild all" for both hardware and software, just in case, but we are not sure if that would definetely prevent the same thing from happening in the future. 

Do you have any idea what happened and what can be done to prevent it from happening again?

Thank you in advance

Onder Yalcin

 Thank you for those inspirational ideas, we will definetely use them. I didn't use any "Error Interrupts" at all because the PLC was very short in resources but after new firmware I might be able to. 

By the way, the same error occured again after I posted here, same except now the device that caused to fault is listed as PLC_1 / PLC_1 which is meaningful since AI1 signal board is on PLC_1. Before that the device was listed as PLC_1 / SM 1223 DI16/DQ16 which have caused all the confusion because the device did not exist in configuration. 

We might explain  that it is the same error caused by a physical malfunction that effects AI1 signal module, but the fact that it shows this error like it happened on a non-existent module is just a minot glitch maybe?

0_Einstein

What logic do you have in OB82 (Diag Error Interrupt OB)?  You should be able to detect and RETAIN instances of device (or module or IO point) failure, perhaps using simple counts.  Once a device or module reports an error, you can then use DEVICE_STATES or similar block to diagnose and save (RETAIN) the problem in a meaningful way.

Also, it may be helpful to save configuration changes with time/date stamps to a RETAINED DB.

OnderYalcin

Hello dear colleagues, 

We have been reported by one of our clients that a PLC on their system has switched to STOP mode and could only be set to RUN mode manually by switching the power OFF and ON. We acquired the diagnostics history and noticed a strange series of events. I wanted to ask your opinions before we take any actions. 

First I want to give you a brief explanation of the system to answer some of the questions you might have when you take a look at the diagnostics file:

• We are using configuration control to regonfigure the combination of IO modules according to the scale of the system that the PLC is used. Basically, there are two DI/DO modules with 8 and 16 IO and either none of them, one of each or both of them are used according to the selection of configuration.
• We have an optional PNBacnet gateway device in the configuration but it is not used on that particular system so it is deactivated after every startup but not before initiating a IO device failure. That is not relevant to our topic, at least we think so, but I still needed to mention it to clear some questions that might occur.

Now I will summarize the chain of events that is important to us, in chronological order, from the diagnostics poinf of view:

1. Prior to error, there are no changes for more than 3 days and everyhing is working nice.
2. (Event 25 of 50) We have a New IO access error from IW110, which is addressed on a AI1 signal board, but the diagnostics shows the device of the access error as if its SM 1223 DI16/DQ16, which is not even on the system according to the selected configuration. 
3. (Event 24 of 50) We have a Hardware configuration error which says that the module on the previous event has been removed and that caused the PLC to switch to STOP mode. Again this module should not be on the system according to the selected configuration.

We suspected that maybe the client changed the selection of configuration during runtime and did not want to tell us they did that but it does not also make sense, because even if they change the configuration a "Startup OB" is required for the new configuration to be applied, but there is no sign of a startup of PLC before those errors occured.

The address (IW100) that triggered the error is connected to a pressure sensor on the field, so it is possible that it might malfuntion or cause malfunction on AI1 signal board, so that is something that we can explain to our customer. We will also try "rebuild all" for both hardware and software, just in case, but we are not sure if that would definetely prevent the same thing from happening in the future. 

Do you have any idea what happened and what can be done to prevent it from happening again?

Thank you in advance

Onder Yalcin