In practice, how do you handle errors?

[ijustlovemath]

This question stems from a project I'm working on using a database. Clients pull a job from a database, run it, and commit the results back into the database. I've got a working prototype that does pretty well, except for a few issues (that may be chalked up to bad design).

The first involves race conditions: when a client pulls a job from the database, it first checks the job's parameters, then updates the job status to "Running", at which point the job cannot be pulled by another client. The race condition occurs when two clients pull the same job and try to insert the same job's results into the results table (primary key error). I'm planning to add a "Pending" phase to avoid this.

A second one comes when the DB goes down, for whatever reason. It's on a laptop, so downtime is expected on occasion, but these errors are harder to come up with a strategy to fix.

So now that you see a little where I'm coming from, the main question.

How do you guys set up error handling?

I've been considering using a construct like this:

 

The basic idea is to drop deeper and deeper into Error-No Error case structures, and handle errors by continually executing the code in a while loop until we get no errors. The issue with this approach is mostly that a lack of support for recursion makes this code very very messy. I also am not sure how to determine what error codes a set of VIs uses, in order to make proper handling cases for all of them.

I understand the importance of error handling, and get the basic implementation of it, so the NI docs and Sixclear videos are mostly oversimplified. What I would like to find is an in depth resource for error handling constructs. Is there such a thing?

[Sam_Sharp · Accepted solution]

Have a look at the attached PDF - it shows a pretty comprehensive list of how to handle errors within a loop/application.

It doesn't cover things you might want to do with the error such as notify the user, log the error etc.

What you actually do will depend heavily on your application!

[ijustlovemath]

Thanks, this is extremely helpful to me! Do you have any suggestions for looking up error codes for a given VI? Sometimes I just dive into the block diagram, but it's often the case that the low low level subVIs are locked.

[Sam_Sharp]

No, there isn't any documentation that I know of that tells you what errors could be returned by a VI. There are some error codes built into LabVIEW and it's toolkits and I think there is documentation for these online (search 'LabVIEW error codes') but there is always the chance that you might get some unexpected error code (e.g. third party code) and you need to think about what to do in these cases - probably log the error and maybe you'll want to update your software to handle that error code in the future or just treat it as an 'unhandled exception'.

[PatrickLye]

This is similar to an issue I was working on, but they changed my priorities on me again...

What I've come up with is that that:

1. When using the database toolkit the errors are VERY VERY delayed.

2. Some of the VIs in the toolkit wouldn't even output their errors until I put a simple error handler on the error cluster output. (I have no idea what that was all about. Tech support said, "That is odd." Yep, I agree...

3. I use a Functional global to pass data throughout the program.

I have been experimenting with a case structure in a separate loop. In this case structure I have the following cases:

Connect - Loop until connection is made. If connection takes too long go to error case

Read or Write data - Do required function, loop back to this case, or on error go to error case.

Error - This is pretty complex. I ping for connection to required items to perform ANY or ALL of the needed error functions. Restart a system or systems, log to SQL table, Email users from the correct users in all user groups selected from a database table, local alarming at the system in the factory, log alarm to local log file, handle undone error functions when the correct system is reconnected and can be pinged again, write an error to a database table (This table is monitored by computers around the factory. These computers will alarm to notify people of the problem. This program also allows them to select to go to a responsive help website I've created on both the industrial and office intranets. This website opens with the exact help they need within the LabView alarm monitoring program by use of an html5 accordion. I find this type of a help system to be best as I can easily modify it to change as needed. Our personnel tend to need exact step by step instructions with pictures for the things that come up on the computer and automation systems. Then they can follow the steps needed to correct the problem.

Disconnect - This will probably end up in the Error case eventually, but it currently disconnects from the database and sends things on to the connect case.

My eventual plan is to use the Error handling routine I'm developing in this case structure as reusable code that can be dropped in as a VI and easily configured for any needed use. I also want to add in functionality for text messaging to certain groups of users listed in the SQL database. I've found that using a simple error case structure isn't good enough to handle many of the things that I run into, but I do use this, as shown in the SixClear and NI training videos when it is appropriate. Honestly I've never found a need for the simple error handler to actually pop up for the user outside of when I'm debugging code.

My goal is to keep things running in the factory whenever possible. In order to do this I have to look at what can be done to automatically restart connections, computers, PLCs, etc... whenever possible. Many of the people in the factory won't even look at an alarm. This forces me to alarm in multiple places to get someone to actually do anything about it. I actually have it set up to email me at my personal email if a problem isn't dealt with for too long. Then I can VPN into the factory networks and see what needs to be done... It's a bit exasperating at times, but this is the ocean that I get to continually tread water in... Hopefully you won't have to go to the depths that I need to in your facility, but some of these kinds of functions are most likely useful in a lot of other cases.

[ijustlovemath]

Needless to say, that sounds...interesting. If it's not under NDA and you're willing to share, I'd love to take a peek at your error handling. Some of it sounds like exactly what I need. I think I'm going to end up needing to use stored procedures to grab my values properly, without those delays you're talking about. I'd also love to see how you set up that html5 page, as I was thinking of having my DB laptop host a web server that would run queries on DB health and other such things. In any case, good luck on that project!

[Bob_Schor]

A number of the Sample Projects that ship with LabVIEW (open LabVIEW, Create New Project, and look for those that talk about Measurement) use a Queued Message Handler model that includes pretty comprehensive Error Handling.  As I recall, an Error Message can be generated with several "flavors", Major, Minor, Critical, etc. and either be logged, ignored, or cause the program to "gracefully" come to a safe stop.

It might be instructive to create a New Project and choose one (or more) of these examples, print out the VIs, and see which model best fits your situation.  You can, of course, freely adapt the ideas here to your existing code, but the ideas are worth studying.  Central to the idea is that when an Error occurs, the local condition decides on the severity of the Error and what needs to be done immediately, then clears the error and sends an "Error Message" (with appropriate parameters and flags) to the Message Handler's "Error" State for further disposition.

Bob Schor

[PatrickLye]

The delay is in how long the vi sends out an alarm after conneciton is lost. Stored procedures won't help on this. A way to test is to unplug a network cable from a system and then see how long it is before your read or write throws an error. I suspect this is a delay in the underlying code not in the LabView code.

I've attached one of the webpages. It uses the bootstrap framework. Then the program that calls in the help section simply passes something like this "fm_alarms.html#1" to the web server. The webserver then serves the page with the correct accordion section opened. Around here, whenever possible, I try to get the users to the exact help they need without having to browse to it. It helps a lot. This is easy because the web page string is stored in the SQL database, and it is written from the program that writes the error to the database, so that I can use a combo box, populated from an array of errors to select the correct alarm to view.

Sadly I've been pulled off of this project. I hope to at least get the error handling portion completed soon, but I'm currently putting in power monitoring systems throughout the factory that logs electricity usage to a Microsoft SQL database so that the owner can begin to get a better view of how much it actually cost to produce his products.

When I get it down to the reusable VI stage I may be able to post the error handling, but everything is so interrelated I'm not certain how much that will help. Hopefully as I move forward it will get refined to a point that it will be easy to show people.

I'll keep that in mind as I move forward.

[aputman]

If you look in VIPM, there is a library called Structured Error Handler (SEH).  It is an express VI that allows you to handle a specific or generic error code and perform a series of actions (retry, run a VI, notify user, etc).  A separate error loop will handle some macros for safe shutdown, logging, etc. 

[Petru_Tarabuta]

Hi Sam,

"Have a look at the attached PDF - it shows a pretty comprehensive list of how to handle errors within a loop/application." - Thanks for attaching the useful LabVIEW Error Responses.pdf.

Question: Do you know the original location of the PDF? Was it originally attached to an NI Knowledge Base article? I have not been able to find the original source of the document.

There is a slightly different version of the same PDF attached to the following community document: LabVIEW Error Responses.pdf. But that doesn't seem to be the original source either.

Thanks,
Petru