serial port hang/freeze

[mattkav]

On rare occasions my application freezes.  What I believe is happening is that the VISA Read or VISA Write vi never returns.  (The program is an executable, so I can’t go to the diagram to be sure.  I just released a new version that will help me isolate it further, but it takes time to propagate new software to users.)  I do have the timeout set to 500ms, and I know this works because it timeouts when I have no hardware attached.

When the freeze happens, I have to go to the Task Manager to kill my application; however, it never actually quits.  It goes away on the Applications tab, but won’t leave the Processes tab.  Selecting End Process on the Processes tab won't make it go away either.  I can get it to go away, however, by unplugging the USB cable.  I then plug the cable back in, relaunch my application, and all is OK.  (A restart works, too.)  I am using the FTDI 2232 USB to dual serial port chip (set up for RS485) in my system.

My first question is whether anyone has seen this before or has a suggestion to get this work.

My second question is how do I figure out what is actually hanging?  My LabVIEW 8.5.1 executable calls a VISA 4.41 driver.  The VISA driver calls a Windows COM port driver.  The Windows COM port driver calls the FTDI driver (Version 2.4.6).  The FTDI driver calls the USB driver.  We are running Windows XP with all updates installed.

I have about 50 of these systems running continuously.  I see about one freeze like this once every few days on a random system.  I am not able to reproduce it on demand.  Any debug solution will have to be deployable to all 50 systems; otherwise, it is possible that I won't see the bug for weeks, if ever.

Thanks in advance for the help.
Matt K.

[Wiebe@CARYA]

There are some advanced debugging tricks.

First, you *can* make an exe with diagrams! If you enable debugging, and
don't remove the diagrams, you still get your exe, but with a LabVIEW
development system you can debug the application. You should even be able to
connect to a remote PC, but I have no experiance with that. Debugging
probably won't work when the hangs in a VISA node though, but if it's a
while loop that hangs, you might find it. The exe will become a lot larger,
and the IP isn't safe either. Perhaps you can be selective about what you
remove and what not...

Another neet trick is to use the command line. You can make external VI's,
and run them in the exe. When the exe runs, go the it's path on a command
line and type the exe's name followed by the path to the external VI. The vi
will be opened in the executable (if all dependancies are found). So why is
this usefull? Well, the VI could contain some nifty VI server techniques,
that might be just what you need to get some information. Do note that in an
exe, even the front panels are removed from most of the VI's. The external
VI can also read buffers used in the program.

So, my last idea is to write everything that happens to some place. A file,
event log, database or an internal buffer (read as explained above). You do
need to prevent the log from growing too big.

I'm sure there are more ways...

Regards,

Wiebe.

[mattkav]

Thanks for the suggestions Wiebe.

Your first suggestion led me to the help page titled "Debugging Applications and Shared Libraries".  I will look into that.  However, if I do isolate the problem to VISA Read or VISA Write, how do I take it to the next step to determine what driver is the problem?

Matt K.

[Wiebe@CARYA]

"mattkav" <x@no.email> wrote in message
news:1234386609046-851400@exchange.ni.com...
> Thanks for the suggestions Wiebe.Your first suggestion led me
> to the help page titled &quot;Debugging Applications and Shared
Libraries&quot;.&nbsp;
> I will look into that.&nbsp; However, if I do isolate the problem to VISA
> Read or VISA Write, how do I take it to the next step to determine what
> driver is the problem?Matt K.

Well, I think it's unlikely that the VISA nodes themselves are hanging. They
have been used a lot, and there aren't reports that about any problems like
yours.

The only advise seems a bit stupid, I admit. If it hangs when you do
something, don't do that something!

My guess is that the driver chokes in some situations. Drivers are very
divers in how they are build, ranging from absolutely crap to excellent.
They also have to be suited for the application that uses them, and the same
range applies to the applications. It's not as simple as "well written
drivers never choke". They have to be tested, and errors like yours (hanging
once every month or whatever) are very hard to find. It's hard to tell
without seeing any code.

You might try a different approach. If you could detect the hanging in the
program itself, you should be able to reset the hanging code. Depending on
your structure, this is very easy or a lot of work... If you started your
acquisition code dynamically, and it is frosen (detect this with a watchdog
construction), simply stop the code and restart it! Just an idea. Not as
nice as actually fixing the problem.

Regards,

Wiebe.

[BlueTwo]

Hi Matt,

We are developing some DAQ/IO for a custom application using the FTDI 232 (not the dual). I have experienced the same behaviour when running more than one device at once. I have not managed to detect it programatically, let me know if you manage to. I have spent a long time working through all the various time-out settings and port cycling options etc. without success.

It does not cause the VI to crash, but, as you describe, hang, maybe once every few days. As soon as the USB is unplugged the VI will continue. I had a dialogue with FTDI support who high-lighted a few possible causes. The most likely seemed to be that when waiting for a response from the FTDI, the handle for a particular device  became invalid, leading to the VI waiting. Not exactly convinced by this, as setting timeouts did not solve it. Maybe the power was spiking enough to momentarily brown out the ftdi without making it appear disconnected?

The only cause that we have been able to tie it to is either poor power supply to the DAQ, or noise. We have made the DAQ self powered, isolated the usb supply from the DAQ supply, and optically isolated the switched devices (motors etc) on the outputs. we have also implemented a routine in the DAQ which will reboot the FTDI if it seems to loose contact. This seems to have  resolved the issue, though it as not explained why it was happening.

We are looking at using the 2232, is it as straight forward as the 232? 

Blue 

[Wiebe@CARYA]

You used to get problems if you called Serial Init while another serial port
was in use. I'm not sure if this is still a problem, since the serial VI's
are all rewritten to use VISA, but it just might...

Regards,

Wiebe.

[BlueTwo]

Just dug out the email from FTDI support....

Blue 

Hello,

If it is an application problem then what could happen is you ask to read 10
bytes but only receive 5. The application would appear to hang waiting for
the remaining 5 bytes unless you have set timeouts to allow the read to
abort.


The fact you need to unplug and re-insert suggests the driver needs to
reload.

This could in theory be a result of a surprise disconnect.
What this means is the device handle is open and the device disappears from
the PC (like unplugging - but could be an EMC/noise event).
When the device re-appears to the PC the handle and the hardware no longer
match as the stack is not maintained during a disconnect.
This would result in you not talking to the actual device but some "rogue"
device handle which could theoretically hang the application.

Some ideas for debug:

1.
Verify the driver version installed is 2.04.06
http://www.ftdichip.com/Drivers/CDM/CDM%202.04.06%20WHQL%20Certified.zip

2.
Ensure you have set read and write timeouts in your application

3.
Use WM_DeviceChange to determine if you are disconnecting.
http://msdn.microsoft.com/en-us/library/aa363480.aspx

4.
Try increasing the resetpiperetrycount to make the device more immune to
noise events.
http://www.ftdichip.com/Documents/AppNotes/AN_AdvancedDriverOptions_(FT_0000
73).pdf

[LV_Pro]

In a probably unrelated instance, I had a program that would hang on random ports of a multiport card (regular PCI plugin card) at a VISA write if my program had launched an instance of Windows Media Player previously. I tracked the problem down to the version of the AVI codec I was using, the system that didn't exhibit the problem having an older version. Didn't go much further than that, figure that some part of the serial card's memory space was getting stepped on. In my case, as we were still in the development mode I was able to run more and more of the program until the problem occured.  I think, if possible, that if you can create a vi that writes the current step to a file, and then salt these all over your program, before and after writes, you might get some useful info. In my case when the program hung only the specific vi was hung, and when probbed it showed that it was stuck in the VISA write. The only way to get it out of there was to use the Windows Task Manager to kill the LabVIEW instance.

Good luck, these intermittents require creative troubleshooting.

Please keep us informed of your progress!

[mattkav]

Blue

I don't recall what happens if I unplug the USB before I quit the software, so I will try it next time I see the problem.

The 2232 works for the most part other than the random disconnects that seem to appear.

Thanks for the forward from FTDI.  I already addressed 1 and 2 in an earlier posting, but I am curious about 3.  If the USB device is disconnected, shouldn't I see that in the Device Manager?  I don't hear the sound Windows makes when the USB disconnects until I actually unplug it.  I have looked in the Device Manager and the ports are still listed.  However, I haven't looked in the USB section, so I will do that next time.
I also wonder about number 4.  Do you know if there is a way to check if the retries are exceeded?  50 seems like a lot, so I would be worried about blindly changing that number and hoping it fixes it.  Even if it does, it seems like the proper solution is to reduce the noise.  I supposed I will ask FTDI if they know a way to check.  (We have already had some discussion about this issue.

Matt K.

[BlueTwo]

Hello Matt,

I am no expert in this, but apparently it need not show up on the desktop if it does not properly disconnect. I did not get as far as trying to implement what was on the MSDN page as it is beyond my current understanding...

I don't know how you check how many pipe retries were required, though i tried large and small numbers with no observable difference, so i assumed it was not related to the issue i had. One of the status flags maybe?

Sorry, a bit of a case of groping around in the dark. 

Do post if you get any new info from FTDI 

Cheers,

Blue