DMA problems with PCI 6115 Analog Input

[DennisWingo]

I am running a PCI 6115 through a PCI to PCI express bridge to a Mac Pro computer. At sample rates over about 50k samples per second the computer completely locks up. This is not an OS related issue as the Mac can recover from that, this freezes the entire machine, requiring a hard reset to recover.  This to me is a DMA arbitration issue.  Question is:  Who out there is using the PCI 6115 at high data rate acquisition and how is it doing and do you have any suggestions.

The 6115 has the 64 megasample memory onboard. 

I have already verified the following.

The bridge card is properly working as a PCI express x1 lane card and the OS says that it is running at Max throughput.

The system configuration that I have is the following 

Mac Pro quad core 2.66 GHz.

1 X16 Graphics card.

5 Gigabytes of RAM.

Thanks 

[G_Lo]

Hi Dennis,

What is the version of the OS and NI-DAQmxBase (or NI-DAQ) being used?  Which environment were you programming in?  If this testing was done with personalized code can you check if the same behavior is exhibited in an example program.  Testing this on a PCI slot (if available) may also tell us if the bridge is the issue.  Also, was this behavior apparent during acquisition or generation with the 6115?

[DennisWingo]

G

I am using DAQmxBase version 3.2, and the latest National Instruments Labview 8.6 that we just purchased.  I have done further debugging and what happens is what whenever I do a continuious acquisition using your own example from the DAQmxBase example )Cont Acq&Graph Voltage-Int Clk-Analog Start.vi) the acquistion will run fine until it is stopped once and then if you restart it, It hard locks up the MacIntosh computer, requring a power on reset.  If after I do the continious acquisition and stop it, I close the vi and then reopen it, it will work fine.  This tells me that there is a DMA setup problem with the PCI-6115 in the driver part of your vi or hardware physical layer initialization software/firmware.  Also, If I try and go above about 2 MHz acquisition rate, I get time outs with an undefined error.  I would really appreciate help as I am ready to toss your hardware and software and end an 18 year relationship with your company.

[DennisWingo]

Actually I would request that someone who is really knowledgeable about your software, data acq hardware, and the design of computers call me to work through this issue (as well as familarity with a Mac OS version of your software).  When I get to 2 Mhz acquistion rate I am getting DMA overflows. This is not something for a rookie to address.

[DennisWingo]

Update:

The DAQmxBase vi Cont Acq&Graph Voltage-Int Clk.vi does not lock up the computer.  However, I am getting a DMA overflow (Error 42) Possible reason(s):

The DMA buffer overflowed because data was not read from the buffer as fast as the DMA channel wrote to the buffer.  Which DMA channel is being talked about here?  I need some way to look at the register set of the PCI-6115 and I need a detailed engineering description of the board and its internals so that I can figure this problem out.

This is happening at 1.85 million samples/sec but at least it is not crashing the Mac CPU now.

[Tom_W_[DE]]

Hi Dennis-

I am looking through the DAQmx Base code for possible causes, but as yet I have not been able to reproduce the failure you're seeing.  I'd like to ask for more info about your setup to help narrow it down:

1.  What type of PCIe bridge card are you using?  What type of PCI-PCIe bridge does it actually contain (if you know)?

2.  When the machine was locked up previously, what effect (if any) did killing the LabVIEW process or aborting the LabVIEW VI (with the "stop sign" button) have?

3.  For both the Continuous and Finite examples, which values are you setting for the other parameters besides sample rate?  Please provide a list of all parameters or screenshots of your front panels while running.

    - one possible cause for a user interface lockup (though not an actual deadlock) would be that we spin on polling the DMA buffer for the requested number of samples in DAQmx Base with no delay in the polling loop.  The idea is that a properly-chosen sample rate / samples to read combination would make extended spinning of that loop unnecessary so the lack of loop delay should increase performance without negatively affecting the user experience in most cases.  But, if a large number of samples to read with a comparitively low sample rate is chosen, that loop could make the user interface appear non-responsive.  You could try adding a short delay to SSeries-- AI Scarab DMA Read Samples.vi in vi.lib\_DAQmxBase\Internals\SSeries\SSeries VIs\Ai.  This might increase the possibility of seeing overflows, but it would let us know for sure if the lockup is a user-interface lockup or a system deadlock.

    - the finite overflows may also be related to your samples per read size.  You can see in AI.ConfigureDMABuffer.vi from \vi.lib\_DAQmxBase\Internals\SSeries\Settings\Ai that the software DMA buffer is artificially limited to 2096896 Bytes due to the limitation that some Unix-based systems (read: Linux) could balk at allocating contiguous buffers larger than that.  You could try increasing that upper limit (or removing it) to see if it alleviates the buffer overflows you're seeing in the finite case.

Thanks-

[DennisWingo]

I am using the PLX Chipset, integrated into Magma's PCIe to PCI bridge.  I took a look at the CPU load on the Mac and it is only about 1.8 Megasamples/sec and about 29% CPU load with Safari and other software running at the same time just to push the system a bit.  I opened another version of the DAQmxBase examples and it is not crashing the CPU now. However, I am getting DMA overruns if i set the sample rate over 1.8 megasamples/sec.  There are buffer sizes that you can play with but so far I have not been able to get the VI to accept the buffer size changes.

What I think that I need is some detailed information on the various VI's in the DAQmx Base as I see some things in there that could be helpful.  I have ordered a plethora of books on Labview as it has been a few years since I used it extensively.  It might help if you have any VI's where I can stream the data to disk rather than to the display to see if we can offload the overhead from the display and run it out to my SATA drives.  I have a 7200 RPM SATA drive so that should be sufficient for this application.

When the machine locks up it is in hard lock, power on reset is the only thing that works.   

Here is the screen shot of the DMA overflow 

[Tom_W_[DE]]

Hi Dennis-

Your acquisition rate is about 1.9MS/s, but your 'Samples to Read' is only 1kS.  This presents a problem, as there is non-trivial overhead with each call to DAQmx Base Read and your current settings are really not making the most out of that fixed cost.  By increasing your samples to read, you still incur the fixed cost of the 'read' call in general, but you can grab much larger chunks of samples and save time on a per sample basis.

With your current settings, your read/process/graph loop has to run 1.9M / 1k = 1900 times per second.  If your loop rate falls behind, the software buffer holding the samples DMA'd from the board is likely to fall behind.  If you increase the samples to read you can dial down the number of required loop iterations without taxing your system too much.  I usually choose a samples to read value that corresponds to 10-20 loop iterations per second.  You may also increase performance by increasing the buffer size upper limit using the method I described in my previous post ( AI.ConfigureDMABuffer.vi ).

These changes should at least make it possible to increase your sample rate before you see overflows.  Please let me know how much of a difference they make.

Thanks-

Message Edited by Tom W [DE] on 08-26-2008 09:29 AM

[DennisWingo]

Tom

Thanks a bunch!  I realized I had not answered your question about the bridge.  The bridge is using the PLX chipset and a x1 lane PCIe card from Magma.

http://www.magma.com/products/pci/1PCI/

We are closing in on recovering some "lost" NASA Data and this will hit the news big time when it happens.

[DennisWingo]

Tom

That helped.  I am running at about 2.8 megasamples/sec now and the CPU usage is down to about 12% (safari is using almost that much).  I am getting DMA underruns now when I go over 3 megasamples/sec so I am seeing the balance between the parameters.  I read somewhere that NI had a Spectrum Analyzer VI.  Is that part of the full development package or do I have to buy that?

Will keep working and update.