Dual Core DAQmx Acquisition Glitch Problems on Athlon64

[billings11]

I have a problem with dual core Athlon 64 4200 machines on ASUS AV8 mainboards.  I have PCI-6251's to acquire data.  There is definitely a timing problem in DAQmx when used with fast dual core mahcines.  This is simply continuous simultaneous acquisition and generation on ao0 and ai0. 

The 6251 is plugged into a CB-68 with nothing connected.  Normally I should just be picking up white noise and some coupling from the ao, but every so often I get those huge noise spikes.  They are not periodic.  The spikes are only there if I am generating and acquiring at the same time.  If I just acquire they go away.  They aren't there if I run the same code on a single core machine.  Also if I go in the BIOS and set the CPU speed to 1500MHz instead of 2100MHz the problem goes away.  The spikes are smaller if I connect the acquisition channel to a lower source impedance, but they are still there on the order of several millivolts even if I short the ai to ground.  That is certainly enough to make them useless to me for acquiring wavforms.  I have tried the same 6251 cards in single core machines and they do not have the problem.  It only happens when I run them on my new dual core machines.

I am normally using DAQmx 8.3, with labview 8.2, but I have downloaded and tried this with DAQmx 8.6 and it didn't make a difference.  This problem exists not just with my acquisition code, but with any express vi acquisition or example vi for simultaneous generation and acquisition.

[billings11]

I also wanted to add that sometimes I have seen these spikes be several volts in amplitude and not just a few mV.  This is also happening in development and executable environments.  For these plots I was acquiring at 500kHz.

Message Edited by billings11 on 09-07-2007 08:37 AM

[billings11]

On the newer computers with the Asus M2V mother board and the Dual Core AMD Athlon 4000+ (2.1 GHz internal clock) I did the following:

1. Placed the 6251 card in each available slot – Still failed
2. With Task Manager I set the software to use only Core 0 – Still failed, but somewhat less noise.
3. Checked for a grounding issue with the PC by:
1. Moving the computer to the same location as a slower one without the problem – Still failed.
2. Remove the ground on the computer power cable – Still failed.
4. Used NIDaq 7.5, 8.3, 8.5, and 8.6 – Still failed.
5. Used only a single memory chip (Disables dual bus feature for memory) – Still failed.
6. Disabled all possible components and drivers not absolutely nessesary – Still failed.
7. Changed the BIOS of the computer from Auto Detecting the processor type to manually setting – Still failed at recommended setting.

It is only when I manually configured the processor to it lowest setting and incrementally increased the speed that the system worked reliably.   At 2 Ghz internal the system started showing a small amount of noise while at the max speed of 2.1 GHz the system showed more.  I would assume that a faster processor will only compound the problem in the future.

Here are the two boards and processors we are using that are having the issue:

Asus A8V with Dual Core AMD Athlon 3800+ (2 GHz internal clock)

Asus M2V with Dual Core AMD Athlon 4000+ (2.1 GHz internal clock)

[stilly32]

Hi,

I appreciate the screenshots and details on your post, it really helped me to see what you are experiencing. You may have already found this, but we did come across a known issue with our drivers and AMD dual core machines:

AMD Multi-Core/CPU Systems Can Cause Unexpected Behavior in NI Driver Software

If the applying the patch and using DAQmx 8.6 does not work for you, please post back and let us know. I would also appreciate it if you post back if it does work, since I have not run across these specific symptoms.

Thanks,

Andrew S

National Instruments 

Message Edited by stilly32 on 09-10-2007 02:16 PM

[billings11]

We Integrated the AMD drivers right into XP Setup so no previous AMD drivers were loaded (prevents possible upgrade issues)

• Only installed NI Daq 8.6 (again to prevent any possible upgrade issues)

I still got the same results.  There were a lot of spikes.

After that I tried one more recommendation which was to use the "/onecpu" flag in the Boot.ini file so Windows only used one CPU for everything from the start.  This did reduce the spikes, but they were still present just like setting the affinity to one core reduced the spiking, but still had the problem.

According to the NI site either of these adjustments is supposed to fix the problem if it is a RTS timing issue. 

We now have also seen the same problem on a single core AMD Athlon 4.0GHz machine.  And on the dual core machine, even when we disable a core entirely we still see glitches until we slow the remaining core down.  On my 2.1Ghz single core Athlon there is no problem, but on faster Athlon machines we are sometimes seeing this glitch effect.  So at this point we are starting to suspect that DAQmx has timing problems with any fast Athlon machines, not just dual core.  Maybe its because of the AMD integrated memor controller, but at this point it looks like developers need to use AMD or NI, but not both.

Message Edited by billings11 on 09-12-2007 09:33 AM

[Matt_W1]

Have you considered the motherboard being the problem, both motherboards listed use the same chipset and are from the same manufacturer (although asus typically does a good job). So it may be a quirk with that chipset. Or maybe the power supply and/or motherboard isn't filtering the power well enough (you could try running them on a different circuit or replacing the power supplies, but I'd doubt that would fix it). You could try updating the motherboard drivers and bios, to see if that helps. Only way to be sure is run a motherboard with a different chipset, preferably not a VIA, since the quirk (if there is one) might be in more of their chipsets.

How did you lower the frequency of the CPU via the multiplier or bus speed? Bus speed can have some subtle affects on the chipset (certain hidden timings vary with the bus speed, at least on the Intel side of things).

Have you tried a different PCI-6251 (I would assume so).

Matt W

[Jordan N.]

It very well could be the motherboard is the problem so that is why we tried another, but got the same results.  Still could very well be the MFG (Asus), but we do not have a new faster board readily available to test.

But the boards do not have the same chipset or design.  The Asus A8V is based on the VIA K8T800 Pro chipset while the M2V is using the VIA K8T890 chipset.  A8V is using DDR memory controller while M2V is using DDR2.  They are only both from VIA Technologies, but have very different design like one supporting 939 pin processors while the other is AM2.

We have also considered and tried both power supplies and the wall socket they are plugged into.  We moved computers that were having the problem to the physical location where there was a computer not having the problem.  We only replaced the box and left everything else in place (monitor, keyboard, cables, etc.) to see if it was the physical location.  That did not help.

When we replaced the power supply it was with a model that was completely different and was a lot more powerful than needed for these workstations.  We used a 450W unit that is powerful enough for a server.

We were able to lower the CPU freq by going into the bios and reducing the multiplier on the CPU.  On the AMD 4000+ we reduced the multiplier from 10.5x (2100Mhz) to 9x (1800Mhz).  This enabled us to make use of the card.  We also tried to reducing the memory speed from 677MHz to 400MHz on the Asus M2V but that did not help on that test.  Only reducing the CPU multiplier is all that has worked so far.

Seeing how reducing the memory speed did not fix the problem and that it happens on the two chipsets it does not appear to be a memory controller or memory timing issue.  Also using the /onecpu flag in the boot.ini file along with all the other recommendations in the link below did not help so we don't appear to be dealing with that issue either. 

AMD Multi-Core/CPU Systems Can Cause Unexpected Behavior in NI Driver Software

Jordan Nolan

[billings11]

Jordan and I are colleagues working on the problem together..

We have also switched around many 6251 cards and the problem stays with the computer regardless of the card.

Message Edited by billings11 on 09-13-2007 09:21 AM

[Matt_W1]

The current for sale models of those motherboards use the same chipsets (K8M890 and K8T890).

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Description=A8V+&x=0&y=0
http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Description=m2v&x=0&y=0

And the main difference between the 939 and AM2 is for support DDR2 (since the memory controller is on the CPU with AMD), so the only big differences between the two is the memory and that one uses an updated chipset (I assuming since the numbers aren't all that different). So I would try another motherboard with a different chipset manufacturer (whatever the current nVidia or AMD chipset is), if at all possible (A new suitable motherboard should be ~$50, of course most of the cost will be paying someone to put it together).

The old way to fix the TSC problem (when I last had the problem) was to use this hotfix and add /usepmtimer to the boot.ini. But that was years ago, so I don't know if that'll work with the current AMD drivers (since they likely contain that hotfix), if it'll create more problems.

Matt W.

[Jordan N.]

Thank you for your suggestion but I don't think that we can play musical motherboards or take wild guesses at what motherboard we can use next because NI hasn't fixed a problem.  Especially a problem that you indicate they have known about for years.  In all of the posts I have read about this issue I have not seen any reference so far to a chipset or brand and we have already tried this on two chipsets. 

It seems like a small thing to get another mobo for $50, but that very well might not be the quality board we are use to or from a MFG that we have had great luck with for about 7 years running.  Despite the good relationship I would definetly through them under a bus in a heart beat if we learned that they had an inherent problem in their product, but that not being the case we need an NI solution for what appears to be an NI problem.  I never want to get into a situation where NI starts telling us that we can only use certain computer components if we want to use their product. 

I recall that at one point I also tried the PM switch you mentioned but that did not help either.  Thanks for that tip as well.