Error Status Code -88705: PXIe-6341 with PXIe-8370/PCIe-8371 and PXIe-1071

[uranc]

We're using NI PXIe-1071 based system for Neuropixels control system with custom IMEC acquisition Module, PXIe-6341 I/O Module. Communication is via PXIe-8370/PCIe-8371 Remote Control Module and it is a HP Z4 workstation which offers PCIe x4 only via PHC HP Z4 G4 White Paper .

Expansion Slot 1, PCIe3 x16: GPU nvidia Quadro P2000
Expansion Slot 2, PCH x4 : empty
Expansion Slot 3, PCIe3 x16: Infiniband adapter
Expansion Slot 4, PCH x4 : empty
Expansion Slot 5, PCIe3 x8 : NI 8371 (PCIe x4)

We have NI PXI platform services version 18.5 and NI-DAQmx 18.5 ( https://billkarsh.github.io/SpikeGLX/help/NIDriverInstall/NI_driver_installation/ ).

Even though IMEC acquisition Module works perfectly fine, PXIe-6341 I/O Module is often not detected in device manager, with error status code -88705 described here: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000000P6iKSAS&l=en-US

 

After restarting the computer PXIe-6341 I/O Module eventually works, but sometimes the error continues after 5 - 10 restarts, and is not reliable. When it doesn't work, systems gets stuck with interrupts either at startup or anytime the MXI cable is plugged in, system recovers after unplugging the MXI cable.

We've followed several NI/PXI guidelines but no luck so far:

• Mitigating MXI-Express PC Incompatibility 
• MXI-Express Compatibility and Connectivity Troubleshooting Guide 
  • Symptom 1: Computer does not boot or hangs when PXI(e) chassis is connected to PCI(e) MXI card
    Computer gets consistent Blue Screens of Death (BSoD)

What works:

• Replace PXIe-8370/PCIe-8371 with PXIe-8380/PCIe-8381, PXIe-6341 is detected in device manager, works reliably.

What doesn't work:

• Refresh NI-MAX by going to View >> Refresh
• Power-cycle the system. This error is sometimes reported if the computer has not been restarted after the installation of an NI driver. Please try restarting your computer.
• The NI Device Loader Windows Service (nidevldu) and/or NI Configuration Manager Service is not running, please start these services as described here. If the NI Device Loader still does not start automatically after restarting computer, you can try to uninstall all antivirus software in your computer and restart again. 
• Occasionally, the NI-MAX database can become corrupted due to a software install/update or other events impacting the registry files owned by MAX. Reset the database, restart your computer and devices, and try again.
• Make sure you are not exceeding the maximum working voltage for your device.
• Determine the Number and Range of PCI/PCIe Root Bus Devices 
• Ensure your computer is running the latest BIOS version.
• Disable Active State Power Management (ASPM) in your BIOS Settings by consulting your motherboard's manual. If your BIOS does not allow you to disable ASPM, configure it through your Windows Settings.
• NI MXI-Express BIOS Compatibility Software 18.0 Readme does not find the device when it's in BIOS debug mode.

The problem seems to be the hardware PXIe-8370/PCIe-8371 or the MXI cable - as the PXIe-8380/PCIe-8381 works perfectly fine. 

Any clues? Thanks in advance.

Cem

[Deus_ex_Machina]

Dear Uranc,

I was wondering if you have seen this NI article regarding the -88705 error:

Error -88705 or 80040383 with DAQmx or GPIB Device in NI-MAX

[uranc]

Hey Harvey,

indeed, that was the first thing we tried, services are running all the time. In fact by plugging out the PC/PXI cable and plugging it in again we can see that the NI services are active.

• Once the PC starts and "hangs" it looks like windows is trying to work with the NI Card and it eventually shows up in the device manager. CPU load goes up to and beyond 50%, Windows becomes unresponsive. We have seen "Interrupts" to eat up 40% of the CPU load!
• On restart of the PXI chassis or unplugging the cable, the load immediately drops to zero. In one occasion a pop-up appeared for the audio device. This could be in indicator, that the Interrupt lines are not properly separated.

[Deus_ex_Machina]

Hi dear,

Have you checked this article?

High CPU Usage When Using MXI-Express with NI-DAQmx and NI MAX Test Panels

If you have already solved the issue with PXIe-8370/PCIe-8371, can you share us the solution?

Best,

Harvey

[onecore]

Did you ever find a solution? I too would like to use the PXIe-8370 with the PCIe-8371 for neuropixels.

[uranc]

Hi,

short answer is you can use it, it works if it works. Long answer below:

We were using NI PXIe-1071 based system for Neuropixels control system with custom IMEC acquisition Module, PXIe-6341 I/O Module. Communication is via PXIe-8370/PCIe-8371 Remote Control Module.

It worked fine for some time but the PXIe-6341 card would not be detected on NI MAX or device manager sometimes, and after restarting 3-5 times it would work until the next time the computer will be restarted.

In short, Windows would not boot properly if the NI PXI-6341 I/O Module is plugged-in or it is not detected in NI MAX. IMEC Card works fine otherwise.  It causes the system to freeze on startup (or restart) and the system recovers as soon as we turn off the NI PXIe-1071 Chassis, or unplug the interfacing cable between NI-PCIe8371/PXIe-8370. It was a HP Z4 Workstation which is known to have PCIe lane issues (they had 7 BIOS upgrades in the past 7 months), but we have not found a stable BIOS configuration. The chassis freezes the CPU with PCIe interrupts (we had the audio driver pop up a few times during debugging) so it fits the NI BIOS Compatibility issue.

We have another setup with the next-gen NI PCIe-8381/PXI-8381 with PXIe-1071 Chassis and the PXI-6341 I/O module where we never had any issues. To test, we plugged in the PCIe-8381/PXI-8381 from the working setup instead of NI-PCIe8371/PXIe-8370 and it seemed to solve the problem - we had 1 week of smooth recordings with no boot up or detection problems . Hence we ordered the next gen 8381 and has endured a week of test runs. Assuming things are back to normal, we enabled the audio drivers in BIOS (which we have disabled before during debugging) and since then we had no luck fixing the issue.

Next, since the working setup had an older HP WS Z640 we setup a fresh system with HP WS620, which worked perfectly for a few days. Thinking that the problem is fixed with the new WS we ran some tests and plugged in a SSD hard disk to start with recordings. And since then we have the freezing issue again.

At some point it stopped working again and after checking various NI threads and other labs with similar issues we tried:

• 5 different computers (Dell, HP, Lenovo – old/new) since some NI devices have BIOS compatibility issues (some due to new Intel chipsets, some due to PCIe lanes)
• Turn off Virtualization / VT for Direct I/O - Kernel DMA Protection: https://forums.ni.com/t5/PXI/PXIe-8031-in-PXIe-1082-causing-Driver-Verifier-DMA-Violation/td-p/40061...
• Try a PCIe x4 slot instead of PCIe x8 slot for the PCIe-8381/PXI-8381 seems to solve exact same problem for 2 other labs

The issue still might be pointing to BIOS/motherboard/chipset issues with the chassis PXIe-1071

• https://forums.ni.com/t5/PXI/pxie-8381-pcie-8381-PXIE-1071-chassis/td-p/4242358
• https://www.ni.com/de/support/documentation/supplemental/14/understanding-mxi-express-enumeration-an...

or PXIe-1071 chassis might be damaged:

• https://forums.ni.com/t5/PXI/PXIE-8381-not-working/td-p/4241695

Currently, the system never works with the 6341 is unplugged in and it begins as soon as the NI drivers are installed. If you flip the dip-switch on the PCIe card the card is visible on NI-MAX but that's about it.

Overall, all such problems end up in computer freezing due to actual physical or compatibility problems and the problem is it is extremely difficult without an extreme amount of time and effort and NI support is insufficient. There are known cases of both Windows-NI hardware compatibility/software issues but also in our case since the rest of the devices works fine (when the 6341 is unplugged) and in the windows logs the system seems to freeze after bridging with the NI -6341 I/O device our IT thinks that the card is defect. 

However, the problem is NI delivery times are extremely slow in the last years and there's not much support when you try to fix things. One advice would be to use PXI Thunderbolt to overcome possible compatibility issues and is physically safer without the direct access to PCIe lanes (even though the bandwidth is lower). In case it is a problem with the PXIe-1071 a better option would be NI-PXIe 1082. There are also ADLink chassis options but I do not know much about that. 

Most of this information and advice is based on IMEC support and the Neuropixels Slack channel and I suggest you look for advise there instead.

Good luck!

Cem