Issue with GPU Analysis Toolkit and the Application Builder

[ColeVV]

Hello Everyone,

I think I have found an issue with the GPU Analysis toolkit and the Application Builder. Do demonstrate this, I have created a new project with the following .vi which initializes, loads the cuBLAS library and does a 2x2 matrix multiplication.

I have attached a .zip file with this project to this post. This vi works just fine in the development environment, however when I build the application and try to run it I get the following runtime error, which is generated right at the beginning after the "Initialize Device" vi.

---------

??? in lvcublas.lvlib:Initialize Library.vi:1-> Simple GPU Test.vi<APPEND>

<b>GPU Analysis Information:</b>  A GPU resource is invalid. 

<b>Cause:</b> Creating a GPU resource failed.

<b>Solution:</b> Make sure the resource request was for a valid device. If so, check to see if the device supports the type of resource being created or has the amount of physical resources available to honor the request. 

<b>Cause:</b> A GPU resource has been deleted explicitly.

<b>Solution:</b> Only delete the GPU resource after all processes using that resource are finished with it. 

<b>Cause:</b> A GPU resource has been deleted implicitly.

<b>Solution:</b> Avoid replacing an existing GPU resource with a newly created one in situations where the resource is being cached (e.g. passed to a shift register).  Once a GPU resource is no longer used, it may be scheduled for deletion internally to release system resources and avoid memory leaks.

---------

Does anyone have any thoughts as to why this might be?

Some additional info

----

- Windows 7 64-bit

- LabVIEW 2012

- Graphics card is an NVIDIA GeForce GTX 680

- NVIDIA Driver version 9.18.13.1106

- The output of "nvcc --version" to give the CUDA version is: 5.0, V0.2.1121

- Same behaviour regardless of whether or not the built executable and/or LabVIEW is run with administrator priviledges

- System information report from MAX:

Generated report

****************************

Operating System(OS)                    Windows 7 Professional

OS Version                      

       6.01.7601
OS Info                                 Service Pack 1
  Processor                               Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz / Intel64 Family 6 Model 45 Stepping 7 / GenuineIntel / 3599 MHz
Number of Processors                    8
Physical Memory                         31.9 GB of RAM
  Drive C:\                               329 GB of 698 GB free
Drive E:\                               1.49 TB of 1.81 TB free
Drive F:\                               192 GB of 195 GB free
                                       
  National Instruments Software:          Version:
                                       
CVI Run-Time                            10.0.1.434
NI-DAQmx Device Driver                  9.6.2f0
NI-DAQmx ADE Support                    9.6.2
  NI-DAQmx MAX Configuration              9.6.2
NI-488.2 Software                       3.1.1
NI-IMAQ Software                        4.7.0
NI I/O Trace                            3.0.2f0
IVI Compliance Package                  4.5
  LabVIEW 2012 SP1 f2 (64-bit)            12.0.1
     GPU Analysis Toolkit               12.0.0
LabVIEW Run-Time 2010 SP1               10.0.1
LabVIEW Run-Time 2011 SP1 f2 (64-bit)   11.0.1
Measurement & Automation Explorer       5.4.0f0
  Measurement Studio                      Visual Studio 2005 Support - See individual versions below.
     DotNET                             
          Common                        12.0.20.253
Measurement Studio                      Visual Studio 2008 Support - See individual versions below.
       DotNET                             
          Common                        12.0.35.253
          Common (64-bit)               12.0.35.253
Measurement Studio                      Visual Studio 2010 Support - See individual versions below.
       DotNET                             
          Common                        13.0.40.188
          Common (64-bit)               13.0.40.188
NI-USI                                  2.0.0.4901
NI-IMAQdx                               4.0.0
  NI PXI Platform Services                3.2.1
NI-PAL Software                         2.9.1
NI-Serial Software                      3.9.1
LabVIEW SignalExpress                   6.0
NI System Configuration                 5.4.0f0
  NI-VISA                                 5.3
     NiVisaServer.exe                   5.3.0.49152
     NIvisaic.exe                       5.1.2.49152
NI-VISA Runtime                         5.3
LabVIEW Run-Time 2011 SP1 f2            11.0.1
  LabVIEW Run-Time 2012 SP1               12.0.1
LabVIEW Run-Time 8.2.1                  8.2.1
LabVIEW Run-Time 8.6.1                  8.6.1
LabVIEW Run-Time 2009 SP1 f6            9.0.1
LabVIEW Run-Time 2012 SP1 (64-bit)      12.0.0
  ****************************

[MathGuy]

One reason for this error is that the CUDA runtime library was not found during initialization. This will result in an invalid resource (downstream).

Are you running the app from the same system you developed it on?

[ColeVV]

Hi MathGuy,

Yes, I am running the app on the same system that it was developed on. Were you able to open the project I attached? I would be interested if you see the same error.

Cole

[MathGuy]

Which version of LV did you use to dev & build the app?

[ColeVV]

I had printed out the details of all of the versions of LV in the original post. But here are what you asked for again,

  LabVIEW 2012 SP1 f2 (64-bit)            12.0.1

     GPU Analysis Toolkit               12.0.0

LabVIEW Run-Time 2012 SP1 (64-bit)      12.0.0

[MathGuy]

I've checked against both 32- and 64-bit LV builds on Windows 7 x64. The issue is tied to a support dll (lvcuda32.dll or lvcuda64.dll) used as part of Initialize Device VI. The error you receive is due to a device initialization failure and not a CUBLAS library failure. I'll try to summarize why the error happens and give at least one way to fix it.

Why the error?

Building a stand-alone application with LabVIEW that includes GPU computing uses external libraries but it only knows about those libraries called from CLNs (Call Library Function Nodes) where the library name (and possibly path) are hard-coded in the node. For the GPU toolkit  to support multiple CUDA versions and different Windows platforms (i.e. 32- and 64-bit versions) from a single API, certain libraries are identified at run-time and passed to the CLN for execution. This is (a) how an lvcuda*.dll library is used in the LVCUDA API and (b) why they do not show up as a dependency like other external libraries (e.g. lvgpu32.dll or lvgpu64.dll).

The Fix

The GPU toolkit installs lvcuda*.dll in a lvgpu folder under LabVIEW's resource directory. This is part of the default path LabVIEW uses (in the development environment) to locate external binaries. No such default directory exists once you build a stand-alone application. However, there is still a search path used by the executable to find binaries it needs.

The details regarding this can be found here. Sometimes it's valuable to find out what dependencies these binaries have (e.g. a library which is itself linked to another library). How to track such dependencies are discussed here.

While these resources describe a few ways to make external libraries work from a stand-alone application, copying the external libraries to the system directory appears to be the most straightforward and yields success 100% of the time as long as you put the DLLs in the proper place.

For a 32-bit app, copy lvcuda32.dll to <windows_root>\system\lvgpu. For a 64-bit app, copy lvcuda64.dll to <windows_root>\System32\lvgpu. The 'lvgpu' subdirectory is required as this is the folder structure used inside LabVIEW's resource directory. This may be why your prior attempts to copy these DLLs to the system folder failed.

Note that you do not have to do anything special with lvgpu32.dll or lvgpu64.dll. These libraries are recognized as dependencies and the VIs that call them will automatically look for them in their new location based on the build specification.

Alternatives

If copying files to the system directory is not an option, the executable can look for libraries using the PATH environment variable. Appending the library location to the PATH can have unintended consequences or encounter other problems (e.g. too many characters on 32-bit OSes) so it's not always a fail-safe approach.

I've also been told that copying the libraries into the run-time engines folder may also work but this is modifying a Program Files folder. Whether these changes persist during some installation modifications is undefined. In addition, you'd have to copy the libraries to each version of the run-time engine your application is built against.

I hope this gets you to a solution that fits your needs. I have noted the issue and will be posting a KB on the topic once we've arrived at a suitable solution.

Darren

[ColeVV]

Darren,

Thank you for taking the time to put down such a thorough analysis. I will give your solution a try and look into the contents of the KB articles you linked to. I'll post here how it went.

Thanks again!

[MathGuy]

Here's an additional resource that goes into detail on the search path used to find shared libraries:

http://digital.ni.com/public.nsf/websearch/EBAE870D564CBBE78625788B004E76D3

Since your instance is not tied to the development directory, you'll want to follow the attached linked for MSDN to get the details for the search path on Windows used by your stand-alone application.

[ColeVV]

Your original fix worked great for my test program! Now I will test on my real application and if there are any issues then I'll follow up after that.

Great work! Thanks again!