With your code as it currently stands, there's no need for the DLL you wrote - all the functions you call can be called directly from LabVIEW. I would start there, see if removing one layer of DLL improves performance. Of course, since your callback does not currently do anything, there's no need for it nor for the calls to set it up. Without your LabVIEW code it is impossible to know if your LabVIEW code is not well-written, or if the problem is the library.
If you have not already done so, you should search this forum for OpenHaptics. You're not the first person to try to use this software with LabVIEW, and you may find that someone has already done the work and is willing to share.
A callback is a function that some other library or application calls on your behalf. You pass a pointer to the function, and the application or library calls your function at the right time, usually with the option to pass some data with it. In your case, I think the callback can be a simple function in C that calls PostLVUserEvent with the a pointer to the position array (you'll have to test that part). In LabVIEW, set up a user event with a data type of a cluster containing 3 integers, matching the position array (there are no fixed-size arrays in LabVIEW, so you'll need to use a cluster instead). You will also need to use the LabVIEW event structure, and this may not be intuitive to a new LabVIEW user; it also may not be supported in the evaluation version, I'm not sure.
Putting a DLL call in its own loop is not sufficient to make it run in a separate thread. You need to set the DLL call to run in any thread (reentrant). Make sure your DLL is thread safe; or, if it is not, that you only call it in one location to avoid accidentally calling it concurrently.
