Using labview to simulate a touch monitor

[Nightwork]

Hello all,

I was hoping someone could point me in the right direction. I just got started using labview. My company designs a product which uses a touch monitor for control. I would like to automate several of our tests using labview. Currently an operator has to go through our software and press the touchscreen. I would like to use labview to send the touch screen input to the system. Currently the touch monitor connects with a video cable and a USB. My guess is the USB is sending the touch input data. I was hoping to unplug the usb from our system and plug a laptop in running a labview VI. Labview would simulate the touch monitor and we could automate a lot of our tests.

Is this a reasonable way to solve this problem? What should I start researching to do this? I have tried using google and everything I find is about how to use a touch monitor to control labview. Which isn't really what I am trying to acomplish.

Thank you in advance,

Josh

[Bob_Schor]

About 7-8 years ago, I actually wrote a LabVIEW program that utilized a TouchScreen.  I can't quite recall what the problem was, and it may be that touch screens have improved a lot since then, but there were things that were not intuitive, like "where is the cursor when you aren't "clicking" somewhere".

If what you want to do is simulate "Touching the Button on the Touch Screen", you can simply "simulate" the "finger on the screen" with the "mouse on the desktop" (or, if you have a PC with a TouchScreen, the "finger on the TouchScreen".  You could put a Boolean and say "Press the "Go" button", but you could also put a static Picture on the screen and detect finger position and press location.  

But I'm not sure I understand what it is you want to do.  Nevertheless, I think LabVIEW could probably do it.

Bob Schor

[tst]

Let's start with a couple of assumption:

1. You're talking about software running on a PC with Windows.
2. The touch screen is connected to that PC and acts as a monitor and as a mouse. You should be able to check if it appears as a mouse in the Windows Device Manager. You can plug and unplug it to see it appear in the list of devices.

Assuming this is true, then you have a couple of options which should probably work:

1. Call an OS function to simulate the click. In Windows, this can be SendInput, and there should be examples of how to call it. The main issue with this would be that you need to specify where to click and that might be something that's not fixed.
2. Use a program like AutoIt or AutoHotKey to do this, with or without LV. This might work better.

If these assumptions are not correct, then I would start by talking to the team which created the device to understand how the touch screen works. Sending custom data over USB is possible in principle, but is more complicated. I personally never did it. It's also possible they might have or can build another mechanism for you to send the same data (e.g. using a web service or custom commands over a TCP or serial connection).

[Nightwork]

Thank you tst for your reply. Your assumptions are mostly correct. The monitor acts like a monitor and a mouse. I spoke to someone in our software development team and they said that the touch monitor is basically off the shelf and they dont do anything special to use it. Just installed the necessary driver and it works. The system I am trying to control is an ubuntu system. Sorry for not mentioning that at the beginning. 

After your replies and doing some research I think was I want to be able to do is send mouse click data over usb. I would like to use my windows laptop to run Labview and connect this laptop to an ubuntu computer with a usb cable. Then  I would like the labview program to simulate a mouse click on the ubuntu computer. Can someone point me in the right direction for how to do that?

I am lucky that the I am trying to automate runs full screen and the buttons never move location. So once i figure out the locations I want to sent it should be fairly easy to script after that.

Nightwork

[LVNinja]

You may be better off using a uC like Teensy to generate the mouse USB messages 

https://www.pjrc.com/teensy/td_mouse.html

This uC has already implemented the libraries that generate the USB mouse data. In case you raise the question, I don't think you will  be able to program TEENsy with Labivew for this application. 

[LLindenbauer]

---

@Nightwork wrote:

... connect this laptop to an ubuntu computer with a usb cable. Then  I would like the labview program to simulate a mouse click on the ubuntu computer. Can someone point me in the right direction for how to do that?

---

This solution in particular is most certainly not possible due to how USB works. On a USB tree, there can only be one root host. Some chipsets allow for switching between host and device, this is called USB on-the-go https://en.wikipedia.org/wiki/USB_On-The-Go - The Teensy linked above probably does this. Some SBCs in the Raspberry line can do this as well. A lot of smartphones can do it. I don't know of a single PC board that has USB OTG.

With that out of the way: Linux has lots of options to inject input data. One popular tool would be xdotool: https://manpages.ubuntu.com/manpages/trusty/man1/xdotool.1.html

 You can dial in to your ubuntu box using any old method and let xdotool do the clicks.

This leads to a more fundamental question: What do you mean when you say

"trying to automate runs full screen"? What problem are you trying to solve?

There surely should be more suitable options to do what you want. Does your target program not have a remote interface? Can you not connect to the remote desktop to the Ubuntu machine? It will most certainly be simpler then involving all kinds of additional technology on top.

[Nightwork]

Thank you both for your replies. This has been very helpful for me to understand why I was not having an easy time using google to find what I was trying to do. This was one of those cases where I was thinking something would be simple because I didnt know enough to realize the difficulties.

Not that I understand that my initial idea is not possible due to how USB works. I wonder if another approach is better. Perhaps I can use some NI hardware to simulate the mouse data? 

I will have to look into the remote desktop option as well. If I was able to do the remote desktop option that could be a very nice solution if it would work the way I am invisioning. I could have a laptop running Labview, and use the same laptop to connect via a remote desktop. Then have labview click on the remote desktop to simulate the clicks on the device under test. 

I am not sure if this will be practical or not. I know that we have many things like this locked down for security reasons.But I will investige this further.

thank you for you help so far.

Josh

[LVNinja]

Check this link 
Simulate Mouse Click on a LabVIEW control - NI Community

You may need to have it running on the computer that runs your application. It's worth trying if would work with your remote desktop idea.