Methods for acquiring live images from a video game for real-time proceesing

[Wez]

I am trying to grab a continuous sequence of images from a game that is running on the same machine as Labview and to have the VI perform edge and object tracking on the live images.

I have tried using third party screen capture programs but couldn't read the created AVI file whilst recording was in progress. I then tried accessing the game through an ActiveX web browser container but the game doesn't seem to support that method. I then thought about a VI which would continuously 'print screen'. I used a .net constructor and a graphics invoke node to call CopyFromScreen (This was achieved by modifying M.Kuznetsov's Continuous Screen Capture Example (https://decibel.ni.com/content/docs/DOC-37135).

The CopyFromScreen VI works but every 2-3 seconds there is a very short flash of white. While these white frames can be ignored programtically I would really like some advice on whether there is a better approach that I may be missing. I'm hoping that somewhere in that huge heirachy of .net functions within Labview there may be some way of embeddding an open Windows window into the front panel itself or intercepting the game's display data and processing that stream. Can I pass the game's Windows handle by reference to Labview and open it within some type of container? (I admit that I don't know if that previous sentence makes any sense at all!)

Another idea I have but with no clue as to how I would implement it, would be for a VI to be detected by the operating system as a second display and I could then choose to run the game on this virtual monitor VI.

Many thanks,

Wes

[tst]

---

@Wez wrote:

The CopyFromScreen VI works but every 2-3 seconds there is a very short flash of white. While these white frames can be ignored programtically I would really like some advice on whether there is a better approach that I may be missing. I'm hoping that somewhere in that huge heirachy of .net functions within Labview there may be some way of embeddding an open Windows window into the front panel itself or intercepting the game's display data and processing that stream. Can I pass the game's Windows handle by reference to Labview and open it within some type of container? (I admit that I don't know if that previous sentence makes any sense at all!)

---

The flash of white is probably not actually there as data. At least when I run the VI here, it doesn't appear in the actual images. That means you can just ignore it. You can also probably get rid of it by unchecking the Erase First option on the picture control, which tells the indicator to clear itself before drawing the new image, something which isn't needed if you're drawing over its entire area.

While you certainly can embed one window inside another (look at the Windows API SetParent method and you can find some examples here), that won't help you as you want the image data and you already have something that gives you that.

[Sam_Sharp]

Interesting idea - here are a couple of ideas I can think of:

1) I think it's possible to 'hook' into DirectX/OpenGL to pull in the frame data but you'll probably need to write some C++/C# to manipulate the data into something you can use with LabVIEW. Unless you can find a library that does that for you (I found some talk about it here: https://encodingtalk.com/threads/how-does-fraps-work.1405/).

2) Re-route the output back to a capture card which you can then use with LabVIEW (e.g. as an IMAQdx device so you can use NI Vision)

3) Take screenshots

What sort of frame rate do you need to capture at? These methods will have different amounts of latency as well as frequencies.

[Wez]

Unchecking EraseFirst from the picture box's properties has removed the white flashes, thanks for the tip. The other problem is that if I want to display the results of the machine vision processes on the screen then my print screen function will start feeding the processed images back into the image processor and corrupting the results. I found a Windows function called PrintWindow that can capture hidden windows but I get an error when I try to include the dll in my VI as the dll is 64 bit but my Labview version is 32 bit. I'm thinking I'll have the VI pass some lines of code in a command window but I'm struggling to find an example of how to print screen/window from cmd line.

[Wez]

---

@Sam_Sharp wrote:

Interesting idea - here are a couple of ideas I can think of:

1) I think it's possible to 'hook' into DirectX/OpenGL to pull in the frame data but you'll probably need to write some C++/C# to manipulate the data into something you can use with LabVIEW. Unless you can find a library that does that for you (I found some talk about it here: https://encodingtalk.com/threads/how-does-fraps-work.1405/).

2) Re-route the output back to a capture card which you can then use with LabVIEW (e.g. as an IMAQdx device so you can use NI Vision)

3) Take screenshots

What sort of frame rate do you need to capture at? These methods will have different amounts of latency as well as frequencies.

---

I think you may be right about hooking into DirectX with some C++. After spending the day becoming a Labview .net expert I can now confirming that PrintWindow does not produce an image of the game window. The window that receives the print message must be able to process it and as most games do not use WS_Paint calls there is no way to have the game copy itself to the clipboard.

I am interested in reading about how screen capture software actually works but it seems that the link you posted Sam is broken and a Google for 'How FRAPS Works doesn't provide any indepth knowledge. I am now going to try to become an expert on everything DirectX overnight!

Another solution that may be worth investigating is how display mirroring works and what would be involved to have a LV picture box (or some other container type) treated as a second monitor by the operating system. Guess I'd better keep flicking through MSDN then!

I was really hoping that there would be a way of having LV play the game in full screen but I guess having the pre and post processed images side by side may not be such a bad thing. 

[Hatef.fouladi]

all thing you need is
1)to acquire a VGA video signal from a computer video card output using some thing like NI-1409
2) divide your desktop into two monitor by right click on desktop and use screen options
now your second output VGA from your computer is belong to second monitor
and in first monitor you can acquire signal and save

[Blokk]

---

@Hatef wrote:
all thing you need is
1)to acquire a VGA video signal from a computer video card output using some thing like NI-1409
2) divide your desktop into two monitor by right click on desktop and use screen options
now your second output VGA from your computer is belong to second monitor
and in first monitor you can acquire signal and save

---

1. The NI-1409 is obsolete product
2. Sam_Sharp already mentioned rerouting as an option: "2) Re-route the output back to a capture card which you can then use with LabVIEW (e.g. as an IMAQdx device so you can use NI Vision)"
3. I have the strong feeling that the OP well knows how to set up 2 monitors for a PC 🙂

Please only reply to posts in the forum when you have some relevant info to share, hunting to be top poster purely by the number of posts is meaningless (and right now I just broke this rule 🙂 )...

[Sam_Sharp]

For some reason the NI forums likes to eat links when I put them in brackets - this is the correct link: https://encodingtalk.com/threads/how-does-fraps-work.1405/

You could also have a look at some of the streaming software that's available and see if you can figure out how those work and use the knowledge to your advantage (XSplit / oBroadcast - they're used for streaming games live to the internet e.g. Twitch) and there's also now the Steam broadcasting / Steam link which broadcasts either to the internet or local to your network (basically allows you to play a game on another PC by streaming it from your gaming PC).

I'm sure there must be a library you can use that someone has written to capture the frames - finding it and then being able to hook into it with LabVIEW is probably where you'll have the difficulty (writing a wrapper etc.).

[Wez]

Just a little update as it has been a few days. I am still trying to copy images to Labview from Grand Theft Auto.

Over the last two days I have exhausted my efforts to get .Net function calls to acquire a sequece of images from hidden/background windows. I have learnt how to access existing dynamic link libraries and also created my own for use within a VI. I believe these processes completed successfully but all attempts resulted in a purely black rectangle being imported from the clipboard. I believe this has something to do with the fact that the target is a DirectX application.

I have now redirected my interest towards understanding and implimenting 'hooking'. I admit I am now well in over my head. I believe screen capture software works by redirecting a DirectX application's display functions through itself and copying them before returning them to the display stream. I am beginning to feel (although this subject interests me greatly) that I should just be running Fraps to automatically take 5 shots every second and have Labview read Fraps' screenshot folder upon detection of a new image. Something inside me really wants to get Labview to handle the entire proecess in-house though!

As always, I welcome any ideas.

P.S. Sam, I found that link very interesting and had already stumbled upon the project it discusses. I have run that project and am able to see my game window within the application! I am now trying to understand how the hell it works!! Give me a few days and I will gladly give everyone a crash course on hooking (maybe).

Wez

[Wez]

After fiddling with my previous attempts at calling a custom DLL I have had some success. It appears I had some function parameters round the wrong way and some others of the wrong type. My DLL function borrows heavily from the example C++ code posted in the solution of this forum thread:

http://stackoverflow.com/questions/7292757/how-to-get-screenshot-of-a-window-as-bitmap-object-in-c

I read through the official Labview custom DLL examplles and tutorials once more and spotted some errors in my function type and also how I called the function from Labview. At the moment the title of the window I am attempting to grab is hard coded into the DLL function itself, at some point in the future I will adapt the code to be able to accept a string from a labview input.

I am glad that I can now import the display data of any window which is open and not minimized into Labview! I have attached some images of my desktop running the VI and also the DLL code that the VI library call node executes.

Wez

(AKA very happpy bunny)