Methods for acquiring live images from a video game for real-time proceesing

[Sam_Sharp]

Great news - glad you got it sorted out. If you want to / can, it would be great if you would share your code with the NI Community (e.g. here on the forums, or create an article about it over on the NI Developer Community).

Also, feel free to mark the solution as appropriate and give kudos to any posts that you found helpful 🙂

[Kyle97330]

I'm actually curious as to your end goal here.  Are you trying to create a bot that can play GTA5 based on visual input?

[Wez]

The block diagram above prodeuces a VI that hangs on a blank screen after a random ammount of time, obviously this is not desireable. The fault is fairly obvious, do not create a new IMAQ image every iteration of the while loop! I moved the IMAQ create outside the loop and all is fine.

[Wez]

---

@Kyle97330 wrote:

I'm actually curious as to your end goal here.  Are you trying to create a bot that can play GTA5 based on visual input?

---

Yes Kyle I am attempting to have a VI drive around inside Grand Theft Auto using edge detection to maintain its postion within the bounds of the road way and also blob tracking to maneuver around traffic and pedestrians.

I am working within a team of four developing a system that must contain three physically seperate Labview based applications that share data to accomplish some task as part of a university assigned coursework. An additional aspect of the design specification is that we must also interface with two "Car Chassis". These chassis' are microcontrollers with various LEDs mounted on a car shaped piece of plastic. Onboard is also a resistor next to a thermistor and a DC motor driven fan. The idea being that events happening in the game should be mimicked by the hardware, i.e. at night time the headlights come on, or the bandit car has been spotted so switch on blues and twos, or about to turn left so switch on the indicator, driving at speed for some time should heat up the resistor (engine) and high 'engine' temp. be detected by the thermistor which activates the fan motor which cools the resistor etc.

We have decided to attempt to have Labview play cops and robbers. Two of the systems will be police patrol vehicles and the third will be some sort of "dispatcher" chase controller with a God's eye view (from a helicopter perhaps) of the game world. The idea being that a game engine or human controlled target vehicle can be identified by any one of the police sub-systems on sight and chase commentary used by the other units to plan the interception of the bandit vehicle. 

We are running Multi Theft Auto on top of Grand Theft Auto San Andreas. MTA is open source which allows us to program a custom map, misson parameters and binocular camera angles etc. It is also trivial to have game data parsed to the VI's, although we will be keeping this borrowing of data to a minimum. For example speed data will be wired up to a gauge on the patrol vehicle front panel. I want a proper cockpit view front panel with as many functioning instruments as possible. The game data will not be used to complete the mission directly, the VI's should be able to apprehend the criminals as autonomously as possible.

We have a little under three months until deadline and now I've actually written out what we need to do I'm starting to panic! If it's okay with the admins I will continue to update this thread with an ever evolving AI Patrolman. Does any one have any experience with supervised machine learning??

As always please feel free to comment and advise as I am way beyond my comfort zone.

Wez

[Wez]

---

@Sam_Sharp wrote:

Great news - glad you got it sorted out. If you want to / can, it would be great if you would share your code with the NI Community (e.g. here on the forums, or create an article about it over on the NI Developer Community).

Also, feel free to mark the solution as appropriate and give kudos to any posts that you found helpful 🙂

---

I posted some screenshots of the code. The VI is a very simple demo so I didn't think I needed to attach the actual program. The custom dynamic linked library (DLL) contains a call to FindWindow which takes the title of the window you wish to screen shot (The title is written on the top left of the window border on all windowed applications. It can also be found by using Spy++) as one of its parameters. In my case I am screen shotting an instance of multi theft auto with the title "MTA: San Andreas". The other parameter is the Windows class of the application, in my case I left it as NULL which searches all active applications on the desktop.

FindWindow returns the windows handle (a C++ type hWnd) of the target and this handle can be used to call PrintWindow. The rest of the program is concerned with preparing the clipboard to accept a bitmap image with enough memory assigned to contain the entire image depending upon the resolution of the target window. Make sure resources are released upon completion of the function to avoid memory leaks! The clipboard must be closed after writing to allow other applications to access it.

I wanted to print directly to a Labview object but couldn't as Windows doesn't see Labview objects as seperate entities and as such only the parent window can be assigned a display context but thats no good! I couldn't find any method that would allow me to avoid having to use the clipboard (This doesn't mean it isn't possible!). If I had time I would continue reading about DirectX functions and hooking.

I really struggled to get Labview to behave properly with Windows types, pointers and references. I found the only solution was to have Labview call a library containing a purely C++ solution. This is something that many forums and NI examples state as the only course of action when attempting to impliment .Net functionality that relie upon references, pointers and special Windows class types.

Once I have a more functional program I won't hesitate to share. Still a long way to go though. Would it be wise to start a new thread?

Wez

[Wez]

I feel it would be a bit cheeky to accept my solution as a definitive answer to this problem so will give it a few more days and hopefully someone that knows what they are doing will pop in and put me straight!

[tst]

If your post has the solution then it's perfectly fine to mark it as the solution, which can also help people looking for this in the future (although this is sufficiently complex that I would expect people to read everything).

You can certainly update this thread with info as you have more, but I would suggest other platforms for that. If you want to keep it local, you can create a document or a blog or even a group (if you have a few collaborators and think you have enough content) over at the communities section, which is where people generally post examples (you can find various write-ups of student projects there). Just put a link to it here when you have it up. It sounds like an interesting project, so I would probably read it.

As a side point, good job on being organized and doing things properly and on time (which is not a given even on paying jobs). It's refreshing to see students actually taking on a challenge and doing the work to find the solution. While it's not universal, the type of students who come here to ask questions are often the ones stuck at a basic level and trying to get someone to do their homework.