Memory usage jumps but never returns to normal

[markza]

HI guys,

I have a cRIO application with a Host PC for talking to the cRIO.  The cRIo logs data (data stored in 1min intervals 24/7) to weekly tdms files, and the host pc then ftp's the files and places the data into a local MS Access database.  The user is then able to pull data reports on the host PC via the database.

So during a process of pulling a report from the database, the following process occurs:

1) HostPC checks the last entry in the access database to know how much data it needs from the cRIO

2) It then ftp's the necessary datafiles (tdms) across to update the database with all the logged data to date

3) It parses the tdms files, and stores the data to the database

4) It then does the database query (basically get all data between date X and now) and stores it to a csv file and saves it.

Now during this process the memory usage jumps up (and can jump by hundreds of MBs) on the PC and just never returns to a normal amount?  If I do a report of 20days worth of data, it can go over 200MB memory usage.  I mean, its not that much data ever?!

I am using Labview 2010, and using the database connectivity toolkit.

The full database is about 7MB, and each 20day report is roughly 400KB. 

I have tried to use the "Request deallocation" vi in some of my subvi's, but that doesn't really do anything.   I just don't understand why it needs 200MB for anything?  To me thats excessive!

Hopefully you guys can help me out if any of you have seen something similar or have any suggestions.  Appreciate any help.

[Cory_K]

You may not directly have 200 MB of data, but your code may still be storing it.

For example, if you have 50 MB of data, and 4 buffers are created along the way (for data coersions, etc), you will have 4 x 50 MB = 200 MB.

Could you possibly post your code? I would like to see if this is occurring anywhere.

[markza]

Ok, I'm just creating a similar extract from my main project to illustrate this issue, and I'll post that code within the next few minutes.

[markza]

Hi guys,

Here is my code for reproducing the memory issue I'm having.

The mani.vi basically does a database query for all the records between a start date and now, and saves it to a comma separated file (so save it to something like a temp.csv via the dialog).

If you just run the vi straight, it'll get ALL the info from the database...thats over a months worth of data I think, and your memory usage will go mad.

Any help would be appreciated...please help me find a better solution to do this.  Hopefully I'm doing something stupid.

[Yamaeda]

In the advanced tab in DB toolkit you can Execute query, Fetch elements and Move Cursor to next. That way you'll only request 1 row at a time from the DB. (If the others doesn't help)

50k rows with that many fields, converted to variants and then converted to strings quickly adds up to alot of memory.

You can create a cluster of the correct datatypes and convert the data to an array of that instead of an array of strings.

/Y

[markza]

Hi, thanks for responding

Ja, i guess thats possible, but to do it all one row by row seems ridiculous, and it'll be so time consuming and sluggish surely.  I mean, for a months worth of data (which is realistic) thats 44640 individual queries.  If push comes to shove, then I'll have to try that for sure.  

You can see by the example that I'm saving it to a text file, in csv format.  So it needs to be a string array, a cluster won't be of much help I dont think.

The only other way I can think of is to break it up into more manageable chunks...maybe pull each column separately in a for loop and build up a 2D array like that until the spreadsheet storing.  

[Yamaeda]

---

@markza wrote:

Hi, thanks for responding

Ja, i guess thats possible, but to do it all one row by row seems ridiculous, and it'll be so time consuming and sluggish surely.  I mean, for a months worth of data (which is realistic) thats 44640 individual queries.  If push comes to shove, then I'll have to try that for sure.  

You can see by the example that I'm saving it to a text file, in csv format.  So it needs to be a string array, a cluster won't be of much help I dont think.

The only other way I can think of is to break it up into more manageable chunks...maybe pull each column separately in a for loop and build up a 2D array like that until the spreadsheet storing.  

---

You only do 1 query, but instead of Fetching All (as the Select does) you'll use the cursor to step through the data.

You can use Format to String or Write Spreadsheet fire with doubles.

You can break it down to get the data day by day instead of a full month at once. 🙂

/Y

[markza]

AH I see now, cool, all great suggestions, I'll give them a go and see how long it takes.  

I unfortunately have to do a months worth of data at a time because the client is able to specify custom dates for the select query.  So it is kind of time critical, I dont want the poor bugger sitting there for 5mins while this thing chuggs away at the database.

I did change it to query for individual columns in a for loop, and that reduced the memory usage by about 65%...which is significant.

[ChrisReed · Accepted solution]

I had the same problem as Mark did except it was retrieving data from an MS SQL 2008 database and I posted the issue here:

http://forums.ni.com/t5/LabVIEW/LabVIEW-does-not-exit-in-executable/td-p/1304980

I think it was Ravens Fan post who guided me to a LVProject called "Task Manager in LabVIEW" by Cosmin (see link)

https://decibel.ni.com/content/docs/DOC-9292 (and I used them to make up my own VI - see image below).

As you can see it automatically obtains the application EXE reference, so you can place it anywhere in your application where ever your memory usage jumps too high and is not being released.

Chris

[markza]

Excellent, thanks Chris. Your solution (to deallocate memory) combined with smaller database data requests (reduce max memory usage) works out perfectly.