Not Enough Memory with my array

[ds_1]

Hi Guys,

              I'm having some trouble with my dataset, I guess the operation I'm performing being in a loop is the problem as I'm making extra memory copies of my array. I get a Not Enough Memory error when I run this. I'm using LV8.5, I've added /3G switch and increased virtual memory.

           Is there an easier way I should be doing this?

           Thanks

[Norbert_B]

I don't understand the purpose of the whole algorithm, nevertheless:

you can improve performance by using "split number" in the data manipulation palette (numeric).

Keep in mind that arrays have to be in closed memory spaces. So i asume that you are seeing memory fragmentation.

Your VI needs 4*2621439+4+32+16+16+16*2621439 Bytes in memory only for handling your array (the very least). Please note that each number in the addition has to be in a close memory space. So your output, the boolean array, needs about 42 MB without any gap. And this is only for the computation.

Display creates additional copies, so your algorithm eats up about twice the stated amount of RAM, so in all about 110MB for this single VI. You need two packages of 42MB closed memory and two of 10MB. If the OS fails to deliver that memory, you will get the "Not enough memory" error regardless of available memory as sum.....

hope this helps,

Norbert 

[ds_1]

Thanks for the response Norbert,

        Sorry I haven't been able to respond till now, though this problem still seems to haunt me, I am investigating the split numbers solution although I don't know if it will suit my needs. My project is a data acquisition system, I need to process large sets of 16,18 or 24 bit data, in fact at this point I need to process a 30 bit word:

2x 16 bit reads with some 0's padding, the bits of interest show up in specific locations, hence the need to convert to boolean to allow me to check all is correct, it is also necessary to reverse the bus read correctly from the hardware so reversing is necessary.

I realise that this conversion of a number array to boolean array consumes a lot of RAM but I'm not sure how to easily recreate the same function with the number array.

        As the processing of this data becomes more complex I am more often running into this memory situation...

      Any more suggestions are welcome..

         Thanks,

         ds

[GerdW]

Hi ds,

I cannot look at your vi right now, but I would like mention the possibility to use boolean functions with (integer) numbers. This way you can avoid to convert to boolean arrays...

[altenbach]

Stay away from anything green!

Since you only use 16 bits, I am not even sure why your input is I32. Why not make it e.g. U16. If you simply want to reverse the bits, you can do that with a 16 bit lookup table and keep it U16. This would be orders of magnitude more efficient.

Back to the boolean array. What processing do you do with it later? Boolean arrays are very inefficient. You start out with 16 bits if information per array element and you convert it to 16 bytes per element of the the original array. Basically you are inflating your data structures by a factor of eight. That's why you run out of memory!

[ds_1]

Hi altenbach,

                     Thanks for the suggestions, the data can be either 12,14,16,18 24 or 30 bit so the lookup table may get a little complicated to implement, particulatrly in the case of the 30bit where it is a middle 7 bits only that i need to reverse, but I think I may need to try it out!

               Thanks for info on the boolean. I suspected that they were innefficient but I didn't realise the full extent. I think perhaps that I should just change my numeric indicators to binary "display format" so I can debug in this manner, a little trickier to to visually debug stuff but more efficient I guess.

[Yamaeda]

Reversing the middle 7 bits should be easy:

Number to boolean array (1 30-bit word now instead of the full array)

Array subset (middle 7 bits)

Reverse array

Replace Array Subset

Boolean array to number

That should do it (i first thought up a bit shifting version, but i like the idea behind this)

/Y

[GerdW]

@Yamaeda:

again I would note to use boolean functions directly on the integer numbers, thus avoiding that number to boolean array to number conversion!

You can AND a number to mask the needed bits and OR to join several bit patterns!

[Yamaeda]

@Gerd

I started off like that, but as i didn't find any "reverse boolean" i did that bit through number to boolean array ... than i realized the other could be optimized if i did them both that way also ...

Else i generally perfer some bit shuffling, as i started off in assembler long time ago on the C64. 😉

/Y

[GerdW]

Hi Yamaeda,

even on C64 it was faster to use AND and ORA mnemonics with full bytes (aka U8) than to work on single bits