LabVIEW
Converting maps to array slow?
Hi,
I'm working on logger module which works like this: The user can send in the LogEntry cluster which has a logpath and a logmessages array which belongs to the given path. The messages array in most cases has only one element, but it could have more. I'd like to keep the last 200 LogEntry clusters for each file. Because I dont know how many files will be used I thought its good idea to create a map in which the Key is the filename and the Value is the whole LogEntry cluster. Because LogEntry has the filename as well this means that the Value also has the Key value, but that doesnt really matter for the example.
Once the VI is called I add the incoming LogEntry cluster to the right Key of the map and then check if the key has more than 200 elements. If so then I delete the first element of the array. By this the size of the key array keeps growing from 0 to 200 and then remains 200 as the new element throws out the oldest. This part works alright.
The problem is that later in this VI I need take all the log entries out of the map and combine them into an array. Its clear that if I work with one file only (incoming log entries use the same file) then I have maximum 200 entries. If I work with two files then I'll have maximum 400 entries and so on. So thats a relatively low number. Nonetheless my colleauge started to complain about the speed of this VI and it turn out that converting the map to an array takes ridiculously long.
I have attached the VI I'm talking about in a stripped version I created for debugging purposes. There are two timers one measures the exec time of the code snippet above and the other measures the map to array conversion time. You'll see that the latter takes extremely more time than the first.
Things get even worse if you work with more than one file, so try this:
I'm very much surprized about this behaviour and I'm wondering if I do something completely non-sense.
Thx!
EDIT: attached the missing file
Hi 1984,
for me it takes ~1.2s to concatenate all our map elements. That makes 0.12ms per iteration: why do you think this would be slow?
Right now you stress the LabVIEW memory manager as you CONCATENATE 1D arrays of varying length - so I think the measured time is relatively small.
Btw. your example VI is not really useful as there is no real LogPath data involved: you either get "whatever" or "empty path"…
Its the unbundling that takes most of the time, and then the concatenating. Converting to array is fast:
Thanks for responding. The performance gets significantly worse very quick.
If I have 10 paths (so max total of 2000 clusters) then the total exec time of 10000 iterations is 22200 on my PC as opposed to 1600 when I had only one path. From this point adding a single element takes 2-4ms and thats something I consider slow, not to mention that the LogEntry cluster in my real appliation has more elements which makes it even slower. (and I mean it could go VERY slow like 100ms+)
I changed my VI a bit, to avoid concatenation and the exec time decreased by about 40% which is great but still find it quite high. Not having the VI open (just running in the background as a subVI still gains some speed).
I have attached a new version of this test VI.
1. set iteration number to 1 and the init true
2. run the vi
3. turn off init and set lets say 10k for the iterations.
(note: the VI is ugly as this is hacked inside out. The original is clean)
I'm unsure that I can achieve the speed I'd like to have, so I'm wondering how else I can achieve the same functionality.
Delete from array is slow, if you change that to a Rotate array and Replace array the first part goes from ~400 to 21ms on my computer.
You're right that that unbundling is surprisingly slow. It has something to do with the unbundle, if you just extract the cluster it's instant.
I think I made an error in my previous implementation as I have initialized an array, but then used the insert into array instead of the replace array subset VI. The correct implementation (I guess) is this:
Another thing is that the Initialize array function also takes good amount of time. Just for curiousity I initialized an array of 3000 elements outside of the for loop and that decreased my exec time a lot (to 7000ms). Of course this doesnt have to be initialized outside just detect if the map size grew from the last iteration and if so just add another 200 elements to this array. By this I avoided to init the array again and again in every iteration. I gotta run now, but I will do a better implementation and then let you guys know how the exec time changed.
@1984 wrote:
...
I changed my VI a bit, to avoid concatenation and the exec time decreased by about 40% which is great but still find it quite high. Not having the VI open (just running in the background as a subVI still gains some speed).
...
Why are you making the array 200 times bigger than required? Just unbundling and concatenating is a lot faster at least:
Why are you making the array 200 times bigger than required?
I dont make it 200 times bigger. I make it exactly the right size to be able to store (collection size)x200 log entries.
Just unbundling and concatenating is a lot faster at least:
Quite the contrary: its a lot slower.
