How to improve the performance when open big data file

[alex.]

I am using the attached vi to open a data file, and later will process it.

But just in the open stage, the operation is already very slow.

Is there any way to improve it? Thanks.

[Yarmouth]

I am using LabVIEW8.5 so I can't open your VIs. But I ever met the same problem. Using shift registers rather than channels can improve the efficiency.You can also divide the data into several parts and then deal with them separately.

[alex.]

The opened array size is 12,000,000, the data file size is 11M.

May I know what is channels. And if divide into different parts, how to manage them with more efficience.

Thanks.

[RavensFan]

I have no idea what Yarmouth is talking about when he talks about channels.  It is not very helpful advice considering he even stated he wasn't able to open your VI's.

You are using a Read Characters from File VI which is a rather old VI.  LV context help says it is no longer supported.  Try using Read Text File on the File I/O palette instead.

You might want to put in some timing functions to see which part of your VI is taking the longest.  When you say it is very slow, how slow is it?

I suspect that the Match Pattern function is what is taking a long time,  is it really necessary to do that?   Suppose you read the file in smaller portions inside a loop until you find that "#9", then once find it take the data after it and read the rest of the file.

[alex.]

On old PC may take half min, on new PC only 3 second, but after that, all the operation will be very slow.

All the data processing part still not programed yet.

I tried to run it by bubble, and found every part is slow on old PC. The data file is 11M cannot upload here, any where else can upload.

Thanks,

[RavensFan]

No, there is no good place to upload the file.  How old is the old PC?  How much RAM, what kind of hard drive, CPU and speed.  3 seconds vs. 30 seconds is pretty significant difference.  Three doesn't seem horrible.  Thirty seconds is pretty bad, so using an old PC might be a bad idea.

As I said, first benchmark your code.  Break up the pieces into flat sequence structures with the timer function in the frames in between the real operation, and do some subtraction to see what is taking a long time.

[Guruthilak]

I believe reading a file that too 11M will defenetly hog your CPU. Try reading the file in a smaller chunks and process this data everytime you read the chunk.

Also as mentioned, The RAM , processor everything counts on the performance

Guru

[alex.]

The old PC is pentium 4 2x2.8G, 512M memory, and the new one is hp server with quad cpu.

The final one will not be so powerful as the new one.

[alex.]

The file is here.

https://www.yousendit.com/download/VGlmZGVjTkxLVlVLSkE9PQ

[RavensFan]

How much RAM is in the new PC?

I downloaded your file and tried it out with your VI.  It seemed to run very fast.  About 80 milliseconds in total.  My PC is a quad core AMD with 4GB of RAM on Vista Home premium.

I noticed that after I ran your VI, the VI properties showed about 222 MB of memory being used.  That is such a large portion of the 512 MB on your older PC, I'm not surprised it is running slow.  Windows eats up a lot of RAM.  I imagine that your hard drive is thrashing during those 30 seconds trying to move around the virtual memory.

You may also want to look at this knowledge base article that is often linked to  Managing Large Data Sets in LabVIEW

Try increasing the memory on the old PC if you can to at least 1GB.  See if that shows improvement.