Can i optimize write of large data array into text file ?

[mhed]

Thanks all for your answers and your explanations

I've modified my VI using Mythilt and mcduff examples.

By writting block into my file, i divided my time about 4-5. That's very helpfull for my application...and the user who will to use it 🙂

I've also tried the example of Bob_Schor, it is also very fast. However, i cannot use it, because, the value 41M, that i gave for my data array, is only an example.

In my final application, i will have large array (like 48M, 41M) or very small ones. The size is depending on the use of my application.

But thanks for your help anyway.

[Blokk]

I am just courious: what is the reason you need to use ASCII file? At such sizes a binary (TDMS very useful) file is much simpler to handle, and also to read the data out of it later on... Of course if you have a special reason, like the file needs to be read by another software which you cannot modify...?

[Bob_Schor]

The point of my "example" was to test an algorithm using a specific example.  If it flat-out failed, you stop there.  If it looks promising ("it is also very fast"), then you spend a little bit of time making it "generalizable", i.e. adding the bells and whistles.

Bob Schor

[mhed]

Hi,

@Bob_Schor

Don't misunderstood me, i didn't say your example is not good. I've tried it with a few tests.

However, i can't have so much time to modify it to be more used with various size of data array. Well, by 'no much time', i mean that with my skills in LV it will take me some time that i don't really have. 

But i don't forget you help me, and i really appreciate it, and i don't forget your example, because maybe i will come back later on it, if i need.

@blokk

You're right. In other applications i've made, i used TDMS file, it is much quicker and easier to handle. However, in my current application, the file is wrote by LV but it used (read) by another application (in fact a Perl script).

[Blokk]

Any computer language should use binary file ops at such file sizes, much faster. You can read tdms from PERL too...

http://digital.ni.com/public.nsf/allkb/A3663DE39D6A2C5A86257204005C11CA

Another way is to create your own binary file structure, and then create a reader function at the PERL side. This should be easy to implement using the LV low level binary file functions. 

[Blokk]

Ok, I just realised PERL is just a script language...so maybe in overall you are stucked with ASCII files...

[mhed]

Hi

Well, i'm not sure i'm stucked with ASCII file with PERL. It is possible to read binary file with this language.

However, i've not been involved in the creation of this script, i'm only using it. It requires input text file, i need to write ASCII file.

As i say earlier, it will be possible to change this mechanism, but my LV application is for a customer project in progress, and mostly based on reuse part. 

So i can't modify everything i want, or everything it will be. I need to adapt my appliction to those parts.

But, maybe for another use, in another project i try to update the Perl script to read binary, if i gain disk space and reduce the time to read/write the file.

[.aCe.]

In the interest of keeping it fast. Here is one that takes 17s compared to 42s (mcduff's version) on my machine.

[mcduff]

Nice work!!

I did not know writing to an ASCII file could be done in parallel. Did you check that everything is written in the right order?

Lastly, the speed comes at a price of more memory, but if thatmay not be a problem for the OP.

Cheers,

mcduff

[altenbach]

---

@mcduff wrote:

I did not know writing to an ASCII file could be done in parallel. Did you check that everything is written in the right order?

---

I would expect that the right order is not guaranteed using a parallel FOR loop.