Improve the performance of data processing

[GeorgeCT]

Hi all,

In my text file, there's index of x and y such as

index 1:x

50 50.1 50.2 50.3 50.4 50.5 50.6 50.7 50.8 50.9 51...500(neither every index starts at 50 nor with interval of 0.1)

index 1:y

2000 2000 2000 0 2000 12000 18000 22000 40000 48000 46000...62000

index 2:x

50.2 50.3 50.4 50.5 50.6 50.7 50.8 50.9 51 51.1 51.2...500

index 2:y

6000 2000 4000 22000 36000 26000 30000 34000 44000 34000 34000...58000

and I want to get the maximum y when x = 50.5 among all the indexes, so the answer above is 22000.

I want to find more than 60 maximum y of the x in each file.

I have more than 50,000 text file, and there're more than 600 indexes in one file.

My code is working but it's pretty slow when execute a real file, is there any way could speed it up?

I attached a 4 indexes file for example and my LabVIEW code(v12.0/v16.0/v17.0).

Any help is sincerely appreciated! Thank you all!

[altenbach]

Just from a 10 second glance at the code, there are glaring errors:

• What is the size of the "appended array" compared to the array of paths? (the shorter one determines the number of iterations).
• You can use a conditional tunnel to create the "output array". No case structure of shift register needed.
• There are glaring race conditions, for example the local variable "output array" will most likely get read before the indicator terminal has been updated.
• The "array-to-cluster-unbundle" dance is plain silly.
• I don't understand your algorithm to find maxima or even how you define one.
• I see a lot of Rube Goldberg constructs.

[altenbach · Accepted solution]

OK, while I don't understand your algorithm (and I don't think is sound*(see note)), here's equivalent code for the part inside your out FOR loop (i.e. processing one file). It gives the same result as yours but feels much faster.

Note: *Threshold array expects a non-descending array, and your data is not. Can you explain what you are trying to find in the data?

Some comments:

• To read the first (and only) column of a file, set the read to "transpose" and get the 1D output. No need to reshape and such.
• You process each string with the Spreadsheet string to array 120 times. once is enough. So put that before the second loop.
• Many of your successive FOR loops can be combined into one.
• I am sure there is tons of slack left. This is just a quick attempt ...

[GeorgeCT]

Thank you for your help,  altenbach.

That helps a lot, I tried one of my real text file, my original code ran 120 secs, but yours just ran less than 1.5 secs.

Thank you for your comments and code, let me learn a lot.

I used Threshold array because the "x" in some of my file looks like below:

50.1610183716 50.2241020203 50.2871780396 50.3502578735 50.4133338928 50.4764175415 50.5395011902 50.6025733948 50.6656570435

and for example I want to find the closest one to 50.5

[altenbach]

---

@GeorgeCT wrote:

I used Threshold array because the "x" in some of my file looks like below:

50.1610183716 50.2241020203 50.2871780396 50.3502578735 50.4133338928 50.4764175415 50.5395011902 50.6025733948 50.6656570435

and for example I want to find the closest one to 50.5

---

But that's not what threshold array does at all! Can you explain it graphically?

(Also note that you can probably process several files in parallel..)