Simple Script Execution: Taking days and only using 15% of CPU power

[Skipper0815]

Dear community,

i have a data-file (.tdm) with 17 waveform channels each containing 13742499 data points. I want to delete all the data points which are no actual load points. I already build a channel ("LoadTrigger") which is a trigger for the points of interest. With the following script i tried to erase the datapoints im not interested in:

Dim RowCount

For RowCount = 1 to GlobalChnLength

If ChnVal (RowCount, "/LoadTrigger") = 0 THEN

Call DataBLDel("1-17", RowCount,1)

End If

Next

The script is now running for over 20h without any results. Can somebody please tell me if I have a major mistake in my script or if there is a way to make it faster?

Additionally i'm wondering why DIADEM is not using more of my cpu-power? As you can see on the attached picture DIADEM is mainly using one core and never more than 15% of the CPU.

Best regards and hoping for construcitve answers!

[AndreasK · Accepted solution]

If you try to remove rows based on coditions you should use code like this:

Option Explicit
CreateExample

' make sure the correct group is active
data.Root.ChannelGroups(1).Activate
' use chncalculate to set the values that match a condition to NOVALUE
call chncalculate("Ch(1) = Ch(1) + CTNV(0 = Ch(1))")
' Delete the rows that contain a NOVALUE in a bunch of channels
Call ChnNovHandle("[1]/[1]","'[1]/[2]' - '[1]/[3]'","Delete","XY",1,0,0)



sub CreateExample()
  data.Root.Clear
  dim g : set g = data.Root.ChannelGroups.Add("group")
  dim c1 : set c1 = g.Channels.Add("c1",DataTypeFloat64)
  dim c2 : set c2 = g.Channels.Add("c2",DataTypeFloat64)
  dim c3 : set c3 = g.Channels.Add("c3",DataTypeFloat64)
  dim i : for i = 1 to 100
    c1(i) = i mod 2
    c2(i) = i
    c3(i) = i * 0.00001
  Next
end sub

which will remove all rows that contain a 0 in the first channel.

will be converted to

[Skipper0815]

Hey Andreas,

thanks for your quick reply. The solution is working fine except the channels are converted into numeric channels which isnt a big issue.

Do you know why there is such a big difference between using calculate and my solution?

Anyways thanks alot and have a nice day!

[AndreasK]

There are multiple reasons but the main reason is because its done as one operation in the kernel.

While in your implementation the data is intact after each loop the DIAdem kernel has the freedom to

optimize. Only after finishing the data has to be consistent.

In additional it can be done in a loop working on the memory directly using C++.

[Brad_Turpin]

Hi Skipper,

One other reason is that VBScript is an interpreted language-- it has no compiler, so a FOR loop over millions of iterations will always perform slowly.  Notice that Andreas' solution has just a few VBScript lines executing, even though potentially millions of rows are being evaluated.  Of course underneath SOME code has to consider each row, but with compiled C++ code that runs MUCH faster.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Skipper0815]

Thanks for all your explanation on my topic. Is there any general rule which solutions for processing long or just huge datasets, is the fastest option? Like using implemented functions is always faster?

On top of that i'm still curious why diadem is only using 15% CPU power?

I'm running DIAdem 2015 64bit under Windows 7 with an i7-3770 and 8gb ram.

Best regards,

Phil!

[AndresID]

Hello Skipper,

The reason Diadem is only using around 12.5% of the processor is because you chip has 4 cores with 8 threads. Your code is not multithreaded so it only works in one thread. It might take 100% of the processing of that thread, which means in total you are using 100 / 8 = 12.5.

Regards,

Andrés

[Otmar]

---

Skipper0815 wrote:

On top of that i'm still curious why diadem is only using 15% CPU power?

I'm running DIAdem 2015 64bit under Windows 7 with an i7-3770 and 8gb ram.

Best regards,

Phil!

---

Hello Phil,

DIAdem is a single threaded application, as was already mentioned by another poster here.

DIAdem offers a way to use multiple threads, by using "worker" instances. I recommend having a look at this example in DIAdem 2015 (the "worker" ability has been around for a few versions now, but this example is new as of 2015):

If you analyze long channels or multiple files, the worker ability can save you a good amount of time. It's not what I would recommend to total novices, but with the example you should be able to make some progress on having DIAdem parallel process multiple channels or files on up to eight threads on your PC (mine have 4 actual and 4 virtual cores, so I get to run a thread on each of my 8 cores).

Let us know if you need some advice/help after checking out the example ...

Have a great weekend,

      Otmar

[Skipper0815]

Hey Otmar,

thanks for your advice! I already found the example and i'm working myself through the code because the documentation is really weak.

If somebody knows some ressources for that specific topic in Diadem i would be happy if you share it with me.

Have a nice weekend!