Graph refresh is very slow with large datasets

[heel]

When graphing large datasets in DIAdem, the building of the graph is slow (3M pts takes 30sec). Fair enough - the problem is however that whatever little change you subsequently make to the graph, it will refresh all over again, and during that time you cannot do anything else with DIAdem.

Any way to ease this?

[AndreasK]

Which version of DIAdem are you using?

It might speed up if you change the loading behavior if your files are fragmented.

Please have a try.

[heel]

Thanks Andreas,

Its DIAdem 15.0 64bit version.

I tried your suggestion (Load data on first access), but it does not help. I think it only applies for reading data from storage - that may be a good optimization, but I have the problem in the report tab where data probably are in memory already. It is painstaking to edit the report - whatever little action on the report layout may take half a minute. Almost useles to work with.

[Otmar]

Hello heel,

I just tried making a 2D graph with about 200,000,000+ points, and it takes about two seconds to plot on my ageing (approaching 4 years old) laptop (8GB RAM, 2.6GHz Intel i7, and SSD).

Are you creating a 3D graph?3D graphs perform hidden-line calculations and may take time to draw when you use very large data sets.

If you just have a 2D graph, some setting must be causing the long draw time, 30 seconds sound like an excessive amount of time, especially given that you use the 64-bit version.

If you are editing parts of the REPORT that are not benefiting from actually seeing the data (like adding or rearranging text or other items), you can turn off the data layer in this menu:

That will show all the axis systems with no data inside, good for making edits that do not require you to see the data in the axes.

I would still like to know what causes the issues with the redraw speed though. Can you post an image of the report you are creating (please remove any confidential items from the report before posting it here). We may be able to narrow down the issue by looking at the type of report you are trying to generate ...

Best regards,

    Otmar

[heel]

Hello Otmar,

Thanks for your tip. Indeed removing the data layer activation makes it much easier to do what I am doing on the report page: Adding text and graphics and rearranging plots. 

The plots are simple 2D plots (amplitude versus time) looking like this:

The time axis holds 3M points.

I am accessing the computer via remote desktop on a fast link. I will test tomorrow when I get closer to the machine how the performance is on the console. Will report back then.

Thanks once more - now it is much more convenient to work with.

[heel]

I tried the same operations on the local console of the Laptop. The speed for updating the report graphs is a bit faster there, but still very slow - in the order of 15-

20sec to refresh the display. I should maybe add that I did not restart the program. The laptop is less than a year old model 8GB ram, SSD disk. The data are on a networkshare.

[AndreasK]

> The data are on a networkshare

I really believe it is related to the Load Mode of the Data.

Please set it to

• Load Bulk on first access
• Always load bulk

And make sure you reload your file before testing. (It is just changing the behavior for the future)

You have to save your Desktop.ddd to make the setting persitent.

P.S.: If set to "Always lod bulk" it is made sure that all data is cached before you start reporting at all. So you will see the Impact of Report without having to move data from share to local memory.

[heel · Accepted solution]

Problem appears to be solved - restart of DIAdem restored the refresh time to acceptable levels, or at least it appears that the restart was the only change made.

I tried subsequently with both settings of reload mentioned by AndreasK and the both perform equally well.

I tried access from remote desktop and it works equally well - also starting DIAdem from remote (to see if it would be a graphics driver issue)

I feel kind of silly not to be able to pinpoint what was wrong and thank you for your help.