DQMH Consortium Toolkits Feature Requests

I’d like to suggest making the DQMH scripting tools API public. This would allow new events or modules to be generated programmatically.

My use case: I’d like to take any non-DQMH library and generate a DQMH layer for the library’s public API. With some clever coding and discipline, this could automate the creation of a TestStand API based on DQMH, which would be incredibly useful.

Moreover, public APIs often encourage the development of supporting tools and promote the technology, especially within creative communities.

Many DQMH developers (including myself!) encounter the stumbling block of firing requests before a module is "ready". We assume that the following code is sufficient for starting and synchronizing a module:



The 'Synchronize Module Events' VI ensures that request event references are valid, but it doesn't do anything about ensuring that the module is actually ready. One very common scenario is a module Main VI that includes code in the 'Initialize' frame of the MHL that fires private events to initialize Helper Loops. If the VI that called Start Module isn't waiting on the Module Did Init broadcast, then it could immediately fire a request after calling Synchronize Module Events, and if a helper loop is registered for that event, then helper loop could run the event frame for that fired event before the MHL has had a chance to fire the private event to initialize the helper loop. There are many other possible pitfalls, but this is one of the more common ones I've seen.

To solve this issue, I think DQMH should ship with a Public API VI that is a wrapper for Start Module + Synchronize Module Events, but includes the extra step of waiting for 'Module Did Init'. The module author needs to ensure that the Module Did Init broadcast is called after all necessary initialization code... if they do so, then it is "safe" to begin calling events externally.

Here's my proposed interface for the Start Module and Wait for Initialization VI:

And my proposed implementation:

If this VI shipped with the DQMH singleton and cloneable module templates, it would help us avoid module synchronization + initialization issues like what I describe above.

When creating a new DQMH event, I'd like this

When the DQMH Consortium introduced the support for private requests with DQMH 7.0, a conscious decision was made not to support synchronous requests with reply, to avoid dead-lock scenarios.

With the way we at HSE start additional DQMH modules from any given DQMH module (by sending a request from the MHL to the EHL to start and register for modules, then reply back to the calling MHL case, so the whole process becomes synchronous), it would be super helpful if we could script those private "start submodule" requests the exact same way as all other private requests.

TL;DR: Add support for private Request and Wait for Reply.

Edit: After internal discussion, let me add that I know how to make a regular request a private one by setting the access scope. I don’t want to do that - I want the private Request with Reply to be the same as all other private events, and not a modified public request.

Edit2: If you scroll down, you'll see that Manu added a vital piece of information. We usually communicate with one of the EHLs/Helper Loops, not the MHL. So scripting should not create another case in the MHL but keep the reply in the targeted EHL/Helper Loop.

When creating a new Event, it would be empowering to be able to automatically call a VI immediately after the scripting completes. The idea is based on the behavior of the Pre- and Post-Build VI calling ability of the application build process. The implementation should provide an option to specify a VI to call and an option to generate a VI with the proper controls already on the connector pane. A proposed set of such controls would be VI references or paths to the Event VI or VIs (Broadcast or Request... or both for Round Trip) and VI references or paths to the Tester VI and RT Tester VI (if there is one).

Adding the feature would allow developers using DQMH to create their own extensions to the built-in scripting. For example, if the other feature request here is not accepted, this one would allow a developer to implement it on our own.

Once a Broadcast happens, only modules that are registered for it can access the value contained in the Broadcast. If a module starts execution and registers for Broadcasts from another module after that module has sent a Broadcast, the registering module must wait until the Broadcast happens again before it has the latest value. We have encountered some use cases where it is beneficial to know the most recent value of a particular Broadcast. In those cases, we have created workarounds such as a "Refresh" Request that can force a module to repeat all or a subset of its Broadcasts. Of course, that requires the addition of a Request and the code to handle it, etc.

The feature request is to have the Broadcast scripting create two additional VIs for each Broadcast. One VI would be Private, and it would be a data store for the value of the Broadcast event (message). The other VI would be Public, and it would be a Read Accessor for that value. I am picturing the Public VI calling the Private one, and the Private one is a Functional Global. In the Broadcast event VI, the value would get written to the Private VI's Functional Global.

Having a Public VI that contains the most recent value for the Broadcast make it easy to write small state machines or action engines that can access Broadcast value data without Event Structures.

There are other messaging architectures that implement the "Read Global" ability, and it has proven to have value for us in development of some systems.

For Cloneable modules, the "Read Accessor" could take a Module ID as an input.

Today DQMH scripters add comments with specific hashtags. But some of these hashtags names are very generic (for example #CodeNeeded or #CodeRecommended).

Adding a 'DQMH_' prefix would allow a better isolation of DQMH related hashtags in the bookmark manager (ie. #DQMH_CodeNeeded).

Actually some DQMH tags already have this prefix (#DQMH_HowTo).

When adding an helper loop from the scripter, add a checkbox that would allow creating a subVI that contains the helper loop.

This checkbox should be unchecked by default.

If checked, a (private) VI is create and contains the helper loop.

The subVI is then dropped inside the Main.vi where the helper loop would have been created.

For some reason I need to know the module name of the module that I start.

Module name constant is a private VI and therefore cannot be called in a cller VI diagram.

Having the Start Module VI returning this information could be useful (instead of specifically creating a R&R just for this purpose).

Today when creating adding a module template it is not possible to set a 'human readable' name for template.

The template name is the module's library name. Having a field to specify a more friendly name could be useful (could be set by default withe the module's library name though).

In our current practice, we encapsulate the content of DQMH MHL events within subVIs to maintain a tidy diagram and especially to facilitate the seamless propagation of changes to similar projects that use the same implementation. 

I think It would greatly benefit the DQMH framework if it could autonomously generate these subVIs as part of its MHL event scripting process.

Current situation:

Expected situation:

For DQMH modules written prior to DQMH 7, helper loops will unlikely have the "DQMH_HELPER_LOOP_NAME" string constant wired to the error helper VI as shown below.

This idea is to propose a new test be added to DQMH Validate Module to check that all helper loops have this string constant present.

When preparing a project for Lumos tracking...

Lumos adds "Trace VIs", when migrating this code to another PC without Lumos installed
When opening DQMH module it asks for all the "Trace VIs" and breaks the code...

It be great to have another option to Remove or Un-prepare project for Lumos Tracking and be able

to distribute this code without Lumos installed

Thanks! Great tool!

As users of SVN for source code control, we rely on SVN-Locks to maintain a conflict-free development environment when collaborating with multiple project team members.

When working on DQMH Projects as a team, it would greatly enhance our workflow if we could selectively apply SVN locks to specific subsets of the project.

Currently, the DQMH scripting necessitates write access to the entire project, limiting flexibility in group collaborations. 

Is it feasible to consider modifying this behavior to provide increased flexibility and efficiency for group work?

All DQMH modules come with a Module Timeout--constant.vi:

This value defaults to 5 seconds, but I've worked with many modules over the years (usually having to do with serial devices) where this value is modified. I find it tedious to have to dig into the timeout VI any time I want to find out what the timeout actually is. Especially when I'm working with multiple modules at the same time that may have different (legitimate) timeout values. So I propose the following:

1) The default modules that ship with DQMH should include the timeout value in their icons:

2) There should be a Validate test that ensures the Timeout VI icon displays the correct value, and provide a fixer that will update the icon if it does not.

We have identified that, if multiple clonable module instances are executed and a specific module (e.g. the first launched module) is stopped (no waiting) and afterward all open modules are stopped at once (incl. wait) the "stop module.vi" an error 1 is returning.

Steps to reproduce

1. Create a project and add a new clonable module
2. Create a tester VI and implement the code above 
3. Run the VI and see error 1 at second Stop Module.

What is happening in the Background

Situation 1 - "Stall Data Flow" = 0

1. "Stop Module.vi" 2 runs into "Wait on Stop Sync.vi" and synchronizes stop over rendezvous.
   The acquired rendezvous size is 3 (Module 1 which is at stopping, Module 2 and Stop Module)
2. Module 1 is waiting in "Safe to Destroy Refnums.vi"
3. Module 2 runs into "Close Module.vi"
   

    

   1. Last clone instance is fire at (1)
   2. Releasing the Semaphore (2) will wake up module 1 that it is now safe to destroy refnums now.
      Module 2 runs into "Wait on Stop Sync.vi" (3) and synchronizes over rendezvous.
   3. "Stop Module.vi" and Module 2 waiting for a third participation to join the rendezvous.
   4. Module executes case to destroy Master reference.... and executes "Wait on Stop Sync.vi" (3) with no synchronization as the boolean "Wait for Module to stop?" is on false.
   5. Module 1 executes "Destroy Sync Refnums.vi" (4) and is destroying the rendezvous.
   6. Module 2 and "Stop Module.vi" will be release from the waiting of the rendezvous as the reference is now invalid and returning error 1.

Situation 2 - "Stall Data Flow" = e.g. 2000ms

1. In compare to the situation 1 the first module is already removed here. The obtained rendezvous has the expected size of 2.
2. When module 2 enters rendezvous synchronization in "Wait on Stop Sync.vi" (3) the expected amount of participant is reached, and the execution can continue.
3. In most of our tests, this situation worked fine and did not create an error.
4. For some situation, we had the behavior that shutdown of the first module seems to be faster as the wake-up from rendezvous of the second module. The module main of module 2 opened and showed error 1. Module 1 seems to destroy the references to early. 

Situation 3 - First Module will be stopped with last "Stop Module.vi" call

The shutdown of a module is for this scenario delayed (add a wait 1000ms to exit case of the module)

1. Stop Module 2 waits for 11 rendezvous participations. (10 module and itself)
2. One of the previously closed module will destroy the synchronization events and makes the rendezvous reference invalid. => Error 1 at "Stop Module.vi"

Potential Fix

Spoiler

The following screenshots are showing an extension of the "Stop Module.vi" and the "Close Module.vi".

The idea is to use a single element queue (SEQ) containing a map of sets. The key of the map refers to a "Stop Module.vi" which waits for stopping all module at the time when the "Stop Module.vi" is executed. The Set contains all Module ID's which should be stopped. Each module checks in its close condition if the SEQ is existing. If so, the module ID will be removed from the sets which containing the module ID. An empty set refers to all required modules have stopped and a notifier which is used for synchronization will be fired.
Close Module.vi extention
Stop Module.vi extention

 

With those extensions, all three described scenarios should be fixed. In addition, should it be possible to stop all module and launch in the background new ones, the stop and wait will wait until all those modules ID run at the stop execution are finished.

 

I added the project which the extensions and tests to the post.

Addition

I'm not sure, but I think that with the described change, destroy of the Module's Semaphore (1) should be done with the boolean condition of the First & Last Instance (2). (Red line)

Please let me know if you need any additional information and details.

It would be awesome if the DQMH validation tool could go back more than one version.

eg.  If you open an old piece of code (that was written in say DQMH 4.2), the only way to upgrade it to 6.1 is to one version at a time, upgrade the code step by step.  This would mean the following:

 - Install DQMH 5.

 - Run Validation tool

 - Install DQMH 6

 - Run Validation tool

 - Install DQMH 6.1

 - Run Validation tool

This is a rather painful process, and when faced with this, I have decided to simply implement the module again starting from the latest version, and porting the code over rather than go through that.

Neither approach is obviously ideal.

We really need a tool that can convert an older module up to the latest standard.  This will become a more common problem now that we have DQMH-based code out there from 2018 onwards.  The more people use it, the more this problem will surface.

I have 'barebones' singleton and cloneable module templates that I always use when creating new DQMH modules for projects. I started with the shipping module templates and removed:

- 'Do Something' events

- All #Bookmarks

- Simulate EHL/MHL error buttons

I propose DQMH ship with these simple module templates. They help save several minutes when creating a new module. I'm not suggesting changing the project template, but rather simply adding two new options in the Add New DQMH Module drop-down:

I'm open to suggestions for different names. 🙂

I am aware of the note that is being shown on the "Remove DQMH Event" that states: 

" NOTE 2: You cannot remove the last private event from a DQMH module. Once a private request has been created, the module requires at least one private event to be present." 

The request i believe would come in handy, is that instead of "hiding" or "not showing" the one and only private request, to instead show it as available for deletion, and when pressing OK, populate the information on a popup that shows what NOTE 2 said. 

This with the objective of not giving the user the notion that DQMH scripting tools are broken, especially the ones just beginning to use DQMH.