CLI tools for sequence analysis and diff

[Phil.Lundeen]

We're trying to add some Review/Continuous Integration tooling to help with our teams' Teststand sequence development. Among the goals are to automatically generate diffs of sequence files (using the FileDiffer.exe utility) and to automatically generate Sequence Analyzer reports, such that these artifacts could be referenced as part of a code review process (e.g. in pull requests). Both utilities purport to having a command line interface, but the support seems flaky at best.

The Sequence Analyzer (AnalyzerApp.exe) works most of the time, but every once in a while, we find that the process hangs indefinitely.

The Sequence Diff Tool (FileDiffer.exe) works when run in an interactive session, but when it runs under the SERVICE session (e.g. headlessly) as part of our CI pipeline, it appears to run and then pause at the end without actually generating a report (so, again, hanging indefinitely).

Both hang scenarios seem likely related to running the process headlessly; I suspect there is some sort of dialog message that is thrown and awaiting user feedback, but there is no way to access it.

Has anyone out there had any successes with using these utilities? Any lessons learned that we could try?

I suspect this is another situation where the CLI support was an afterthought, and probably not adequately tested/dogfood'ed internally at NI. But it would be nice if we could incorporate some of these tools to bring in software engineering best practices into Teststand development.

[Anand_Jain]

HI,

I know a few customers who have been successful with the workflows you describe below with TestStand. 

Regarding the hang with the Sequence Analyzer, it might be a bug that should be looked at. Have you contacted contact support for this issue? They can request for a dump file to help diagnose the issue further. 

Regarding the Sequence File Differ, have you seen the page linked below and made the changes described in the article under section -"Setting up a Build Machine for use with TestStand". Note some of the changes are not required for newer versions of TS. What version of TestStand are you using?

Setup TestStand With Jenkins - NI

Regards, 

Anand Jain

NI

[Phil.Lundeen]

---

@Anand_Jain wrote:

Regarding the Sequence File Differ, have you seen the page linked below and made the changes described in the article under section -"Setting up a Build Machine for use with TestStand". Note some of the changes are not required for newer versions of TS. What version of TestStand are you using?

Setup TestStand With Jenkins - NI

We are running Teststand 2019.

We are using Gitlab CI/CD for our automation, not Jenkins, so a lot of the details in that link are not applicable. But I suspect the steps recommending running under an interactive account (and not the "SYSTEM" account) are germane. I can look into whether it is possible to have the Gitlab CI Runner run under some specific user account, but it feels like a workaround for a more fundamental problem with the tools if there is a CLI that can't run headlessly.

I'll see about opening a support request.