Best practices for file I/O within producer/consumer loops

[jamoore84]

Ben,

Thanks for your revisions - I'll step through it and see if I have any questions.

Is the reason you pulled the majority of the initialization case out of the state machine and  put it in a sequqence structure just so you could properly route errors?

thanks again,

jimmy

[Ben]

---

jamoore84 wrote:

Ben,

Thanks for your revisions - I'll step through it and see if I have any questions.

Is the reason you pulled the majority of the initialization case out of the state machine and  put it in a sequqence structure just so you could properly route errors?

thanks again,

jimmy

---

No.

There was no guarentee the Init state would execute before the values were being read. Putting them in a seq frame and then using the error cluster to seq the other loops made sure they were in the correct state before any of the loops got at them.

As I osted in my previous, you are looking a "quick hack" code. I'd probably move the read stuff into the "check" state and toss my "trick" (Note to others: The trick was not to execute only one action. If play-back was active when checked, it execute one more case before returning to the caller.)

The important take home message is

THe State of the AE is inside the AE and only inside the AE.

Methods can be used to chage and query states but the master copy is in the AE.

Ben

[jamoore84]

Ben,

I’ve had some time to look at the code and I have a few questions.  I’ll break them questions down case-by-case:
1.     Initialize Case:  No questions
2.    Record Start Case:  Why do you have an inner case structure to differentiate between     “Recording” and “Idle” cases?  I cannot think of a situation where you would enter the     “Record Start” case when already recording.  The enabling/disabling of buttons should     account for that.  This leads to my next question…
3.    Playback Start Case:  If there was a nested case structure to differentiate between     Recording and Idle inner states for the Recording Start Case, why not again in the Playback     Start Case?
4.    Record Case:  No questions.
5.    Playback Case: Is there a reason why this case is missing the AE state check that the     Record Case has?  Also, is there a reason to transition to the ‘Check State’ case?  Does     this offer anything I’m not immediately seeing? Can I replace the “Check State” case  with     having the Playback state call itself(until the EOF error occurs)?
6.    Close Record File & Close Playback File – No questions.
7.    Check State case:  Inside this case there is a case structure to check the state of the AE.      Why should the “Check State” case do anything more than output what state the AE is in?      Also, I do not immediately see why there would be cases for Idle and Playback, but not for     Recording.  Is there a reason why you would not want to call the Record case if the AE     state was at “record”? 

In a previous post you mentioned that you would move the read functionality into the Check State Case.  Is this so that you could just call “check state”, and then output playback data if it is playing back?  If so, that would explain why, from “check State”, you call the playback state if the AE is at “Playback”.  I may have just answered #7 myself.

Lastly, If I click ‘Record Start’ and let the file dialog box sit, it appears that the generated data is being stored in some buffer; it’s as if labview starts caching data the moment I click Record Start.  Where is this being cached?  How large is the buffer?  I could this causing a problem if the dialog box is left open for a long amount of time.  In the real program I’m revising, there is potential for file streaming rates at/near the HD bandwidth limit.  I can forsee this data accumulation during the “Record start/record commit” period becoming a problem.  However, this question effectively takes us away from Action Engines and might be better suited for a different thread.

Thanks for all the help thus far – this will unclutter my earlier code a great deal!

jimmy

[Ben]

---

jamoore84 wrote:
Ben,

I’ve had some time to look at the code and I have a few questions.  I’ll break them questions down case-by-case:
1.     Initialize Case:  No questions
2.    Record Start Case:  Why do you have an inner case structure to differentiate between     “Recording” and “Idle” cases?  I cannot think of a situation where you would enter the     “Record Start” case when already recording.  The enabling/disabling of buttons should     account for that.  This leads to my next question…
3.    Playback Start Case:  If there was a nested case structure to differentiate between     Recording and Idle inner states for the Recording Start Case, why not again in the Playback     Start Case?
4.    Record Case:  No questions.
5.    Playback Case: Is there a reason why this case is missing the AE state check that the     Record Case has?  Also, is there a reason to transition to the ‘Check State’ case?  Does     this offer anything I’m not immediately seeing? Can I replace the “Check State” case  with     having the Playback state call itself(until the EOF error occurs)?
6.    Close Record File & Close Playback File – No questions.
7.    Check State case:  Inside this case there is a case structure to check the state of the AE.      Why should the “Check State” case do anything more than output what state the AE is in?      Also, I do not immediately see why there would be cases for Idle and Playback, but not for     Recording.  Is there a reason why you would not want to call the Record case if the AE     state was at “record”? 

In a previous post you mentioned that you would move the read functionality into the Check State Case.  Is this so that you could just call “check state”, and then output playback data if it is playing back?  If so, that would explain why, from “check State”, you call the playback state if the AE is at “Playback”.  I may have just answered #7 myself.

Lastly, If I click ‘Record Start’ and let the file dialog box sit, it appears that the generated data is being stored in some buffer; it’s as if labview starts caching data the moment I click Record Start.  Where is this being cached?  How large is the buffer? 

😎

 I could this causing a problem if the dialog box is left open for a long amount of time.  In the real program I’m revising, there is potential for file streaming rates at/near the HD bandwidth limit.  I can forsee this data accumulation during the “Record start/record commit” period becoming a problem.  However, this question effectively takes us away from Action Engines and might be better suited for a different thread.

Thanks for all the help thus far – this will unclutter my earlier code a great deal!

jimmy

---

2)

That is defensive programming in action. By putting that code inside the AE I was defend the AE from misuse. Relying on the enable disable of controls that are not part of the AE would be opening myslef up for trouble.

3)

I was already planning on moving that code and eleminating that action.

5)

I believe the tunnel for that case that controls iterating was un-wired so only that state was executed. This waht part of my quick hack.

7)

If I just  used two action one after the other to first check the sate and if palying back then doa a playback, there would be apossiblility that the state could be changing between when I checked and when I did the playb-back. By chaining those actions together, I enusred that I would not run into this situation. I think you answered your own Q.

Your data is being buffered in the un-read Q. Unless you limit the Q size when you create it, the size is limited my memory.

😎

The dialog being in the AE, hung the AE durring the dialog and threre stalled all other callers. Either move the dialog out of the AE (good idea not to have dialogs in high performance AEs anyway) or flush the queue to toss old readings.

Please clean-up that code and feel free to mod it to fit your needs. I would feel bad if you used that code as is in any real application. Go back through that proceedure I posted earlier and rewrite that code properly.

Have fun!

Ben

[jamoore84]

---

 

😎

The dialog being in the AE, hung the AE durring the dialog and threre stalled all other callers. Either move the dialog out of the AE (good idea not to have dialogs in high performance AEs anyway) or flush the queue to toss old readings.

Please clean-up that code and feel free to mod it to fit your needs. I would feel bad if you used that code as is in any real application. Go back through that proceedure I posted earlier and rewrite that code properly.

---

Ben,

I've spent some time with the code and have been able to get it working the way I would like.  Since this is my first exposure to Action Engine's, I'm not sure where the weak points of ths implementation lie.  I've added case structure states and user-defined error messages to let the user know if the action engine is accessing cases when in an incorrect state, but the only performance change I can think to make is your already-suggested  moving of the the AE Playback case code into the Playing case of the inner case structure inside the AE Check State case.

You had also mentioned other improvements - such as removing the playback start case (in response to my question 3).  Could you explain where this would go?

Did your misgivings for using this code in a "real application" stem from the  lack of robust code, VI performance, or both?  The action engine itself is the only piece I am concerned with.

Thanks again,

jimmy

PS:  How can I separate the file dialog prompt from the AE during the Record Start and Playback Start cases?  The only method that comes to mind is to stick the "File Dialog" function in its own sub VI.  is there a better way?

[Ben]

---

jamoore84 wrote:

---

 

😎

The dialog being in the AE, hung the AE durring the dialog and threre stalled all other callers. Either move the dialog out of the AE (good idea not to have dialogs in high performance AEs anyway) or flush the queue to toss old readings.

Please clean-up that code and feel free to mod it to fit your needs. I would feel bad if you used that code as is in any real application. Go back through that proceedure I posted earlier and rewrite that code properly.

---

Ben,

I've spent some time with the code and have been able to get it working the way I would like.  Since this is my first exposure to Action Engine's, I'm not sure where the weak points of ths implementation lie.  I've added case structure states and user-defined error messages to let the user know if the action engine is accessing cases when in an incorrect state, but the only performance change I can think to make is your already-suggested  moving of the the AE Playback case code into the Playing case of the inner case structure inside the AE Check State case.

You had also mentioned other improvements - such as removing the playback start case (in response to my question 3).  Could you explain where this would go?

Did your misgivings for using this code in a "real application" stem from the  lack of robust code, VI performance, or both?  The action engine itself is the only piece I am concerned with.

Thanks again,

jimmy

PS:  How can I separate the file dialog prompt from the AE during the Record Start and Playback Start cases?  The only method that comes to mind is to stick the "File Dialog" function in its own sub VI.  is there a better way?

---

Hi Jimmy,

THe code I posted was intended to help you and others learn about exploiting AE's and was unfortunatly a rush job. I outlined earlier the process I'd go through to do this job myself. If you do this I believe you will find

that many of your actions are not longer required, so they should be tossed,

 the seperate ref's for record and playback are not req since you will never be doing both ath the same time,

BUT....

I am still not sure an AE is the one and only answer!

I'd probably still use the AE but it would only be used in the logging loop. All of the AE actions that  communicate start and stop file spec etc may not be required if you go to a Master/Salve architecture.

Queue to transfer commands from GUI loop to logger.

The queue data should be a command (type def'd enum) with commands like "Start Logging" Stop logging" Set Log FIle path",....

Alos include a file path so you can pass the file spec.

The Event struture for start loging queeus up a command to the logger that says start logging. The logging loop can prompt for the file namae and then invoke the AE.

Backing up...

THe AE as we have it structured at my last update was used for logging AND communications. Rather than do both jobs use the queues to pass the commands, and let the logger deal with the AE.

AS far as not using it in a "real application"

I develop auditable code for a living. If I know it will not pass VI Analyzer scrutany as well as peer review,  and thorough testing on my part, then it is not ready for a "real applicaiton". The code I posted for you would get beat-up by the VI Analyzer, peer review, and my short tests already showed there was an issue when the play-back ended.

Next post will try to illustrate just one of the methods I have used....

Ben

[Ben]

Here are some screen shots of an app that used queues for commands and the AE shared the data amoung many clients (about 12 so Queues were not used to pass the data).

DAQ always keeps an AE update with the most recent set of readings.

Logger watches for commands coming in via a queue and changes it running state (idle or logging) based on the command

 

If no command is waiting and it is time to updat the log then read from the AE and write to file.

Again, the design analysis is important if you want to get the interaction right.

I sure hope this helps!

Ben

Message Edited by Ben on 03-25-2009 08:21 AM

[jamoore84]

---

The Event struture for start loging queeus up a command to the logger that says start logging. The logging loop can prompt for the file namae and then invoke the AE.

---

It almost sounds like the logger loop could become it's own AE, whose sole function is to initialize, open, and close file references.  I assume your suggesting to make the logger it's own While Loop, building off of the Master/Slave architecture you mentioned above.

How do you enqueue type-def enum data in a queue?

[Ben]

Not near a machine with that code any more so...

I use a queue whos data is a cluster. In the cluster I place the enum (commands) anlong with any other data I need to pass (file spec, or Sample rate). THat cluster is used when the queue is created so when its time to pass a command I queue up the cluster the way you were queueing (an array?).

Ben

[jamoore84]

Something like this?  It updates rather slowly - what am I doing wrong?