Losing counts using asynchronous reads

[KrisJa]

Hi guys,

I have been working on an application to allow me to do imaging on a microscope. Using an external clock source I need to "bin" photons coming from my sample for a set period of time.

To do these measurements I devised an APD object in C#. The code to this opject is attached.

When I want to measure I call the following code to instantiate the APDs I need;

Identifier (Not Daq related, used by my code elsewhere),

Channel number (Not Daq related, used by my code elsewhere),

Device (a 6601 or 6602),

The counter delimiting bintime,

Timebase (20 or 80),

The line taking in triggers,

The counter counting photons,

The line taking in the actual photon TTLs

 this.m_apdAPD1 = new KUL.MDS.Hardware.APD("APD1", 0, "Dev1", "Ctr2", "80MHzTimebase", "PFI27", "Ctr1", "PFI39");
 this.m_apdAPD2 = new KUL.MDS.Hardware.APD("APD2", 1, "Dev1", "Ctr4", "80MHzTimebase", "PFI31", "Ctr3", "PFI35");

Where "this" is the form that hosts my APDs.

I start actual measurement like so: 

this.m_apdAPD1.StartAPDAcquisition(2, 128*128, 128);

Meaning that I will bin photons for a 2 ms period, that my image consists of 128*128 pixels and that I will read those pixels line per line. Once this method is called on the APDs they will sit around and wait until triggers come in on PFI27 or PFI31 (indicating a new position of the sample reached) and then they will count photon TTLs coming in from either PFI39 or 35...

Internally the APD object uses BeginRead and EndRead sets to read the actual photon counts. TheStartAPDAcquisition is called on the UI thread and for the Reader I set SynchronizeCallbacks = true; Therefore, also the CallBack is fired on the UI thread, which means all measured variables are updated on the UI thread (not that I think it matters here, but still).

All this works fine most of the time, however, in a seemingly random fashion it will happen that APD1 does not register all triggers coming in on PFI27 and thus never collects all 128*128 pixels (and thus times out). What is strange is that if I switch around PFI27 and PFI31 like so:

 this.m_apdAPD1 = new KUL.MDS.Hardware.APD("APD1", 0, "Dev1", "Ctr2", "80MHzTimebase", "PFI31", "Ctr1", "PFI39");
 this.m_apdAPD2 = new KUL.MDS.Hardware.APD("APD2", 1, "Dev1", "Ctr4", "80MHzTimebase", "PFI27", "Ctr3", "PFI35");

it is  still the first APD that will time out.

Furthermore, I am 100% certain the triggers are indeed coming in on these lines from my external source. Also, when I was using my APD objects synchronously in a previous version of the code (or rather, when they relied on Synchronous Reads internally), all counts also registered perfectly. Sadly enough I cannot keep working synchronously for various reasons.

The randomness of the issue makes me suspect I might be suffering from a bug related to threading/async operations where things happen in the expected order most of the time, yet not always.

This issue is a big showstopper for me so any help would be greatly appreciated!

Message Edited by KrisJa on 09-21-2009 05:01 AM

[KrisJa]

Nobody? 🙂

[Dan_K]

Hello KrisJa,

I took a look at your code and I think I understand how everything is working. I have a couple of questions.

1. You mentioned that you are missing triggers. Could you find out how many triggers you are missing in a given acquisition.
2. Could you post the behavior you are expecting to see in a timing diagram, detailing two trigger signals, the resulting gating pulse and the signal you are using as the timebase for counting.
3. And could you specify your trigger rate and the timebase you are using when you are seeing the missed triggers.

The answers to these questions will help to narrow down where your problem lies.

[KrisJa]

Hi Dan,

Thanks already for the interest. First off, I tweaked the code for my APD class just a little bit (commented out some stuff I think was superfluous and explicitly start one of my tasks). The new version is attached, together with the source to PhotoDiode, another detector class that also suffers from the problem.

I attached a scheme for the triggering/timing as well, I hope it is somewhat clear, if not I can provide more detailed info.

The APDs are instantiated on a form like so;

// Board, Binpulsegen, Timebase, Triggerline, TTLCounter, Inputline
this.m_apdAPD1 = new KUL.MDS.Hardware.APD("APD1", 0, "Dev1", "Ctr6", "80MHzTimebase", "PFI31", "Ctr5", "PFI39");
this.m_apdAPD2 = new KUL.MDS.Hardware.APD("APD2", 1, "Dev1", "Ctr4", "80MHzTimebase", "PFI27", "Ctr3", "PFI35");

Subsequently, their events are hooked up and they are started;

this.m_apdAPD1.StartAPDAcquisition(
                    _docDocument.TimePPixel,                                             // For example 2, indicating 2 ms

                    _docDocument.PixelCount,                                             // For example 128 * 128                   

                    _docDocument.PixelCount / _docDocument.ImageWidthPx); // Mostly 128, to read in 128 value chunks but can be -1 also this.m_apdAPD2.StartAPDAcquisition(
                    _docDocument.TimePPixel,
                    _docDocument.PixelCount,
                    _docDocument.PixelCount / _docDocument.ImageWidthPx);

When they are started they will sit and wait for triggers coming in from an external Piezo controller (see attached scheme). They will timeout after 5 second.

The eventhandler for the BufferUpdated event is;

 private void det_BufferUpdated(object __oSender, EventArgs __evargsE)
        {
            KUL.MDS.Hardware.IDetector _idetDetector = (KUL.MDS.Hardware.IDetector)__oSender;    // Get to props/methods of the sender
            ScanDocument _docDocument = this.Document as ScanDocument;                                 // Class that holds data

            if (_idetDetector.Type == DetectorType.APD)
            {
                //_docDocument.StoreChannelData(1, this.m_apdAPD2.APDBuffer);
                _docDocument.StoreChannelData(_idetDetector.Channel, ((APD)_idetDetector).APDBuffer);
            }
            if (_idetDetector.Type == DetectorType.PD)
            {
               // PhotoDiodes get treated differently, this eventhandler is used by all IDetectors...
            }
        }

The AcquisitionDone event is currently not used...

 Finally, my form also holds a Forms.Timer that ticks every 800ms, the Tick handler is as follows;

private void m_tmrUITimer_Tick(object __oSender, EventArgs __evargsE)
        {
            PaintToScreen();                                                       // Paint the recorded images from both APDs to screen

            UpdateUI();

            if (this.m_apdAPD2.IsDone & this.m_apdAPD1.IsDone)
            {
                if (this.m_Stage.IsScanning)
                {
                    this.m_Stage.Stop();
                }

                this.m_tmrUITimer.Stop();

                this.m_Stage.Home();

                // Handle auto-save, omitted for clarity.              

                // Enable all controls again.
                EnableCtrls();

                UpdateUI();

                // Unhook some events here... deleted for clarity.
            }
        }

I attached the code for the form where all this happens for completeness, it is more messy than my other source files though, because I am constantly tinkering it.

Finally, what I mean by missed counts is that in 8 out of ten cases, when I set out to record a 128*128 image both APDs actually accumulate 128*128 pulsewidth measurements (=photon counts for every pixel) as expected. However, every so often APD1 (the one that is started first in code) fails to accumulate the full amount of points. After 5 seconds it obviously times out.

I did a number of tests;

1. I disabled each of the APD's one at a time in code and used Measurement and Automation to check if the piezo controller was actually feeding the correct amount of triggers to the terminals used by the disabled APD. As far as I could tell (I tried 10 times for each APD) this was always the case, so the problem is not with actual missing HW triggers
2. I changed the counters and PFI's used by each instance of my APD class. This also had no effect. 8 out of 10 measurements were ok butevery so often APD1 failed ( so again the instance that was started first ) even though it was now hooked up to collect the signals that were previously going to the APD2 instance (I hope this makes sense).
3. I set the APD's up to do -1 reads (all samples available) and the amount of lost pixels is way less than 128 and appears random.

So in conclusion, the bug is quite random. Mostly my app works fine, but sometimes it does not. AFAIK the HW is fine. The randomness of it all makes my suspect something to do with concurrency (in the async calls???) but I do not see how... This is a really frustrating issue!!

Anyway, I hope my ramblings make sense to you! If you need info, please let me know!

[KrisJa]

The form that holds APDs and the general experiment actions... It might be messy though!

[KrisJa]

Hi Dan,

I will also prepare a minimal test case for you with detailed instructions. I think this is the only way to get to the bottom of this problem. I have been testing for hours now and it misses samples on a totally random basis.

I will make sure my minimal case contains all necessary elements of the larger app and that it manifests the same problem back here.

It will take some time to do this but I'll keep you posted.

[KrisJa]

Hello Dan,

I finished work on my minimal testcase. How could I make it available to you? I think it might be a bit big to put on the formum...

Cheers,

Kris

[KrisJa]

Ok, in case you' re still following up on this, please Email me at

k r i s.s p a m b o x@g m a i l.com (without the spaces obviously) and I will provide you with a link to my repro for the problem.

Cheers,

Kris

[TJWhit]

There's nothing like replying to a 5-year-old post but I was having a similar problem and your post popped up in my search. I was using LabVIEW with a PXIe-6612 counter/timer to count fringes in a Michelson interferometer. I needed to check the count so I could know the location of the travelling mirror. "Checking the count" means I sent a pulse to the counter to dump the count to a buffer so it could be read. I noticed when I put a 3 ms delay in the loop that checked the counter, few if any error occurred in a short duration count but if I let the loop go as fast as possible or I kept the counter going for a long period of time (or worse, both), errors would always occur. Further checking showed that the slower I checked the counter, the fewer errors I had.  Errors always manifested as a count that was too low - I was missing counts. 

I am convinved now that this happens when the request to read the count coincides with the actual event arriving at the counter. This is a hardware issue and the only fix I can see is to implement synchronous reads that are separated from the actual event by some minimum time. When I did this, I got zero errors. The "safe" time would have to be determined by experimentation or perhaps a call to NI. I recognize my solution my not be feasible in your experiment but hopefully knowing the cause will give you some insight how to get around it.

[KrisJa]

Hi TJWhit,

That would make sense actually.

I have long since moved to other ways to achieve similar behavior in Daqmx using C# but it is good to keep in mind that this might happen.

Really weird that Dan didn't get back to this though 🙂 he had a few years to come up with an answer.