Computer pause every 2 seconds while writing to hard drive.

[rickford66]

Running LV2010 on a W7 Pro 64 bit machine with 3.3GHz i5-2500 CPU and 8G RAM.  The machine is about 6 months old.

I have an application in which I am reading from a USB device and writing to the hard drive.  The USB device doesn't know when my application writes to the hard drive, so I really doubt the problem is anywhere in there.  My app continuously gathers and graphs data from the USB device.  The device has a buffer in it and the status of the buffer is sent up with the data in every packet.  The status is always low, as the computer has no trouble reading the buffer faster than the device can load it up.  That is, unless the PC is writing the data to the hard drive.  When that happens, it operates perfectly for about 2 seconds, then the buffer suddenly fills up and overflows, then it just as quickly read back down to zero.  This happens over and over, at repeatable intervals, which are about 2 seconds apart.  The only reason the buffer would overflow is that the PC is not reading it fast enough.  On slower computers of the past, I would see this happen as the buffer would inch its way up slowly, when the PC was just barely too slow to keep up.  Back then, it happened, recording to the HD or not.  In this case, it shoots up quickly and back down, as if the PC just took a quick coffee break.  I've even set the priority to highest setting hoping the OS wouldn't take a break, but alas, it makes no difference.  Does anyone know why a PC would do this while recording to the HD?  Thanks.

[RavensFan · Accepted solution]

Is the code that is writing to the hard drive in the same loop that is acquiring the data?  Writing to the hard drive is always going to be slower than other processes within the PC.  Post your code so we can see what you are doing.  You should be using a producer/consumer architecture to pass the data via queues from the data acquisition loop to the file writing loop.

Is it possible that this PC has some virus scanning software running on it that is taking way too long to scan every file that the PC is writing to the hard drive?

[rickford66]

I am writing to the hard drive in the same loop that is acquiring the data.  I know it's slower, but it has always been fast enough because my sample rates are not that fast.  I doubt I could post the code because it is very complex.  I doubt anyone would take enough time to completely understand what's happening, but the basic idea is, I send a USB command to send back data... then, while that's happening in the background, I process the data from the last cycle, then I read the data I just asked for, then repeat.  I've heard about this producer/consumer thing before, but I've never gotten my feet wet with it.  I am self taught, never formally trained, so there are likely a lot of holes in my knowledge.

The PC does have virus scanning software on it.  I will see if I can disable it.  In the past, it was locked out so we couldn't, but with the new machine, who knows.  I'm not saving many files... this is just one big file that is continuously being appended too.  That is, one file per test.

[rickford66]

I was able to turn off the virus protection and the problem persists.

Is it possible that the OS is not actually writing the data to the HD as I'm acquiring it, but instead is gathering up a bunch of it and writing it all at once?

[johnsold]

Who knows what the OS is doing with the file writes?

As your file grows the OS need to re-allocate space on the disk or fragment the file. Slow processes. Pre-allocating a file and then writing the data to it should avoid those problems. Or breaking the file into multiple smaller files might help.

The TDMS file VIs are optimized for streaming data to file. That might be a better way.

Lynn

[rickford66]

One other thing... the task manager shows the CPU usage at about 10%, and when the buffer spikes, the CPU usage remains the same.

[rickford66]

How do I preallocate the file?

[rickford66]

I guess I should have looked that up before asking off the cuff.  I used the set file size vi to preallocate the file, but this didn't help.  I've seen some other posts of people having trouble with high data rates.  Compared to some I've seen, I'm recording quite slow... around 80k bytes per second.  If computers these days are capable of recording many MB per second, it should have no problem with 80k bytes per second.

[johnsold]

That does seem like a rather modest data rate. 

I agree with RavensFan that a Producer/Consumer architecture is worth considering.  That allows isolation of data processing and file writing from the data acquisition task.

Although modern computers are much faster, the OSes have gotten more complex.  Some things just do not seem to get much better.

Lynn

[billko]

Certain "Green" Western Digital drives park their heads within 8 seconds or less of being idle - part of being "green," I guess - so when you do a write to the hard drive, it needs to spin up first if it has been idle for more than a few seconds.  Maybe it's something similar?

Anyways, maybe it's time to look into producer/consumer architecture.  That way the queue can absorb the overflow until the delay abates...