Upload to S3 Timeouts

[ChaseP]

Has anyone here had issues uploading large files ( >1Gb)? I have tryed using both the upload by parts and singles upload vi's but i still receive Error 56  from the LabviewHTTPClient:PUTBuffer.vi.

[ChaseP]

Here is a screen shot of the error i am getting. I have found I can avoid this by reducing my "chunk size" but then i get a different error on Upload Part .vi if my chunk size is below 5120

[peter_smith]

Hi Chase!

Could you please provide some more details? Giving the following information might help to solve your problem easier:

• Screenshot of the Block diagram, or the VI itself
• Where does the bucket reside (geographically) you try to upload to?
• Do you see the same error code when you modify chunk size?
• Did you always use the Put Object from file (Multipart).vi?
• Does the method you use work for smaller, but large files (eg. 800MB)?

Regards,

Peter

[ChaseP]

Hi Peter,

Thanks for the quick reply.

• The VI i was was using was "Upload a File.vi" from the Labview s3 examples.
  
  
• US Standard
  
  
• If I reduce the Chunk size bellow 5120 KB, I don't get the time out error, however i get a different error from "Complete Multi-part Upload" saying the chuck size is too small.
  
  
• Currently I only use Multipart upload for files larger than 5 Mb
  
  
• With more testing I found this error happened even with files that are only 50 MB.

I have found a work around by increasing the HTTP timeout value located in "LL_PUT Object (Core).vi"

After doing this, I can upload any size file I need.

[peter_smith]

Chase,

I've just checked the upload to S3 using the same example VI. Upload of a 980 MB file with chunk size set to 5120 kB (default) and a 5 GB file with chunck size set to 20480 kB both worked.

Setting chunk size to lower than 5120 kB is not allowed by Amazon itself, so that's why you get that error.

I'm happy you managed to solve the problem by setting the timeout on the PUTBuffer.vi itself. I did not wire it in the toolkit, because the default setting of 10 secs should be more than enough for any webserver to respond. Well, it looks like I just was wrong. Such a large timout is very strange however, so I suspect there might be some other problems with your setup (network, Internet provider, etc...).

Just as a reference, could you try the same operation at some other place (eg. with a different internet connection)?

Regards,

Peter

[chops88]

I'm was having the same issue: small files (< 5MB?) would upload without issue. Anything above that file size will return multiple 56 errors. I was only able to get it and others to upload by changing the timeout in Put Core to >16000 and keeping the chunk size back at 5120. Then in watching the probe window, the result was coming back from Http Put at 16sec intervals. A little math and speedtest.net - my upload speed clocks in at 2.5Mbps which = 312.5KBytes/s, and 5120KB/312.5KBs = ~16seconds. Maybe the timeout needs to be hard-coded much higher for slower connection speeds? 10 second timeout is 512KB/s for 5MB or 4Mbps - pretty fast upload speed for most people. Or am I missing something?

[peter_smith]

Hey chops88,

Pretty ncie debugging! I looked at the PUT Core VI and you were right with your assumptions. The problem truely comes from the default timeout of the HTTPClient's PUT.vi combined with your connection speed. I really overlooked this when designing the toolkit, as I have a 100/80Mbps connection at home, so I never ran into such problems. Now I tried it here at work and I've also been able to reproduce it.  As a workaround altering the toolkit VIs one can change the timeout to suit their needs. In the next release I'll use a higher default timeout value and wire out that input to outer VIs as well.

Sorry for the inconvenience.

Regards,

Peter

[chops88]

I was actually thinking "this guy must have the most INSANE upload speeds and quite possibly be from the future". I can't purchase upload speeds like that in my town. Out of curiosity just had a peak and if I lived in a bigger city I could get 300/300 for $338/mo haha.

[phelhe]

Can you explain more in depth how to do this? I am using internet that is sometimes fast and sometimes slow, and when it is slow I get an error even if the file is not that big (15mb). I found the PUT Core VI but I do not see where to change the timeout. Thanks!