GZip De-Compression failure, possible buffer oversize? File unzips in 7zip but not with the OpenG library.

[eli.barber]

https://forums.ni.com/t5/Example-Code/GZip-Compression/ta-p/3534832

Using this code, gzipped file only partially uncompressed. Its a large XML File. I am wondering if because it works partially that means there is some sort of buffer ratio I can change within the OpenG library?

The file unzippes perfectly when using 7zip. But I need a way to extract this using Labview. I cannot guarantee 7zip is installed on every machine. What other options do I have?

[DavidBoyd]

Eli,

I'm nearly certain that this issue is for @RolfK to address.  He has worked diligently over the years to keep the OpenG GZip code cross-platform, bitness-agnostic, and file-size-tolerant to the greatest possible extent.  Your post may have arrived past working hours today in the Netherlands.

EDIT: Just checking, are you using the latest release available from VIPM?  I see v5.0.4-1 released 09 September 2024

Dave

[eli.barber]

Dave,

I have updated to the latest version in VIPM, v5.0.4-1. Unfortunately, I am still getting the degradation issue. Let me know if there is anything else you need.

[tst]

---

@eli.barber wrote:

...I need a way to extract this using Labview. I cannot guarantee 7zip is installed on every machine. What other options do I have?

---

Don't know about your actual question, but it should be possible to include 7-Zip with your program without installing it and call it from LV with the relevant command line switches (or call into the DLL, but that would be harder). If you go this route, I suggest you start with something which lets you test easily (e.g. call a custom command line in a custom directory) and if that works, build the code to match.

[Yamaeda]

---

@eli.barber wrote:

https://forums.ni.com/t5/Example-Code/GZip-Compression/ta-p/3534832

Using this code, gzipped file only partially uncompressed. Its a large XML File. I am wondering if because it works partially that means there is some sort of buffer ratio I can change within the OpenG library?

The file unzippes perfectly when using 7zip. But I need a way to extract this using Labview. I cannot guarantee 7zip is installed on every machine. What other options do I have?

---

Add 7zip.exe to the project/installer so you always have it. 😛

[rolfk]

I'm sorry but the OpenG ZIP library works for ZIP files, that is not the same as gzip. 7Zip contains support for 7 compression and at least 39 decompression formats and many subvariants of them.

While I could make the OpenG ZIP library also support gzip, I haven't found the need and energy to do so yet.

The Inflate algorithm indeed has to make some estimations about the expected size of the decompressed stream. The code as presented in the provided Example code link seems to contain information about the expected end size in the header so the deflate algorithm should not run into any problems like that. If the deflate runs out of memory space it does do a number of retries with an exponentially bigger reallocated buffer. That should fix the problem unless the stream is extremely compressed beyond like 96%.

And you call about degradation. Are you saying it did work correctly in the past? I never tested it like that and would be surprised if it worked in old versions and doesn't now.

[eli.barber]

Rolf Kalbermatter,

The file I attached in my first post is a compressed XML file specifically. 872kb compressed and 17,615kb uncompressed. It looks very close to the 96% threshold.

What I mean by degradation is that 7-zip can unzip the file entirely--And all the files XML tags appear orderly and proper.

The OpenG library, (See screenshots from 2nd post), exhibits unusual behavior. At first, the decompression looks fine (1st screenshot). As the library further decompresses, the test starts to display incorrect XML tags. Finally, (3rd screenshot) the decompressed file becomes only a gibberish of numbers.

I can't attach the file as it is >10MB, but if you decompress the file from my first post, and try to decompress with the example code, you will see what I am talking about.

[rolfk · Accepted solution]

The routine in the linked article does a number of things wrong.

First it prepends a zlib header (0x789C) to the deflate stream. This tells the inflate algorithm that it is a valid zlib stream and that it should use the adler32 algorithm for the crc calculation. But it is a gzip stream which should use the crc32 algorithm for checking. The inflate routine also checks the actual last 4 bytes in the stream to be a correct crc, so that of course will fail, which is why the routine returns -3 (DATA_ERROR) which is translated into LabVIEW error 42.

The inflate algorithm internally also would allow direct gzip stream decoding including the header but this is not currently exposed to the API. I do have an idea how that can be done but it will take a few days.

However the attached VI works on my computer (with oglib_lvzip-5.0.4-1.ogp) but generates of course the error 42 because of the CRC mismatch. So I'm curious if you still see corrupted output.

[eli.barber]

Wow that VI converts it perfectly!

I don't think I really even need the CRC check anyway. Since it is XML and when I convert it to a cluster, I should be able to tell if anything is missing?

This is perfect thank you. This helps me as I can use this on a cRIO if I decide to go that route. If not, this is still a  much better solution then calling 7zip from Command Prompt.

thank you,

[rolfk]

---

@eli.barber wrote:

Wow that VI converts it perfectly!

I don't think I really even need the CRC check anyway. Since it is XML and when I convert it to a cluster, I should be able to tell if anything is missing?

This is perfect thank you. This helps me as I can use this on a cRIO if I decide to go that route. If not, this is still a  much better solution then calling 7zip from Command Prompt.

---

The cRIO will have to wait a little more. I'm working on getting the necessary shared libraries build and put in a package that can then be installed on the NI Linux RT RIO hardware. Once that is available you can install the according ipk file to your cRIO and then deploy LabVIEW programs that use the OpenG ZIP Library to you cRIO.

I already build a 5.0.5 version that allows for gzip header streams. Just need some more testing and a few cosmetics. It's a minor release, the release with cRIO support will be 5.1.0.