Writing file more 4GB

[Valentin87]

Hi.

I would like write datalog file data that contains array (size=1M of dbl data), strings and numeric. But when I reach size of file = 4GB I have an error that it's max file size. Is there a way to write such data and have output small size file? Thanks.

[GerdW]

Hi Valentin,

you can write data to a file continuously, even with more than 4GB in total.

But you should just use the basic FileOpen/Write/Close operations…

write datalog file

Use basic file operations instead of these Datalog file operations…

[wiebe@CARYA]

---

@Valentin87 wrote:

I would like write datalog file data that contains array (size=1M of dbl data), strings and numeric. But when I reach size of file = 4GB I have an error that it's max file size. Is there a way to write such data and have output small size file? Thanks.

---

That should work OoTB.

The max file size could be limited by your file system?

Are you simply attaching data, or do you keep an internal pointer? That pointer would need to be 64 bit.

Also, some express VIs might not use 64 bit pointers internally, and most of us wouldn't use them anyway. So if you do, that could be the cause.

It would help if you posted some code...

[billko]

FAT32 has a 4 GB file size limit.  Maybe that's what you're running into?

[BertMcMahan]

Are you using LV 32 bit or 64 bit? Any chance you're running into memory issues?

[wiebe@CARYA]

---

wiebe@CARYA wrote:

Also, some express VIs might not use 64 bit pointers internally, and most of us wouldn't use them anyway. So if you do, that could be the cause.

---

The Write File+ (string).vi found in the deeper catacombs of some express VIs has a 32-bit file pointer. the input is not used in the express VIs I looked at, but it might be used in some.

Post some code, so we can help you better...

[alexderjuengere]

---

wiebe@CARYA wrote:

---

wiebe@CARYA wrote:

Also, some express VIs might not use 64 bit pointers internally, and most of us wouldn't use them anyway. So if you do, that could be the cause.

---

The Write File+ (string).vi found in the deeper catacombs of some express VIs has a 32-bit file pointer. the input is not used in the express VIs I looked at, but it might be used in some.

Post some code, so we can help you better...

---

it looks like, write binary file is affected by this - I want to write an array of clusters as one file, containing the number of array elements in the filename e.g. dump_150000, dump 160000

in this scenario, when the file size is greater than 4 GB, the resulting binary file is much smaller than 4 GB

in this particular case, this limit is somewhere between 150.000 and 160.000 rows:

I hope, this is not a bug - so if this is expected behavior, does there exist a workaround?

I am coding on Windows 10 using Labview 2023 64 Bit

[alexderjuengere]

---

@alexderjuengere wrote:

---

I hope, this is not a bug - so if this is expected behavior, does there exist a workaround?

---

yes, there it is: https://forums.ni.com/t5/LabVIEW/LabVIEW-quot-Write-to-Binary-File-Questions-quot/m-p/3818462#M10788...

[GerdW]

Hi Alex,

---

@alexderjuengere wrote:
I hope, this is not a bug - so if this is expected behavior, does there exist a workaround?

---

What happens when you write to the file in chunks of 10000 rows?

What happens when you open the file once and use the file reference for all following file operations?

[alexderjuengere]

---

@GerdW wrote:

---

What happens when you write to the file in chunks of 10000 rows?

---

in my particular scenario, a dump of 10.000 rows will work as fine as 150.000 rows

---

@GerdW wrote:

What happens when you open the file once and use the file reference for all following file operations?

---

This was actually the crucial point - using  "open/create file" once and then iteratively "set file position, write binary file, get file position", keeping the file refnum from open/create in a shift register and closing it after the loop has executed.