File-system options and data safety

[QFang]

Hi,

I hijacked a thread that was related, but I think this topic deserves its own thread.

I'll try to summarize the relevant content of the other thread here.

Essentially, the old style cRIO's by default use the (seriously outdated) Reliance file system from Datalight. (Datalight has since moved to Reliance NITRO but no plans exist for NI to bump to that version, citing concerns over backwards compatibility).

For attached storage (SD cards, USB sticks) on the new Linux based targets, you have a few choices of file-systems:

• FAT (16 and 32)
• ext2
• ext3

The only one of those choices that are even remotely "safe" for use in an application that can and will experience hard stops (power suddenly removed, user pressing physical reset or restart button) is ext3, and it is not as safe as one might think, in fact it was never designed for use in environments where the system could go down "mid-write" as it were.

To someone like myself that deal with systems in the oil and gas industry, data safety is very important, and our cRIO systems are operating in a headless (no PC side application), stand-alone manner.  We typically need to carry up to several GB's of historic data logs.  In our current legacy systems, we use the 9014 (2GB onboard Solid state), but the newer devices that we wish to migrate to typically only have ~512MB onboard storage, offering instead SD card slots for expanded storage.

It is very much desireable (for us anyway) to see support for more robust file systems to be supported out of the box for the new linux based RT targets.

ext4 would offer a slight improvement over ext3, but it still leaves me a bit skeptical.

Does anyone have experience with ZFS?  -Is it even possible to extend/add ZFS support to the NI RT Linux version (ZFS for attached storage, not talking about changing the "internal" system (UBIFS I think?))?

It seems likely that we could extend and subsequently mount SD / USB using Datalight Reliance NITRO, the problem with that path is mainly one of cost as their SDK runs in the 4-digit USD, and the first digit 'aint 1 or 0, and then they also have deployment level licensing, so for us it has proven cost prohibitive, though the geek in me would LOVE to be able to use Reliance NITRO as it seems awesome and I trust it from all the years we have used Reliance (on cRIO's Reliance is default and presumably paid for by NI).

[BradM]

1. ext4 has possibly *worse* issues than ext3 regarding data integrity.
2. For the past releases, NI has released the source code for the kernel as it was used to create the kernel that is used with these targets. It is not for the faint of heart, but you strike me as someone who has technical experience to be successful in this endeavor. You may want to consider patching with ZFS support (or reconfiguring the kernel to support FUSE and use ZFSFuse) if data integrity on a system with such unstable power is paramount.
   • An unfortunate situation that you can run afoul of: changing the 2013/2013sp1 kernel will in all likelihood leave the NI hardware (including the FPGA) unusable since the kernel modules will no longer load. This has been fixed (with some effort on your side) with 2014 kernels. Installing more recent RIO software and doing a custom install will give you the more recent kernel, even with LabVIEW 2013/sp1
3. You may also want to consider a battery-backing solution (and sensing circuitry to alert the controller to the situation) to give the controller a chance to bring things to a safe state prior to power being lost. Note that you don't really need a huge battery, just enough to give the system enough time to unmount the drive.
4. Keep in mind, with a lot of the flash parts out there, the hardware-provided FTL may be the enemy that no filesystem can combat regarding corruption on surprise powerloss. You should be working with the vendor of your flash parts to get some level of verification of behavior on certain commands, etc. etc.

[Andreev_S.V.]

Hello QFang,

I have had exactly the same task with test results integrity in any power

failure. A complete solution has been achieved by the following approach:

1. Create a big single file at the beginning of the test. It should be

large enough to keep ALL the expected test results.

2. Fill that file by all zeroes, flush buffers close the file and reopen it

for read/write.

3. Do not resize file during tests.

4. Write test data to that file in chunks (like by 4Kb), following by

flushing buffers, closing and reopening the file.

5. You could add some header/ footer in each block to simplify processing

of stored test data by end users.

6. CRC is recommended for each written data block. You could place it in

the header/footer.

7. File system must be journaled. This approach does not work on FAT.

I have been testing this method on NTFS for a long time and by applying

severe power losses. There were no problems during tests both in lab and at

the clients' factory.

BTW, it would be great to have an official support of NTFS in NI Linux RT.

Are there any plans from National Instruments on this?

[metux]

(Datalight has since moved to Reliance NITRO but no plans exist for NI to bump to that version, citing concerns over backwards compatibility).

Correct, there's no kernel support for that yet. We have bttrfs (so nobody was bored enough to write a nitrofs driver).

For attached storage (SD cards, USB sticks) on the new Linux based targets, you have a few choices of file-systems:

• FAT (16 and 32)
• ext2
• ext3

That's bad, that NI still didn't enable vital/standard filesystems like bttrfs, ext4, ...

in fact it was never designed for use in environments where the system could go down "mid-write" as it were.

Actually, that was one of the main design goals: extend ext3 w/ journalling for exactly this reason. But that's long ago, ext3 is old legacy, we have ext4 since aeons.

To someone like myself that deal with systems in the oil and gas industry, data safety is very important, and our cRIO systems are operating in a headless (no PC side application), stand-alone manner.  

Interesting. I've evaluated the cRIOs for exactly that area (eg. needed atex certification, etc). But I had to find out that the devices are completely useless for us. No usable drivers, no documentation, not even a stable bus interface to the cards, no help from NI (even after several mail conversations and calls) - so we dropped NI from the vendor list. This company doesn't want to sell a thousand cRIOs ...

We typically need to carry up to several GB's of historic data logs. 

Don't try it w/ the cRIO. Instead send the data to some extra server outside the danger zone. Making sure the cRIO is running reliably (including fully automatic re-deployment, etc) is hard enough, not even speaking about HA storage ...

ext4 would offer a slight improvement over ext3, but it still leaves me a bit skeptical.

Use bttrfs.

You'll have to rebuild your kernel. Yes, that's the point where deployment and management of these boxes get relly ugly (w/ a standard GNU/Linux system, eg. Debian, or something ptxdist-based, it would be pretty trivial). 

By the way: https://forums.ni.com/t5/Additional-NI-Software-Idea/cRIO-image-development-kit/idc-p/3819979

Does anyone have experience with ZFS? 

Yes. But unless you're dealing w/ legacy volumes (or shared w/ solaris machines), it's not worth the effort. We currently only have userland implementation (FUSE-based) - ugly to operate, especially in embedded (= operator-less) environments. But we have bttrfs.

Is it even possible to extend/add ZFS support to the NI RT Linux version (ZFS for attached storage,

Theoretically possible. But why introducing all that complexity ? We have bttrfs.

not talking about changing the "internal" system (UBIFS I think?)?

Tricky. You'd need a different boot/rootfs partition (or initramfs), which complicates the boot process and increases bootup times.

It seems likely that we could extend and subsequently mount SD / USB using Datalight Reliance NITRO,

Even worse: they only offer a userland library. IOW: you have to change your application. And it's bad for performance and RT scheduling. There free code is very limited anyways. And the commercial version is really expensive, and I doubt you'll get the full source code. Linux drivers w/o source code = completely unusable.

[metux]

ext4 has possibly *worse* issues than ext3 regarding data integrity.

This was an ancient bug, back when ext4 was considered unstable. Fixed aeons ago. Don't spread FUD.

You may want to consider patching with ZFS support

Don't try OOT modules, if you're not a kernel hacker and willing to do all the maintenance work. Especially not for industrial systems w/o operator.

Installing more recent RIO software and doing a custom install will give you the more recent kernel, even with LabVIEW 2013/sp1

Still an old, unmaintained one. If a device doesn't run w/ mainline kernel, and you don't have a kernel hacker at your disposal, don't buy it. Period.

NI folks have proven themselves to be rookies in the Linux world:

They don't even know how to build rpm packages properly (w/o creating a massive vulnerability) ... http://seclists.org/fulldisclosure/2018/Jul/67