NTFS alternate data streams

[ilyak]

Hello:

I'm trying to write code that works with alternate data streams in NTSF. Similar to what this example from Microsoft does. The attached VI does a simple example code - open the second stream 'metadata' and write something there. However, LabVIEW (2009 SP1) open VI always returns error when I try to open the stream. 

So, is this known that LabVIEW (at least 2009) cannot work with NTFS streams? If it can, please tell me how.

Thank you,

-Ilya.

[Ben_J1]

Ilya,

   Is there a way you could post a screenshot of the error? Also the file you are trying to open? This would help people help you.

Regards from Austin,

[SimonH]

Ilya,

I don't think LabVIEW's file I/O VIs will allow you to directly interface with ADS in NTFS.  The :stream syntax doesn't look like a normal file path and that is what would generate Error 1.

There are alternate ways though, here's a couple I know of:

1. If you want to store text data you can use System Exec.vi and a short command to create your ADS.  Here's your attached VI rewritten this way:
   
   
   
   
2. Use an ActiveX/.NET or DLL based implementation.  If you are looking to set the standard "Title", "Subject", "Author" type metadata you might find this link useful: http://digital.ni.com/public.nsf/allkb/9A6985F314D04FA48625732400719E54?OpenDocument

I hope this helps.

~Simon

[ilyak]

Hi Simon:

Yes, I figured that core LabVIEW uses standard C library functions that probably don't support NTFS specific stuff.

1. I can use the console redirection tat you suggested.
2. I can call CreateFile directly from kernel32.dll.
3. The dsofile.dll can be used also, as described in this post. However, I keep running into some weird effects, especially if the files are not MS-Office files. One of the problems is copy:

 Maybe it depends on the target file system. However, none of the Office files exibit the same error.

Do you know what might be cuasing this and what can be done to fix it?

Thank you,

-Ilya.

[ilyak]

Hi Ben:

I believe it is Error -1 - invalid file name.

-Ilya.

[SimonH]

Office files are special cases where they actually don't use an ADS to store that data -- I believe it's part of the DOM for the file and the Windows Explorer integration knows how to read the data out.

Dsofile.dll will first attempt to store metadata inside a file using OLE (which is what it does for Office docs) and if that fails it then uses an ADS.  That would explain why you are seeing different behavior for non-OLE documents (like text and wave files).

Any data stored in an NTFS ADS will obviously be lost if you move the file to another file system (FAT or anything natively Mac/UNIX) or even if you use certain programs that are not aware of ADS (like most email clients).  Do you know if you were copying the file to another NTFS-formatted drive?

Personally, I'd be scared that I'd inadvertantly lose the information if I stored it in this fashion since ADS unaware applications may fail to copy the data without telling you.

~Simon

[ilyak]

Hi Simon:

1. Yes, it is likely that I was copying files from NTFS to ext3 or something like that.
2. Ah, using ADS (even with dsofile) would have been such an elegant solution. Basically, I have a load of binary files. The idea is to keep the files simple and just contain the pure data in order to maximaize the read performance. However, I'll need to store, somewhere, the information about the file contents. Making the file names formatted is one way, using this metadata is another.

I suppose using DSO is one way. Just have to be careful during file transfer.

-Ilya.

[SimonH · Accepted solution]

Ilya,

I agree - ADS is tantalizing close to a very elegant solution.  I hadn't actually looked into using them for something useful -- I've always considered them a security flaw and a hold-over from the old Mac resource fork days.

Another suggestion that may or may not work for you:

Have you looked into TDMS files?  They were designed to be a solution to this type of problem - metadata is stored along with the binary data so you don't have to sacrifice performance for the convenience of keeping everything together.  I believe the way LabVIEW implements TDMS read and write is faster than you'd generally be able to acheive since we are able to bypass some of the regular windows caching etc.  TDMS is not specific to LabVIEW though so you can open the files in Excel or offline analysis tools such as DIAdem.

You can find more info about TDMS here if you're interested - http://zone.ni.com/devzone/cda/tut/p/id/3727

Anyway, best of luck.

~Simon

[ilyak]

Hi Simon:

Thank you for the point to TDMS. I didn't know about this solution. It might actually do everything that is needed. I was actually leaning towards just allocating some fixed chunk in each data file and just storing my metadat there. Then reading the actual binary data can be done very efficiently from some fixed offset. But if NI has the ready-made solution - even better.

Thanks for your help.

-Ilya.