Overcoming maximum path length

[BowenM]

Here's the short version:

I need to run some analysis on historic data saved as .TDMS files. That data is massive in size (several TB of data) and archived deep into some share that occasionally causes the path to be > 260 characters.

My options are:

1. Move all the data somewhere else with a shorter path, run my analysis.

2. Move individual files that exceed max path length and run those separately.

3. Find a way to overcome the max path length restriction.

I'm probably going to be stuck with option 1 or 2... but either would be a long, very time consuming process riddled with bureaucracy. So this post is my "hail Mary" hoping someone has a clever way around the need to move my data elsewhere.

[Hooovahh]

Modern file systems allow mapping locations to another one and I suspect this could work around your issue.  If you have a long path like "C:\Documents and Settings\Users\Hooovahh\Documents\Subversion\Projects\LabVIEW Reuse"

You can map that to an empty folder like "C:\LabVIEW Reuse" using mklink.  Then the folder on the root of your C drive will be mapped to that far off location, and it should work just fine.  I'm not sure if it needs too be a hard link, or a junction but you can play around and see what works.  Just make sure you delete the symbolic link when you are done, and not the actual files.

[Kevin_Price · Accepted solution]

Just had to do a little something like this to enable a build with a deeply nested structure of long-named inheritance trees.

I told my colleagues I fixed it with a little DOS-FU.

1. Open a CMD shell prompt (or likely PowerShell too)

2. issue the SUBST command.   Start with SUBST /? to get the online help

3. You'll end up with something like:

SUBST X: "C:\Data\Project Long Names\SubFolder\..."

Thereafter, Windows will mount that long pathname as the virtual drive letter X, and you can refer to files with a much-shorted name like "X:\File0001.tdms"

-Kevin P, old timer

[AnalogKid2DigitalMan]

Not sure if this would help:

https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

-AK2DM

[Petru_Tarabuta]

Are you sure that it's an issue? In LabVIEW, do you get an error when opening the TDMS files that have a path longer than 260 characters?

My understanding is that the 260 character max path length is a limitation of File Explorer, not of the NTFS file system itself. In other words, the NTFS Windows file system can deal with long paths. But File Explorer throws an error when trying to create a new file at a long path (e.g. when pasting a file copied from somewhere else).

For example, the robocopy utility doesn't suffer from the limitation. You can use it to copy or move files and/or folders into long path locations. The following article contains more information about robocopy: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy

In short, perhaps LabVIEW can already open the files?

[paul_cardinale]

Within LabVIEW, the limitation is no more than 255 characters per path element. I don't think there's a limit on the number of elements.

[BowenM]

@ AK2DM, Petru_Tarabuta, and paul_cardinale  - read here. It is a real constraint, and LongPaths are not supported.

I do think mklnk or subst will work though!

My first attempt was just mapping the share through net use, but it only saved me a few characters and I still had a lot of files that exceeded the limitations. Thank you all

[Yamaeda]

Do a hard link with "Subst V: c:\project\sdlkjfsldsdf" (or similar) and it'll shorten your path. 🙂

[billko]

---

@Kevin_Price wrote:

Just had to do a little something like this to enable a build with a deeply nested structure of long-named inheritance trees.

I told my colleagues I fixed it with a little DOS-FU.

1. Open a CMD shell prompt (or likely PowerShell too)

2. issue the SUBST command.   Start with SUBST /? to get the online help

3. You'll end up with something like:

SUBST X: "C:\Data\Project Long Names\SubFolder\..."

Thereafter, Windows will mount that long pathname as the virtual drive letter X, and you can refer to files with a much-shorted name like "X:\File0001.tdms"

-Kevin P, old timer

---

OMG - I had completely forgotten about this command.  It was so helpful back in the day.

[rolfk]

---

@paul_cardinale wrote:

Within LabVIEW, the limitation is no more than 255 characters per path element. I don't think there's a limit on the number of elements.

---

As Bowen pointed out it is more complicated than this.

LabVIEW paths itself have technically two limits. Each path element only can be at most 255 characters, and there are at most 2^15 path elements possible as the according integer that encodes the number of path elements is in fact a 16 bit integer.

The NTFS system driver (and the according OS disk manager) has a limit of 2^15 (UTF16) characters for paths. Sounds all peachy but it isn't.

Microsoft Windows until recently only supported those 2^15 characters when using the Unicode system APIs. The 8-bit ANSI disk APIs only support 259 character long file paths. There is nowadays this "experimental" registry setting that enables long path names even in the ANSI API (basically it actually disables the path length check on those APIs and the APIs are all supposedly rewritten in a way or at least reviewed to properly translate even longer path names to the according Unicode path before eventually calling the Unicode API anyways.

The problem is that the LabVIEW path needs to be converted into a platform specific string before calling the system APIs and LabVIEW has hardcoded limits in those conversion routines that will simply refuse to try to convert longer paths into a platform specific string, since the used underlaying OS API anyhow doesn't support longer names (for over 30 years until recently). Changing that would be possible but is a lot more complicated than just removing that check in the LabVIEW routines, since most of those conversions are done on the stack. That was a prudent thing to do, since stack space is completely free in terms of runtime performance but a scarce resource that can't be wasted for 32K buffers every time. Since paths couldn't get longer than 259 bytes anyways, this was however an easy to do sacrifice.

To support such huge potential paths the code will have to be rewritten to:

- preparse (with according transformations necessary for the actual platform) the LabVIEW path to determine how long the resulting path will get

- allocate a large enough memory buffer 

- reparse the path and format it into the memory buffer

- call the Windows API

- deallocate the memory buffer

This is costing extra runtime performance, for pre-transform and memory allocation and deallocation but more importantly is pretty much a complete rewrite of the according path conversion routine. And it's not just a single function as the whole path handling is a little more complex than that it could be all handled everywhere with one function only.

So while LabVIEW paths could theoretically handle paths that are up to 2^15 * 255 characters long, and Windows nowadays can handle 2^15 character long paths (unless you try to access a FAT(32) volume or some old  network share), there is this LabVIEW to Windows translation that can not.

In my opinion this should have been long ago changed already and by directly accessing the Windows Unicode APIs for these functions. In that case LabVIEW could have supported long path names already 20 years ago (and would be prepared to support Unicode path names once Unicode support is fully available in LabVIEW). The extra overhead of dynamic memory allocations for every path when having to call Windows APIs using a path, might have been a valid concern back when LabVIEW was ported to Windows 3.1, but has since been long marginalized by better memory managers and CPUs that perform a million times more operations per second than back then.

Use of symbolic links is a valid and working option to circumvent long path names. Unfortunately they are still a suspect entity under Windows unlike Unix platforms who know them pretty much since inception. In order to create or delete symbolic links under Windows, a user needs to have elevated rights or one must enable a developer feature in the Control Panel settings which allows creation of symbolic links as normal user. Creation of hardlinks or mount points requires in any case elevated privileges, which I can understand up to a certain point as they are a pretty deep going modification in the actual filesystem.