Calculate size of folder (including sub folders) quickly

[Darren Rose]

Hi

Found this project and I am wondering how it could be modified to get me just the size of a given folder (including sub folders) quicky?

At the moment I am using it with GetFiles or EnumerateFiles which is certainly quick, but I then have to loop through the IList or IEnumerable to total the .Length of each file found which slows the process down somewhat

Any thoughts please?

[Opulos Inc.]

In the internal FastFileInfo constructor, you could try commenting out everything except for the this.Length = ...; line, and see how the speed compares. In addition to that, you could also try eliminating creating new FastFileInfo objects, and just maintain a long TotalLength value. Other than that, you could search the WinAPI to see if there's a DLL call that only gets the Length and none of the other file properties, but I'm not sure one exists. If you do some performance testing, please post your results here.

[Darren Rose]

Hi, thank you for your response

Commenting out the other lines other than 'length' doesn't really make any difference to speed.

Your code is extremely quick at getting me a list / enumerable of all files in a given directory e.g. (using VB i'm afraid!) running either of these on C:\Windows takes about 16 seconds.

Dim files As IEnumerable(Of FastFileInfo) = FastFileInfo.EnumerateFiles("C:\Windows", ".", SearchOption.AllDirectories)

or

Dim files As IList(Of FastFileInfo) = FastFileInfo.GetFiles("C:\Windows", ".", SearchOption.AllDirectories)

And a bonus is that with your code it handles the errors such as System.UnauthorizedAccessException I would get if trying to use standard Directory.EnumerateFiles / DirectoryInfo.EnumerateFiles etc

But now I have a list (or enumerable) of all the files in chosen location I have to loop through that list again to then add up all the sizes (length) e.g.

Dim size As Long

For Each f As FastFileInfo In files
size = size + f.Length
Next f

This then takes just as long again if not longer than it did to get the list

What would be nice is if the initial FastFileInfo.EnumerateFiles could be totalling the size as it generates the list, hence giving me the value I need and doing it much quicker!!

Thoughts please?

[Opulos Inc.]

Something in your testing is not correct. Summing up the total length takes a fraction of a second. E.g:

        DateTime now = DateTime.UtcNow;
        IList<FastFileInfo> list = FastFileInfo.GetFiles(@"C:\Windows", "*.*", SearchOption.AllDirectories);
        double ts1 = (DateTime.UtcNow - now).TotalSeconds;
        now = DateTime.UtcNow;
        long totalLength = 0;
        for (int i = list.Count - 1; i >= 0; i--) {
            totalLength += list[i].Length;
        }
        double ts2 = (DateTime.UtcNow - now).TotalSeconds;
        System.Diagnostics.Debug.WriteLine(totalLength + "  " + ts1 + "  " + ts2 + "  " + list.Count);

• totalLength = 30,104,229,635
• ts1 = 188.271
• ts2 = 0.004
• numFiles = 154,344

Windows caches all the file property information, so I'm looking into how to clear the file system cache safely before more tests can be done.

[Darren Rose]

I have run the tests multiple times on two different computers, with reboots in between tests and it is always roughly the same...

[Opulos Inc.]

To clear the file system cache I used:
https://docs.microsoft.com/en-us/sysinternals/downloads/rammap
From the menubar: Empty > Empty Standby List

TotalLength ts1 ts2 NumFiles
Test #2: 30104229635  131.5281854  0.0156138  154344
Test #3: 30104229635  148.1681834  0.0156301  154344

commenting out everything except assigning Length (no improvement):
Test #4: 30104229635  152.9615534  0  154344

The FINDEX_INFO_LEVELS parameter passed into the FindFirstFileEx(...) method doesn't have an option to only query the file length. So at this point, ask your question on a forum like stackoverflow.

[Darren Rose]

Okay thanks, think my testing code may well have been the issue. But handy to know how to clear the cache so I don't have to keep rebooting between tests :)

[Darren Rose]

Confirmed it was my test code, well actually the ListView control that was showing my folders and the size, for some reason it was firing multiple times for each entry which was then confusing the results - using your example above outputting times to console allowed me to quickly see it was firing more than it should causing the delay. Thank you for help, your code is working perfectly for me now