Need CLI version. I know it`s too brazen.))

[Anonymous]

Yep, SwiftSearch is incomparably fast even on just now loaded Windows OS.
Do you plan to assemble noGUI version ? maybe with search output result list to clipboard ?
And it would be great to have an option to include in result list fullpathes only, no date, no attrib, etc.
Thank you. Cool app!

[-]

Hi! Have you seen the existing console version? It's not perfect (it has minor bugs) but it might still get the job done for what you'd like to do. If it doesn't, though, unfortunately writing a better one based on the same codebase requires more time investment than I have at the moment so I'm not sure if/when I might get the chance.

[Anonymous]

I see.
Will wait for new CLI. ))
Good Luck!

[dzmitry.lahoda]

Console version would be killer (and please rename SwiftSearch) into somthing shorter. I have renamed into search. May be ss would be good name.

When I am in NON admin cmder console I type (how to run SwiftSearch as non Admin? Everything works that way) :

ss "MyProject.log"

I get in outut

C:/Server/MyProject.log
D:/Client/Somewhere/MyProject.log

So by default search returns only paths., so I can filter this and pass into cd.

I will donate 10 USD for such feature if there will be donate button.

[-]

I added a console version. :)

Example usage:

SwiftSearch "**\Users\**\AppData\**" "C:|D:" ""

You can put "*" in the 3rd argument instead of "" to get all the fields.
As usual, you can omit the quotes if the arguments don't contain spaces or other special characters like |.
Note that the output format is preliminary and may change in future versions without notice. (For example, columns may be swapped/added/removed/reformatted, command-line argument order or specifications may change, etc.)

I'm not inclined to change the name though. You can rename it locally if you'd like.

[Robert Nio]

Hi,

thanks for the HARD work to get this done.

Q: Is there a DLL or another way to access the tool from within another tool ... right now I fork another CMD to use the cli version.

Q: How can I get the extended ATTRIBUTES of files (xrw ... etc.)

Thanks again for this wonderfull tool :)

Robert

[-]

Hi,

There's no way to call it as a DLL unfortunately. I didn't really intend it to be something to build more tooling on top of.

By "extended attributes" Are you talking about $EA_INFORMATION, about WSL file attributes (chmod etc.), or about regular file attributes (FILE_ATTRIBUTE_NORMAL)? Although, I guess I don't really have much support for most of them... what exactly are you looking to be able to do?

[Robert Nio]

Wow ... what a FAST reply :) The FILE_ATTRIBUTE_NORMAL would do.

I am working on a tool ... traversing the disk and collecting basic info for each file.

Your approach is the fastest .. having done a lot of device drivers in the past ... I guess it was not too difficult to put this tool together ( a LOT of work though).

Right now I use your command line tool ... and build a WRAPPER around it ... catching its output.

If I had a DLL … I could just call the various procedures from you … and catch the results very easily for further processing.

In the mean time I have been trying to “rebuild” your tool chain. Unfortunately, there are a lot of older tools and thus no more available (e.g. Win SDK 5.0 / 6.0A or WinDDK v.3790 etc.).

No sure if you ever updated those … any other ideas on how to replicate your build environment?

Thanks again for all your help !

Robert

[-]

Haha, it was absolutely difficult to put this tool together. ;) I worked on it for many, many years. It wasn't even in C++ in the beginning; I had to rewrite it (multiple times I believe, but my memory has faded). :-) Finding documentation on NTFS was much more difficult when I wrote it too. And getting it to have such high performance took a long, long time.

For the attributes, I'll have to see if I can add them... though I'm not sure where/how. I don't know if I even keep them all around in memory!

For my toolchain, I don't think you need older Windows SDKs? I'm not even sure if you need the DDK; I think you should be able to just do a normal Release build. Feel free to let me know what errors you see if a normal Release build doesn't work. If you want the DDK though, I think you might be able to download it here.

A DLL interface is difficult to provide partly because the in-memory representation is complex (to minimize memory usage), and I don't really want to have to provide an ABI for all the data structures (I've changed them a lot in the past, and they probably will change in the future). It's also a fair bit of work, with very few users to justify it, so you might need to make do with the current state for now...

[Robert Nio]

[EDIT]
Found “C++ Windows XP Support for VS 2017 (v141) tools”:

Visual Studio INSTALLER:

Visual Studio VS2019 "Individual components" > "Compilers, build tools, and runtimes" > "C++ Windows XP Support for VS 2017 (v141) tools [Deprecated]"

After getting the DDK as well as the c++ v141_xp build tools installed ... I still get an error for the missing ATL (? Active Template Library) ... Any Idea, what is best to get the missing items?

The error is triggered through the "stdafx.h":

For:

#include <atlbase.h>
#include <atlwin.h>

I installed (see above):
C++ ATL for v141 build tools with Spectre Mitigations (x86 & x64)

... from "stdafx.h" these are NOT resolved: ...still missing ...:

#ifdef __cplusplus

#include <atlapp.h>
#include <atlcrack.h>
#include <atlmisc.h>
extern WTL::CAppModule _Module;
#include <atldlgs.h>
#include <atlframe.h>
#include <atlctrls.h>
#include <atlctrlx.h>
#include <atltheme.h>

#endif

[-]

Ah, you need WTL for that. Try checking out this revision: https://sourceforge.net/p/wtl/code/467/

[Robert Nio]

We are making small progress ..

I INCLUDED this directory in my VS project properties (Additional Include Directories and Additional #using Directories)

C:\wtl-code-r636-trunk\wtl\Include

Here are the ERROR msg I get:

Severity Code Description Project File Line Suppression State
Error C1083 Cannot open include file: 'atlapp.h': No such file or directory SwiftSearch d:\google drive\protect\important\nim\swift_search\swiftsearch-code\stdafx.h 196

Severity Code Description Project File Line Suppression State
Error (active) E0135 namespace "WTL" has no member "CAppModule" SwiftSearch D:\Google Drive\Protect\IMPORTANT\NIM\Swift_Search\swiftsearch-code\stdafx.h 199

Many of these:

Severity Code Description Project File Line Suppression State
Error (active) E1696 cannot open source file "atltheme.h" SwiftSearch D:\Google Drive\Protect\IMPORTANT\NIM\Swift_Search\swiftsearch-code\stdafx.h 205

[-]

Ah, don't add it to those. Open Project.props in a text editor, look for lines that say <IncludePath ...>, then add the following line after them:

<IncludePath>C:\wtl-code-r636-trunk\wtl\Include;$(IncludePath)</IncludePath>

I didn't have this in the project because I have WTL in my user-level includes (globally), sorry about that.

[-]

I think you might have the wrong revision though. (Your folder suggests you have r636 instead of r467.) It might still work, but it might not.

[Robert Nio]

I was NOT able to get it working with the <includepath ...="">, line you suggested … I did NOT have any include lines in my Project.props:</includepath>

<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release (Windows 2003 DDK)|x64|FALSE'">
    <ClCompile>
      <IncludePath>D:\Google Drive\Protect\IMPORTANT\NIM\wtl-code-r636-trunk\wtl\Include;$(IncludePath)</IncludePath>
      <AdditionalOptions>-Wno-unused-command-line-argument -fmsc-version=1300 %(AdditionalOptions) /Os</AdditionalOptions>
    </ClCompile>
  </ItemDefinitionGroup>

So I just copied the WTL include files to the root source directory ... and changed the <include> to "include" in the "stdafx.h" file ... now I do not get these lost reference errors anymore ...</include>

... but NEW ones :)

c1xx : fatal error C1083: Cannot open source file: 'libs\regex\src\regex_debug.cpp': No such file or directory

I downloaded the https://www.boost.org/users/download/ package and now have the libs on my disk … Q: How / where do I tell VS the location ?

Thanks again for all your help

Robert

[-]

Ah sorry. Just add it after the <XPDeprecationWarning> line instead.

For Boost, extract it somewhere (like C:\Boost; I think version 1.72 should work, but you can try other versions) and change the line:

<PropertyGroup Label="UserMacros" />

to the following (make sure to include the final backslash):

<PropertyGroup Label="UserMacros">
  <BOOST_ROOT>C:\Boost\</BOOST_ROOT>
</PropertyGroup>

[Robert Nio]

Thanks ... that helped VS to find the CPP files of REGEX ...

Now the error is within those files:

#include <boost/config.hpp>

Error   C1083   Cannot open include file: 'boost/config.hpp': No such file or directory SwiftSearch c:\program files\boost\boost_1_73_0\libs\regex\src\cpp_regex_traits.cpp 20  

[-]

Hmm, I think that one might require following the Boost setup instructions? On my machine there is a boost subdirectory that has symlinks inside it pointing to the individual contents of the libs subdirectory. I think it might've been generated with b2.exe or bjam.exe, but you might be able to just run mklink /D C:\Boost\boost libs and get it to work that way, I'm not sure.

[Robert Nio]

SUCCESS! I was able to create a clean build of Swift Search !

Thanks for all your help.

Here is what was needed:
NOTE: some steps might be redundant … but it works 😊

Visual Studio INSTALLER:

Visual Studio VS2019 "Individual components" > "Compilers, build tools, and runtimes" > "C++ Windows XP Support for VS 2017 (v141) tools [Deprecated]"

Install the WINDDK: https://download.microsoft.com/download/9/0/f/90f019ac-8243-48d3-91cf-81fc4093ecfd/1830_usa_ddk.iso

Install the vcpkg: a C++ package manager for Windows.
vcpkg install boost-regex
vcpkg install boost-algorithm
vcpkg install wtl
vcpkg integrate install  letting VS know what you just installed.

Download and install the WTL source from https://sourceforge.net/projects/wtl/
Download BOOST from https://www.boost.org/

EDIT Project.props in source directory:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ImportGroup Label="PropertySheets" />
  <PropertyGroup Label="UserMacros" >
  <BOOST_ROOT>---insert BOOST root location ---\</BOOST_ROOT>
    <XPDeprecationWarning>false</XPDeprecationWarning>
    <IncludePath>--- insert location of WTL library ---wtl\Include;$(IncludePath)</IncludePath>
  </PropertyGroup>

[Robert Nio]

Looking @ the code I do not see a "switch" for a console version ... I guess the code is separate ... where can I get a copy of that ?

Thanks again for all your help & support ...

[-]

The console version is _tmain, the GUI version is _tWinMain, both are there. It's just a question of whether you link with /Subsystem:Console or /Subsystem:Windows. :-)

[Robert Nio]

while trying out the new build ... I ran into an ABORT error:

abort() has been called

I guess its an UNHANDLED EXEPTION ? ... x800000003

So I made sure I use the exact version of WTL as you do ... What version of BOOST are you using?

running the debugger I get:

Unhandled exception at 0x76D24402 in SwiftSearch.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x05DCE15C. occurred

    [External Code] 
>   SwiftSearch.exe!__scrt_throw_std_bad_alloc() Line 36    C++
    SwiftSearch.exe!operator new(unsigned int size=35928932) Line 53    C++
    SwiftSearch.exe!std::_Default_allocate_traits::_Allocate(const unsigned int _Bytes=35928932) Line 52    C++
    SwiftSearch.exe!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>(const unsigned int _Bytes=35928893) Line 100  C++
    SwiftSearch.exe!std::_Allocate<8,std::_Default_allocate_traits,0>(const unsigned int _Bytes=35928893) Line 184  C++
    SwiftSearch.exe!std::allocator<NtfsIndex::Record>::allocate(const unsigned int _Count=466609) Line 997  C++
    SwiftSearch.exe!std::vector<NtfsIndex::Record,std::allocator<NtfsIndex::Record>>::_Resize<NtfsIndex::Record * <lambda>(NtfsIndex::Record *, const unsigned int)>(const unsigned int _Newsize=311074, std::vector<NtfsIndex::Record,std::allocator<NtfsIndex::Record>>::resize::__l2::NtfsIndex::Record * <lambda>(NtfsIndex::Record *, const unsigned int) _Udefault_or_fill=NtfsIndex::Record * <lambda>(NtfsIndex::Record * _Dest, const unsigned int _Count){...}) Line 1441 C++
    SwiftSearch.exe!std::vector<NtfsIndex::Record,std::allocator<NtfsIndex::Record>>::resize(const unsigned int _Newsize=311074) Line 1483  C++
    SwiftSearch.exe!vector_with_fast_size<NtfsIndex::Record,std::allocator<NtfsIndex::Record>>::resize(const unsigned int size=311074) Line 1727    C++
    SwiftSearch.exe!NtfsIndex::at(const unsigned int frs=264337, std::_Vector_iterator<std::_Vector_val<std::_Simple_types<NtfsIndex::Record>>> * const existing_to_revalidate=0x00000000) Line 2015    C++
    SwiftSearch.exe!NtfsIndex::load(const unsigned __int64 virtual_offset=270663680, void * const buffer=0x4ac6b088, const unsigned int size=1048576, const unsigned __int64 skipped_begin=0, const unsigned __int64 skipped_end=0) Line 2201   C++
    SwiftSearch.exe!OverlappedNtfsMftReadPayload::ReadOperation::operator()(const unsigned int size=1048576, const unsigned int __formal=20244520) Line 4257    C++
    SwiftSearch.exe!IoCompletionPort::worker(const unsigned long timeout=4294967295) Line 3893  C++
    SwiftSearch.exe!OleIoCompletionPort::worker(const unsigned long timeout=4294967295) Line 4075   C++
    SwiftSearch.exe!IoCompletionPort::worker() Line 3877    C++
    SwiftSearch.exe!OleIoCompletionPort::worker() Line 4063 C++
    SwiftSearch.exe!IoCompletionPort::iocp_worker(void * me=0x00f9f27c) Line 3847   C++
    SwiftSearch.exe!invoke_thread_procedure(unsigned int(__stdcall*)(void *) procedure=0x00b99f70, void * const context=0x00f9f27c) Line 92 C++
    SwiftSearch.exe!thread_start<unsigned int (__stdcall*)(void *)>(void * const parameter=0x01318de0) Line 115 C++
    [External Code] 

am I using the wrong ATL ?

[-]

My current local Boost version (which seems to work fine) was 1.72 as I mentioned earlier, but I think I might've been using Boost 1.68 for the last release? You'll have to debug it yourself :-) have fun debugging!

[Robert Nio]

I got every thing up and running ... get similar "results" (speed / size) as the original binary.

I was able to fix the POINTER ITERATOR TAGS which has throwing the exception.

Upgraded everything to x64 and using the latest SDK (Win 10.0.18362) / boost (1.73) and VS2017 Platform Toolset 141

I had to keep wtl-code-r467

For the COMAND LINE VERSION ? I just compiled to ".com" extention ... works now :)

BTW, now that I had a little more time to look at your code ... I understand WHY it took many years to get it to run as fast as it does ... LOTS of work!! THANKS AGAIN :)

[-]

Glad to hear! And sorry the code is a little (very) messy ;)