Menu ▾ ▴

Home

lqm678

Liberate & Digitize daily works by 2 exe : Data Mining; Map-Reduce; Pipe Endless.

Most time Just 1 command line to solve your daily text processing work.

Match/Search/Replace: msr.exe/msr-Win32.exe/msr.cygwin/msr.gcc**/msr-i386.gcc**

  • Match/Search/Replace/Execute/* Files/Pipe Lines/Blocks.
  • Filter/Load/Extract/Transform/Stats/* Files/Pipe Lines/Blocks.

Not-IN-latter: nin.exe/nin-Win32.exe/nin.cygwin/nin.gcc**/nin-i386.gcc**

  • Get Exclusive/Mutual Line-Set or Key-Set;
  • Remove Line-Set or Key-Set matched in latter file/pipe;
  • Get Unique/Mutual/Distribution/Stats/* Files/Pipe Line-Set or Key-Set.

MSR overview: (usage/examples: readme.txt (github.com) )

Performance comparison msr > findstr, msr ~ grep :

Compare findstr/grep on Windows Compare findstr/grep on Cygwin
Colorful screenshot Colorful screenshot
Summary table Summary table

Vivid Colorful Demo/examples: Run windows-test.bat (github.com) without parameters.

Almost no learning cost:

  • You can use plain text to search/replace (-x/-ix search-text to -o replace-to) if you're not farmiliar with Regex.
  • Using general Regex in C++, C#, Java, Scala, needless to learn strange Regex syntax like FINDSTR, Awk and Sed, etc.
  • Most of the time only use searching(Regex: -t/-i -t, Plain text: -x/-i -x);
  • Some of the time use searching with replacing-to(-o);
  • Just use -PAC to get pure result as same as other tools (no Path-number: -P, no Any-info : -A, no Color: -C)
  • All options are optional and no order and effective mean while; Free with abbreviations/full-names.

Usage screenshots

  • Also can see tools/readme.txt (github.com)
  • Zoom out following screenshots to 90% or smaller if it's not tidy or comfortable.

msr on Windows/Linux

nin on Windows/Linux

Demo and test screenshots

Zoom out following screenshots to 90% or smaller if it's not tidy or comfortable.

Powerful

  • Single exe for multiple platforms: Windows/Linux/Cygwin/Ubuntu/CentOS/Fedora .
  • Smart Loading files with 8 composable kinds of filters:
    • 5 pairs of file attribute filters
    • File name patterns (-f/--nf)
    • Directory patterns(-d/--nd)
    • Full path patterns(--pp/--np)
    • Size range(--s1,--s2)
    • Write-time range(--w1,--w2)
    • 3 kinds of file row / block filters to start/stop/skip reading/replacing each files/pipe:
    • Row/line number begin/end (-L, -N);
    • Block begin/end patterns (-b, -Q);
    • Normal begin/end patterns (b, -q).
  • Process pipe (output of self/other commands) endless as you want.
  • Two composable single exe: msr.exe/cygwin/gcc* (github.com) especially powerful with nin.exe/cygwin/gcc* (github.com).
  • 54/21 composable options for msr / nin (just run them without parameters to get colorful usage/examples or see readme.txt (github.com)) for further extractions/mining.
    msr --help # same as : msr -h / msr nin --help # same as : nin -h / nin msr | msr -t "^\s*-{1,2}\S+" -q "^\s*-h\s+" nin | msr -t "^\s*-{1,2}\S+" -q "^\s*-h\s+"

One limitation:

  • Cannot process Unicode files/pipe so far; Fine with UTF-8 and ANSI etc.

Just run the exe, you'll get the usage and examples.

Besides, some script/batch/shell files are also examples.

Search/Replace text by msr.exe / msr.gcc / msr.cygwin**

  • in files or from pipe
    • pipe (command line result);
    • files recursively in directories and multiple root paths separated by comma or semicolon.
  • with
    • Normal/Plain text searching/replacing;
    • Regex text searching/replacing, and with single-line/multi-line mode.
  • with excluding and including syntax meanwhile for
    • Filtering file-name/directory-name/full-path-string;
    • Filtering include text and exclude text;
    • Filtering files by write-time range and file size.
  • Finally Sort result text by specified time or text (regex pattern);
  • Output with hierarchy colors by searching-regex and enhancing-regex.

Typical scenarios of msr: Coding/Deploying/Test/Operation/Log-Mining

  1. Find text in pipe (command line result) or files (such as code, log)
  2. Replace/Extract/Tranform text from pipe or files recursively in multiple root path.
  3. Search/Replace and get percentage, distribution; further extraction on previous pipe or even command line.
  4. Find processes / Kill processes by regex/pid with colorful matching.
  5. Find files with specified name, in modification time range, size range and other filters, then use the path list to operate (-X is helpful).
  6. Look up a tool's usage with colors and context (Up/Down lines).
  7. Grep a command's result with matching info and time cost, and colorful matched lines or blocks.
  8. Adding colors to your scripts, especially nice to usage and examples section.
  9. Map <--> Reduce : Filter files and load, extract, transform, ... , pipe endless.
    ... ...

With requirements of:

  1. Basic text searching(-x) / replacing-to(-o) , plus case sensitive or not (-i).
  2. General Regex (regular expression) searching(-t)/replacing(-o) : consistent regex syntax with C#/C++/Java, not like strange or limited regex as AWK/GAWK/SED/FINDSTR …
  3. Recursively (-r) search / replace files in paths (-p) (multiple paths separated by "," or ";")
  4. For replacing: replace regex-pattern(-t)/normal-text(-x) to (-o) final-text
    • Preview: no -R
    • Replace : with -R
    • Backup files only if changed/replaced (-K) :

  5. Backup : Original files will be backuped to : {name}--lz-backup--{file-last-write-time}-{N}

    • Such as : myConfig.xml--lz-backup-2013-11-13__11_38_24
    • But if replaced many times in a second :
    • Will be : myConfig.xml--lz-backup-2013-11-13__11_38_24-N (N start from 1 )

  6. Powerful filtering: (can use all of following options meanwhile)

  7. For file : Sorting order by the prior of --wt and --sz if use both.
    • file-name(-f) / directory-name(-d) / path(-pp) + ALL :
    • include/exclude(--nf/--nd/--np) +ALL;
    • file modification time filter: --w1,--w2 : like --w1 2012-09-08 --w2 "2013-03-12 15" (or "2013-03-12 15:00")
    • file size range filter: --s1 , --s2 : like --s1 100KB --s2 1M
    • show file modification time and sort : --wt : useful if list file with -l
    • show file size and unit and sort : --sz : useful if list file with -l

  8. For file row : if not begin or stopped, not output/match/replace even if matched.

    • Regex pattern
    • start reading (-b)
    • stop reading (-q) ignore if has matched start pattern.
    • stop reading if has matched start pattern (-Q)
    • File line number
    • start at row (-L)
    • stop at row (-N)

  9. Matching(-t/-x) and non-matching(--nt/--nx) filter at mean while.(like file filter)

  10. Powerful output control:
  11. Can sort by time if specified time format -F for the logs from multiple paths;
  12. Colorful output and with hierarchy for captured matching group;
  13. Capture(-t/-x) and enhance(-e) and with different colors;
  14. Lines up(-U) and down(-D) as context to the captured row;
  15. Head(-H) and tail(-T) for whole result rows.
  16. Out summary info only if matched (-O)
  17. No any info just pure result (-A)
  18. Out summary info to stderr (-I)
  19. Execute(-X) output lines as commands (if they're callable commands) : because 'for' loop on Windows need escape | to ^| , > to ^> , etc.
  20. Extra and useful output info : if not use -A
  21. When you did it;
  22. What command line (-c) you used; What's the percentage and distribution.
  23. Where the files and rows (if not use -P) you searched/modified and working directory.
  24. How much time cost (so you know to start it at night/lunch if too long)
  25. Matching count and percentages (Also can use -l to get just brief file list and count/percentage)
  26. Use -PAC to get clean result (no path-line, no any info, no color)
  27. Use -PIC to output info to stderr pipe.
  28. Use -PC/-POC/-l -PC/-lPOC >nul(nul on Windows, /dev/null on Linux) to use summary info as source input for latter process/tool.
  29. If has -c in command line, can append any extra text, useful with -O -H 0 or >nul to do further extraction based on summary info:
    • msr -x "D:\data" -p xx.log -O >nul | msr -xxx xxx -c Checking D:\data
  30. Use -z to directly input string to read, avoid using echo which is clumsy in pipe on Windows.
  31. Use -J and -H output/match count to jump out.
MongoDB Logo MongoDB