Recent changes to ReleasesValidator

Discussion for ReleasesValidator page

Lee Thompson — Mon, 08 Apr 2013 22:14:00 -0000

This settings do not match the current code:

ValidOutputPath=    the valid output path
InValidOutputPath=  the invalid output path

Instead use:

OutputPathValid=    the valid output path
OutputPathNotValid= the invalid output path

WikiPage ReleasesValidator modified by Shaun

Shaun — Tue, 05 Mar 2013 12:24:20 -0000

--- v4
+++ v5
@@ -16,7 +16,7 @@

 In the case of RAR password checking just the first segment (part) of the RAR file is downloaded and analyses. This will incur a download data hit of a few hundred KB of data but it allows exclusion of passworded releases which can be a pain when building indexes.

-To extract RAR file data and check for passworded files inside other RAR files the validator uses the freeware unrar tool available from RarLabs [http://www.rarlab.com/rar_add.htm]. To do deep inspection of RAR files you need to download the unrar tool for your platform from the above page and extract it to the same path as the ReleasesValidator.exe.
+To extract RAR file data and check for passworded files inside other RAR files the validator uses the freeware unrar tool available from RarLabs . To do deep inspection of RAR files you need to download the unrar tool for your platform from the above page and extract it to the same path as the ReleasesValidator.exe.

 Once the release has been validated it is copied to one of two output paths, a directory containing valid releases and a directory containing invalid releases. The invalid releases can be deleted on a daily bases is they are no longer needed.

WikiPage ReleasesValidator modified by Shaun

Shaun — Fri, 22 Feb 2013 00:33:02 -0000

--- v3
+++ v4
@@ -22,7 +22,8 @@

 By default ReleasesValidator.exe tries to load config data from "validator.conf" you can also specify the config file name on the command line

-ReleasesValidator.exe "config file name"
+Usage:
+***ReleasesValidator.exe "config file name"***

 The config file has the following options:

WikiPage ReleasesValidator modified by Shaun

Shaun — Fri, 22 Feb 2013 00:20:06 -0000

--- v2
+++ v3
@@ -16,7 +16,7 @@

 In the case of RAR password checking just the first segment (part) of the RAR file is downloaded and analyses. This will incur a download data hit of a few hundred KB of data but it allows exclusion of passworded releases which can be a pain when building indexes.

-To extract RAR file data and check for passworded files inside other RAR files the validator uses the freeware unrar tool available from RarLabs [http://www.rarlab.com/rar_add.htm]. To do deep inspection of RAR files you need to download the unrar tool for you platform from the above page and extract it to the same path as the ReleasesValidator.exe.
+To extract RAR file data and check for passworded files inside other RAR files the validator uses the freeware unrar tool available from RarLabs [http://www.rarlab.com/rar_add.htm]. To do deep inspection of RAR files you need to download the unrar tool for your platform from the above page and extract it to the same path as the ReleasesValidator.exe.

 Once the release has been validated it is copied to one of two output paths, a directory containing valid releases and a directory containing invalid releases. The invalid releases can be deleted on a daily bases is they are no longer needed.

WikiPage ReleasesValidator modified by Shaun

Shaun — Fri, 22 Feb 2013 00:19:24 -0000

--- v1
+++ v2
@@ -15,6 +15,8 @@
 The first few checks are trivial and only require the data that is already available in the release NZB file. The more comprehensive RAR password testing and ultimately any file content testing needs to actually download parts or whole files to be further processed.

 In the case of RAR password checking just the first segment (part) of the RAR file is downloaded and analyses. This will incur a download data hit of a few hundred KB of data but it allows exclusion of passworded releases which can be a pain when building indexes.
+
+To extract RAR file data and check for passworded files inside other RAR files the validator uses the freeware unrar tool available from RarLabs [http://www.rarlab.com/rar_add.htm]. To do deep inspection of RAR files you need to download the unrar tool for you platform from the above page and extract it to the same path as the ReleasesValidator.exe.

 Once the release has been validated it is copied to one of two output paths, a directory containing valid releases and a directory containing invalid releases. The invalid releases can be deleted on a daily bases is they are no longer needed.

WikiPage ReleasesValidator modified by Shaun

Shaun — Fri, 22 Feb 2013 00:10:05 -0000

ReleasesValidator

The job of the validator is to confirm the release is valid, not just from a completeness point of view i.e. all files and parts of files are available as this is done by the extractor. But that the release is useable and fit for use in a system. There are a number of tests that can be run and depending on what your target use is of the releases different tests can be used.

The current implementation is outlined on the [How to Index NNTP Usenet Groups] page.

Release must not contain just a single file that is an NZB file
Check release subject for black listed RegEx matches
Check RAR files for password
- Download first segment
- Check for password header
- Extract file list and look for names starting with *
- Extract files and check for rars, if any check them for passwords

The first few checks are trivial and only require the data that is already available in the release NZB file. The more comprehensive RAR password testing and ultimately any file content testing needs to actually download parts or whole files to be further processed.

In the case of RAR password checking just the first segment (part) of the RAR file is downloaded and analyses. This will incur a download data hit of a few hundred KB of data but it allows exclusion of passworded releases which can be a pain when building indexes.

Once the release has been validated it is copied to one of two output paths, a directory containing valid releases and a directory containing invalid releases. The invalid releases can be deleted on a daily bases is they are no longer needed.

By default ReleasesValidator.exe tries to load config data from "validator.conf" you can also specify the config file name on the command line

ReleasesValidator.exe "config file name"

The config file has the following options:

NntpServer=         your news server
NntpUser=           your user name
NntpPassword=       your password
InputPath=          path to the NZB files to be tested
ValidOutputPath=    the valid output path
InValidOutputPath=  the invalid output path
BlackListedSubject= RegEx with black listed subject text

You can repeat the last config option multiple times

BlackListedSubject=RegEx01
BlackListedSubject=RegEx02
BlackListedSubject=RegEx03

This will load all the black list RegEx in and test each one of a release subject for blacklist matches.