Recent changes to Parameters

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 09:05:44 -0000

--- v7
+++ v8
@@ -26,15 +26,19 @@
 The name of output file. Several files might be generated, e.g., term-document matrix (in one or more files, according to the output format), tokens, some statistics etc. This name will be extended by pre-defined extensions, like arff, names, data, tokens, etc.

 **local_weights (string, mandatory)**
-Specifies what kind of local weighting will be used. Allowed values are listed in table 1.
+Specifies what kind of local weighting will be used. Allowed values are listed in the following table.

 [[img src=local_weights.png]]

 **global_weights (string, optional)**
-Specifies what kind of global weighting will be used. Allowed values are listed in table 2.
+Specifies what kind of global weighting will be used. Allowed values are listed in the following table.
+
+[[img src=global_weights.png]]

 **normalization (string, optional)**
-Specifies what kind of local weighting will be used. Allowed values are listed in table 3.
+Specifies what kind of local weighting will be used. Allowed values are listed in the following table.
+
+[[img src=normalization.png]]

 **min_word_length (integer, optional)**
 Specifies the minimal length (in characters) of a word to appear in the output. Shorter words are filtered out.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 09:04:41 -0000

--- v6
+++ v7
@@ -27,6 +27,8 @@

 **local_weights (string, mandatory)**
 Specifies what kind of local weighting will be used. Allowed values are listed in table 1.
+
+[[img src=local_weights.png]]

 **global_weights (string, optional)**
 Specifies what kind of global weighting will be used. Allowed values are listed in table 2.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 08:58:39 -0000

--- v5
+++ v6
@@ -61,14 +61,14 @@
 **output_format (string, compulsory)**
 Specifies an output format of the term-document matrix. Allowed values are:

-+ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
-+Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
-+C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier), 
-*SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
-*CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
-*CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
-*SVMlight - sparse matrix format for software packages SVMlight,
-*YALE - Yale sparse matrix format.
+* ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
+* Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
+* C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier), 
+* SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
+* CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
+* CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
+* SVMlight - sparse matrix format for software packages SVMlight,
+* YALE - Yale sparse matrix format.

 **n_grams (string, optional)**
 When provided, sequences consisting of the specified numbers of subsequent words will be treated as attributes (for example, for n_grams=2, the attributes derived from this paragraph will include when_provided, provided_sequences, sequences_consisting, etc.). It is possible to specify more n-gram sizes by supplying a comma separated list, e.g. –n_grams=2,3.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 08:57:28 -0000

--- v4
+++ v5
@@ -60,14 +60,15 @@

 **output_format (string, compulsory)**
 Specifies an output format of the term-document matrix. Allowed values are:
-*  ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
-*  Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
-*  C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier), 
-*  SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
-*  CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
-*  CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
-*  SVMlight - sparse matrix format for software packages SVMlight,
-*  YALE - Yale sparse matrix format.
+
++ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
++Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
++C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier), 
+*SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
+*CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
+*CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
+*SVMlight - sparse matrix format for software packages SVMlight,
+*YALE - Yale sparse matrix format.

 **n_grams (string, optional)**
 When provided, sequences consisting of the specified numbers of subsequent words will be treated as attributes (for example, for n_grams=2, the attributes derived from this paragraph will include when_provided, provided_sequences, sequences_consisting, etc.). It is possible to specify more n-gram sizes by supplying a comma separated list, e.g. –n_grams=2,3.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 08:56:25 -0000

--- v3
+++ v4
@@ -60,14 +60,14 @@

 **output_format (string, compulsory)**
 Specifies an output format of the term-document matrix. Allowed values are:
--  ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
--  Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
--  C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier), 
--  SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
--  CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
--  CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
--  SVMlight - sparse matrix format for software packages SVMlight,
--  YALE - Yale sparse matrix format.
+*  ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
+*  Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
+*  C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier), 
+*  SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
+*  CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
+*  CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
+*  SVMlight - sparse matrix format for software packages SVMlight,
+*  YALE - Yale sparse matrix format.

 **n_grams (string, optional)**
 When provided, sequences consisting of the specified numbers of subsequent words will be treated as attributes (for example, for n_grams=2, the attributes derived from this paragraph will include when_provided, provided_sequences, sequences_consisting, etc.). It is possible to specify more n-gram sizes by supplying a comma separated list, e.g. –n_grams=2,3.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 08:55:42 -0000

--- v2
+++ v3
@@ -100,7 +100,7 @@
 When the logarithm is used in calculations of term weights (e.g., global weight inverse document frequency, or local weight logarithm, alternate logarithm, etc.) a user might decide what will be the base of the logarithm. The two allowed options include natural (the base is e) and common (with base 10).

 **tags (string, optional)**
-If the text of the documents is marked by a SGML based markup language (e.g., HTML or XML) only the content of selected elements (e.g., the  element used in the Reuters data set) might be processed. The element names are passed as a comma or space separated list. 
+If the text of the documents is marked by a SGML based markup language (e.g., HTML or XML) only the content of selected elements (e.g., the <text> element used in the Reuters data set) might be processed. The element names are passed as a comma or space separated list. 

 **split_into_elements (Boolean, optional)**
 When the content of selected elements is extracted this parameter determines whether every such extracted content will become a separate document or not.
@@ -110,26 +110,33 @@

 **output_original_texts (Boolean, optional)**
 Original documents satisfying the specified preprocessing rules (e.g., documents from given classes, containing only allowed words) are written to a file with extension original.txt.
-output_tokens (Boolean, optional)
+
+**output_tokens (Boolean, optional)**
 A file with the documents containing only the terms that satisfy the specified preprocessing rules, or the derived terms, like stemmed words or n-grams, are written to a file with extension tokens.txt.
-preprocess_data (Boolean, optional)
+
+**preprocess_data (Boolean, optional)**
 When desired, only the preprocessing phase that finishes before term-document matrix creation is completed. All additional files, like the dictionary, statistics, a file with tokens, etc., are created when a user specifies so. This parameter is suitable when it is necessary to discover some characteristics of the data, like the list of unique terms and their frequencies after all filtering parameters (like minimal word length, minimal or maximal document frequency, stemming, etc.) are applied.
-remove_multiple_characters (Boolean, optional)
+
+**remove_multiple_characters (Boolean, optional)**
 When set to true, all appearances of the same character more than two times in a row are replaced by two appearances. For example, “goooooood” is replaced by “good”.
-preserve_numbers (Booelan, optional)
+
+**preserve_numbers (Booelan, optional)**
 Normally, all non-letters, including numbers, are removed from the texts. When desired, numbers will remain in the texts and are treated as words. Numbers have optional + or – sign, several digits, possibly followed by a decimal point (in that case no digits are necessary before the decimal point) and several decimal places, optionally followed by the semilogarithmic notation (e or E followed by a positive or negative exponent containing at least one digit).
-case (string, optional)
+
+**case (string, optional)**
 Specifies, what case folding transformation should be applied to the documents. Available options include 'no' (original case is preserved),'lower case', and 'upper case'.
-preserve_symbols (string, optional)
+
+**preserve_symbols (string, optional)**
 Normally, all non-letters are removed from the texts. When desired, some of the symbols (characters) might be preserved. These characters are included in the string passed as this parameter. For example, when +:$ is supplied, all plus signs, colons, and dollar signs remain in the texts and are treated as words.

+**preserve_emoticons (Boolean, optional)**
+When desired, all emoticons (strings like :-), :-(, and others) in the texts will be replaced by their textual alternatives. The list of allowed emoticons is located in the file emoticons.txt located in the application directory. The content of the file might be modified so the user defined emoticons will be used. Every row of the file contains one emoticon and its textual representation, e.g., “:D laughing”, or “:| straight face”. All occurrences of “:D” will be then replaced by string “laughing”, “:|” will be replaced by “straight_face”, etc.

+**output_decimal_places (integer, optional)**
+When real numbers appear as the vectors’ components, this parameter specifies the number of places after the decimal point. When not specified, 3 is used as the default value.

-preserve_emoticons (Boolean, optional)
-When desired, all emoticons (strings like :-), :-(, and others) in the texts will be replaced by their textual alternatives. The list of allowed emoticons is located in the file emoticons.txt located in the application directory. The content of the file might be modified so the user defined emoticons will be used. Every row of the file contains one emoticon and its textual representation, e.g., “:D laughing”, or “:| straight face”. All occurrences of “:D” will be then replaced by string “laughing”, “:|” will be replaced by “straight_face”, etc.
-output_decimal_places (integer, optional)
-When real numbers appear as the vectors’ components, this parameter specifies the number of places after the decimal point. When not specified, 3 is used as the default value.
-sort_attributes (string, optional)
+**sort_attributes (string, optional)**
 This parameter defines how the attributes will be sort in all output files (term-document matrix, dictionary, etc.). Allowed values include 'none' (no explicit sorting is used, the attributes are sorted according to the internal representation in computer memory, depending on the data) , 'alphabetically', and 'document frequency'.
-stemming (string, optional)
+
+**stemming (string, optional)**
 When desired, stemming is applied to the input data. Stemming algorithms are implemented through a Perl interface to the C version of the Snowball stemmers. Supported languages (used as parameter values) include 'danish','dutch','english','finnish','french','german', 'hungarian', 'italian','norwegian', 'portuguese', 'romanian', 'russian', 'spanish', 'swedish', and 'turkish'.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 08:53:46 -0000

--- v1
+++ v2
@@ -9,39 +9,56 @@

 **skip_tokens (integer, optional)**
 When specified, the desired number of tokens at the beginning of every document is skipped and excluded from further processing. This is suitable, e.g., in situations when there the documents contain headers or other meta-data at the beginning. In this step, tokens are considered to be strings separated by one or more whitespaces, a comma, or a semicolon.
-class_position (integer, optional)
+
+**class_position (integer, optional)**
 This parameter specifies the position of a token that determines the class of a document. If the value of this parameter is less or equal to the number of skipped tokens, the token will be preserved. In this step, tokens are considered to be strings separated by one or more whitespaces, a comma, or a semicolon.
-randomize (Boolean, optional)
+
+**randomize (Boolean, optional)**
 When selected, the output instances will be randomized and not in the same order as they are on the input.
-processed_classes (string, optional) 
+
+**processed_classes (string, optional) **
 A list of comma or whitespace separated values might be provided. In that case, only documents with these supplied labels will be processed.
-output_dir (string, optional)
+
+**output_dir (string, optional)**
 The name of a directory where the output (term-document matrix and possibly some additional files) will be stored.
-output_file (string, mandatory)
+
+**output_file (string, mandatory)**
 The name of output file. Several files might be generated, e.g., term-document matrix (in one or more files, according to the output format), tokens, some statistics etc. This name will be extended by pre-defined extensions, like arff, names, data, tokens, etc.
-local_weights (string, mandatory)
+
+**local_weights (string, mandatory)**
 Specifies what kind of local weighting will be used. Allowed values are listed in table 1.
-global_weights (string, optional)
+
+**global_weights (string, optional)**
 Specifies what kind of global weighting will be used. Allowed values are listed in table 2.
-normalization (string, optional)
+
+**normalization (string, optional)**
 Specifies what kind of local weighting will be used. Allowed values are listed in table 3.
-min_word_length (integer, optional)
+
+**min_word_length (integer, optional)**
 Specifies the minimal length (in characters) of a word to appear in the output. Shorter words are filtered out.
-max_word_length (integer, optional)
+
+**max_word_length (integer, optional)**
 Specifies the maximal length (in characters) of a word to appear in the output. Longer words are filtered out.
-min_global_frequency (integer, optional)
+
+**min_global_frequency (integer, optional)**
 Specifies the minimal global frequency of a word to appear in the output. Words that appear less times (thus are very rare) are filtered out.
-max_global_frequency (integer, optional)
+
+**max_global_frequency (integer, optional)**
 Specifies the maximal global frequency of a word to appear in the output. Words that appear more times (thus are very common) are filtered out.
-min_local_frequency (integer, optional)
+
+**min_local_frequency (integer, optional)**
 Specifies the minimal local frequency of a word (i.e., in one document) to appear in the output. Words that appear less times (thus are very rare) are filtered out.
-max_local_frequency (integer, optional)
+
+**max_local_frequency (integer, optional)**
 Specifies the maximal local frequency of a word (i.e., in one document) to appear in the output. Words that appear more times (thus are very common) are filtered out.
-min_document_frequency (integer, optional)
+
+**min_document_frequency (integer, optional)**
 Specifies the minimal document frequency of a word (i.e., the number of documents where is the word contained) to appear in the output. Words that appear less times (thus are very rare) are filtered out.
-max_document_frequency (integer, optional)
+
+**max_document_frequency (integer, optional)**
 Specifies the maximal document frequency of a word (i.e., the number of documents where is the word contained) to appear in the output. Words that appear more times (thus are very common) are filtered out.
-output_format (string, compulsory)
+
+**output_format (string, compulsory)**
 Specifies an output format of the term-document matrix. Allowed values are:
 -  ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
 -  Sparse ARFF (Attribute-Relation File Format) used by the Weka  machine learning software, 
@@ -51,35 +68,47 @@
 -  CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
 -  SVMlight - sparse matrix format for software packages SVMlight,
 -  YALE - Yale sparse matrix format.
-n_grams (string, optional)
+
+**n_grams (string, optional)**
 When provided, sequences consisting of the specified numbers of subsequent words will be treated as attributes (for example, for n_grams=2, the attributes derived from this paragraph will include when_provided, provided_sequences, sequences_consisting, etc.). It is possible to specify more n-gram sizes by supplying a comma separated list, e.g. –n_grams=2,3.
-create_dictionary (Boolean, optional)
+
+**create_dictionary (Boolean, optional)**
 When desired, a list of tokens (one token per line) is written to a file with extension dict.

+**create_dictionary_freq (Boolean, optional)**
+When desired, a list of tokens and their global frequencies (one token per line, ordered by the frequency in descending order) is written to a file with extension freq.dict.

-create_dictionary_freq (Boolean, optional)
-When desired, a list of tokens and their global frequencies (one token per line, ordered by the frequency in descending order) is written to a file with extension freq.dict.
-create_dictionary_freq_classes (Boolean, optional)     
+**create_dictionary_freq_classes (Boolean, optional)**
 When desired, a list of tokens and their global frequencies and frequencies in individual classes (one token per line, ordered by the global frequency in descending order) is written to a file with extension freq.dict.
-print_statistics (Boolean, optional)
+
+**print_statistics (Boolean, optional)**
 Some algorithm parameters and data set statistics are written to a file with extension stat.txt. The statistics include the total number of processed documents and unique attributes. For each processed class, the number of documents, unique attributes, and information about term numbers (minimal, maximal, average, variance) in the documents from the given class are provided.
-dictionary_file (string, optional)
+
+**dictionary_file (string, optional)**
 When a dictionary (a list of allowed words) is supplied the words that are not in the dictionary are eliminated.  This is useful, e.g., when creating test data set that must have the same attributes as the training set, or when it makes sense to use a dictionary from a given domain. The name of the file containing the allowed words is passed using this parameter. In the file, there is a single word on each line.
-replacement_rules_file (string, optional)
+
+**replacement_rules_file (string, optional)**
 When supplied, rules for replacing some text with another (e.g., "European Union" -> "EU") are applied. The name of the file containing the allowed words is passed using this parameter. In the file, there is a single rule on each line in the form “ORIGINAL => REPLACE\n”.
-stopwords_file (string, optional)
+
+**stopwords_file (string, optional)**
 When stopwords are supplied they are filtered out before further processing.  The name of the file containing stopwords is passed using this parameter. In the file, there is a single word on each line.
-allowed_symbols__file (string, optional)
+
+**allowed_symbols_file (string, optional)**
 Some expressions (symbols), like abbreviations, might include symbols that would be eliminated (e.g., string “U.S.A.” would be transformed to three terms “U”, “S”, and “A”). When a list of allowed symbols is supplied these symbols are preserved in the text. The name of the file containing allowed symbols is passed using this parameter. In the file, there is a single expression on each line.
-logarithm_type (string, optional)
+
+**logarithm_type (string, optional)**
 When the logarithm is used in calculations of term weights (e.g., global weight inverse document frequency, or local weight logarithm, alternate logarithm, etc.) a user might decide what will be the base of the logarithm. The two allowed options include natural (the base is e) and common (with base 10).
-tags (string, optional)
+
+**tags (string, optional)**
 If the text of the documents is marked by a SGML based markup language (e.g., HTML or XML) only the content of selected elements (e.g., the  element used in the Reuters data set) might be processed. The element names are passed as a comma or space separated list. 
-split_into_elements (Boolean, optional)
+
+**split_into_elements (Boolean, optional)**
 When the content of selected elements is extracted this parameter determines whether every such extracted content will become a separate document or not.
-sentences (string, optional)
+
+**sentences (string, optional)**
 A user might also request splitting the documents into sentences. At this moment, sentence boundaries are simply determined by occurrences of given characters. These characters might be enumerated by the user, for example by passing :!?; as the value of this parameter.
-output_original_texts (Boolean, optional)
+
+**output_original_texts (Boolean, optional)**
 Original documents satisfying the specified preprocessing rules (e.g., documents from given classes, containing only allowed words) are written to a file with extension original.txt.
 output_tokens (Boolean, optional)
 A file with the documents containing only the terms that satisfy the specified preprocessing rules, or the derived terms, like stemmed words or n-grams, are written to a file with extension tokens.txt.

Parameters modified by František Dařena

František Dařena — Fri, 08 Apr 2016 08:51:28 -0000

input, input_file (string, mandatory)
Specifies the name of a file with raw texts to be converted to vectors. There is one document on each line. At the beginning of each line there is a class label followed by a tabulator.

encoding (string, optional)
Specifies the input and output encodings for the data files read and produced by the application. Available values include 'ascii', 'iso-8859-1', 'iso-8859-2', and 'utf8'.

subset, subset_size (integer, optional)
Specifies the size of the generated data set in terms of documents number. When not provided, all documents from the source file are processed.

skip_tokens (integer, optional)
When specified, the desired number of tokens at the beginning of every document is skipped and excluded from further processing. This is suitable, e.g., in situations when there the documents contain headers or other meta-data at the beginning. In this step, tokens are considered to be strings separated by one or more whitespaces, a comma, or a semicolon.
class_position (integer, optional)
This parameter specifies the position of a token that determines the class of a document. If the value of this parameter is less or equal to the number of skipped tokens, the token will be preserved. In this step, tokens are considered to be strings separated by one or more whitespaces, a comma, or a semicolon.
randomize (Boolean, optional)
When selected, the output instances will be randomized and not in the same order as they are on the input.
processed_classes (string, optional)
A list of comma or whitespace separated values might be provided. In that case, only documents with these supplied labels will be processed.
output_dir (string, optional)
The name of a directory where the output (term-document matrix and possibly some additional files) will be stored.
output_file (string, mandatory)
The name of output file. Several files might be generated, e.g., term-document matrix (in one or more files, according to the output format), tokens, some statistics etc. This name will be extended by pre-defined extensions, like arff, names, data, tokens, etc.
local_weights (string, mandatory)
Specifies what kind of local weighting will be used. Allowed values are listed in table 1.
global_weights (string, optional)
Specifies what kind of global weighting will be used. Allowed values are listed in table 2.
normalization (string, optional)
Specifies what kind of local weighting will be used. Allowed values are listed in table 3.
min_word_length (integer, optional)
Specifies the minimal length (in characters) of a word to appear in the output. Shorter words are filtered out.
max_word_length (integer, optional)
Specifies the maximal length (in characters) of a word to appear in the output. Longer words are filtered out.
min_global_frequency (integer, optional)
Specifies the minimal global frequency of a word to appear in the output. Words that appear less times (thus are very rare) are filtered out.
max_global_frequency (integer, optional)
Specifies the maximal global frequency of a word to appear in the output. Words that appear more times (thus are very common) are filtered out.
min_local_frequency (integer, optional)
Specifies the minimal local frequency of a word (i.e., in one document) to appear in the output. Words that appear less times (thus are very rare) are filtered out.
max_local_frequency (integer, optional)
Specifies the maximal local frequency of a word (i.e., in one document) to appear in the output. Words that appear more times (thus are very common) are filtered out.
min_document_frequency (integer, optional)
Specifies the minimal document frequency of a word (i.e., the number of documents where is the word contained) to appear in the output. Words that appear less times (thus are very rare) are filtered out.
max_document_frequency (integer, optional)
Specifies the maximal document frequency of a word (i.e., the number of documents where is the word contained) to appear in the output. Words that appear more times (thus are very common) are filtered out.
output_format (string, compulsory)
Specifies an output format of the term-document matrix. Allowed values are:
- ARFF (Attribute-Relation File Format) used by the Weka machine learning software,
- Sparse ARFF (Attribute-Relation File Format) used by the Weka machine learning software,
- C5 required by C5/See5 software package (a file with extension names describing attributes and classes and a file with extension data containing training data for the classifier),
- SPARSE – sparse matrix format, comma separated list of attribute number-attribute vale pairs, followed by a class label,
- CSV – comma separated list of attribute values followed by a class label, with attribute names on the first row,
- CLUTO – sparse matrix format for software packages Cluto, creating also the rlabel file (file with class labels) when applicable,
- SVMlight - sparse matrix format for software packages SVMlight,
- YALE - Yale sparse matrix format.
n_grams (string, optional)
When provided, sequences consisting of the specified numbers of subsequent words will be treated as attributes (for example, for n_grams=2, the attributes derived from this paragraph will include when_provided, provided_sequences, sequences_consisting, etc.). It is possible to specify more n-gram sizes by supplying a comma separated list, e.g. –n_grams=2,3.
create_dictionary (Boolean, optional)
When desired, a list of tokens (one token per line) is written to a file with extension dict.

create_dictionary_freq (Boolean, optional)
When desired, a list of tokens and their global frequencies (one token per line, ordered by the frequency in descending order) is written to a file with extension freq.dict.
create_dictionary_freq_classes (Boolean, optional)
When desired, a list of tokens and their global frequencies and frequencies in individual classes (one token per line, ordered by the global frequency in descending order) is written to a file with extension freq.dict.
print_statistics (Boolean, optional)
Some algorithm parameters and data set statistics are written to a file with extension stat.txt. The statistics include the total number of processed documents and unique attributes. For each processed class, the number of documents, unique attributes, and information about term numbers (minimal, maximal, average, variance) in the documents from the given class are provided.
dictionary_file (string, optional)
When a dictionary (a list of allowed words) is supplied the words that are not in the dictionary are eliminated. This is useful, e.g., when creating test data set that must have the same attributes as the training set, or when it makes sense to use a dictionary from a given domain. The name of the file containing the allowed words is passed using this parameter. In the file, there is a single word on each line.
replacement_rules_file (string, optional)
When supplied, rules for replacing some text with another (e.g., "European Union" -> "EU") are applied. The name of the file containing the allowed words is passed using this parameter. In the file, there is a single rule on each line in the form “ORIGINAL => REPLACE\n”.
stopwords_file (string, optional)
When stopwords are supplied they are filtered out before further processing. The name of the file containing stopwords is passed using this parameter. In the file, there is a single word on each line.
allowed_symbols__file (string, optional)
Some expressions (symbols), like abbreviations, might include symbols that would be eliminated (e.g., string “U.S.A.” would be transformed to three terms “U”, “S”, and “A”). When a list of allowed symbols is supplied these symbols are preserved in the text. The name of the file containing allowed symbols is passed using this parameter. In the file, there is a single expression on each line.
logarithm_type (string, optional)
When the logarithm is used in calculations of term weights (e.g., global weight inverse document frequency, or local weight logarithm, alternate logarithm, etc.) a user might decide what will be the base of the logarithm. The two allowed options include natural (the base is e) and common (with base 10).
tags (string, optional)
If the text of the documents is marked by a SGML based markup language (e.g., HTML or XML) only the content of selected elements (e.g., the element used in the Reuters data set) might be processed. The element names are passed as a comma or space separated list.
split_into_elements (Boolean, optional)
When the content of selected elements is extracted this parameter determines whether every such extracted content will become a separate document or not.
sentences (string, optional)
A user might also request splitting the documents into sentences. At this moment, sentence boundaries are simply determined by occurrences of given characters. These characters might be enumerated by the user, for example by passing :!?; as the value of this parameter.
output_original_texts (Boolean, optional)
Original documents satisfying the specified preprocessing rules (e.g., documents from given classes, containing only allowed words) are written to a file with extension original.txt.
output_tokens (Boolean, optional)
A file with the documents containing only the terms that satisfy the specified preprocessing rules, or the derived terms, like stemmed words or n-grams, are written to a file with extension tokens.txt.
preprocess_data (Boolean, optional)
When desired, only the preprocessing phase that finishes before term-document matrix creation is completed. All additional files, like the dictionary, statistics, a file with tokens, etc., are created when a user specifies so. This parameter is suitable when it is necessary to discover some characteristics of the data, like the list of unique terms and their frequencies after all filtering parameters (like minimal word length, minimal or maximal document frequency, stemming, etc.) are applied.
remove_multiple_characters (Boolean, optional)
When set to true, all appearances of the same character more than two times in a row are replaced by two appearances. For example, “goooooood” is replaced by “good”.
preserve_numbers (Booelan, optional)
Normally, all non-letters, including numbers, are removed from the texts. When desired, numbers will remain in the texts and are treated as words. Numbers have optional + or – sign, several digits, possibly followed by a decimal point (in that case no digits are necessary before the decimal point) and several decimal places, optionally followed by the semilogarithmic notation (e or E followed by a positive or negative exponent containing at least one digit).
case (string, optional)
Specifies, what case folding transformation should be applied to the documents. Available options include 'no' (original case is preserved),'lower case', and 'upper case'.
preserve_symbols (string, optional)
Normally, all non-letters are removed from the texts. When desired, some of the symbols (characters) might be preserved. These characters are included in the string passed as this parameter. For example, when +:$ is supplied, all plus signs, colons, and dollar signs remain in the texts and are treated as words.

preserve_emoticons (Boolean, optional)
When desired, all emoticons (strings like :-), :-(, and others) in the texts will be replaced by their textual alternatives. The list of allowed emoticons is located in the file emoticons.txt located in the application directory. The content of the file might be modified so the user defined emoticons will be used. Every row of the file contains one emoticon and its textual representation, e.g., “:D laughing”, or “:| straight face”. All occurrences of “:D” will be then replaced by string “laughing”, “:|” will be replaced by “straight_face”, etc.
output_decimal_places (integer, optional)
When real numbers appear as the vectors’ components, this parameter specifies the number of places after the decimal point. When not specified, 3 is used as the default value.
sort_attributes (string, optional)
This parameter defines how the attributes will be sort in all output files (term-document matrix, dictionary, etc.). Allowed values include 'none' (no explicit sorting is used, the attributes are sorted according to the internal representation in computer memory, depending on the data) , 'alphabetically', and 'document frequency'.
stemming (string, optional)
When desired, stemming is applied to the input data. Stemming algorithms are implemented through a Perl interface to the C version of the Snowball stemmers. Supported languages (used as parameter values) include 'danish','dutch','english','finnish','french','german', 'hungarian', 'italian','norwegian', 'portuguese', 'romanian', 'russian', 'spanish', 'swedish', and 'turkish'.