## [4d2259]: Doc / lib / libshlex.tex  Maximize  Restore  History

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 \section{\module{shlex} --- Simple lexical analysis} \declaremodule{standard}{shlex} \modulesynopsis{Simple lexical analysis for \UNIX\ shell-like languages.} \moduleauthor{Eric S. Raymond}{esr@snark.thyrsus.com} \moduleauthor{Gustavo Niemeyer}{niemeyer@conectiva.com} \sectionauthor{Eric S. Raymond}{esr@snark.thyrsus.com} \sectionauthor{Gustavo Niemeyer}{niemeyer@conectiva.com} \versionadded{1.5.2} The \class{shlex} class makes it easy to write lexical analyzers for simple syntaxes resembling that of the \UNIX{} shell. This will often be useful for writing minilanguages, (e.g. in run control files for Python applications) or for parsing quoted strings. \begin{seealso} \seemodule{ConfigParser}{Parser for configuration files similar to the Windows \file{.ini} files.} \end{seealso} \subsection{Module Contents} The \module{shlex} module defines the following functions: \begin{funcdesc}{split}{s\optional{, posix=\code{True}\optional{, spaces=\code{True}}}} Split the string \var{s} using shell-like syntax. If \code{posix} is \code{True}, operate in posix mode. If \code{spaces} is \code{True}, it will only split words in whitespaces (setting the \member{whitespace_split} member of the \class{shlex} instance). \versionadded{2.3} \end{funcdesc} The \module{shlex} module defines the following classes: \begin{classdesc}{shlex}{\optional{instream=\code{sys.stdin}\optional{, infile=\code{None}\optional{, posix=\code{False}}}}} A \class{shlex} instance or subclass instance is a lexical analyzer object. The initialization argument, if present, specifies where to read characters from. It must be a file-/stream-like object with \method{read()} and \method{readline()} methods, or a string (strings are accepted since Python 2.3). If no argument is given, input will be taken from \code{sys.stdin}. The second optional argument is a filename string, which sets the initial value of the \member{infile} member. If the \var{instream} argument is omitted or equal to \code{sys.stdin}, this second argument defaults to stdin''. The \var{posix} argument was introduced in Python 2.3, and defines the operational mode. When \var{posix} is not true (default), the \class{shlex} instance will operate in compatibility mode. When operating in posix mode, \class{shlex} will try to be as close as possible to the posix shell parsing rules. See~\ref{shlex-objects}. \end{classdesc} \subsection{shlex Objects \label{shlex-objects}} A \class{shlex} instance has the following methods: \begin{methoddesc}{get_token}{} Return a token. If tokens have been stacked using \method{push_token()}, pop a token off the stack. Otherwise, read one from the input stream. If reading encounters an immediate end-of-file, \member{self.eof} is returned (the empty string (\code{""}) in non-posix mode, and \code{None} in posix mode). \end{methoddesc} \begin{methoddesc}{push_token}{str} Push the argument onto the token stack. \end{methoddesc} \begin{methoddesc}{read_token}{} Read a raw token. Ignore the pushback stack, and do not interpret source requests. (This is not ordinarily a useful entry point, and is documented here only for the sake of completeness.) \end{methoddesc} \begin{methoddesc}{sourcehook}{filename} When \class{shlex} detects a source request (see \member{source} below) this method is given the following token as argument, and expected to return a tuple consisting of a filename and an open file-like object. Normally, this method first strips any quotes off the argument. If the result is an absolute pathname, or there was no previous source request in effect, or the previous source was a stream (e.g. \code{sys.stdin}), the result is left alone. Otherwise, if the result is a relative pathname, the directory part of the name of the file immediately before it on the source inclusion stack is prepended (this behavior is like the way the C preprocessor handles \code{\#include "file.h"}). The result of the manipulations is treated as a filename, and returned as the first component of the tuple, with \function{open()} called on it to yield the second component. (Note: this is the reverse of the order of arguments in instance initialization!) This hook is exposed so that you can use it to implement directory search paths, addition of file extensions, and other namespace hacks. There is no corresponding close' hook, but a shlex instance will call the \method{close()} method of the sourced input stream when it returns \EOF. For more explicit control of source stacking, use the \method{push_source()} and \method{pop_source()} methods. \end{methoddesc} \begin{methoddesc}{push_source}{stream\optional{, filename}} Push an input source stream onto the input stack. If the filename argument is specified it will later be available for use in error messages. This is the same method used internally by the \method{sourcehook} method. \versionadded{2.1} \end{methoddesc} \begin{methoddesc}{pop_source}{} Pop the last-pushed input source from the input stack. This is the same method used internally when the lexer reaches \EOF on a stacked input stream. \versionadded{2.1} \end{methoddesc} \begin{methoddesc}{error_leader}{\optional{file\optional{, line}}} This method generates an error message leader in the format of a \UNIX{} C compiler error label; the format is \code{'"\%s", line \%d: '}, where the \samp{\%s} is replaced with the name of the current source file and the \samp{\%d} with the current input line number (the optional arguments can be used to override these). This convenience is provided to encourage \module{shlex} users to generate error messages in the standard, parseable format understood by Emacs and other \UNIX{} tools. \end{methoddesc} Instances of \class{shlex} subclasses have some public instance variables which either control lexical analysis or can be used for debugging: \begin{memberdesc}{commenters} The string of characters that are recognized as comment beginners. All characters from the comment beginner to end of line are ignored. Includes just \character{\#} by default. \end{memberdesc} \begin{memberdesc}{wordchars} The string of characters that will accumulate into multi-character tokens. By default, includes all \ASCII{} alphanumerics and underscore. \end{memberdesc} \begin{memberdesc}{whitespace} Characters that will be considered whitespace and skipped. Whitespace bounds tokens. By default, includes space, tab, linefeed and carriage-return. \end{memberdesc} \begin{memberdesc}{escape} Characters that will be considered as escape. This will be only used in posix mode, and includes just \character{\textbackslash} by default. \versionadded{2.3} \end{memberdesc} \begin{memberdesc}{quotes} Characters that will be considered string quotes. The token accumulates until the same quote is encountered again (thus, different quote types protect each other as in the shell.) By default, includes \ASCII{} single and double quotes. \end{memberdesc} \begin{memberdesc}{escapedquotes} Characters in \member{quotes} that will interpret escape characters defined in \member{escape}. This is only used in posix mode, and includes just \character{"} by default. \versionadded{2.3} \end{memberdesc} \begin{memberdesc}{whitespace_split} If true, tokens will only be split in whitespaces. This is useful, for example, for parsing command lines with \class{shlex}, getting tokens in a similar way to shell arguments. \versionadded{2.3} \end{memberdesc} \begin{memberdesc}{infile} The name of the current input file, as initially set at class instantiation time or stacked by later source requests. It may be useful to examine this when constructing error messages. \end{memberdesc} \begin{memberdesc}{instream} The input stream from which this \class{shlex} instance is reading characters. \end{memberdesc} \begin{memberdesc}{source} This member is \code{None} by default. If you assign a string to it, that string will be recognized as a lexical-level inclusion request similar to the \samp{source} keyword in various shells. That is, the immediately following token will opened as a filename and input taken from that stream until \EOF, at which point the \method{close()} method of that stream will be called and the input source will again become the original input stream. Source requests may be stacked any number of levels deep. \end{memberdesc} \begin{memberdesc}{debug} If this member is numeric and \code{1} or more, a \class{shlex} instance will print verbose progress output on its behavior. If you need to use this, you can read the module source code to learn the details. \end{memberdesc} \begin{memberdesc}{lineno} Source line number (count of newlines seen so far plus one). \end{memberdesc} \begin{memberdesc}{token} The token buffer. It may be useful to examine this when catching exceptions. \end{memberdesc} \begin{memberdesc}{eof} Token used to determine end of file. This will be set to the empty string (\code{""}), in non-posix mode, and to \code{None} in posix mode. \versionadded{2.3} \end{memberdesc} \subsection{Parsing Rules\label{shlex-parsing-rules}} When operating in non-posix mode, \class{shlex} with try to obey to the following rules. \begin{itemize} \item Quote characters are not recognized within words (\code{Do"Not"Separate} is parsed as the single word \code{Do"Not"Separate}); \item Escape characters are not recognized; \item Enclosing characters in quotes preserve the literal value of all characters within the quotes; \item Closing quotes separate words (\code{"Do"Separate} is parsed as \code{"Do"} and \code{Separate}); \item If \member{whitespace_split} is \code{False}, any character not declared to be a word character, whitespace, or a quote will be returned as a single-character token. If it is \code{True}, \class{shlex} will only split words in whitespaces; \item EOF is signaled with an empty string (\code{""}); \item It's not possible to parse empty strings, even if quoted. \end{itemize} When operating in posix mode, \class{shlex} will try to obey to the following parsing rules. \begin{itemize} \item Quotes are stripped out, and do not separate words (\code{"Do"Not"Separate"} is parsed as the single word \code{DoNotSeparate}); \item Non-quoted escape characters (e.g. \character{\textbackslash}) preserve the literal value of the next character that follows; \item Enclosing characters in quotes which are not part of \member{escapedquotes} (e.g. \character{'}) preserve the literal value of all characters within the quotes; \item Enclosing characters in quotes which are part of \member{escapedquotes} (e.g. \character{"}) preserves the literal value of all characters within the quotes, with the exception of the characters mentioned in \member{escape}. The escape characters retain its special meaning only when followed by the quote in use, or the escape character itself. Otherwise the escape character will be considered a normal character. \item EOF is signaled with a \code{None} value; \item Quoted empty strings (\code{""}) are allowed; \end{itemize} `