## [a530bb]: doc / cmucl / cmu-user / cmu-user.tex  Maximize  Restore  History

### 13322 lines (11054 with data), 528.6 kB


CMU Common Lisp User's Manual

Robert A. MacLachlan, Editor

July 1992
CMU-CS-92-161

July 1997
Net Version

School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213

Supersedes Technical Reports CMU-CS-87-156 and CMU-CS-91-108.

Abstract

CMU Common Lisp is an implementation of that Common Lisp runs on various Unix workstations. See the README file in the distribution for current platforms. The largest single part of this document describes the Python compiler and the programming styles and techniques that the compiler encourages. The rest of the document describes extensions and the implementation dependent choices made in developing this implementation of Common Lisp. We have added several extensions, including a source level debugger, an interface to Unix system calls, a foreign function call interface, support for interprocess communication and remote procedure call, and other features that provide a good environment for developing Lisp code.

This research was sponsored by the Defense Advanced Research Projects Agency, Information Science and Technology Office, under the title Research on Parallel Computing issued by DARPA/CMO under Contract MDA972-90-C-0035 ARPA Order No. 7330.

The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency or the U.S. government.

\end{rawhtml} \clearpage \vspace*{\fill} \textbf{Keywords:} lisp, Common Lisp, manual, compiler, programming language implementation, programming environment \clearpage \pagestyle{headings} \pagenumbering{roman} \tableofcontents \clearpage \pagenumbering{arabic} %%\end{iftex} %%\setfilename{cmu-user.info} %%\node Top, Introduction, (dir), (dir) \hide{File:/afs/cs.cmu.edu/project/clisp/hackers/ram/docs/cmu-user/intro.ms} \hide{ -*- Dictionary: cmu-user -*- } \begin{comment} * Introduction:: * Design Choices and Extensions:: * The Debugger:: * The Compiler:: * Advanced Compiler Use and Efficiency Hints:: * UNIX Interface:: * Event Dispatching with SERVE-EVENT:: * Alien Objects:: * Interprocess Communication under LISP:: * Debugger Programmer's Interface:: * Function Index:: * Variable Index:: * Type Index:: * Concept Index:: --- The Detailed Node Listing --- Introduction * Support:: * Local Distribution of CMU Common Lisp:: * Net Distribution of CMU Common Lisp:: * Source Availability:: * Command Line Options:: * Credits:: Design Choices and Extensions * Data Types:: * Default Interrupts for Lisp:: * Packages:: * The Editor:: * Garbage Collection:: * Describe:: * The Inspector:: * Load:: * The Reader:: * Running Programs from Lisp:: * Saving a Core Image:: * Pathnames:: * Filesystem Operations:: * Time Parsing and Formatting:: * Lisp Library:: Data Types * Symbols:: * Integers:: * Floats:: * Characters:: * Array Initialization:: Floats * IEEE Special Values:: * Negative Zero:: * Denormalized Floats:: * Floating Point Exceptions:: * Floating Point Rounding Mode:: * Accessing the Floating Point Modes:: The Inspector * The Graphical Interface:: * The TTY Inspector:: Running Programs from Lisp * Process Accessors:: Pathnames * Unix Pathnames:: * Wildcard Pathnames:: * Logical Pathnames:: * Search Lists:: * Predefined Search-Lists:: * Search-List Operations:: * Search List Example:: Logical Pathnames * Search Lists:: * Search List Example:: Search-List Operations * Search List Example:: Filesystem Operations * Wildcard Matching:: * File Name Completion:: * Miscellaneous Filesystem Operations:: The Debugger * Debugger Introduction:: * The Command Loop:: * Stack Frames:: * Variable Access:: * Source Location Printing:: * Compiler Policy Control:: * Exiting Commands:: * Information Commands:: * Breakpoint Commands:: * Function Tracing:: * Specials:: Stack Frames * Stack Motion:: * How Arguments are Printed:: * Function Names:: * Funny Frames:: * Debug Tail Recursion:: * Unknown Locations and Interrupts:: Variable Access * Variable Value Availability:: * Note On Lexical Variable Access:: Source Location Printing * How the Source is Found:: * Source Location Availability:: Breakpoint Commands * Breakpoint Example:: Function Tracing * Encapsulation Functions:: The Compiler * Compiler Introduction:: * Calling the Compiler:: * Compilation Units:: * Interpreting Error Messages:: * Types in Python:: * Getting Existing Programs to Run:: * Compiler Policy:: * Open Coding and Inline Expansion:: Compilation Units * Undefined Warnings:: Interpreting Error Messages * The Parts of the Error Message:: * The Original and Actual Source:: * The Processing Path:: * Error Severity:: * Errors During Macroexpansion:: * Read Errors:: * Error Message Parameterization:: Types in Python * Compile Time Type Errors:: * Precise Type Checking:: * Weakened Type Checking:: Compiler Policy * The Optimize Declaration:: * The Optimize-Interface Declaration:: Advanced Compiler Use and Efficiency Hints * Advanced Compiler Introduction:: * More About Types in Python:: * Type Inference:: * Source Optimization:: * Tail Recursion:: * Local Call:: * Block Compilation:: * Inline Expansion:: * Byte Coded Compilation:: * Object Representation:: * Numbers:: * General Efficiency Hints:: * Efficiency Notes:: * Profiling:: Advanced Compiler Introduction * Types:: * Optimization:: * Function Call:: * Representation of Objects:: * Writing Efficient Code:: More About Types in Python * More Types Meaningful:: * Canonicalization:: * Member Types:: * Union Types:: * The Empty Type:: * Function Types:: * The Values Declaration:: * Structure Types:: * The Freeze-Type Declaration:: * Type Restrictions:: * Type Style Recommendations:: Type Inference * Variable Type Inference:: * Local Function Type Inference:: * Global Function Type Inference:: * Operation Specific Type Inference:: * Dynamic Type Inference:: * Type Check Optimization:: Source Optimization * Let Optimization:: * Constant Folding:: * Unused Expression Elimination:: * Control Optimization:: * Unreachable Code Deletion:: * Multiple Values Optimization:: * Source to Source Transformation:: * Style Recommendations:: Tail Recursion * Tail Recursion Exceptions:: Local Call * Self-Recursive Calls:: * Let Calls:: * Closures:: * Local Tail Recursion:: * Return Values:: Block Compilation * Block Compilation Semantics:: * Block Compilation Declarations:: * Compiler Arguments:: * Practical Difficulties:: * Context Declarations:: * Context Declaration Example:: Inline Expansion * Inline Expansion Recording:: * Semi-Inline Expansion:: * The Maybe-Inline Declaration:: Object Representation * Think Before You Use a List:: * Structure Representation:: * Arrays:: * Vectors:: * Bit-Vectors:: * Hashtables:: Numbers * Descriptors:: * Non-Descriptor Representations:: * Variables:: * Generic Arithmetic:: * Fixnums:: * Word Integers:: * Floating Point Efficiency:: * Specialized Arrays:: * Specialized Structure Slots:: * Interactions With Local Call:: * Representation of Characters:: General Efficiency Hints * Compile Your Code:: * Avoid Unnecessary Consing:: * Complex Argument Syntax:: * Mapping and Iteration:: * Trace Files and Disassembly:: Efficiency Notes * Type Uncertainty:: * Efficiency Notes and Type Checking:: * Representation Efficiency Notes:: * Verbosity Control:: Profiling * Profile Interface:: * Profiling Techniques:: * Nested or Recursive Calls:: * Clock resolution:: * Profiling overhead:: * Additional Timing Utilities:: * A Note on Timing:: * Benchmarking Techniques:: UNIX Interface * Reading the Command Line:: * Lisp Equivalents for C Routines:: * Type Translations:: * System Area Pointers:: * Unix System Calls:: * File Descriptor Streams:: * Making Sense of Mach Return Codes:: * Unix Interrupts:: Unix Interrupts * Changing Interrupt Handlers:: * Examples of Signal Handlers:: Event Dispatching with SERVE-EVENT * Object Sets:: * The SERVE-EVENT Function:: * Using SERVE-EVENT with Unix File Descriptors:: * Using SERVE-EVENT with the CLX Interface to X:: * A SERVE-EVENT Example:: Using SERVE-EVENT with the CLX Interface to X * Without Object Sets:: * With Object Sets:: A SERVE-EVENT Example * Without Object Sets Example:: * With Object Sets Example:: Alien Objects * Introduction to Aliens:: * Alien Types:: * Alien Operations:: * Alien Variables:: * Alien Data Structure Example:: * Loading Unix Object Files:: * Alien Function Calls:: * Step-by-Step Alien Example:: Alien Types * Defining Alien Types:: * Alien Types and Lisp Types:: * Alien Type Specifiers:: * The C-Call Package:: Alien Operations * Alien Access Operations:: * Alien Coercion Operations:: * Alien Dynamic Allocation:: Alien Variables * Local Alien Variables:: * External Alien Variables:: Alien Function Calls * alien-funcall:: The alien-funcall Primitive * def-alien-routine:: The def-alien-routine Macro * def-alien-routine Example:: * Calling Lisp from C:: Interprocess Communication under LISP * The REMOTE Package:: * The WIRE Package:: * Out-Of-Band Data:: The REMOTE Package * Connecting Servers and Clients:: * Remote Evaluations:: * Remote Objects:: * Host Addresses:: The WIRE Package * Untagged Data:: * Tagged Data:: * Making Your Own Wires:: Debugger Programmer's Interface * DI Exceptional Conditions:: * Debug-variables:: * Frames:: * Debug-functions:: * Debug-blocks:: * Breakpoints:: * Code-locations:: * Debug-sources:: * Source Translation Utilities:: DI Exceptional Conditions * Debug-conditions:: * Debug-errors:: \end{comment} %%\node Introduction, Design Choices and Extensions, Top, Top \chapter{Introduction} CMU Common Lisp is a public-domain implementation of Common Lisp developed in the Computer Science Department of Carnegie Mellon University. \cmucl{} runs on various Unix workstations---see the README file in the distribution for current platforms. This document describes the implementation based on the Python compiler. Previous versions of CMU Common Lisp ran on the IBM RT PC and (when known as Spice Lisp) on the Perq workstation. See \code{man cmucl} (\file{man/man1/cmucl.1}) for other general information. \cmucl{} sources and executables are freely available via anonymous FTP; this software is as is'', and has no warranty of any kind. CMU and the authors assume no responsibility for the consequences of any use of this software. See \file{doc/release-notes.txt} for a description of the state of the release you have. \begin{comment} * Support:: * Local Distribution of CMU Common Lisp:: * Net Distribution of CMU Common Lisp:: * Source Availability:: * Command Line Options:: * Credits:: \end{comment} %%\node Support, Local Distribution of CMU Common Lisp, Introduction, Introduction \section{Support} The CMU Common Lisp project is no longer funded, so only minimal support is being done at CMU. There is a net community of \cmucl{} users and maintainers who communicate via comp.lang.lisp and the cmucl-bugs@cs.cmu.edu \begin{changebar} cmucl-imp@cons.org \end{changebar} mailing lists. This manual contains only implementation-specific information about \cmucl. Users will also need a separate manual describing the \clisp{} standard. \clisp{} was initially defined in \i{Common Lisp: The Language}, by Guy L. Steele Jr. \clisp{} is now undergoing standardization by the X3J13 committee of ANSI. The X3J13 spec is not yet completed, but a number of clarifications and modification have been approved. We intend that \cmucl{} will eventually adhere to the X3J13 spec, and we have already implemented many of the changes approved by X3J13. Until the X3J13 standard is completed, the second edition of \cltltwo{} is probably the best available manual for the language and for our implementation of it. This book has no official role in the standardization process, but it does include many of the changes adopted since the first edition was completed. In addition to the language itself, this document describes a number of useful library modules that run in \cmucl. \hemlock, an Emacs-like text editor, is included as an integral part of the \cmucl{} environment. Two documents describe \hemlock{}: the \i{Hemlock User's Manual}, and the \i{Hemlock Command Implementor's Manual}. %%\node Local Distribution of CMU Common Lisp, Net Distribution of CMU Common Lisp, Support, Introduction \section{Local Distribution of CMU Common Lisp} In CMU CS, \cmucl{} should be runnable as \file{/usr/local/bin/cmucl}. The full binary distribution should appear under \file{/usr/local/lib/cmucl/}. Note that the first time you run Lisp, it will take AFS several minutes to copy the image into its local cache. Subsequent starts will be much faster. Or, you can run directly out of the AFS release area (which may be necessary on SunOS machines). Put this in your \file{.login} shell script: \begin{example} setenv CMUCLLIB "/afs/cs/misc/cmucl/@sys/beta/lib" setenv PATH \${PATH}:/afs/cs/misc/cmucl/@sys/beta/bin \end{example} If you also set \code{MANPATH} or \code{MPATH} (depending on the Unix) to point to \file{/usr/local/lib/cmucl/man/}, then \code{man cmucl}' will give an introduction to CMU CL and \samp{man lisp} will describe command line options. For installation notes, see the \file{README} file in the release area. See \file{/usr/local/lib/cmucl/doc} for release notes and documentation. Hardcopy documentation is available in the document room. Documentation supplements may be available for recent additions: see the \file{README} file. Send bug reports and questions to \samp{cmucl-bugs@cs.cmu.edu}. If you send a bug report to \samp{gripe} or \samp{help}, they will just forward it to this mailing list. %%\node Net Distribution of CMU Common Lisp, Source Availability, Local Distribution of CMU Common Lisp, Introduction \section{Net Distribution of CMU Common Lisp} \subsection{CMU Distribution} Externally, CMU Common Lisp is only available via anonymous FTP. We don't have the manpower to make tapes. These are our distribution machines: \begin{example} lisp-rt1.slisp.cs.cmu.edu (128.2.217.9) lisp-rt2.slisp.cs.cmu.edu (128.2.217.10) \end{example} Log in with the user \samp{anonymous} and \samp{username@host} as password (i.e. your EMAIL address.) When you log in, the current directory should be set to the \cmucl{} release area. If you have any trouble with FTP access, please send mail to \samp{slisp@cs.cmu.edu}. The release area holds compressed tar files with names of the form: \begin{example} \var{version}-\var{machine}_\var{os}.tar.Z \end{example} FTP compressed tar archives in binary mode. To extract, \samp{cd} to the directory that is to be the root of the tree, then type: \begin{example} uncompress \var{value}}. For example, to start up the saved core file mylisp.core use either of the following two commands: \begin{example} \code{lisp -core=mylisp.core lisp -core mylisp.core} \end{example} %%\node Credits, , Command Line Options, Introduction \section{Credits} Since 1981 many people have contributed to the development of CMU Common Lisp. The currently active members are: \begin{display} Marco Antoniotti David Axmark Miles Bader Casper Dik Scott Fahlman * (fearless leader) Paul Gleichauf * Richard Harris Joerg-Cyril Hoehl Chris Hoover Simon Leinen Sandra Loosemore William Lott * Robert A. Maclachlan * \end{display} \noindent Many people are voluntarily working on improving CMU Common Lisp. *'' means a full-time CMU employee, and +'' means a part-time student employee. A partial listing of significant past contributors follows: \begin{display} Tim Moore Sean Hallgren + Mike Garland + Ted Dunning Rick Busdiecker Bill Chiles * John Kolojejchick Todd Kaufmann + Dave McDonald * Skef Wholey * \end{display} \vspace{2 em} \researchcredit \begin{changebar} From 1995, development of CMU Common Lisp has been continued by a group of volunteers. A partial list of volunteers includes the following \begin{table}[h] \begin{center} \begin{tabular}{ll} Paul Werkowski & pw@snoopy.mv.com \\ Peter VanEynde & s950045@uia.ua.ac.be \\ Marco Antoniotti & marcoxa@PATH.Berkeley.EDU\\ Martin Cracauer & cracauer@cons.org\\ Douglas Thomas Crosher & dtc@scrooge.ee.swin.oz.au\\ Simon Leinen & simon@switch.ch\\ Rob MacLachlan & ram+@CS.cmu.edu\\ Raymond Toy & toy@rtp.ericsson.se \end{tabular} \end{center} \end{table} In particular Paul Werkowski completed the port for the x86 architecture for FreeBSD. Peter VanEnyde took the FreeBSD port and created a Linux version. \end{changebar} \hide{File:/afs/cs.cmu.edu/project/clisp/hackers/ram/docs/cmu-user/design.ms} \hide{ -*- Dictionary: cmu-user -*- } %%\node Design Choices and Extensions, The Debugger, Introduction, Top \chapter{Design Choices and Extensions} Several design choices in Common Lisp are left to the individual implementation, and some essential parts of the programming environment are left undefined. This chapter discusses the most important design choices and extensions. \begin{comment} * Data Types:: * Default Interrupts for Lisp:: * Packages:: * The Editor:: * Garbage Collection:: * Describe:: * The Inspector:: * Load:: * The Reader:: * Running Programs from Lisp:: * Saving a Core Image:: * Pathnames:: * Filesystem Operations:: * Time Parsing and Formatting:: * Lisp Library:: \end{comment} %%\node Data Types, Default Interrupts for Lisp, Design Choices and Extensions, Design Choices and Extensions \section{Data Types} \begin{comment} * Symbols:: * Integers:: * Floats:: * Characters:: * Array Initialization:: \end{comment} %%\node Symbols, Integers, Data Types, Data Types \subsection{Symbols} As in \cltl, all symbols and package names are printed in lower case, as a user is likely to type them. Internally, they are normally stored upper case only. %%\node Integers, Floats, Symbols, Data Types \subsection{Integers} The \tindexed{fixnum} type is equivalent to \code{(signed-byte 30)}. Integers outside this range are represented as a \tindexed{bignum} or a word integer (\pxlref{word-integers}.) Almost all integers that appear in programs can be represented as a \code{fixnum}, so integer number consing is rare. %%\node Floats, Characters, Integers, Data Types \subsection{Floats} \label{ieee-float} \cmucl{} supports two floating point formats: \tindexed{single-float} and \tindexed{double-float}. These are implemented with IEEE single and double float arithmetic, respectively. \code{short-float} is a synonym for \code{single-float}, and \code{long-float} is a synonym for \code{double-float}. The initial value of \vindexed{read-default-float-format} is \code{single-float}. Both \code{single-float} and \code{double-float} are represented with a pointer descriptor, so float operations can cause number consing. Number consing is greatly reduced if programs are written to allow the use of non-descriptor representations (\pxlref{numeric-types}.) \begin{comment} * IEEE Special Values:: * Negative Zero:: * Denormalized Floats:: * Floating Point Exceptions:: * Floating Point Rounding Mode:: * Accessing the Floating Point Modes:: \end{comment} %%\node IEEE Special Values, Negative Zero, Floats, Floats \subsubsection{IEEE Special Values} \cmucl{} supports the IEEE infinity and NaN special values. These non-numeric values will only be generated when trapping is disabled for some floating point exception (\pxlref{float-traps}), so users of the default configuration need not concern themselves with special values. \begin{defconst}{extensions:}{short-float-positive-infinity} \defconstx[extensions:]{short-float-negative-infinity} \defconstx[extensions:]{single-float-positive-infinity} \defconstx[extensions:]{single-float-negative-infinity} \defconstx[extensions:]{double-float-positive-infinity} \defconstx[extensions:]{double-float-negative-infinity} \defconstx[extensions:]{long-float-positive-infinity} \defconstx[extensions:]{long-float-negative-infinity} The values of these constants are the IEEE positive and negative infinity objects for each float format. \end{defconst} \begin{defun}{extensions:}{float-infinity-p}{\args{\var{x}}} This function returns true if \var{x} is an IEEE float infinity (of either sign.) \var{x} must be a float. \end{defun} \begin{defun}{extensions:}{float-nan-p}{\args{\var{x}}} \defunx[extensions:]{float-trapping-nan-p}{\args{\var{x}}} \code{float-nan-p} returns true if \var{x} is an IEEE NaN (Not A Number) object. \code{float-trapping-nan-p} returns true only if \var{x} is a trapping NaN. With either function, \var{x} must be a float. \end{defun} %%\node Negative Zero, Denormalized Floats, IEEE Special Values, Floats \subsubsection{Negative Zero} The IEEE float format provides for distinct positive and negative zeros. To test the sign on zero (or any other float), use the \clisp{} \findexed{float-sign} function. Negative zero prints as \code{-0.0f0} or \code{-0.0d0}. %%\node Denormalized Floats, Floating Point Exceptions, Negative Zero, Floats \subsubsection{Denormalized Floats} \cmucl{} supports IEEE denormalized floats. Denormalized floats provide a mechanism for gradual underflow. The \clisp{} \findexed{float-precision} function returns the actual precision of a denormalized float, which will be less than \findexed{float-digits}. Note that in order to generate (or even print) denormalized floats, trapping must be disabled for the underflow exception (\pxlref{float-traps}.) The \clisp{} \w{\code{least-positive-}\var{format}-\code{float}} constants are denormalized. \begin{defun}{extensions:}{float-normalized-p}{\args{\var{x}}} This function returns true if \var{x} is a denormalized float. \var{x} must be a float. \end{defun} %%\node Floating Point Exceptions, Floating Point Rounding Mode, Denormalized Floats, Floats \subsubsection{Floating Point Exceptions} \label{float-traps} The IEEE floating point standard defines several exceptions that occur when the result of a floating point operation is unclear or undesirable. Exceptions can be ignored, in which case some default action is taken, such as returning a special value. When trapping is enabled for an exception, a error is signalled whenever that exception occurs. These are the possible floating point exceptions: \begin{Lentry} \item[\kwd{underflow}] This exception occurs when the result of an operation is too small to be represented as a normalized float in its format. If trapping is enabled, the \tindexed{floating-point-underflow} condition is signalled. Otherwise, the operation results in a denormalized float or zero. \item[\kwd{overflow}] This exception occurs when the result of an operation is too large to be represented as a float in its format. If trapping is enabled, the \tindexed{floating-point-overflow} exception is signalled. Otherwise, the operation results in the appropriate infinity. \item[\kwd{inexact}] This exception occurs when the result of a floating point operation is not exact, i.e. the result was rounded. If trapping is enabled, the \code{extensions:floating-point-inexact} condition is signalled. Otherwise, the rounded result is returned. \item[\kwd{invalid}] This exception occurs when the result of an operation is ill-defined, such as \code{\w{(/ 0.0 0.0)}}. If trapping is enabled, the \code{extensions:floating-point-invalid} condition is signalled. Otherwise, a quiet NaN is returned. \item[\kwd{divide-by-zero}] This exception occurs when a float is divided by zero. If trapping is enabled, the \tindexed{divide-by-zero} condition is signalled. Otherwise, the appropriate infinity is returned. \end{Lentry} %%\node Floating Point Rounding Mode, Accessing the Floating Point Modes, Floating Point Exceptions, Floats \subsubsection{Floating Point Rounding Mode} \label{float-rounding-modes} IEEE floating point specifies four possible rounding modes: \begin{Lentry} \item[\kwd{nearest}] In this mode, the inexact results are rounded to the nearer of the two possible result values. If the neither possibility is nearer, then the even alternative is chosen. This form of rounding is also called round to even'', and is the form of rounding specified for the \clisp{} \findexed{round} function. \item[\kwd{positive-infinity}] This mode rounds inexact results to the possible value closer to positive infinity. This is analogous to the \clisp{} \findexed{ceiling} function. \item[\kwd{negative-infinity}] This mode rounds inexact results to the possible value closer to negative infinity. This is analogous to the \clisp{} \findexed{floor} function. \item[\kwd{zero}] This mode rounds inexact results to the possible value closer to zero. This is analogous to the \clisp{} \findexed{truncate} function. \end{Lentry} \paragraph{Warning:} Although the rounding mode can be changed with \code{set-floating-point-modes}, use of any value other than the default (\kwd{nearest}) can cause unusual behavior, since it will affect rounding done by \llisp{} system code as well as rounding in user code. In particular, the unary \code{round} function will stop doing round-to-nearest on floats, and instead do the selected form of rounding. %%\node Accessing the Floating Point Modes, , Floating Point Rounding Mode, Floats \subsubsection{Accessing the Floating Point Modes} These functions can be used to modify or read the floating point modes: \begin{defun}{extensions:}{set-floating-point-modes}{% \keys{\kwd{traps} \kwd{rounding-mode}} \morekeys{\kwd{fast-mode} \kwd{accrued-exceptions}} \yetmorekeys{\kwd{current-exceptions}}} \defunx[extensions:]{get-floating-point-modes}{} The keyword arguments to \code{set-floating-point-modes} set various modes controlling how floating point arithmetic is done: \begin{Lentry} \item[\kwd{traps}] A list of the exception conditions that should cause traps. Possible exceptions are \kwd{underflow}, \kwd{overflow}, \kwd{inexact}, \kwd{invalid} and \kwd{divide-by-zero}. Initially all traps except \kwd{inexact} are enabled. \xlref{float-traps}. \item[\kwd{rounding-mode}] The rounding mode to use when the result is not exact. Possible values are \kwd{nearest}, \latex{\kwd{positive\-infinity}}\html{\kwd{positive-infinity}}, \kwd{negative-infinity} and \kwd{zero}. Initially, the rounding mode is \kwd{nearest}. See the warning in section \ref{float-rounding-modes} about use of other rounding modes. \item[\kwd{current-exceptions}, \kwd{accrued-exceptions}] Lists of exception keywords used to set the exception flags. The \var{current-exceptions} are the exceptions for the previous operation, so setting it is not very useful. The \var{accrued-exceptions} are a cumulative record of the exceptions that occurred since the last time these flags were cleared. Specifying \code{()} will clear any accrued exceptions. \item[\kwd{fast-mode}] Set the hardware's fast mode'' flag, if any. When set, IEEE conformance or debuggability may be impaired. Some machines may not have this feature, in which case the value is always \false. No currently supported machines have a fast mode. \end{Lentry} If a keyword argument is not supplied, then the associated state is not changed. \code{get-floating-point-modes} returns a list representing the state of the floating point modes. The list is in the same format as the keyword arguments to \code{set-floating-point-modes}, so \code{apply} could be used with \code{set-floating-point-modes} to restore the modes in effect at the time of the call to \code{get-floating-point-modes}. \end{defun} \begin{changebar} To make handling control of floating-point exceptions, the following macro is useful. \begin{defmac}{ext:}{with-float-traps-masked}{traps \ampbody\ body} \code{body} is executed with the selected floating-point exceptions given by \code{traps} masked out (disabled). \code{traps} should be a list of possible floating-point exceptions that should be ignored. Possible values are \kwd{underflow}, \kwd{overflow}, \kwd{inexact}, \kwd{invalid} and \kwd{divide-by-zero}. This is equivalent to saving the current traps from \code{get-floating-point-modes}, setting the floating-point modes to the desired exceptions, running the \code{body}, and restoring the saved floating-point modes. The advantage of this macro is that it causes less consing to occur. Some points about the with-float-traps-masked: \begin{itemize} \item Two approaches are available for detecting FP exceptions: \begin{enumerate} \item enabling the traps and handling the exceptions \item disabling the traps and either handling the return values or checking the accrued exceptions. \end{enumerate} Of these the latter is the most portable because on the alpha port it is not possible to enable some traps at run-time. \item To assist the checking of the exceptions within the body any accrued exceptions matching the given traps are cleared at the start of the body when the traps are masked. \item To allow the macros to be nested these accrued exceptions are restored at the end of the body to their values at the start of the body. Thus any exceptions that occurred within the body will not affect the accrued exceptions outside the macro. \item Note that only the given exceptions are restored at the end of the body so other exception will be visible in the accrued exceptions outside the body. \item On the x86, setting the accrued exceptions of an unmasked exception would cause a FP trap. The macro behaviour of restoring the accrued exceptions ensures than if an accrued exception is initially not flagged and occurs within the body it will be restored/cleared at the exit of the body and thus not cause a trap. \item On the x86, and, perhaps, the hppa, the FP exceptions may be delivered at the next FP instruction which requires a FP \code{wait} instruction (\code{%vm::float-wait}) if using the lisp conditions to catch trap within a \code{handler-bind}. The \code{handler-bind} macro does the right thing and inserts a float-wait (at the end of its body on the x86). The masking and noting of exceptions is also safe here. \item The setting of the FP flags uses the \code{(floating-point-modes)} and the \code{(set (floating-point-modes)\ldots)} VOPs. These VOPs blindly update the flags which may include other state. We assume this state hasn't changed in between getting and setting the state. For example, if you used the FP unit between the above calls, the state may be incorrectly restored! The \code{with-float-traps-masked} macro keeps the intervening code to a minimum and uses only integer operations. %% Safe byte-compiled? %% Perhaps the VOPs (x86) should be smarter and only update some of %% the flags, the trap masks and exceptions? \end{itemize} \end{defmac} \end{changebar} %%\node Characters, Array Initialization, Floats, Data Types \subsection{Characters} \cmucl{} implements characters according to \i{Common Lisp: the Language II}. The main difference from the first version is that character bits and font have been eliminated, and the names of the types have been changed. \tindexed{base-character} is the new equivalent of the old \tindexed{string-char}. In this implementation, all characters are base characters (there are no extended characters.) Character codes range between \code{0} and \code{255}, using the ASCII encoding. \begin{changebar} Table~\ref{tbl:chars}~\vpageref{tbl:chars} shows characters recognized by \cmucl. \end{changebar} \begin{changebar} \begin{table}[tbhp] \begin{center} \begin{tabular}{|c|c|l|l|l|l|} \hline \multicolumn{2}{|c|}{ASCII} & \multicolumn{1}{|c}{Lisp} & \multicolumn{3}{|c|}{} \\ \cline{1-2} Name & Code & \multicolumn{1}{|c|}{Name} & \multicolumn{3}{|c|}{\raisebox{1.5ex}{Alternatives}}\\ \hline \hline \code{nul} & 0 & \code{\#\back{NULL}} & \code{\#\back{NUL}} & &\\ \code{bel} & 7 & \code{\#\back{BELL}} & & &\\ \code{bs} & 8 & \code{\#\back{BACKSPACE}} & \code{\#\back{BS}} & &\\ \code{tab} & 9 & \code{\#\back{TAB}} & & &\\ \code{lf} & 10 & \code{\#\back{NEWLINE}} & \code{\#\back{NL}} & \code{\#\back{LINEFEED}} & \code{\#\back{LF}}\\ \code{ff} & 11 & \code{\#\back{VT}} & \code{\#\back{PAGE}} & \code{\#\back{FORM}} &\\ \code{cr} & 13 & \code{\#\back{RETURN}} & \code{\#\back{CR}} & &\\ \code{esc} & 27 & \code{\#\back{ESCAPE}} & \code{\#\back{ESC}} & \code{\#\back{ALTMODE}} & \code{\#\back{ALT}}\\ \code{sp} & 32 & \code{\#\back{SPACE}} & \code{\#\back{SP}} & &\\ \code{del} & 127 & \code{\#\back{DELETE}} & \code{\#\back{RUBOUT}} & &\\ \hline \end{tabular} \caption{Characters recognized by \cmucl} \label{tbl:chars} \end{center} \end{table} \end{changebar} %%\node Array Initialization, , Characters, Data Types \subsection{Array Initialization} If no \kwd{initial-value} is specified, arrays are initialized to zero. %%\node Default Interrupts for Lisp, Packages, Data Types, Design Choices and Extensions \section{Default Interrupts for Lisp} CMU Common Lisp has several interrupt handlers defined when it starts up, as follows: \begin{Lentry} \item[\code{SIGINT} (\ctrl{c})] causes Lisp to enter a break loop. This puts you into the debugger which allows you to look at the current state of the computation. If you proceed from the break loop, the computation will proceed from where it was interrupted. \item[\code{SIGQUIT} (\ctrl{L})] causes Lisp to do a throw to the top-level. This causes the current computation to be aborted, and control returned to the top-level read-eval-print loop. \item[\code{SIGTSTP} (\ctrl{z})] causes Lisp to suspend execution and return to the Unix shell. If control is returned to Lisp, the computation will proceed from where it was interrupted. \item[\code{SIGILL}, \code{SIGBUS}, \code{SIGSEGV}, and \code{SIGFPE}] cause Lisp to signal an error. \end{Lentry} For keyboard interrupt signals, the standard interrupt character is in parentheses. Your \file{.login} may set up different interrupt characters. When a signal is generated, there may be some delay before it is processed since Lisp cannot be interrupted safely in an arbitrary place. The computation will continue until a safe point is reached and then the interrupt will be processed. \xlref{signal-handlers} to define your own signal handlers. %%\node Packages, The Editor, Default Interrupts for Lisp, Design Choices and Extensions \section{Packages} When CMU Common Lisp is first started up, the default package is the \code{user} package. The \code{user} package uses the \code{common-lisp}, \code{extensions}, and \code{pcl} packages. The symbols exported from these three packages can be referenced without package qualifiers. This section describes packages which have exported interfaces that may concern users. The numerous internal packages which implement parts of the system are not described here. Package nicknames are in parenthesis after the full name. \begin{Lentry} \item[\code{alien}, \code{c-call}] Export the features of the Alien foreign data structure facility (\pxlref{aliens}.) \item[\code{pcl}] This package contains PCL (Portable CommonLoops), which is a portable implementation of CLOS (the Common Lisp Object System.) This implements most (but not all) of the features in the CLOS chapter of \cltltwo. \item[\code{debug}] The \code{debug} package contains the command-line oriented debugger. It exports utility various functions and switches. \item[\code{debug-internals}] The \code{debug-internals} package exports the primitives used to write debuggers. \xlref{debug-internals}. \item[\code{extensions (ext)}] The \code{extensions} packages exports local extensions to Common Lisp that are documented in this manual. Examples include the \code{save-lisp} function and time parsing. \item[\code{hemlock (ed)}] The \code{hemlock} package contains all the code to implement Hemlock commands. The \code{hemlock} package currently exports no symbols. \item[\code{hemlock-internals (hi)}] The \code{hemlock-internals} package contains code that implements low level primitives and exports those symbols used to write Hemlock commands. \item[\code{keyword}] The \code{keyword} package contains keywords (e.g., \kwd{start}). All symbols in the \code{keyword} package are exported and evaluate to themselves (i.e., the value of the symbol is the symbol itself). \item[\code{profile}] The \code{profile} package exports a simple run-time profiling facility (\pxlref{profiling}). \item[\code{common-lisp (cl lisp)}] The \code{common-lisp} package exports all the symbols defined by \i{Common Lisp: the Language} and only those symbols. Strictly portable Lisp code will depend only on the symbols exported from the \code{lisp} package. \item[\code{unix}, \code{mach}] These packages export system call interfaces to generic BSD Unix and Mach (\pxlref{unix-interface}). \item[\code{system (sys)}] The \code{system} package contains functions and information necessary for system interfacing. This package is used by the \code{lisp} package and exports several symbols that are necessary to interface to system code. \item[\code{common-lisp-user (user cl-user)}] The \code{common-lisp-user} package is the default package and is where a user's code and data is placed unless otherwise specified. This package exports no symbols. \item[\code{xlib}] The \code{xlib} package contains the Common Lisp X interface (CLX) to the X11 protocol. This is mostly Lisp code with a couple of functions that are defined in C to connect to the server. \item[\code{wire}] The \code{wire} package exports a remote procedure call facility (\pxlref{remote}). \end{Lentry} %%\node The Editor, Garbage Collection, Packages, Design Choices and Extensions \section{The Editor} The \code{ed} function invokes the Hemlock editor which is described in \i{Hemlock User's Manual} and \i{Hemlock Command Implementor's Manual}. Most users at CMU prefer to use Hemlock's slave \Llisp{} mechanism which provides an interactive buffer for the \code{read-eval-print} loop and editor commands for evaluating and compiling text from a buffer into the slave \Llisp. Since the editor runs in the \Llisp, using slaves keeps users from trashing their editor by developing in the same \Llisp{} with \Hemlock. %%\node Garbage Collection, Describe, The Editor, Design Choices and Extensions \section{Garbage Collection} CMU Common Lisp uses a stop-and-copy garbage collector that compacts the items in dynamic space every time it runs. Most users cause the system to garbage collect (GC) frequently, long before space is exhausted. With 16 or 24 megabytes of memory, causing GC's more frequently on less garbage allows the system to GC without much (if any) paging. \hide{ With the default value for the following variable, you can expect a GC to take about one minute of elapsed time on a 6 megabyte machine running X as well as Lisp. On machines with 8 megabytes or more of memory a GC should run without much (if any) paging. GC's run more frequently but tend to take only about 5 seconds. } The following functions invoke the garbage collector or control whether automatic garbage collection is in effect: \begin{defun}{extensions:}{gc}{} This function runs the garbage collector. If \code{ext:*gc-verbose*} is non-\nil, then it invokes \code{ext:*gc-notify-before*} before GC'ing and \code{ext:*gc-notify-after*} afterwards. \end{defun} \begin{defun}{extensions:}{gc-off}{} This function inhibits automatic garbage collection. After calling it, the system will not GC unless you call \code{ext:gc} or \code{ext:gc-on}. \end{defun} \begin{defun}{extensions:}{gc-on}{} This function reinstates automatic garbage collection. If the system would have GC'ed while automatic GC was inhibited, then this will call \code{ext:gc}. \end{defun} %%\node \subsection{GC Parameters} The following variables control the behavior of the garbage collector: \begin{defvar}{extensions:}{bytes-consed-between-gcs} CMU Common Lisp automatically GC's whenever the amount of memory allocated to dynamic objects exceeds the value of an internal variable. After each GC, the system sets this internal variable to the amount of dynamic space in use at that point plus the value of the variable \code{ext:*bytes-consed-between-gcs*}. The default value is 2000000. \end{defvar} \begin{defvar}{extensions:}{gc-verbose} This variable controls whether \code{ext:gc} invokes the functions in \code{ext:*gc-notify-before*} and \code{ext:*gc-notify-after*}. If \code{*gc-verbose*} is \nil, \code{ext:gc} foregoes printing any messages. The default value is \code{T}. \end{defvar} \begin{defvar}{extensions:}{gc-notify-before} This variable's value is a function that should notify the user that the system is about to GC. It takes one argument, the amount of dynamic space in use before the GC measured in bytes. The default value of this variable is a function that prints a message similar to the following: \begin{display} \b{[GC threshold exceeded with 2,107,124 bytes in use. Commencing GC.]} \end{display} \end{defvar} \begin{defvar}{extensions:}{gc-notify-after} This variable's value is a function that should notify the user when a GC finishes. The function must take three arguments, the amount of dynamic spaced retained by the GC, the amount of dynamic space freed, and the new threshold which is the minimum amount of space in use before the next GC will occur. All values are byte quantities. The default value of this variable is a function that prints a message similar to the following: \begin{display} \b{[GC completed with 25,680 bytes retained and 2,096,808 bytes freed.]} \b{[GC will next occur when at least 2,025,680 bytes are in use.]} \end{display} \end{defvar} Note that a garbage collection will not happen at exactly the new threshold printed by the default \code{ext:*gc-notify-after*} function. The system periodically checks whether this threshold has been exceeded, and only then does a garbage collection. \begin{defvar}{extensions:}{gc-inhibit-hook} This variable's value is either a function of one argument or \nil. When the system has triggered an automatic GC, if this variable is a function, then the system calls the function with the amount of dynamic space currently in use (measured in bytes). If the function returns \nil, then the GC occurs; otherwise, the system inhibits automatic GC as if you had called \code{ext:gc-off}. The writer of this hook is responsible for knowing when automatic GC has been turned off and for calling or providing a way to call \code{ext:gc-on}. The default value of this variable is \nil. \end{defvar} \begin{defvar}{extensions:}{before-gc-hooks} \defvarx[extensions:]{after-gc-hooks} These variables' values are lists of functions to call before or after any GC occurs. The system provides these purely for side-effect, and the functions take no arguments. \end{defvar} %%\node \subsection{Weak Pointers} A weak pointer provides a way to maintain a reference to an object without preventing an object from being garbage collected. If the garbage collector discovers that the only pointers to an object are weak pointers, then it breaks the weak pointers and deallocates the object. \begin{defun}{extensions:}{make-weak-pointer}{\args{\var{object}}} \defunx[extensions:]{weak-pointer-value}{\args{\var{weak-pointer}}} \code{make-weak-pointer} returns a weak pointer to an object. \code{weak-pointer-value} follows a weak pointer, returning the two values: the object pointed to (or \false{} if broken) and a boolean value which is true if the pointer has been broken. \end{defun} %%\node \subsection{Finalization} Finalization provides a hook'' that is triggered when the garbage collector reclaims an object. It is usually used to recover non-Lisp resources that were allocated to implement the finalized Lisp object. For example, when a unix file-descriptor stream is collected, finalization is used to close the underlying file descriptor. \begin{defun}{extensions:}{finalize}{\args{\var{object} \var{function}}} This function registers \var{object} for finalization. \var{function} is called with no arguments when \var{object} is reclaimed. Normally \var{function} will be a closure over the underlying state that needs to be freed, e.g. the unix file descriptor in the fd-stream case. Note that \var{function} must not close over \var{object} itself, as this prevents the object from ever becoming garbage. \end{defun} \begin{defun}{extensions:}{cancel-finalization}{\args{\var{object}}} This function cancel any finalization request for \var{object}. \end{defun} %%\node Describe, The Inspector, Garbage Collection, Design Choices and Extensions \section{Describe} In addition to the basic function described below, there are a number of switches and other things that can be used to control \code{describe}'s behavior. \begin{defun}{}{describe}{ \args{\var{object} \&optional{} \var{stream}}} The \code{describe} function prints useful information about \var{object} on \var{stream}, which defaults to \code{*standard-output*}. For any object, \code{describe} will print out the type. Then it prints other information based on the type of \var{object}. The types which are presently handled are: \begin{Lentry} \item[\tindexed{hash-table}] \code{describe} prints the number of entries currently in the hash table and the number of buckets currently allocated. \item[\tindexed{function}] \code{describe} prints a list of the function's name (if any) and its formal parameters. If the name has function documentation, then it will be printed. If the function is compiled, then the file where it is defined will be printed as well. \item[\tindexed{fixnum}] \code{describe} prints whether the integer is prime or not. \item[\tindexed{symbol}] The symbol's value, properties, and documentation are printed. If the symbol has a function definition, then the function is described. \end{Lentry} If there is anything interesting to be said about some component of the object, describe will invoke itself recursively to describe that object. The level of recursion is indicated by indenting output. \end{defun} \begin{defvar}{extensions:}{describe-level} The maximum level of recursive description allowed. Initially two. \end{defvar} \begin{defvar}{extensions:}{describe-indentation} The number of spaces to indent for each level of recursive description, initially three. \end{defvar} \begin{defvar}{extensions:}{describe-print-level} \defvarx[extensions:]{describe-print-length} The values of \code{*print-level*} and \code{*print-length*} during description. Initially two and five. \end{defvar} %%\node The Inspector, Load, Describe, Design Choices and Extensions \section{The Inspector} \cmucl{} has both a graphical inspector that uses X windows and a simple terminal-based inspector. \begin{defun}{}{inspect}{ \args{\ampoptional{} \var{object}}} \code{inspect} calls the inspector on the optional argument \var{object}. If \var{object} is unsupplied, \code{inspect} immediately returns \false. Otherwise, the behavior of inspect depends on whether Lisp is running under X. When \code{inspect} is eventually exited, it returns some selected Lisp object. \end{defun} \begin{comment} * The Graphical Interface:: * The TTY Inspector:: \end{comment} %%\node The Graphical Interface, The TTY Inspector, The Inspector, The Inspector \subsection{The Graphical Interface} \label{motif-interface} CMU Common Lisp has an interface to Motif which is functionally similar to CLM, but works better in CMU CL. See: \begin{example} \file{doc/motif-toolkit.doc} \file{doc/motif-internals.doc} \end{example} This motif interface has been used to write the inspector and graphical debugger. There is also a Lisp control panel with a simple file management facility, apropos and inspector dialogs, and controls for setting global options. See the \code{interface} and \code{toolkit} packages. \begin{defun}{interface:}{lisp-control-panel}{} This function creates a control panel for the Lisp process. \end{defun} \begin{defvar}{interface:}{interface-style} When the graphical interface is loaded, this variable controls whether it is used by \code{inspect} and the error system. If the value is \kwd{graphics} (the default) and the \code{DISPLAY} environment variable is defined, the graphical inspector and debugger will be invoked by \findexed{inspect} or when an error is signalled. Possible values are \kwd{graphics} and {tty}. If the value is \kwd{graphics}, but there is no X display, then we quietly use the TTY interface. \end{defvar} %%\node The TTY Inspector, , The Graphical Interface, The Inspector \subsection{The TTY Inspector} If X is unavailable, a terminal inspector is invoked. The TTY inspector is a crude interface to \code{describe} which allows objects to be traversed and maintains a history. This inspector prints information about and object and a numbered list of the components of the object. The command-line based interface is a normal \code{read}--\code{eval}--\code{print} loop, but an integer \var{n} descends into the \var{n}'th component of the current object, and symbols with these special names are interpreted as commands: \begin{Lentry} \item[U] Move back to the enclosing object. As you descend into the components of an object, a stack of all the objects previously seen is kept. This command pops you up one level of this stack. \item[Q, E] Return the current object from \code{inspect}. \item[R] Recompute object display, and print again. Useful if the object may have changed. \item[D] Display again without recomputing. \item[H, ?] Show help message. \end{Lentry} %%\node Load, The Reader, The Inspector, Design Choices and Extensions \section{Load} \begin{defun}{}{load}{% \args{\var{filename} \keys{\kwd{verbose} \kwd{print} \kwd{if-does-not-exist}} \morekeys{\kwd{if-source-newer} \kwd{contents}}}} As in standard Common Lisp, this function loads a file containing source or object code into the running Lisp. Several CMU extensions have been made to \code{load} to conveniently support a variety of program file organizations. \var{filename} may be a wildcard pathname such as \file{*.lisp}, in which case all matching files are loaded. If \var{filename} has a \code{pathname-type} (or extension), then that exact file is loaded. If the file has no extension, then this tells \code{load} to use a heuristic to load the right'' file. The \code{*load-source-types*} and \code{*load-object-types*} variables below are used to determine the default source and object file types. If only the source or the object file exists (but not both), then that file is quietly loaded. Similarly, if both the source and object file exist, and the object file is newer than the source file, then the object file is loaded. The value of the \var{if-source-newer} argument is used to determine what action to take when both the source and object files exist, but the object file is out of date: \begin{Lentry} \item[\kwd{load-object}] The object file is loaded even though the source file is newer. \item[\kwd{load-source}] The source file is loaded instead of the older object file. \item[\kwd{compile}] The source file is compiled and then the new object file is loaded. \item[\kwd{query}] The user is asked a yes or no question to determine whether the source or object file is loaded. \end{Lentry} This argument defaults to the value of \code{ext:*load-if-source-newer*} (initially \kwd{load-object}.) The \var{contents} argument can be used to override the heuristic (based on the file extension) that normally determines whether to load the file as a source file or an object file. If non-null, this argument must be either \kwd{source} or \kwd{binary}, which forces loading in source and binary mode, respectively. You really shouldn't ever need to use this argument. \end{defun} \begin{defvar}{extensions:}{load-source-types} \defvarx[extensions:]{load-object-types} These variables are lists of possible \code{pathname-type} values for source and object files to be passed to \code{load}. These variables are only used when the file passed to \code{load} has no type; in this case, the possible source and object types are used to default the type in order to determine the names of the source and object files. \end{defvar} \begin{defvar}{extensions:}{load-if-source-newer} This variable determines the default value of the \var{if-source-newer} argument to \code{load}. Its initial value is \kwd{load-object}. \end{defvar} %%\node The Reader, Stream Extensions, Load, Design Choices and Extensions \section{The Reader} \begin{defvar}{extensions:}{ignore-extra-close-parentheses} If this variable is \true{} (the default), then the reader merely prints a warning when an extra close parenthesis is detected (instead of signalling an error.) \end{defvar} %%\node Stream Extensions, Running Programs from Lisp, The Reader, Design Choices and Extensions \section{Stream Extensions} \begin{defun}{extensions:}{read-n-bytes}{% \args{\var{stream buffer start numbytes} \ampoptional{} \var{eof-error-p}}} On streams that support it, this function reads multiple bytes of data into a buffer. The buffer must be a \code{simple-string} or \code{(simple-array (unsigned-byte 8) (*))}. The argument \var{nbytes} specifies the desired number of bytes, and the return value is the number of bytes actually read. \begin{itemize} \item If \var{eof-error-p} is true, an \tindexed{end-of-file} condition is signalled if end-of-file is encountered before \var{count} bytes have been read. \item If \var{eof-error-p} is false, \code{read-n-bytes reads} as much data is currently available (up to count bytes.) On pipes or similar devices, this function returns as soon as any data is available, even if the amount read is less than \var{count} and eof has not been hit. See also \funref{make-fd-stream}. \end{itemize} \end{defun} %%\node Running Programs from Lisp, Saving a Core Image, The Reader, Design Choices and Extensions \section{Running Programs from Lisp} It is possible to run programs from Lisp by using the following function. \begin{defun}{extensions:}{run-program}{% \args{\var{program} \var{args} \keys{\kwd{env} \kwd{wait} \kwd{pty} \kwd{input}} \morekeys{\kwd{if-input-does-not-exist}} \yetmorekeys{\kwd{output} \kwd{if-output-exists}} \yetmorekeys{\kwd{error} \kwd{if-error-exists}} \yetmorekeys{\kwd{status-hook} \kwd{before-execve}}}} \code{run-program} runs \var{program} in a child process. \var{Program} should be a pathname or string naming the program. \var{Args} should be a list of strings which this passes to \var{program} as normal Unix parameters. For no arguments, specify \var{args} as \nil. The value returned is either a process structure or \nil. The process interface follows the description of \code{run-program}. If \code{run-program} fails to fork the child process, it returns \nil. Except for sharing file descriptors as explained in keyword argument descriptions, \code{run-program} closes all file descriptors in the child process before running the program. When you are done using a process, call \code{process-close} to reclaim system resources. You only need to do this when you supply \kwd{stream} for one of \kwd{input}, \kwd{output}, or \kwd{error}, or you supply \kwd{pty} non-\nil. You can call \code{process-close} regardless of whether you must to reclaim resources without penalty if you feel safer. \code{run-program} accepts the following keyword arguments: \begin{Lentry} \item[\kwd{env}] This is an a-list mapping keywords and simple-strings. The default is \code{ext:*environment-list*}. If \kwd{env} is specified, \code{run-program} uses the value given and does not combine the environment passed to Lisp with the one specified. \item[\kwd{wait}] If non-\nil{} (the default), wait until the child process terminates. If \nil, continue running Lisp while the child process runs. \item[\kwd{pty}] This should be one of \true, \nil, or a stream. If specified non-\nil, the subprocess executes under a Unix \i{PTY}. If specified as a stream, the system collects all output to this pty and writes it to this stream. If specified as \true, the \code{process-pty} slot contains a stream from which you can read the program's output and to which you can write input for the program. The default is \nil. \item[\kwd{input}] This specifies how the program gets its input. If specified as a string, it is the name of a file that contains input for the child process. \code{run-program} opens the file as standard input. If specified as \nil{} (the default), then standard input is the file \file{/dev/null}. If specified as \true, the program uses the current standard input. This may cause some confusion if \kwd{wait} is \nil{} since two processes may use the terminal at the same time. If specified as \kwd{stream}, then the \code{process-input} slot contains an output stream. Anything written to this stream goes to the program as input. \kwd{input} may also be an input stream that already contains all the input for the process. In this case \code{run-program} reads all the input from this stream before returning, so this cannot be used to interact with the process. \item[\kwd{if-input-does-not-exist}] This specifies what to do if the input file does not exist. The following values are valid: \nil{} (the default) causes \code{run-program} to return \nil{} without doing anything; \kwd{create} creates the named file; and \kwd{error} signals an error. \item[\kwd{output}] This specifies what happens with the program's output. If specified as a pathname, it is the name of a file that contains output the program writes to its standard output. If specified as \nil{} (the default), all output goes to \file{/dev/null}. If specified as \true, the program writes to the Lisp process's standard output. This may cause confusion if \kwd{wait} is \nil{} since two processes may write to the terminal at the same time. If specified as \kwd{stream}, then the \code{process-output} slot contains an input stream from which you can read the program's output. \item[\kwd{if-output-exists}] This specifies what to do if the output file already exists. The following values are valid: \nil{} causes \code{run-program} to return \nil{} without doing anything; \kwd{error} (the default) signals an error; \kwd{supersede} overwrites the current file; and \kwd{append} appends all output to the file. \item[\kwd{error}] This is similar to \kwd{output}, except the file becomes the program's standard error. Additionally, \kwd{error} can be \kwd{output} in which case the program's error output is routed to the same place specified for \kwd{output}. If specified as \kwd{stream}, the \code{process-error} contains a stream similar to the \code{process-output} slot when specifying the \kwd{output} argument. \item[\kwd{if-error-exists}] This specifies what to do if the error output file already exists. It accepts the same values as \kwd{if-output-exists}. \item[\kwd{status-hook}] This specifies a function to call whenever the process changes status. This is especially useful when specifying \kwd{wait} as \nil. The function takes the process as a required argument. \item[\kwd{before-execve}] This specifies a function to run in the child process before it becomes the program to run. This is useful for actions such as authenticating the child process without modifying the parent Lisp process. \end{Lentry} \end{defun} \begin{comment} * Process Accessors:: \end{comment} %%\node Process Accessors, , Running Programs from Lisp, Running Programs from Lisp \subsection{Process Accessors} The following functions interface the process returned by \code{run-program}: \begin{defun}{extensions:}{process-p}{\args{\var{thing}}} This function returns \true{} if \var{thing} is a process. Otherwise it returns \nil{} \end{defun} \begin{defun}{extensions:}{process-pid}{\args{\var{process}}} This function returns the process ID, an integer, for the \var{process}. \end{defun} \begin{defun}{extensions:}{process-status}{\args{\var{process}}} This function returns the current status of \var{process}, which is one of \kwd{running}, \kwd{stopped}, \kwd{exited}, or \kwd{signaled}. \end{defun} \begin{defun}{extensions:}{process-exit-code}{\args{\var{process}}} This function returns either the exit code for \var{process}, if it is \kwd{exited}, or the termination signal \var{process} if it is \kwd{signaled}. The result is undefined for processes that are still alive. \end{defun} \begin{defun}{extensions:}{process-core-dumped}{\args{\var{process}}} This function returns \true{} if someone used a Unix signal to terminate the \var{process} and caused it to dump a Unix core image. \end{defun} \begin{defun}{extensions:}{process-pty}{\args{\var{process}}} This function returns either the two-way stream connected to \var{process}'s Unix \i{PTY} connection or \nil{} if there is none. \end{defun} \begin{defun}{extensions:}{process-input}{\args{\var{process}}} \defunx[extensions:]{process-output}{\args{\var{process}}} \defunx[extensions:]{process-error}{\args{\var{process}}} If the corresponding stream was created, these functions return the input, output or error file descriptor. \nil{} is returned if there is no stream. \end{defun} \begin{defun}{extensions:}{process-status-hook}{\args{\var{process}}} This function returns the current function to call whenever \var{process}'s status changes. This function takes the \var{process} as a required argument. \code{process-status-hook} is \code{setf}'able. \end{defun} \begin{defun}{extensions:}{process-plist}{\args{\var{process}}} This function returns annotations supplied by users, and it is \code{setf}'able. This is available solely for users to associate information with \var{process} without having to build a-lists or hash tables of process structures. \end{defun} \begin{defun}{extensions:}{process-wait}{ \args{\var{process} \ampoptional{} \var{check-for-stopped}}} This function waits for \var{process} to finish. If \var{check-for-stopped} is non-\nil, this also returns when \var{process} stops. \end{defun} \begin{defun}{extensions:}{process-kill}{% \args{\var{process} \var{signal} \ampoptional{} \var{whom}}} This function sends the Unix \var{signal} to \var{process}. \var{Signal} should be the number of the signal or a keyword with the Unix name (for example, \kwd{sigsegv}). \var{Whom} should be one of the following: \begin{Lentry} \item[\kwd{pid}] This is the default, and it indicates sending the signal to \var{process} only. \item[\kwd{process-group}] This indicates sending the signal to \var{process}'s group. \item[\kwd{pty-process-group}] This indicates sending the signal to the process group currently in the foreground on the Unix \i{PTY} connected to \var{process}. This last option is useful if the running program is a shell, and you wish to signal the program running under the shell, not the shell itself. If \code{process-pty} of \var{process} is \nil, using this option is an error. \end{Lentry} \end{defun} \begin{defun}{extensions:}{process-alive-p}{\args{\var{process}}} This function returns \true{} if \var{process}'s status is either \kwd{running} or \kwd{stopped}. \end{defun} \begin{defun}{extensions:}{process-close}{\args{\var{process}}} This function closes all the streams associated with \var{process}. When you are done using a process, call this to reclaim system resources. \end{defun} %%\node Saving a Core Image, Pathnames, Running Programs from Lisp, Design Choices and Extensions \section{Saving a Core Image} A mechanism has been provided to save a running Lisp core image and to later restore it. This is convenient if you don't want to load several files into a Lisp when you first start it up. The main problem is the large size of each saved Lisp image, typically at least 20 megabytes. \begin{defun}{extensions:}{save-lisp}{% \args{\var{file} \keys{\kwd{purify} \kwd{root-structures} \kwd{init-function}} \morekeys{\kwd{load-init-file} \kwd{print-herald} \kwd{site-init}} \yetmorekeys{\kwd{process-command-line}}}} The \code{save-lisp} function saves the state of the currently running Lisp core image in \var{file}. The keyword arguments have the following meaning: \begin{Lentry} \item[\kwd{purify}] If non-NIL (the default), the core image is purified before it is saved (see \funref{purify}.) This reduces the amount of work the garbage collector must do when the resulting core image is being run. Also, if more than one Lisp is running on the same machine, this maximizes the amount of memory that can be shared between the two processes. \item[\kwd{root-structures}] \begin{changebar} This should be a list of the main entry points in any newly loaded systems. This need not be supplied, but locality and/or GC performance will be better if they are. Meaningless if \kwd{purify} is \nil. See \funref{purify}. \end{changebar} \item[\kwd{init-function}] This is the function that starts running when the created core file is resumed. The default function simply invokes the top level read-eval-print loop. If the function returns the lisp will exit. \item[\kwd{load-init-file}] If non-NIL, then load an init file; either the one specified on the command line or \w{\file{init.}\var{fasl-type}}'', or, if \w{\file{init.}\var{fasl-type}}'' does not exist, \code{init.lisp} from the user's home directory. If the init file is found, it is loaded into the resumed core file before the read-eval-print loop is entered. \item[\kwd{site-init}] If non-NIL, the name of the site init file to quietly load. The default is \file{library:site-init}. No error is signalled if the file does not exist. \item[\kwd{print-herald}] If non-NIL (the default), then print out the standard Lisp herald when starting. \item[\kwd{process-command-line}] If non-NIL (the default), processes the command line switches and performs the appropriate actions. \end{Lentry} \end{defun} To resume a saved file, type: \begin{example} lisp -core file \end{example} \begin{defun}{extensions:}{purify}{ \args{\var{file} \keys{\kwd{root-structures} \kwd{environment-name}}}} This function optimizes garbage collection by moving all currently live objects into non-collected storage. Once statically allocated, the objects can never be reclaimed, even if all pointers to them are dropped. This function should generally be called after a large system has been loaded and initialized. \begin{Lentry} \item[\kwd{root-structures}] is an optional list of objects which should be copied first to maximize locality. This should be a list of the main entry points for the resulting core image. The purification process tries to localize symbols, functions, etc., in the core image so that paging performance is improved. The default value is NIL which means that Lisp objects will still be localized but probably not as optimally as they could be. \var{defstruct} structures defined with the \code{(:pure t)} option are moved into read-only storage, further reducing GC cost. List and vector slots of pure structures are also moved into read-only storage. \item[\kwd{environment-name}] is gratuitous documentation for the compacted version of the current global environment (as seen in \code{c::*info-environment*}.) If \false{} is supplied, then environment compaction is inhibited. \end{Lentry} \end{defun} %%\node Pathnames, Filesystem Operations, Saving a Core Image, Design Choices and Extensions \section{Pathnames} In \clisp{} quite a few aspects of \tindexed{pathname} semantics are left to the implementation. \begin{comment} * Unix Pathnames:: * Wildcard Pathnames:: * Logical Pathnames:: * Search Lists:: * Predefined Search-Lists:: * Search-List Operations:: * Search List Example:: \end{comment} %%\node Unix Pathnames, Wildcard Pathnames, Pathnames, Pathnames \subsection{Unix Pathnames} \cpsubindex{unix}{pathnames} Unix pathnames are always parsed with a \code{unix-host} object as the host and \code{nil} as the device. The last two dots (\code{.}) in the namestring mark the type and version, however if the first character is a dot, it is considered part of the name. If the last character is a dot, then the pathname has the empty-string as its type. The type defaults to \code{nil} and the version defaults to \kwd{newest}. \begin{example} (defun parse (x) (values (pathname-name x) (pathname-type x) (pathname-version x))) (parse "foo") \result "foo", NIL, :NEWEST (parse "foo.bar") \result "foo", "bar", :NEWEST (parse ".foo") \result ".foo", NIL, :NEWEST (parse ".foo.bar") \result ".foo", "bar", :NEWEST (parse "..") \result ".", "", :NEWEST (parse "foo.") \result "foo", "", :NEWEST (parse "foo.bar.1") \result "foo", "bar", 1 (parse "foo.bar.baz") \result "foo.bar", "baz", :NEWEST \end{example} The directory of pathnames beginning with a slash (or a search-list, \pxlref{search-lists}) is starts \kwd{absolute}, others start with \kwd{relative}. The \code{..} directory is parsed as \kwd{up}; there is no namestring for \kwd{back}: \begin{example} (pathname-directory "/usr/foo/bar.baz") \result (:ABSOLUTE "usr" "foo") (pathname-directory "../foo/bar.baz") \result (:RELATIVE :UP "foo") \end{example} %%\node Wildcard Pathnames, Logical Pathnames, Unix Pathnames, Pathnames \subsection{Wildcard Pathnames} Wildcards are supported in Unix pathnames. If \code{*}' is specified for a part of a pathname, that is parsed as \kwd{wild}. \code{**}' can be used as a directory name to indicate \kwd{wild-inferiors}. Filesystem operations treat \kwd{wild-inferiors} the same as\ \kwd{wild}, but pathname pattern matching (e.g. for logical pathname translation, \pxlref{logical-pathnames}) matches any number of directory parts with \code{**}' (see \pxlref{wildcard-matching}.) \code{*}' embedded in a pathname part matches any number of characters. Similarly, \code{?}' matches exactly one character, and \code{[a,b]}' matches the characters \code{a}' or \code{b}'. These pathname parts are parsed as \code{pattern} objects. Backslash can be used as an escape character in namestring parsing to prevent the next character from being treated as a wildcard. Note that if typed in a string constant, the backslash must be doubled, since the string reader also uses backslash as a quote: \begin{example} (pathname-name "foo$$\backslash\backslash$$*bar") => "foo*bar" \end{example} %%\node Logical Pathnames, Search Lists, Wildcard Pathnames, Pathnames \subsection{Logical Pathnames} \cindex{logical pathnames} \label{logical-pathnames} If a namestring begins with the name of a defined logical pathname host followed by a colon, then it will be parsed as a logical pathname. Both \code{*}' and \code{**}' wildcards are implemented. \findexed{load-logical-pathname-defaults} on \var{name} looks for a logical host definition file in \w{\file{library:\var{name}.translations}}. Note that \file{library:} designates the search list (\pxlref{search-lists}) initialized to the \cmucl{} \file{lib/} directory, not a logical pathname. The format of the file is a single list of two-lists of the from and to patterns: \begin{example} (("foo;*.text" "/usr/ram/foo/*.txt") ("foo;*.lisp" "/usr/ram/foo/*.l")) \end{example} \begin{comment} * Search Lists:: * Search List Example:: \end{comment} %%\node Search Lists, Predefined Search-Lists, Logical Pathnames, Pathnames \subsection{Search Lists} \cindex{search lists} \label{search-lists} Search lists are an extension to Common Lisp pathnames. They serve a function somewhat similar to Common Lisp logical pathnames, but work more like Unix PATH variables. Search lists are used for two purposes: \begin{itemize} \item They provide a convenient shorthand for commonly used directory names, and \item They allow the abstract (directory structure independent) specification of file locations in program pathname constants (similar to logical pathnames.) \end{itemize} Each search list has an associated list of directories (represented as pathnames with no name or type component.) The namestring for any relative pathname may be prefixed with \var{slist}\code{:}'', indicating that the pathname is relative to the search list \var{slist} (instead of to the current working directory.) Once qualified with a search list, the pathname is no longer considered to be relative. When a search list qualified pathname is passed to a file-system operation such as \code{open}, \code{load} or \code{truename}, each directory in the search list is successively used as the root of the pathname until the file is located. When a file is written to a search list directory, the file is always written to the first directory in the list. %%\node Predefined Search-Lists, Search-List Operations, Search Lists, Pathnames \subsection{Predefined Search-Lists} These search-lists are initialized from the Unix environment or when Lisp was built: \begin{Lentry} \item[\code{default:}] The current directory at startup. \item[\code{home:}] The user's home directory. \item[\code{library:}] The \cmucl{} \file{lib/} directory (\code{CMUCLLIB} environment variable.) \item[\code{path:}] The Unix command path (\code{PATH} environment variable.) \item[\code{target:}] The root of the tree where \cmucl{} was compiled. \end{Lentry} It can be useful to redefine these search-lists, for example, \file{library:} can be augmented to allow logical pathname translations to be located, and \file{target:} can be redefined to point to where \cmucl{} system sources are locally installed. %%\node Search-List Operations, Search List Example, Predefined Search-Lists, Pathnames \subsection{Search-List Operations} These operations define and access search-list definitions. A search-list name may be parsed into a pathname before the search-list is actually defined, but the search-list must be defined before it can actually be used in a filesystem operation. \begin{defun}{extensions:}{search-list}{\var{name}} This function returns the list of directories associated with the search list \var{name}. If \var{name} is not a defined search list, then an error is signaled. When set with \code{setf}, the list of directories is changed to the new value. If the new value is just a namestring or pathname, then it is interpreted as a one-element list. Note that (unlike Unix pathnames), search list names are case-insensitive. \end{defun} \begin{defun}{extensions:}{search-list-defined-p}{\var{name}} \defunx[extensions:]{clear-search-list}{\var{name}} \code{search-list-defined-p} returns \true{} if \var{name} is a defined search list name, \false{} otherwise. \code{clear-search-list} make the search list \var{name} undefined. \end{defun} \begin{defmac}{extensions:}{enumerate-search-list}{% \args{(\var{var} \var{pathname} \mopt{result}) \mstar{form}}} This macro provides an interface to search list resolution. The body \var{forms} are executed with \var{var} bound to each successive possible expansion for \var{name}. If \var{name} does not contain a search-list, then the body is executed exactly once. Everything is wrapped in a block named \nil, so \code{return} can be used to terminate early. The \var{result} form (default \nil) is evaluated to determine the result of the iteration. \end{defmac} \begin{comment} * Search List Example:: \end{comment} %%\node Search List Example, , Search-List Operations, Pathnames \subsection{Search List Example} The search list \code{code:} can be defined as follows: \begin{example} (setf (ext:search-list "code:") '("/usr/lisp/code/")) \end{example} It is now possible to use \code{code:} as an abbreviation for the directory \file{/usr/lisp/code/} in all file operations. For example, you can now specify \code{code:eval.lisp} to refer to the file \file{/usr/lisp/code/eval.lisp}. To obtain the value of a search-list name, use the function search-list as follows: \begin{example} (ext:search-list \var{name}) \end{example} Where \var{name} is the name of a search list as described above. For example, calling \code{ext:search-list} on \code{code:} as follows: \begin{example} (ext:search-list "code:") \end{example} returns the list \code{("/usr/lisp/code/")}. %%\node Filesystem Operations, Time Parsing and Formatting, Pathnames, Design Choices and Extensions \section{Filesystem Operations} \cmucl{} provides a number of extensions and optional features beyond those require by \clisp. \begin{comment} * Wildcard Matching:: * File Name Completion:: * Miscellaneous Filesystem Operations:: \end{comment} %%\node Wildcard Matching, File Name Completion, Filesystem Operations, Filesystem Operations \subsection{Wildcard Matching} \label{wildcard-matching} Unix filesystem operations such as \code{open} will accept wildcard pathnames that match a single file (of course, \code{directory} allows any number of matches.) Filesystem operations treat \kwd{wild-inferiors} the same as\ \kwd{wild}. \begin{defun}{}{directory}{\var{wildname} \keys{\kwd{all} \kwd{check-for-subdirs}} \morekeys{\kwd{follow-links}}} The keyword arguments to this \clisp{} function are a CMU extension. The arguments (all default to \code{t}) have the following functions: \begin{Lentry} \item[\kwd{all}] Include files beginning with dot such as \file{.login}, similar to \code{ls -a}''. \item[\kwd{check-for-subdirs}] Test whether files are directories, similar to \code{ls -F}''. \item[\kwd{follow-links}] Call \code{truename} on each file, which expands out all symbolic links. Note that this option can easily result in pathnames being returned which have a different directory from the one in the \var{wildname} argument. \end{Lentry} \end{defun} \begin{defun}{extensions:}{print-directory}{% \args{\var{wildname} \ampoptional{} \var{stream} \keys{\kwd{all} \kwd{verbose}} \morekeys{\kwd{return-list}}}} Print a directory of \var{wildname} listing to \var{stream} (default \code{*standard-output*}.) \kwd{all} and \kwd{verbose} both default to \false{} and correspond to the \code{-a}'' and \code{-l}'' options of \file{ls}. Normally this function returns \false{}, but if \kwd{return-list} is true, a list of the matched pathnames are returned. \end{defun} %%\node File Name Completion, Miscellaneous Filesystem Operations, Wildcard Matching, Filesystem Operations \subsection{File Name Completion} \begin{defun}{extensions:}{complete-file}{% \args{\var{pathname} \keys{\kwd{defaults} \kwd{ignore-types}}}} Attempt to complete a file name to the longest unambiguous prefix. If supplied, directory from \kwd{defaults} is used as the working directory'' when doing completion. \kwd{ignore-types} is a list of strings of the pathname types (a.k.a. extensions) that should be disregarded as possible matches (binary file names, etc.) \end{defun} \begin{defun}{extensions:}{ambiguous-files}{% \args{\var{pathname} \ampoptional{} \var{defaults}}} Return a list of pathnames for all the possible completions of \var{pathname} with respect to \var{defaults}. \end{defun} %%\node Miscellaneous Filesystem Operations, , File Name Completion, Filesystem Operations \subsection{Miscellaneous Filesystem Operations} \begin{defun}{extensions:}{default-directory}{} Return the current working directory as a pathname. If set with \code{setf}, set the working directory. \end{defun} \begin{defun}{extensions:}{file-writable}{\var{name}} This function accepts a pathname and returns \true{} if the current process can write it, and \false{} otherwise. \end{defun} \begin{defun}{extensions:}{unix-namestring}{% \args{\var{pathname} \ampoptional{} \var{for-input}}} This function converts \var{pathname} into a string that can be used with UNIX system calls. Search-lists and wildcards are expanded. \var{for-input} controls the treatment of search-lists: when true (the default) and the file exists anywhere on the search-list, then that absolute pathname is returned; otherwise the first element of the search-list is used as the directory. \end{defun} %%\node Time Parsing and Formatting, Lisp Library, Filesystem Operations, Design Choices and Extensions \section{Time Parsing and Formatting} \cindex{time parsing} \cindex{time formatting} Functions are provided to allow parsing strings containing time information and printing time in various formats are available. \begin{defun}{extensions:}{parse-time}{% \args{\var{time-string} \keys{\kwd{error-on-mismatch} \kwd{default-seconds}} \morekeys{\kwd{default-minutes} \kwd{default-hours}} \yetmorekeys{\kwd{default-day} \kwd{default-month}} \yetmorekeys{\kwd{default-year} \kwd{default-zone}} \yetmorekeys{\kwd{default-weekday}}}} \code{parse-time} accepts a string containing a time (e.g., \w{"\code{Jan 12, 1952}"}) and returns the universal time if it is successful. If it is unsuccessful and the keyword argument \kwd{error-on-mismatch} is non-\FALSE, it signals an error. Otherwise it returns \FALSE. The other keyword arguments have the following meaning: \begin{Lentry} \item[\kwd{default-seconds}] specifies the default value for the seconds value if one is not provided by \var{time-string}. The default value is 0. \item[\kwd{default-minutes}] specifies the default value for the minutes value if one is not provided by \var{time-string}. The default value is 0. \item[\kwd{default-hours}] specifies the default value for the hours value if one is not provided by \var{time-string}. The default value is 0. \item[\kwd{default-day}] specifies the default value for the day value if one is not provided by \var{time-string}. The default value is the current day. \item[\kwd{default-month}] specifies the default value for the month value if one is not provided by \var{time-string}. The default value is the current month. \item[\kwd{default-year}] specifies the default value for the year value if one is not provided by \var{time-string}. The default value is the current year. \item[\kwd{default-zone}] specifies the default value for the time zone value if one is not provided by \var{time-string}. The default value is the current time zone. \item[\kwd{default-weekday}] specifies the default value for the day of the week if one is not provided by \var{time-string}. The default value is the current day of the week. \end{Lentry} Any of the above keywords can be given the value \kwd{current} which means to use the current value as determined by a call to the operating system. \end{defun} \begin{defun}{extensions:}{format-universal-time}{ \args{\var{dest} \var{universal-time} \\ \keys{\kwd{timezone}} \morekeys{\kwd{style} \kwd{date-first}} \yetmorekeys{\kwd{print-seconds} \kwd{print-meridian}} \yetmorekeys{\kwd{print-timezone} \kwd{print-weekday}}}} \defunx[extensions:]{format-decoded-time}{ \args{\var{dest} \var{seconds} \var{minutes} \var{hours} \var{day} \var{month} \var{year} \\ \keys{\kwd{timezone}} \morekeys{\kwd{style} \kwd{date-first}} \yetmorekeys{\kwd{print-seconds} \kwd{print-meridian}} \yetmorekeys{\kwd{print-timezone} \kwd{print-weekday}}}} \code{format-universal-time} formats the time specified by \var{universal-time}. \code{format-decoded-time} formats the time specified by \var{seconds}, \var{minutes}, \var{hours}, \var{day}, \var{month}, and \var{year}. \var{Dest} is any destination accepted by the \code{format} function. The keyword arguments have the following meaning: \begin{Lentry} \item[\kwd{timezone}] is an integer specifying the hours west of Greenwich. \kwd{timezone} defaults to the current time zone. \item[\kwd{style}] specifies the style to use in formatting the time. The legal values are: \begin{Lentry} \item[\kwd{short}] specifies to use a numeric date. \item[\kwd{long}] specifies to format months and weekdays as words instead of numbers. \item[\kwd{abbreviated}] is similar to long except the words are abbreviated. \item[\kwd{government}] is similar to abbreviated, except the date is of the form day month year'' instead of month day, year''. \end{Lentry} \item[\kwd{date-first}] if non-\false{} (default) will place the date first. Otherwise, the time is placed first. \item[\kwd{print-seconds}] if non-\false{} (default) will format the seconds as part of the time. Otherwise, the seconds will be omitted. \item[\kwd{print-meridian}] if non-\false{} (default) will format AM'' or PM'' as part of the time. Otherwise, the AM'' or PM'' will be omitted. \item[\kwd{print-timezone}] if non-\false{} (default) will format the time zone as part of the time. Otherwise, the time zone will be omitted. %%\item[\kwd{print-seconds}] %%if non-\false{} (default) will format the seconds as part of %%the time. Otherwise, the seconds will be omitted. \item[\kwd{print-weekday}] if non-\false{} (default) will format the weekday as part of date. Otherwise, the weekday will be omitted. \end{Lentry} \end{defun} %% New stuff \begin{changebar} \section{Random Number Generation} \cindex{random number generation} \clisp{} includes a random number generator as a standard part of the language; however, the implementation of the generator is not specified. Two random number generators are available in \cmucl{}, depending on the version. \subsection{Original Generator} \cpsubindex{random number generation}{original generator} The default random number generator uses a lagged Fibonacci generator given by \begin{displaymath} z[i] = z[i - 24] - z[i - 55] \bmod 536870908 \end{displaymath} where$z[i]$is the$i$'th random number. This generator produces small integer-valued numbers. For larger integer, the small random integers are concatenated to produce larger integers. For floating-point numbers, the bits from this generator are used as the bits of the floating-point significand. \subsection{New Generator} \cpsubindex{random number generation}{new generator} In some versions of \cmucl{}, the original generator above has been replaced with a subtract-with-borrow generator combined with a Weyl generator.\footnote{The generator described here is available if the feature \kwd{new-random} is available.} The reason for the change was to use a documented generator which has passed tests for randomness. The subtract-with-borrow generator is described by the following equation \begin{displaymath} z[i] = z[i + 20] - z[i + 5] - b \end{displaymath} where$z[i]$is the$i$'th random number, which is a \code{double-float}. All of the indices in this equation are interpreted modulo 32. The quantity$b$is carried over from the previous iteration and is either 0 or \code{double-float-epsilon}. If$z[i]$is positive,$b$is set to zero. Otherwise,$b$is set to \code{double-float-epsilon}. To increase the randomness of this generator, this generator is combined with a Weyl generator defined by \begin{displaymath} x[i] = x[i - 1] - y \bmod 1, \end{displaymath} where$y = 7097293079245107 \times 2^{-53}$. Thus, the resulting random number$r[i]$is \begin{displaymath} r[i] = (z[i] - x[i]) \bmod 1 \end{displaymath} This generator has been tested by Peter VanEynde using Marsaglia's diehard test suite for random number generators; this generator passes the test suite. This generator is designed for generating floating-point random numbers. To obtain integers, the bits from the significand of the floating-point number are used as the bits of the integer. As many floating-point numbers as needed are generated to obtain the desired number of bits in the random integer. For floating-point numbers, this generator can by significantly faster than the original generator. \end{changebar} %%\node Lisp Library, , Time Parsing and Formatting, Design Choices and Extensions \section{Lisp Library} \label{lisp-lib} The CMU Common Lisp project maintains a collection of useful or interesting programs written by users of our system. The library is in \file{lib/contrib/}. Two files there that users should read are: \begin{Lentry} \item[CATALOG.TXT] This file contains a page for each entry in the library. It contains information such as the author, portability or dependency issues, how to load the entry, etc. \item[READ-ME.TXT] This file describes the library's organization and all the possible pieces of information an entry's catalog description could contain. \end{Lentry} Hemlock has a command \F{Library Entry} that displays a list of the current library entries in an editor buffer. There are mode specific commands that display catalog descriptions and load entries. This is a simple and convenient way to browse the library. \hide{File:/afs/cs.cmu.edu/project/clisp/hackers/ram/docs/cmu-user/debug.ms} %%\node The Debugger, The Compiler, Design Choices and Extensions, Top \chapter{The Debugger} \hide{-*- Dictionary: cmu-user -*-} \begin{center} \b{By Robert MacLachlan} \end{center} \cindex{debugger} \label{debugger} \begin{comment} * Debugger Introduction:: * The Command Loop:: * Stack Frames:: * Variable Access:: * Source Location Printing:: * Compiler Policy Control:: * Exiting Commands:: * Information Commands:: * Breakpoint Commands:: * Function Tracing:: * Specials:: \end{comment} %%\node Debugger Introduction, The Command Loop, The Debugger, The Debugger \section{Debugger Introduction} The \cmucl{} debugger is unique in its level of support for source-level debugging of compiled code. Although some other debuggers allow access of variables by name, this seems to be the first \llisp{} debugger that: \begin{itemize} \item Tells you when a variable doesn't have a value because it hasn't been initialized yet or has already been deallocated, or \item Can display the precise source location corresponding to a code location in the debugged program. \end{itemize} These features allow the debugging of compiled code to be made almost indistinguishable from interpreted code debugging. The debugger is an interactive command loop that allows a user to examine the function call stack. The debugger is invoked when: \begin{itemize} \item A \tindexed{serious-condition} is signaled, and it is not handled, or \item \findexed{error} is called, and the condition it signals is not handled, or \item The debugger is explicitly invoked with the \clisp{} \findexed{break} or \findexed{debug} functions. \end{itemize} {\it Note: there are two debugger interfaces in CMU CL: the TTY debugger (described below) and the Motif debugger. Since the difference is only in the user interface, much of this chapter also applies to the Motif version. \xlref{motif-interface} for a very brief discussion of the graphical interface.} When you enter the TTY debugger, it looks something like this: \begin{example} Error in function CAR. Wrong type argument, 3, should have been of type LIST. Restarts: 0: Return to Top-Level. Debug (type H for help) (CAR 3) 0] \end{example} The first group of lines describe what the error was that put us in the debugger. In this case \code{car} was called on \code{3}. After \code{Restarts:} is a list of all the ways that we can restart execution after this error. In this case, the only option is to return to top-level. After printing its banner, the debugger prints the current frame and the debugger prompt. %% %%\node The Command Loop, Stack Frames, Debugger Introduction, The Debugger \section{The Command Loop} The debugger is an interactive read-eval-print loop much like the normal top-level, but some symbols are interpreted as debugger commands instead of being evaluated. A debugger command starts with the symbol name of the command, possibly followed by some arguments on the same line. Some commands prompt for additional input. Debugger commands can be abbreviated by any unambiguous prefix: \code{help} can be typed as \code{h}, \code{he}, etc. For convenience, some commands have ambiguous one-letter abbreviations: \code{f} for \code{frame}. The package is not significant in debugger commands; any symbol with the name of a debugger command will work. If you want to show the value of a variable that happens also to be the name of a debugger command, you can use the \code{list-locals} command or the \code{debug:var} function, or you can wrap the variable in a \code{progn} to hide it from the command loop. The debugger prompt is \var{frame}\code{]}'', where \var{frame} is the number of the current frame. Frames are numbered starting from zero at the top (most recent call), increasing down to the bottom. The current frame is the frame that commands refer to. The current frame also provides the lexical environment for evaluation of non-command forms. \cpsubindex{evaluation}{debugger} The debugger evaluates forms in the lexical environment of the functions being debugged. The debugger can only access variables. You can't \code{go} or \code{return-from} into a function, and you can't call local functions. Special variable references are evaluated with their current value (the innermost binding around the debugger invocation)\dash{}you don't get the value that the special had in the current frame. \xlref{debug-vars} for more information on debugger variable access. %% %%\node Stack Frames, Variable Access, The Command Loop, The Debugger \section{Stack Frames} \cindex{stack frames} \cpsubindex{frames}{stack} A stack frame is the run-time representation of a call to a function; the frame stores the state that a function needs to remember what it is doing. Frames have: \begin{itemize} \item Variables (\pxlref{debug-vars}), which are the values being operated on, and \item Arguments to the call (which are really just particularly interesting variables), and \item A current location (\pxlref{source-locations}), which is the place in the program where the function was running when it stopped to call another function, or because of an interrupt or error. \end{itemize} %% \begin{comment} * Stack Motion:: * How Arguments are Printed:: * Function Names:: * Funny Frames:: * Debug Tail Recursion:: * Unknown Locations and Interrupts:: \end{comment} %%\node Stack Motion, How Arguments are Printed, Stack Frames, Stack Frames \subsection{Stack Motion} These commands move to a new stack frame and print the name of the function and the values of its arguments in the style of a Lisp function call: \begin{Lentry} \item[\code{up}] Move up to the next higher frame. More recent function calls are considered to be higher on the stack. \item[\code{down}] Move down to the next lower frame. \item[\code{top}] Move to the highest frame. \item[\code{bottom}] Move to the lowest frame. \item[\code{frame} [\textit{n}]] Move to the frame with the specified number. Prompts for the number if not supplied. \begin{comment} \key{S} [\var{function-name} [\var{n}]] \item Search down the stack for function. Prompts for the function name if not supplied. Searches an optional number of times, but doesn't prompt for this number; enter it following the function. \item[\key{R} [\var{function-name} [\var{n}]]] Search up the stack for function. Prompts for the function name if not supplied. Searches an optional number of times, but doesn't prompt for this number; enter it following the function. \end{comment} \end{Lentry} %% %%\node How Arguments are Printed, Function Names, Stack Motion, Stack Frames \subsection{How Arguments are Printed} A frame is printed to look like a function call, but with the actual argument values in the argument positions. So the frame for this call in the source: \begin{lisp} (myfun (+ 3 4) 'a) \end{lisp} would look like this: \begin{example} (MYFUN 7 A) \end{example} All keyword and optional arguments are displayed with their actual values; if the corresponding argument was not supplied, the value will be the default. So this call: \begin{lisp} (subseq "foo" 1) \end{lisp} would look like this: \begin{example} (SUBSEQ "foo" 1 3) \end{example} And this call: \begin{lisp} (string-upcase "test case") \end{lisp} would look like this: \begin{example} (STRING-UPCASE "test case" :START 0 :END NIL) \end{example} The arguments to a function call are displayed by accessing the argument variables. Although those variables are initialized to the actual argument values, they can be set inside the function; in this case the new value will be displayed. \code{\amprest} arguments are handled somewhat differently. The value of the rest argument variable is displayed as the spread-out arguments to the call, so: \begin{lisp} (format t "~A is a ~A." "This" 'test) \end{lisp} would look like this: \begin{example} (FORMAT T "~A is a ~A." "This" 'TEST) \end{example} Rest arguments cause an exception to the normal display of keyword arguments in functions that have both \code{\amprest} and \code{\&key} arguments. In this case, the keyword argument variables are not displayed at all; the rest arg is displayed instead. So for these functions, only the keywords actually supplied will be shown, and the values displayed will be the argument values, not values of the (possibly modified) variables. If the variable for an argument is never referenced by the function, it will be deleted. The variable value is then unavailable, so the debugger prints \code{} instead of the value. Similarly, if for any of a number of reasons (described in more detail in section \ref{debug-vars}) the value of the variable is unavailable or not known to be available, then \code{} will be printed instead of the argument value. Printing of argument values is controlled by \code{*debug-print-level*} and \varref{debug-print-length}. %% %%\node Function Names, Funny Frames, How Arguments are Printed, Stack Frames \subsection{Function Names} \cpsubindex{function}{names} \cpsubindex{names}{function} If a function is defined by \code{defun}, \code{labels}, or \code{flet}, then the debugger will print the actual function name after the open parenthesis, like: \begin{example} (STRING-UPCASE "test case" :START 0 :END NIL) ((SETF AREF) \#\back{a} "for" 1) \end{example} Otherwise, the function name is a string, and will be printed in quotes: \begin{example} ("DEFUN MYFUN" BAR) ("DEFMACRO DO" (DO ((I 0 (1+ I))) ((= I 13))) NIL) ("SETQ *GC-NOTIFY-BEFORE*") \end{example} This string name is derived from the \w{\code{def}\var{mumble}} form that encloses or expanded into the lambda, or the outermost enclosing form if there is no \w{\code{def}\var{mumble}}. %% %%\node Funny Frames, Debug Tail Recursion, Function Names, Stack Frames \subsection{Funny Frames} \cindex{external entry points} \cpsubindex{entry points}{external} \cpsubindex{block compilation}{debugger implications} \cpsubindex{external}{stack frame kind} \cpsubindex{optional}{stack frame kind} \cpsubindex{cleanup}{stack frame kind} Sometimes the evaluator introduces new functions that are used to implement a user function, but are not directly specified in the source. The main place this is done is for checking argument type and syntax. Usually these functions do their thing and then go away, and thus are not seen on the stack in the debugger. But when you get some sort of error during lambda-list processing, you end up in the debugger on one of these funny frames. These funny frames are flagged by printing \code{[}\var{keyword}\code{]}'' after the parentheses. For example, this call: \begin{lisp} (car 'a 'b) \end{lisp} will look like this: \begin{example} (CAR 2 A) [:EXTERNAL] \end{example} And this call: \begin{lisp} (string-upcase "test case" :end) \end{lisp} would look like this: \begin{example} ("DEFUN STRING-UPCASE" "test case" 335544424 1) [:OPTIONAL] \end{example} As you can see, these frames have only a vague resemblance to the original call. Fortunately, the error message displayed when you enter the debugger will usually tell you what problem is (in these cases, too many arguments and odd keyword arguments.) Also, if you go down the stack to the frame for the calling function, you can display the original source (\pxlref{source-locations}.) With recursive or block compiled functions (\pxlref{block-compilation}), an \kwd{EXTERNAL} frame may appear before the frame representing the first call to the recursive function or entry to the compiled block. This is a consequence of the way the compiler does block compilation: there is nothing odd with your program. You will also see \kwd{CLEANUP} frames during the execution of \code{unwind-protect} cleanup code. Note that inline expansion and open-coding affect what frames are present in the debugger, see sections \ref{debugger-policy} and \ref{open-coding}. %% %%\node Debug Tail Recursion, Unknown Locations and Interrupts, Funny Frames, Stack Frames \subsection{Debug Tail Recursion} \label{debug-tail-recursion} \cindex{tail recursion} \cpsubindex{recursion}{tail} Both the compiler and the interpreter are properly tail recursive.'' If a function call is in a tail-recursive position, the stack frame will be deallocated \i{at the time of the call}, rather than after the call returns. Consider this backtrace: \begin{example} (BAR ...) (FOO ...) \end{example} Because of tail recursion, it is not necessarily the case that \code{FOO} directly called \code{BAR}. It may be that \code{FOO} called some other function \code{FOO2} which then called \code{BAR} tail-recursively, as in this example: \begin{example} (defun foo () ... (foo2 ...) ...) (defun foo2 (...) ... (bar ...)) (defun bar (...) ...) \end{example} Usually the elimination of tail-recursive frames makes debugging more pleasant, since these frames are mostly uninformative. If there is any doubt about how one function called another, it can usually be eliminated by finding the source location in the calling frame (section \ref{source-locations}.) For a more thorough discussion of tail recursion, \pxlref{tail-recursion}. %% %%\node Unknown Locations and Interrupts, , Debug Tail Recursion, Stack Frames \subsection{Unknown Locations and Interrupts} \label{unknown-locations} \cindex{unknown code locations} \cpsubindex{locations}{unknown} \cindex{interrupts} \cpsubindex{errors}{run-time} The debugger operates using special debugging information attached to the compiled code. This debug information tells the debugger what it needs to know about the locations in the code where the debugger can be invoked. If the debugger somehow encounters a location not described in the debug information, then it is said to be \var{unknown}. If the code location for a frame is unknown, then some variables may be inaccessible, and the source location cannot be precisely displayed. There are three reasons why a code location could be unknown: \begin{itemize} \item There is inadequate debug information due to the value of the \code{debug} optimization quality. \xlref{debugger-policy}. \item The debugger was entered because of an interrupt such as \code{$\hat{ }C$}. \item A hardware error such as \code{bus error}'' occurred in code that was compiled unsafely due to the value of the \code{safety} optimization quality. \xlref{optimize-declaration}. \end{itemize} In the last two cases, the values of argument variables are accessible, but may be incorrect. \xlref{debug-var-validity} for more details on when variable values are accessible. It is possible for an interrupt to happen when a function call or return is in progress. The debugger may then flame out with some obscure error or insist that the bottom of the stack has been reached, when the real problem is that the current stack frame can't be located. If this happens, return from the interrupt and try again. When running interpreted code, all locations should be known. However, an interrupt might catch some subfunction of the interpreter at an unknown location. In this case, you should be able to go up the stack a frame or two and reach an interpreted frame which can be debugged. %% %%\node Variable Access, Source Location Printing, Stack Frames, The Debugger \section{Variable Access} \label{debug-vars} \cpsubindex{variables}{debugger access} \cindex{debug variables} There are three ways to access the current frame's local variables in the debugger. The simplest is to type the variable's name into the debugger's read-eval-print loop. The debugger will evaluate the variable reference as though it had appeared inside that frame. The debugger doesn't really understand lexical scoping; it has just one namespace for all the variables in a function. If a symbol is the name of multiple variables in the same function, then the reference appears ambiguous, even though lexical scoping specifies which value is visible at any given source location. If the scopes of the two variables are not nested, then the debugger can resolve the ambiguity by observing that only one variable is accessible. When there are ambiguous variables, the evaluator assigns each one a small integer identifier. The \code{debug:var} function and the \code{list-locals} command use this identifier to distinguish between ambiguous variables: \begin{Lentry} \item[\code{list-locals} \mopt{\var{prefix}}]%%\hfill\\ This command prints the name and value of all variables in the current frame whose name has the specified \var{prefix}. \var{prefix} may be a string or a symbol. If no \var{prefix} is given, then all available variables are printed. If a variable has a potentially ambiguous name, then the name is printed with a \code{\#}\var{identifier}'' suffix, where \var{identifier} is the small integer used to make the name unique. \end{Lentry} \begin{defun}{debug:}{var}{\args{\var{name} \ampoptional{} \var{identifier}}} This function returns the value of the variable in the current frame with the specified \var{name}. If supplied, \var{identifier} determines which value to return when there are ambiguous variables. When \var{name} is a symbol, it is interpreted as the symbol name of the variable, i.e. the package is significant. If \var{name} is an uninterned symbol (gensym), then return the value of the uninterned variable with the same name. If \var{name} is a string, \code{debug:var} interprets it as the prefix of a variable name, and must unambiguously complete to the name of a valid variable. This function is useful mainly for accessing the value of uninterned or ambiguous variables, since most variables can be evaluated directly. \end{defun} %% \begin{comment} * Variable Value Availability:: * Note On Lexical Variable Access:: \end{comment} %%\node Variable Value Availability, Note On Lexical Variable Access, Variable Access, Variable Access \subsection{Variable Value Availability} \label{debug-var-validity} \cindex{availability of debug variables} \cindex{validity of debug variables} \cindex{debug optimization quality} The value of a variable may be unavailable to the debugger in portions of the program where \clisp{} says that the variable is defined. If a variable value is not available, the debugger will not let you read or write that variable. With one exception, the debugger will never display an incorrect value for a variable. Rather than displaying incorrect values, the debugger tells you the value is unavailable. The one exception is this: if you interrupt (e.g., with \code{$\hat{ }C$}) or if there is an unexpected hardware error such as \code{bus error}'' (which should only happen in unsafe code), then the values displayed for arguments to the interrupted frame might be incorrect.\footnote{Since the location of an interrupt or hardware error will always be an unknown location (\pxlref{unknown-locations}), non-argument variable values will never be available in the interrupted frame.} This exception applies only to the interrupted frame: any frame farther down the stack will be fine. The value of a variable may be unavailable for these reasons: \begin{itemize} \item The value of the \code{debug} optimization quality may have omitted debug information needed to determine whether the variable is available. Unless a variable is an argument, its value will only be available when \code{debug} is at least \code{2}. \item The compiler did lifetime analysis and determined that the value was no longer needed, even though its scope had not been exited. Lifetime analysis is inhibited when the \code{debug} optimization quality is \code{3}. \item The variable's name is an uninterned symbol (gensym). To save space, the compiler only dumps debug information about uninterned variables when the \code{debug} optimization quality is \code{3}. \item The frame's location is unknown (\pxlref{unknown-locations}) because the debugger was entered due to an interrupt or unexpected hardware error. Under these conditions the values of arguments will be available, but might be incorrect. This is the exception above. \item The variable was optimized out of existence. Variables with no reads are always optimized away, even in the interpreter. The degree to which the compiler deletes variables will depend on the value of the \code{compile-speed} optimization quality, but most source-level optimizations are done under all compilation policies. \end{itemize} Since it is especially useful to be able to get the arguments to a function, argument variables are treated specially when the \code{speed} optimization quality is less than \code{3} and the \code{debug} quality is at least \code{1}. With this compilation policy, the values of argument variables are almost always available everywhere in the function, even at unknown locations. For non-argument variables, \code{debug} must be at least \code{2} for values to be available, and even then, values are only available at known locations. %% %%\node Note On Lexical Variable Access, , Variable Value Availability, Variable Access \subsection{Note On Lexical Variable Access} \cpsubindex{evaluation}{debugger} When the debugger command loop establishes variable bindings for available variables, these variable bindings have lexical scope and dynamic extent.\footnote{The variable bindings are actually created using the \clisp{} \code{symbol-macro-let} special form.} You can close over them, but such closures can't be used as upward funargs. You can also set local variables using \code{setq}, but if the variable was closed over in the original source and never set, then setting the variable in the debugger may not change the value in all the functions the variable is defined in. Another risk of setting variables is that you may assign a value of a type that the compiler proved the variable could never take on. This may result in bad things happening. %% %%\node Source Location Printing, Compiler Policy Control, Variable Access, The Debugger \section{Source Location Printing} \label{source-locations} \cpsubindex{source location printing}{debugger} One of CMU \clisp{}'s unique capabilities is source level debugging of compiled code. These commands display the source location for the current frame: \begin{Lentry} \item[\code{source} \mopt{\var{context}}]%%\hfill\\ This command displays the file that the current frame's function was defined from (if it was defined from a file), and then the source form responsible for generating the code that the current frame was executing. If \var{context} is specified, then it is an integer specifying the number of enclosing levels of list structure to print. \item[\code{vsource} \mopt{\var{context}}]%%\hfill\\ This command is identical to \code{source}, except that it uses the global values of \code{*print-level*} and \code{*print-length*} instead of the debugger printing control variables \code{*debug-print-level*} and \code{*debug-print-length*}. \end{Lentry} The source form for a location in the code is the innermost list present in the original source that encloses the form responsible for generating that code. If the actual source form is not a list, then some enclosing list will be printed. For example, if the source form was a reference to the variable \code{*some-random-special*}, then the innermost enclosing evaluated form will be printed. Here are some possible enclosing forms: \begin{example} (let ((a *some-random-special*)) ...) (+ *some-random-special* ...) \end{example} If the code at a location was generated from the expansion of a macro or a source-level compiler optimization, then the form in the original source that expanded into that code will be printed. Suppose the file \file{/usr/me/mystuff.lisp} looked like this: \begin{example} (defmacro mymac () '(myfun)) (defun foo () (mymac) ...) \end{example} If \code{foo} has called \code{myfun}, and is waiting for it to return, then the \code{source} command would print: \begin{example} ; File: /usr/me/mystuff.lisp (MYMAC) \end{example} Note that the macro use was printed, not the actual function call form, \code{(myfun)}. If enclosing source is printed by giving an argument to \code{source} or \code{vsource}, then the actual source form is marked by wrapping it in a list whose first element is \code{\#:***HERE***}. In the previous example, \w{\code{source 1}} would print: \begin{example} ; File: /usr/me/mystuff.lisp (DEFUN FOO () (#:***HERE*** (MYMAC)) ...) \end{example} %% \begin{comment} * How the Source is Found:: * Source Location Availability:: \end{comment} %%\node How the Source is Found, Source Location Availability, Source Location Printing, Source Location Printing \subsection{How the Source is Found} If the code was defined from \llisp{} by \code{compile} or \code{eval}, then the source can always be reliably located. If the code was defined from a \code{fasl} file created by \findexed{compile-file}, then the debugger gets the source forms it prints by reading them from the original source file. This is a potential problem, since the source file might have moved or changed since the time it was compiled. The source file is opened using the \code{truename} of the source file pathname originally given to the compiler. This is an absolute pathname with all logical names and symbolic links expanded. If the file can't be located using this name, then the debugger gives up and signals an error. If the source file can be found, but has been modified since the time it was compiled, the debugger prints this warning: \begin{example} ; File has been modified since compilation: ; \var{filename} ; Using form offset instead of character position. \end{example} where \var{filename} is the name of the source file. It then proceeds using a robust but not foolproof heuristic for locating the source. This heuristic works if: \begin{itemize} \item No top-level forms before the top-level form containing the source have been added or deleted, and \item The top-level form containing the source has not been modified much. (More precisely, none of the list forms beginning before the source form have been added or deleted.) \end{itemize} If the heuristic doesn't work, the displayed source will be wrong, but will probably be near the actual source. If the shape'' of the top-level form in the source file is too different from the original form, then an error will be signaled. When the heuristic is used, the the source location commands are noticeably slowed. Source location printing can also be confused if (after the source was compiled) a read-macro you used in the code was redefined to expand into something different, or if a read-macro ever returns the same \code{eq} list twice. If you don't define read macros and don't use \code{\#\#} in perverted ways, you don't need to worry about this. %% %%\node Source Location Availability, , How the Source is Found, Source Location Printing \subsection{Source Location Availability} \cindex{debug optimization quality} Source location information is only available when the \code{debug} optimization quality is at least \code{2}. If source location information is unavailable, the source commands will give an error message. If source location information is available, but the source location is unknown because of an interrupt or unexpected hardware error (\pxlref{unknown-locations}), then the command will print: \begin{example} Unknown location: using block start. \end{example} and then proceed to print the source location for the start of the \i{basic block} enclosing the code location. \cpsubindex{block}{basic} \cpsubindex{block}{start location} It's a bit complicated to explain exactly what a basic block is, but here are some properties of the block start location: \begin{itemize} \item The block start location may be the same as the true location. \item The block start location will never be later in the the program's flow of control than the true location. \item No conditional control structures (such as \code{if}, \code{cond}, \code{or}) will intervene between the block start and the true location (but note that some conditionals present in the original source could be optimized away.) Function calls \i{do not} end basic blocks. \item The head of a loop will be the start of a block. \item The programming language concept of block structure'' and the \clisp{} \code{block} special form are totally unrelated to the compiler's basic block. \end{itemize} In other words, the true location lies between the printed location and the next conditional (but watch out because the compiler may have changed the program on you.) %% %%\node Compiler Policy Control, Exiting Commands, Source Location Printing, The Debugger \section{Compiler Policy Control} \label{debugger-policy} \cpsubindex{policy}{debugger} \cindex{debug optimization quality} \cindex{optimize declaration} The compilation policy specified by \code{optimize} declarations affects the behavior seen in the debugger. The \code{debug} quality directly affects the debugger by controlling the amount of debugger information dumped. Other optimization qualities have indirect but observable effects due to changes in the way compilation is done. Unlike the other optimization qualities (which are compared in relative value to evaluate tradeoffs), the \code{debug} optimization quality is directly translated to a level of debug information. This absolute interpretation allows the user to count on a particular amount of debug information being available even when the values of the other qualities are changed during compilation. These are the levels of debug information that correspond to the values of the \code{debug} quality: \begin{Lentry} \item[\code{0}] Only the function name and enough information to allow the stack to be parsed. \item[\code{\w{$>$0}}] Any level greater than \code{0} gives level \code{0} plus all argument variables. Values will only be accessible if the argument variable is never set and \code{speed} is not \code{3}. \cmucl{} allows any real value for optimization qualities. It may be useful to specify \code{0.5} to get backtrace argument display without argument documentation. \item[\code{1}] Level \code{1} provides argument documentation (printed arglists) and derived argument/result type information. This makes \findexed{describe} more informative, and allows the compiler to do compile-time argument count and type checking for any calls compiled at run-time. \item[\code{2}] Level \code{1} plus all interned local variables, source location information, and lifetime information that tells the debugger when arguments are available (even when \code{speed} is \code{3} or the argument is set.) This is the default. \item[\code{3}] Level \code{2} plus all uninterned variables. In addition, lifetime analysis is disabled (even when \code{speed} is \code{3}), ensuring that all variable values are available at any known location within the scope of the binding. This has a speed penalty in addition to the obvious space penalty. \end{Lentry} As you can see, if the \code{speed} quality is \code{3}, debugger performance is degraded. This effect comes from the elimination of argument variable special-casing (\pxlref{debug-var-validity}.) Some degree of speed/debuggability tradeoff is unavoidable, but the effect is not too drastic when \code{debug} is at least \code{2}. \cindex{inline expansion} \cindex{semi-inline expansion} In addition to \code{inline} and \code{notinline} declarations, the relative values of the \code{speed} and \code{space} qualities also change whether functions are inline expanded (\pxlref{inline-expansion}.) If a function is inline expanded, then there will be no frame to represent the call, and the arguments will be treated like any other local variable. Functions may also be semi-inline'', in which case there is a frame to represent the call, but the call is to an optimized local version of the function, not to the original function. %% %%\node Exiting Commands, Information Commands, Compiler Policy Control, The Debugger \section{Exiting Commands} These commands get you out of the debugger. \begin{Lentry} \item[\code{quit}] Throw to top level. \item[\code{restart} \mopt{\var{n}}]%%\hfill\\ Invokes the \var{n}th restart case as displayed by the \code{error} command. If \var{n} is not specified, the available restart cases are reported. \item[\code{go}] Calls \code{continue} on the condition given to \code{debug}. If there is no restart case named \var{continue}, then an error is signaled. \item[\code{abort}] Calls \code{abort} on the condition given to \code{debug}. This is useful for popping debug command loop levels or aborting to top level, as the case may be. \begin{comment} (\code{debug:debug-return} \var{expression} \mopt{\var{frame}}) \item From the current or specified frame, return the result of evaluating expression. If multiple values are expected, then this function should be called for multiple values. \end{comment} \end{Lentry} %% %%\node Information Commands, Breakpoint Commands, Exiting Commands, The Debugger \section{Information Commands} Most of these commands print information about the current frame or function, but a few show general information. \begin{Lentry} \item[\code{help}, \code{?}] Displays a synopsis of debugger commands. \item[\code{describe}] Calls \code{describe} on the current function, displays number of local variables, and indicates whether the function is compiled or interpreted. \item[\code{print}] Displays the current function call as it would be displayed by moving to this frame. \item[\code{vprint} (or \code{pp}) \mopt{\var{verbosity}}]%%\hfill\\ Displays the current function call using \code{*print-level*} and \code{*print-length*} instead of \code{*debug-print-level*} and \code{*debug-print-length*}. \var{verbosity} is a small integer (default 2) that controls other dimensions of verbosity. \item[\code{error}] Prints the condition given to \code{invoke-debugger} and the active proceed cases. \item[\code{backtrace} \mopt{\var{n}}]\hfill\\ Displays all the frames from the current to the bottom. Only shows \var{n} frames if specified. The printing is controlled by \code{*debug-print-level*} and \code{*debug-print-length*}. \begin{comment} (\code{debug:debug-function} \mopt{\var{n}}) \item Returns the function from the current or specified frame. \item[(\code{debug:function-name} \mopt{\var{n}])] Returns the function name from the current or specified frame. \item[(\code{debug:pc} \mopt{\var{frame}})] Returns the index of the instruction for the function in the current or specified frame. This is useful in conjunction with \code{disassemble}. The pc returned points to the instruction after the one that was fatal. \end{comment} \end{Lentry} %% %%\node Breakpoint Commands, Function Tracing, Information Commands, The Debugger \section{Breakpoint Commands} \cmucl{} supports setting of breakpoints inside compiled functions and stepping of compiled code. Breakpoints can only be set at at known locations (\pxlref{unknown-locations}), so these commands are largely useless unless the \code{debug} optimize quality is at least \code{2} (\pxlref{debugger-policy}). These commands manipulate breakpoints: \begin{Lentry} \item[\code{breakpoint} \var{location} \mstar{\var{option} \var{value}}] %%\hfill\\ Set a breakpoint in some function. \var{location} may be an integer code location number (as displayed by \code{list-locations}) or a keyword. The keyword can be used to indicate setting a breakpoint at the function start (\kwd{start}, \kwd{s}) or function end (\kwd{end}, \kwd{e}). The \code{breakpoint} command has \kwd{condition}, \kwd{break}, \kwd{print} and \kwd{function} options which work similarly to the \code{trace} options. \item[\code{list-locations} (or \code{ll}) \mopt{\var{function}}]%%\hfill\\ List all the code locations in the current frame's function, or in \var{function} if it is supplied. The display format is the code location number, a colon and then the source form for that location: \begin{example} 3: (1- N) \end{example} If consecutive locations have the same source, then a numeric range like \code{3-5:} will be printed. For example, a default function call has a known location both immediately before and after the call, which would result in two code locations with the same source. The listed function becomes the new default function for breakpoint setting (via the \code{breakpoint}) command. \item[\code{list-breakpoints} (or \code{lb})]%%\hfill\\ List all currently active breakpoints with their breakpoint number. \item[\code{delete-breakpoint} (or \code{db}) \mopt{\var{number}}]%%\hfill\\ Delete a breakpoint specified by its breakpoint number. If no number is specified, delete all breakpoints. \item[\code{step}]%%\hfill\\ Step to the next possible breakpoint location in the current function. This always steps over function calls, instead of stepping into them \end{Lentry} \begin{comment} * Breakpoint Example:: \end{comment} %%\node Breakpoint Example, , Breakpoint Commands, Breakpoint Commands \subsection{Breakpoint Example} Consider this definition of the factorial function: \begin{lisp} (defun ! (n) (if (zerop n) 1 (* n (! (1- n))))) \end{lisp} This debugger session demonstrates the use of breakpoints: \begin{example} common-lisp-user> (break) ; Invoke debugger Break Restarts: 0: [CONTINUE] Return from BREAK. 1: [ABORT ] Return to Top-Level. Debug (type H for help) (INTERACTIVE-EVAL (BREAK)) 0] ll #'! 0: #'(LAMBDA (N) (BLOCK ! (IF # 1 #))) 1: (ZEROP N) 2: (* N (! (1- N))) 3: (1- N) 4: (! (1- N)) 5: (* N (! (1- N))) 6: #'(LAMBDA (N) (BLOCK ! (IF # 1 #))) 0] br 2 (* N (! (1- N))) 1: 2 in ! Added. 0] q common-lisp-user> (! 10) ; Call the function *Breakpoint hit* Restarts: 0: [CONTINUE] Return from BREAK. 1: [ABORT ] Return to Top-Level. Debug (type H for help) (! 10) ; We are now in first call (arg 10) before the multiply Source: (* N (! (1- N))) 3] st *Step* (! 10) ; We have finished evaluation of (1- n) Source: (1- N) 3] st *Breakpoint hit* Restarts: 0: [CONTINUE] Return from BREAK. 1: [ABORT ] Return to Top-Level. Debug (type H for help) (! 9) ; We hit the breakpoint in the recursive call Source: (* N (! (1- N))) 3] \end{example} %% %%\node Function Tracing, Specials, Breakpoint Commands, The Debugger \section{Function Tracing} \cindex{tracing} \cpsubindex{function}{tracing} The tracer causes selected functions to print their arguments and their results whenever they are called. Options allow conditional printing of the trace information and conditional breakpoints on function entry or exit. \begin{defmac}{}{trace}{% \args{\mstar{option global-value} \mstar{name \mstar{option value}}}} \code{trace} is a debugging tool that prints information when specified functions are called. In its simplest form: \begin{example} (trace \var{name-1} \var{name-2} ...) \end{example} \code{trace} causes a printout on \vindexed{trace-output} each time that one of the named functions is entered or returns (the \var{names} are not evaluated.) Trace output is indented according to the number of pending traced calls, and this trace depth is printed at the beginning of each line of output. Printing verbosity of arguments and return values is controlled by \vindexed{debug-print-level} and \vindexed{debug-print-length}. If no \var{names} or \var{options} are are given, \code{trace} returns the list of all currently traced functions, \code{*traced-function-list*}. Trace options can cause the normal printout to be suppressed, or cause extra information to be printed. Each option is a pair of an option keyword and a value form. Options may be interspersed with function names. Options only affect tracing of the function whose name they appear immediately after. Global options are specified before the first name, and affect all functions traced by a given use of \code{trace}. If an already traced function is traced again, any new options replace the old options. The following options are defined: \begin{Lentry} \item[\kwd{condition} \var{form}, \kwd{condition-after} \var{form}, \kwd{condition-all} \var{form}] If \kwd{condition} is specified, then \code{trace} does nothing unless \var{form} evaluates to true at the time of the call. \kwd{condition-after} is similar, but suppresses the initial printout, and is tested when the function returns. \kwd{condition-all} tries both before and after. \item[\kwd{wherein} \var{names}] If specified, \var{names} is a function name or list of names. \code{trace} does nothing unless a call to one of those functions encloses the call to this function (i.e. it would appear in a backtrace.) Anonymous functions have string names like \code{"DEFUN FOO"}. \item[\kwd{break} \var{form}, \kwd{break-after} \var{form}, \kwd{break-all} \var{form}] If specified, and \var{form} evaluates to true, then the debugger is invoked at the start of the function, at the end of the function, or both, according to the respective option. \item[\kwd{print} \var{form}, \kwd{print-after} \var{form}, \kwd{print-all} \var{form}] In addition to the usual printout, the result of evaluating \var{form} is printed at the start of the function, at the end of the function, or both, according to the respective option. Multiple print options cause multiple values to be printed. \item[\kwd{function} \var{function-form}] This is a not really an option, but rather another way of specifying what function to trace. The \var{function-form} is evaluated immediately, and the resulting function is traced. \item[\kwd{encapsulate \mgroup{:default | t | nil}}] In \cmucl, tracing can be done either by temporarily redefining the function name (encapsulation), or using breakpoints. When breakpoints are used, the function object itself is destructively modified to cause the tracing action. The advantage of using breakpoints is that tracing works even when the function is anonymously called via \code{funcall}. When \kwd{encapsulate} is true, tracing is done via encapsulation. \kwd{default} is the default, and means to use encapsulation for interpreted functions and funcallable instances, breakpoints otherwise. When encapsulation is used, forms are {\it not} evaluated in the function's lexical environment, but \code{debug:arg} can still be used. \end{Lentry} \kwd{condition}, \kwd{break} and \kwd{print} forms are evaluated in the lexical environment of the called function; \code{debug:var} and \code{debug:arg} can be used. The \code{-after} and \code{-all} forms are evaluated in the null environment. \end{defmac} \begin{defmac}{}{untrace}{ \args{\amprest{} \var{function-names}}} This macro turns off tracing for the specified functions, and removes their names from \code{*traced-function-list*}. If no \var{function-names} are given, then all currently traced functions are untraced. \end{defmac} \begin{defvar}{extensions:}{traced-function-list} A list of function names maintained and used by \code{trace}, \code{untrace}, and \code{untrace-all}. This list should contain the names of all functions currently being traced. \end{defvar} \begin{defvar}{extensions:}{max-trace-indentation} The maximum number of spaces which should be used to indent trace printout. This variable is initially set to 40. \end{defvar} \begin{comment} * Encapsulation Functions:: \end{comment} %%\node Encapsulation Functions, , Function Tracing, Function Tracing \subsection{Encapsulation Functions} \cindex{encapsulation} \cindex{advising} The encapsulation functions provide a mechanism for intercepting the arguments and results of a function. \code{encapsulate} changes the function definition of a symbol, and saves it so that it can be restored later. The new definition normally calls the original definition. The \clisp{} \findexed{fdefinition} function always returns the original definition, stripping off any encapsulation. The original definition of the symbol can be restored at any time by the \code{unencapsulate} function. \code{encapsulate} and \code{unencapsulate} allow a symbol to be multiply encapsulated in such a way that different encapsulations can be completely transparent to each other. Each encapsulation has a type which may be an arbitrary lisp object. If a symbol has several encapsulations of different types, then any one of them can be removed without affecting more recent ones. A symbol may have more than one encapsulation of the same type, but only the most recent one can be undone. \begin{defun}{extensions:}{encapsulate}{% \args{\var{symbol} \var{type} \var{body}}} Saves the current definition of \var{symbol}, and replaces it with a function which returns the result of evaluating the form, \var{body}. \var{Type} is an arbitrary lisp object which is the type of encapsulation. When the new function is called, the following variables are bound for the evaluation of \var{body}: \begin{Lentry} \item[\code{extensions:argument-list}] A list of the arguments to the function. \item[\code{extensions:basic-definition}] The unencapsulated definition of the function. \end{Lentry} The unencapsulated definition may be called with the original arguments by including the form \begin{lisp} (apply extensions:basic-definition extensions:argument-list) \end{lisp} \code{encapsulate} always returns \var{symbol}. \end{defun} \begin{defun}{extensions:}{unencapsulate}{\args{\var{symbol} \var{type}}} Undoes \var{symbol}'s most recent encapsulation of type \var{type}. \var{Type} is compared with \code{eq}. Encapsulations of other types are left in place. \end{defun} \begin{defun}{extensions:}{encapsulated-p}{% \args{\var{symbol} \var{type}}} Returns \true{} if \var{symbol} has an encapsulation of type \var{type}. Returns \nil{} otherwise. \var{type} is compared with \code{eq}. \end{defun} %% \begin{comment} section{The Single Stepper} \begin{defmac}{}{step}{ \args{\var{form}}} Evaluates form with single stepping enabled or if \var{form} is \code{T}, enables stepping until explicitly disabled. Stepping can be disabled by quitting to the lisp top level, or by evaluating the form \w{\code{(step ())}}. While stepping is enabled, every call to eval will prompt the user for a single character command. The prompt is the form which is about to be \code{eval}ed. It is printed with \code{*print-level*} and \code{*print-length*} bound to \code{*step-print-level*} and \code{*step-print-length*}. All interaction is done through the stream \code{*query-io*}. Because of this, the stepper can not be used in Hemlock eval mode. When connected to a slave Lisp, the stepper can be used from Hemlock. The commands are: \begin{Lentry} \item[\key{n} (next)] Evaluate the expression with stepping still enabled. \item[\key{s} (skip)] Evaluate the expression with stepping disabled. \item[\key{q} (quit)] Evaluate the expression, but disable all further stepping inside the current call to \code{step}. \item[\key{p} (print)] Print current form. (does not use \code{*step-print-level*} or \code{*step-print-length*}.) \item[\key{b} (break)] Enter break loop, and then prompt for the command again when the break loop returns. \item[\key{e} (eval)] Prompt for and evaluate an arbitrary expression. The expression is evaluated with stepping disabled. \item[\key{?} (help)] Prints a brief list of the commands. \item[\key{r} (return)] Prompt for an arbitrary value to return as result of the current call to eval. \item[\key{g}] Throw to top level. \end{Lentry} \end{defmac} \begin{defvar}{extensions:}{step-print-level} \defvarx[extensions:]{step-print-length} \code{*print-level*} and \code{*print-length*} are bound to these values while printing the current form. \code{*step-print-level*} and \code{*step-print-length*} are initially bound to 4 and 5, respectively. \end{defvar} \begin{defvar}{extensions:}{max-step-indentation} Step indents the prompts to highlight the nesting of the evaluation. This variable contains the maximum number of spaces to use for indenting. Initially set to 40. \end{defvar} \end{comment} %% %%\node Specials, , Function Tracing, The Debugger \section{Specials} These are the special variables that control the debugger action. \begin{changebar} \begin{defvar}{debug:}{debug-print-level} \defvarx[debug:]{debug-print-length} \code{*print-level*} and \code{*print-length*} are bound to these values during the execution of some debug commands. When evaluating arbitrary expressions in the debugger, the normal values of \code{*print-level*} and \code{*print-length*} are in effect. These variables are initially set to 3 and 5, respectively. \end{defvar} \end{changebar} %% \hide{File:/afs/cs.cmu.edu/project/clisp/hackers/ram/docs/cmu-user/compiler.ms} %%\node The Compiler, Advanced Compiler Use and Efficiency Hints, The Debugger, Top \chapter{The Compiler} \hide{ -*- Dictionary: cmu-user -*-} \begin{comment} * Compiler Introduction:: * Calling the Compiler:: * Compilation Units:: * Interpreting Error Messages:: * Types in Python:: * Getting Existing Programs to Run:: * Compiler Policy:: * Open Coding and Inline Expansion:: \end{comment} %%\node Compiler Introduction, Calling the Compiler, The Compiler, The Compiler \section{Compiler Introduction} This chapter contains information about the compiler that every \cmucl{} user should be familiar with. Chapter \ref{advanced-compiler} goes into greater depth, describing ways to use more advanced features. The \cmucl{} compiler (also known as \Python{}) has many features that are seldom or never supported by conventional \llisp{} compilers: \begin{itemize} \item Source level debugging of compiled code (see chapter \ref{debugger}.) \item Type error compiler warnings for type errors detectable at compile time. \item Compiler error messages that provide a good indication of where the error appeared in the source. \item Full run-time checking of all potential type errors, with optimization of type checks to minimize the cost. \item Scheme-like features such as proper tail recursion and extensive source-level optimization. \item Advanced tuning and optimization features such as comprehensive efficiency notes, flow analysis, and untagged number representations (see chapter \ref{advanced-compiler}.) \end{itemize} %% %%\node Calling the Compiler, Compilation Units, Compiler Introduction, The Compiler \section{Calling the Compiler} \cindex{compiling} Functions may be compiled using \code{compile}, \code{compile-file}, or \code{compile-from-stream}. \begin{defun}{}{compile}{ \args{\var{name} \ampoptional{} \var{definition}}} This function compiles the function whose name is \var{name}. If \var{name} is \false, the compiled function object is returned. If \var{definition} is supplied, it should be a lambda expression that is to be compiled and then placed in the function cell of \var{name}. As per the proposed X3J13 cleanup compile-argument-problems'', \var{definition} may also be an interpreted function. The return values are as per the proposed X3J13 cleanup compiler-diagnostics''. The first value is the function name or function object. The second value is \false{} if no compiler diagnostics were issued, and \true{} otherwise. The third value is \false{} if no compiler diagnostics other than style warnings were issued. A non-\false{} value indicates that there were serious'' compiler diagnostics issued, or that other conditions of type \tindexed{error} or \tindexed{warning} (but not \tindexed{style-warning}) were signaled during compilation. \end{defun} \begin{defun}{}{compile-file}{ \args{\var{input-pathname} \keys{\kwd{output-file} \kwd{error-file} \kwd{trace-file}} \morekeys{\kwd{error-output} \kwd{verbose} \kwd{print} \kwd{progress}} \yetmorekeys{\kwd{load} \kwd{block-compile} \kwd{entry-points}} \yetmorekeys{\kwd{byte-compile}}}} The \cmucl{} \code{compile-file} is extended through the addition of several new keywords and an additional interpretation of \var{input-pathname}: \begin{Lentry} \item[\var{input-pathname}] If this argument is a list of input files, rather than a single input pathname, then all the source files are compiled into a single object file. In this case, the name of the first file is used to determine the default output file names. This is especially useful in combination with \var{block-compile}. \item[\kwd{output-file}] This argument specifies the name of the output file. \true{} gives the default name, \false{} suppresses the output file. \item[\kwd{error-file}] A listing of all the error output is directed to this file. If there are no errors, then no error file is produced (and any existing error file is deleted.) \true{} gives \w{"\var{name}\code{.err}"} (the default), and \false{} suppresses the output file. \item[\kwd{error-output}] If \true{} (the default), then error output is sent to \code{*error-output*}. If a stream, then output is sent to that stream instead. If \false, then error output is suppressed. Note that this error output is in addition to (but the same as) the output placed in the \var{error-file}. \item[\kwd{verbose}] If \true{} (the default), then the compiler prints to error output at the start and end of compilation of each file. See \varref{compile-verbose}. \item[\kwd{print}] If \true{} (the default), then the compiler prints to error output when each function is compiled. See \varref{compile-print}. \item[\kwd{progress}] If \true{} (default \false{}), then the compiler prints to error output progress information about the phases of compilation of each function. This is a CMU extension that is useful mainly in large block compilations. See \varref{compile-progress}. \item[\kwd{trace-file}] If \true{}, several of the intermediate representations (including annotated assembly code) are dumped out to this file. \true{} gives \w{"\var{name}\code{.trace}"}. Trace output is off by default. \xlref{trace-files}. \item[\kwd{load}] If \true{}, load the resulting output file. \item[\kwd{block-compile}] Controls the compile-time resolution of function calls. By default, only self-recursive calls are resolved, unless an \code{ext:block-start} declaration appears in the source file. \xlref{compile-file-block}. \item[\kwd{entry-points}] If non-null, then this is a list of the names of all functions in the file that should have global definitions installed (because they are referenced in other files.) \xlref{compile-file-block}. \item[\kwd{byte-compile}] If \true{}, compiling to a compact interpreted byte code is enabled. Possible values are \true{}, \false{}, and \kwd{maybe} (the default.) See \varref{byte-compile-default} and \xlref{byte-compile}. \end{Lentry} The return values are as per the proposed X3J13 cleanup compiler-diagnostics''. The first value from \code{compile-file} is the truename of the output file, or \false{} if the file could not be created. The interpretation of the second and third values is described above for \code{compile}. \end{defun} \begin{defvar}{}{compile-verbose} \defvarx{compile-print} \defvarx{compile-progress} These variables determine the default values for the \kwd{verbose}, \kwd{print} and \kwd{progress} arguments to \code{compile-file}. \end{defvar} \begin{defun}{extensions:}{compile-from-stream}{% \args{\var{input-stream} \keys{\kwd{error-stream}} \morekeys{\kwd{trace-stream}} \yetmorekeys{\kwd{block-compile} \kwd{entry-points}} \yetmorekeys{\kwd{byte-compile}}}} This function is similar to \code{compile-file}, but it takes all its arguments as streams. It reads \llisp{} code from \var{input-stream} until end of file is reached, compiling into the current environment. This function returns the same two values as the last two values of \code{compile}. No output files are produced. \end{defun} %% %%\node Compilation Units, Interpreting Error Messages, Calling the Compiler, The Compiler \section{Compilation Units} \cpsubindex{compilation}{units} \cmucl{} supports the \code{with-compilation-unit} macro added to the language by the proposed X3J13 with-compilation-unit'' compiler cleanup. This provides a mechanism for eliminating spurious undefined warnings when there are forward references across files, and also provides a standard way to access compiler extensions. \begin{defmac}{}{with-compilation-unit}{% \args{(\mstar{\var{key} \var{value}}) \mstar{\var{form}}}} This macro evaluates the \var{forms} in an environment that causes warnings for undefined variables, functions and types to be delayed until all the forms have been evaluated. Each keyword \var{value} is an evaluated form. These keyword options are recognized: \begin{Lentry} \item[\kwd{override}] If uses of \code{with-compilation-unit} are dynamically nested, the outermost use will take precedence, suppressing printing of undefined warnings by inner uses. However, when the \code{override} option is true this shadowing is inhibited; an inner use will print summary warnings for the compilations within the inner scope. \item[\kwd{optimize}] This is a CMU extension that specifies of the global'' compilation policy for the dynamic extent of the body. The argument should evaluate to an \code{optimize} declare form, like: \begin{lisp} (optimize (speed 3) (safety 0)) \end{lisp} \xlref{optimize-declaration} \item[\kwd{optimize-interface}] Similar to \kwd{optimize}, but specifies the compilation policy for function interfaces (argument count and type checking) for the dynamic extent of the body. \xlref{optimize-interface-declaration}. \item[\kwd{context-declarations}] This is a CMU extension that pattern-matches on function names, automatically splicing in any appropriate declarations at the head of the function definition. \xlref{context-declarations}. \end{Lentry} \end{defmac} \begin{comment} * Undefined Warnings:: \end{comment} %%\node Undefined Warnings, , Compilation Units, Compilation Units \subsection{Undefined Warnings} \cindex{undefined warnings} Warnings about undefined variables, functions and types are delayed until the end of the current compilation unit. The compiler entry functions (\code{compile}, etc.) implicitly use \code{with-compilation-unit}, so undefined warnings will be printed at the end of the compilation unless there is an enclosing \code{with-compilation-unit}. In order the gain the benefit of this mechanism, you should wrap a single \code{with-compilation-unit} around the calls to \code{compile-file}, i.e.: \begin{lisp} (with-compilation-unit () (compile-file "file1") (compile-file "file2") ...) \end{lisp} Unlike for functions and types, undefined warnings for variables are not suppressed when a definition (e.g. \code{defvar}) appears after the reference (but in the same compilation unit.) This is because doing special declarations out of order just doesn't work\dash{}although early references will be compiled as special, bindings will be done lexically. Undefined warnings are printed with full source context (\pxlref{error-messages}), which tremendously simplifies the problem of finding undefined references that resulted from macroexpansion. After printing detailed information about the undefined uses of each name, \code{with-compilation-unit} also prints summary listings of the names of all the undefined functions, types and variables. \begin{defvar}{}{undefined-warning-limit} This variable controls the number of undefined warnings for each distinct name that are printed with full source context when the compilation unit ends. If there are more undefined references than this, then they are condensed into a single warning: \begin{example} Warning: \var{count} more uses of undefined function \var{name}. \end{example} When the value is \code{0}, then the undefined warnings are not broken down by name at all: only the summary listing of undefined names is printed. \end{defvar} %% %%\node Interpreting Error Messages, Types in Python, Compilation Units, The Compiler \section{Interpreting Error Messages} \label{error-messages} \cpsubindex{error messages}{compiler} \cindex{compiler error messages} One of \Python{}'s unique features is the level of source location information it provides in error messages. The error messages contain a lot of detail in a terse format, to they may be confusing at first. Error messages will be illustrated using this example program: \begin{lisp} (defmacro zoq (x) (roq (ploq (+ ,x 3)))) (defun foo (y) (declare (symbol y)) (zoq y)) \end{lisp} The main problem with this program is that it is trying to add \code{3} to a symbol. Note also that the functions \code{roq} and \code{ploq} aren't defined anywhere. \begin{comment} * The Parts of the Error Message:: * The Original and Actual Source:: * The Processing Path:: * Error Severity:: * Errors During Macroexpansion:: * Read Errors:: * Error Message Parameterization:: \end{comment} %%\node The Parts of the Error Message, The Original and Actual Source, Interpreting Error Messages, Interpreting Error Messages \subsection{The Parts of the Error Message} The compiler will produce this warning: \begin{example} File: /usr/me/stuff.lisp In: DEFUN FOO (ZOQ Y) --> ROQ PLOQ + ==> Y Warning: Result is a SYMBOL, not a NUMBER. \end{example} In this example we see each of the six possible parts of a compiler error message: \begin{Lentry} \item[\w{\code{File: /usr/me/stuff.lisp}}] This is the \var{file} that the compiler read the relevant code from. The file name is displayed because it may not be immediately obvious when there is an error during compilation of a large system, especially when \code{with-compilation-unit} is used to delay undefined warnings. \item[\w{\code{In: DEFUN FOO}}] This is the \var{definition} or top-level form responsible for the error. It is obtained by taking the first two elements of the enclosing form whose first element is a symbol beginning with \code{DEF}''. If there is no enclosing \w{\var{def}mumble}, then the outermost form is used. If there are multiple \w{\var{def}mumbles}, then they are all printed from the out in, separated by \code{$=>$}'s. In this example, the problem was in the \code{defun} for \code{foo}. \item[\w{\code{(ZOQ Y)}}] This is the \i{original source} form responsible for the error. Original source means that the form directly appeared in the original input to the compiler, i.e. in the lambda passed to \code{compile} or the top-level form read from the source file. In this example, the expansion of the \code{zoq} macro was responsible for the error. \item[\w{\code{--$>$ROQ PLOQ +}} ] This is the \i{processing path} that the compiler used to produce the errorful code. The processing path is a representation of the evaluated forms enclosing the actual source that the compiler encountered when processing the original source. The path is the first element of each form, or the form itself if the form is not a list. These forms result from the expansion of macros or source-to-source transformation done by the compiler. In this example, the enclosing evaluated forms are the calls to \code{roq}, \code{ploq} and \code{+}. These calls resulted from the expansion of the \code{zoq} macro. \item[\code{==$>$Y}] This is the \i{actual source} responsible for the error. If the actual source appears in the explanation, then we print the next enclosing evaluated form, instead of printing the actual source twice. (This is the form that would otherwise have been the last form of the processing path.) In this example, the problem is with the evaluation of the reference to the variable \code{y}. \item[\w{\code{Warning: Result is a SYMBOL, not a NUMBER.}}] This is the \var{explanation} the problem. In this example, the problem is that \code{y} evaluates to a \code{symbol}, but is in a context where a number is required (the argument to \code{+}). \end{Lentry} Note that each part of the error message is distinctively marked: \begin{itemize} \item \code{File:} and \code{In:} mark the file and definition, respectively. \item The original source is an indented form with no prefix. \item Each line of the processing path is prefixed with \code{--$>$}. \item The actual source form is indented like the original source, but is marked by a preceding \code{==$>$} line. This is like the macroexpands to'' notation used in \cltl. \item The explanation is prefixed with the error severity (\pxlref{error-severity}), either \code{Error:}, \code{Warning:}, or \code{Note:}. \end{itemize} Each part of the error message is more specific than the preceding one. If consecutive error messages are for nearby locations, then the front part of the error messages would be the same. In this case, the compiler omits as much of the second message as in common with the first. For example: \begin{example} File: /usr/me/stuff.lisp In: DEFUN FOO (ZOQ Y) --> ROQ ==> (PLOQ (+ Y 3)) Warning: Undefined function: PLOQ ==> (ROQ (PLOQ (+ Y 3))) Warning: Undefined function: ROQ \end{example} In this example, the file, definition and original source are identical for the two messages, so the compiler omits them in the second message. If consecutive messages are entirely identical, then the compiler prints only the first message, followed by: \begin{example} [Last message occurs \var{repeats} times] \end{example} where \var{repeats} is the number of times the message was given. If the source was not from a file, then no file line is printed. If the actual source is the same as the original source, then the processing path and actual source will be omitted. If no forms intervene between the original source and the actual source, then the processing path will also be omitted. %% %%\node The Original and Actual Source, The Processing Path, The Parts of the Error Message, Interpreting Error Messages \subsection{The Original and Actual Source} \cindex{original source} \cindex{actual source} The \i{original source} displayed will almost always be a list. If the actual source for an error message is a symbol, the original source will be the immediately enclosing evaluated list form. So even if the offending symbol does appear in the original source, the compiler will print the enclosing list and then print the symbol as the actual source (as though the symbol were introduced by a macro.) When the \i{actual source} is displayed (and is not a symbol), it will always be code that resulted from the expansion of a macro or a source-to-source compiler optimization. This is code that did not appear in the original source program; it was introduced by the compiler. Keep in mind that when the compiler displays a source form in an error message, it always displays the most specific (innermost) responsible form. For example, compiling this function: \begin{lisp} (defun bar (x) (let (a) (declare (fixnum a)) (setq a (foo x)) a)) \end{lisp} Gives this error message: \begin{example} In: DEFUN BAR (LET (A) (DECLARE (FIXNUM A)) (SETQ A (FOO X)) A) Warning: The binding of A is not a FIXNUM: NIL \end{example} This error message is not saying there's a problem somewhere in this \code{let}''\dash{}it is saying that there is a problem with the \code{let} itself. In this example, the problem is that \code{a}'s \false{} initial value is not a \code{fixnum}. %% %%\node The Processing Path, Error Severity, The Original and Actual Source, Interpreting Error Messages \subsection{The Processing Path} \cindex{processing path} \cindex{macroexpansion} \cindex{source-to-source transformation} The processing path is mainly useful for debugging macros, so if you don't write macros, you can ignore the processing path. Consider this example: \begin{lisp} (defun foo (n) (dotimes (i n *undefined*))) \end{lisp} Compiling results in this error message: \begin{example} In: DEFUN FOO (DOTIMES (I N *UNDEFINED*)) --> DO BLOCK LET TAGBODY RETURN-FROM ==> (PROGN *UNDEFINED*) Warning: Undefined variable: *UNDEFINED* \end{example} Note that \code{do} appears in the processing path. This is because \code{dotimes} expands into: \begin{lisp} (do ((i 0 (1+ i)) (#:g1 n)) ((>= i #:g1) *undefined*) (declare (type unsigned-byte i))) \end{lisp} The rest of the processing path results from the expansion of \code{do}: \begin{lisp} (block nil (let ((i 0) (#:g1 n)) (declare (type unsigned-byte i)) (tagbody (go #:g3) #:g2 (psetq i (1+ i)) #:g3 (unless (>= i #:g1) (go #:g2)) (return-from nil (progn *undefined*))))) \end{lisp} In this example, the compiler descended into the \code{block}, \code{let}, \code{tagbody} and \code{return-from} to reach the \code{progn} printed as the actual source. This is a place where the actual source appears in explanation'' rule was applied. The innermost actual source form was the symbol \code{*undefined*} itself, but that also appeared in the explanation, so the compiler backed out one level. %% %%\node Error Severity, Errors During Macroexpansion, The Processing Path, Interpreting Error Messages \subsection{Error Severity} \label{error-severity} \cindex{severity of compiler errors} \cindex{compiler error severity} There are three levels of compiler error severity: \begin{Lentry} \item[Error] This severity is used when the compiler encounters a problem serious enough to prevent normal processing of a form. Instead of compiling the form, the compiler compiles a call to \code{error}. Errors are used mainly for signaling syntax errors. If an error happens during macroexpansion, the compiler will handle it. The compiler also handles and attempts to proceed from read errors. \item[Warning] Warnings are used when the compiler can prove that something bad will happen if a portion of the program is executed, but the compiler can proceed by compiling code that signals an error at runtime if the problem has not been fixed: \begin{itemize} \item Violation of type declarations, or \item Function calls that have the wrong number of arguments or malformed keyword argument lists, or \item Referencing a variable declared \code{ignore}, or unrecognized declaration specifiers. \end{itemize} In the language of the \clisp{} standard, these are situations where the compiler can determine that a situation with undefined consequences or that would cause an error to be signaled would result at runtime. \item[Note] Notes are used when there is something that seems a bit odd, but that might reasonably appear in correct programs. \end{Lentry} Note that the compiler does not fully conform to the proposed X3J13 compiler-diagnostics'' cleanup. Errors, warnings and notes mostly correspond to errors, warnings and style-warnings, but many things that the cleanup considers to be style-warnings are printed as warnings rather than notes. Also, warnings, style-warnings and most errors aren't really signaled using the condition system. %% %%\node Errors During Macroexpansion, Read Errors, Error Severity, Interpreting Error Messages \subsection{Errors During Macroexpansion} \cpsubindex{macroexpansion}{errors during} The compiler handles errors that happen during macroexpansion, turning them into compiler errors. If you want to debug the error (to debug a macro), you can set \code{*break-on-signals*} to \code{error}. For example, this definition: \begin{lisp} (defun foo (e l) (do ((current l (cdr current)) ((atom current) nil)) (when (eq (car current) e) (return current)))) \end{lisp} gives this error: \begin{example} In: DEFUN FOO (DO ((CURRENT L #) (# NIL)) (WHEN (EQ # E) (RETURN CURRENT)) ) Error: (during macroexpansion) Error in function LISP::DO-DO-BODY. DO step variable is not a symbol: (ATOM CURRENT) \end{example} %% %%\node Read Errors, Error Message Parameterization, Errors During Macroexpansion, Interpreting Error Messages \subsection{Read Errors} \cpsubindex{read errors}{compiler} The compiler also handles errors while reading the source. For example: \begin{example} Error: Read error at 2: "(,/\back{foo})" Error in function LISP::COMMA-MACRO. Comma not inside a backquote. \end{example} The \code{at 2}'' refers to the character position in the source file at which the error was signaled, which is generally immediately after the erroneous text. The next line, \code{(,/\back{foo})}'', is the line in the source that contains the error file position. The \code{/\back{} }'' indicates the error position within that line (in this example, immediately after the offending comma.) When in \hemlock{} (or any other EMACS-like editor), you can go to a character position with: \begin{example} M-< C-u \var{position} C-f \end{example} Note that if the source is from a \hemlock{} buffer, then the position is relative to the start of the compiled region or \code{defun}, not the file or buffer start. After printing a read error message, the compiler attempts to recover from the error by backing up to the start of the enclosing top-level form and reading again with \code{*read-suppress*} true. If the compiler can recover from the error, then it substitutes a call to \code{cerror} for the unreadable form and proceeds to compile the rest of the file normally. If there is a read error when the file position is at the end of the file (i.e., an unexpected EOF error), then the error message looks like this: \begin{example} Error: Read error in form starting at 14: "(defun test ()" Error in function LISP::FLUSH-WHITESPACE. EOF while reading # \end{example} In this case, \code{starting at 14}'' indicates the character position at which the compiler started reading, i.e. the position before the start of the form that was missing the closing delimiter. The line \w{"\code{(defun test ()}"} is first line after the starting position that the compiler thinks might contain the unmatched open delimiter. %% %%\node Error Message Parameterization, , Read Errors, Interpreting Error Messages \subsection{Error Message Parameterization} \cpsubindex{error messages}{verbosity} \cpsubindex{verbosity}{of error messages} There is some control over the verbosity of error messages. See also \varref{undefined-warning-limit}, \code{*efficiency-note-limit*} and \varref{efficiency-note-cost-threshold}. \begin{defvar}{}{enclosing-source-cutoff} This variable specifies the number of enclosing actual source forms that are printed in full, rather than in the abbreviated processing path format. Increasing the value from its default of \code{1} allows you to see more of the guts of the macroexpanded source, which is useful when debugging macros. \end{defvar} \begin{defvar}{}{error-print-length} \defvarx{error-print-level} These variables are the print level and print length used in printing error messages. The default values are \code{5} and \code{3}. If null, the global values of \code{*print-level*} and \code{*print-length*} are used. \end{defvar} \begin{defmac}{extensions:}{def-source-context}{% \args{\var{name} \var{lambda-list} \mstar{form}}} This macro defines how to extract an abbreviated source context from the \var{name}d form when it appears in the compiler input. \var{lambda-list} is a \code{defmacro} style lambda-list used to parse the arguments. The \var{body} should return a list of subforms that can be printed on about one line. There are predefined methods for \code{defstruct}, \code{defmethod}, etc. If no method is defined, then the first two subforms are returned. Note that this facility implicitly determines the string name associated with anonymous functions. \end{defmac} %% %%\node Types in Python, Getting Existing Programs to Run, Interpreting Error Messages, The Compiler \section{Types in Python} \cpsubindex{types}{in python} A big difference between \Python{} and all other \llisp{} compilers is the approach to type checking and amount of knowledge about types: \begin{itemize} \item \Python{} treats type declarations much differently that other Lisp compilers do. \Python{} doesn't blindly believe type declarations; it considers them assertions about the program that should be checked. \item \Python{} also has a tremendously greater knowledge of the \clisp{} type system than other compilers. Support is incomplete only for the \code{not}, \code{and} and \code{satisfies} types. \end{itemize} See also sections \ref{advanced-type-stuff} and \ref{type-inference}. %% \begin{comment} * Compile Time Type Errors:: * Precise Type Checking:: * Weakened Type Checking:: \end{comment} %%\node Compile Time Type Errors, Precise Type Checking, Types in Python, Types in Python \subsection{Compile Time Type Errors} \cindex{compile time type errors} \cpsubindex{type checking}{at compile time} If the compiler can prove at compile time that some portion of the program cannot be executed without a type error, then it will give a warning at compile time. It is possible that the offending code would never actually be executed at run-time due to some higher level consistency constraint unknown to the compiler, so a type warning doesn't always indicate an incorrect program. For example, consider this code fragment: \begin{lisp} (defun raz (foo) (let ((x (case foo (:this 13) (:that 9) (:the-other 42)))) (declare (fixnum x)) (foo x))) \end{lisp} Compilation produces this warning: \begin{example} In: DEFUN RAZ (CASE FOO (:THIS 13) (:THAT 9) (:THE-OTHER 42)) --> LET COND IF COND IF COND IF ==> (COND) Warning: This is not a FIXNUM: NIL \end{example} In this case, the warning is telling you that if \code{foo} isn't any of \kwd{this}, \kwd{that} or \kwd{the-other}, then \code{x} will be initialized to \false, which the \code{fixnum} declaration makes illegal. The warning will go away if \code{ecase} is used instead of \code{case}, or if \kwd{the-other} is changed to \true. This sort of spurious type warning happens moderately often in the expansion of complex macros and in inline functions. In such cases, there may be dead code that is impossible to correctly execute. The compiler can't always prove this code is dead (could never be executed), so it compiles the erroneous code (which will always signal an error if it is executed) and gives a warning. \begin{defun}{extensions:}{required-argument}{} This function can be used as the default value for keyword arguments that must always be supplied. Since it is known by the compiler to never return, it will avoid any compile-time type warnings that would result from a default value inconsistent with the declared type. When this function is called, it signals an error indicating that a required keyword argument was not supplied. This function is also useful for \code{defstruct} slot defaults corresponding to required arguments. \xlref{empty-type}. Although this function is a CMU extension, it is relatively harmless to use it in otherwise portable code, since you can easily define it yourself: \begin{lisp} (defun required-argument () (error "A required keyword argument was not supplied.")) \end{lisp} \end{defun} Type warnings are inhibited when the \code{extensions:inhibit-warnings} optimization quality is \code{3} (\pxlref{compiler-policy}.) This can be used in a local declaration to inhibit type warnings in a code fragment that has spurious warnings. %% %%\node Precise Type Checking, Weakened Type Checking, Compile Time Type Errors, Types in Python \subsection{Precise Type Checking} \label{precise-type-checks} \cindex{precise type checking} \cpsubindex{type checking}{precise} With the default compilation policy, all type assertions\footnote{There are a few circumstances where a type declaration is discarded rather than being used as type assertion. This doesn't affect safety much, since such discarded declarations are also not believed to be true by the compiler.} are precisely checked. Precise checking means that the check is done as though \code{typep} had been called with the exact type specifier that appeared in the declaration. \Python{} uses \var{policy} to determine whether to trust type assertions (\pxlref{compiler-policy}). Type assertions from declarations are indistinguishable from the type assertions on arguments to built-in functions. In \Python, adding type declarations makes code safer. If a variable is declared to be \w{\code{(integer 3 17)}}, then its value must always always be an integer between \code{3} and \code{17}. If multiple type declarations apply to a single variable, then all the declarations must be correct; it is as though all the types were intersected producing a single \code{and} type specifier. Argument type declarations are automatically enforced. If you declare the type of a function argument, a type check will be done when that function is called. In a function call, the called function does the argument type checking, which means that a more restrictive type assertion in the calling function (e.g., from \code{the}) may be lost. The types of structure slots are also checked. The value of a structure slot must always be of the type indicated in any \kwd{type} slot option.\footnote{The initial value need not be of this type as long as the corresponding argument to the constructor is always supplied, but this will cause a compile-time type warning unless \code{required-argument} is used.} Because of precise type checking, the arguments to slot accessors are checked to be the correct type of structure. In traditional \llisp{} compilers, not all type assertions are checked, and type checks are not precise. Traditional compilers blindly trust explicit type declarations, but may check the argument type assertions for built-in functions. Type checking is not precise, since the argument type checks will be for the most general type legal for that argument. In many systems, type declarations suppress what little type checking is being done, so adding type declarations makes code unsafe. This is a problem since it discourages writing type declarations during initial coding. In addition to being more error prone, adding type declarations during tuning also loses all the benefits of debugging with checked type assertions. To gain maximum benefit from \Python{}'s type checking, you should always declare the types of function arguments and structure slots as precisely as possible. This often involves the use of \code{or}, \code{member} and other list-style type specifiers. Paradoxically, even though adding type declarations introduces type checks, it usually reduces the overall amount of type checking. This is especially true for structure slot type declarations. \Python{} uses the \code{safety} optimization quality (rather than presence or absence of declarations) to choose one of three levels of run-time type error checking: \pxlref{optimize-declaration}. \xlref{advanced-type-stuff} for more information about types in \Python. %% %%\node Weakened Type Checking, , Precise Type Checking, Types in Python \subsection{Weakened Type Checking} \label{weakened-type-checks} \cindex{weakened type checking} \cpsubindex{type checking}{weakened} When the value for the \code{speed} optimization quality is greater than \code{safety}, and \code{safety} is not \code{0}, then type checking is weakened to reduce the speed and space penalty. In structure-intensive code this can double the speed, yet still catch most type errors. Weakened type checks provide a level of safety similar to that of safe'' code in other \llisp{} compilers. A type check is weakened by changing the check to be for some convenient supertype of the asserted type. For example, \code{\w{(integer 3 17)}} is changed to \code{fixnum}, \code{\w{(simple-vector 17)}} to \code{simple-vector}, and structure types are changed to \code{structure}. A complex check like: \begin{example} (or node hunk (member :foo :bar :baz)) \end{example} will be omitted entirely (i.e., the check is weakened to \code{*}.) If a precise check can be done for no extra cost, then no weakening is done. Although weakened type checking is similar to type checking done by other compilers, it is sometimes safer and sometimes less safe. Weakened checks are done in the same places is precise checks, so all the preceding discussion about where checking is done still applies. Weakened checking is sometimes somewhat unsafe because although the check is weakened, the precise type is still input into type inference. In some contexts this will result in type inferences not justified by the weakened check, and hence deletion of some type checks that would be done by conventional compilers. For example, if this code was compiled with weakened checks: \begin{lisp} (defstruct foo (a nil :type simple-string)) (defstruct bar (a nil :type single-float)) (defun myfun (x) (declare (type bar x)) (* (bar-a x) 3.0)) \end{lisp} and \code{myfun} was passed a \code{foo}, then no type error would be signaled, and we would try to multiply a \code{simple-vector} as though it were a float (with unpredictable results.) This is because the check for \code{bar} was weakened to \code{structure}, yet when compiling the call to \code{bar-a}, the compiler thinks it knows it has a \code{bar}. Note that normally even weakened type checks report the precise type in error messages. For example, if \code{myfun}'s \code{bar} check is weakened to \code{structure}, and the argument is \false{}, then the error will be: \begin{example} Type-error in MYFUN: NIL is not of type BAR \end{example} However, there is some speed and space cost for signaling a precise error, so the weakened type is reported if the \code{speed} optimization quality is \code{3} or \code{debug} quality is less than \code{1}: \begin{example} Type-error in MYFUN: NIL is not of type STRUCTURE \end{example} \xlref{optimize-declaration} for further discussion of the \code{optimize} declaration. %% %%\node Getting Existing Programs to Run, Compiler Policy, Types in Python, The Compiler \section{Getting Existing Programs to Run} \cpsubindex{existing programs}{to run} \cpsubindex{types}{portability} \cindex{compatibility with other Lisps} Since \Python{} does much more comprehensive type checking than other Lisp compilers, \Python{} will detect type errors in many programs that have been debugged using other compilers. These errors are mostly incorrect declarations, although compile-time type errors can find actual bugs if parts of the program have never been tested. Some incorrect declarations can only be detected by run-time type checking. It is very important to initially compile programs with full type checks and then test this version. After the checking version has been tested, then you can consider weakening or eliminating type checks. \b{This applies even to previously debugged programs.} \Python{} does much more type inference than other \llisp{} compilers, so believing an incorrect declaration does much more damage. The most common problem is with variables whose initial value doesn't match the type declaration. Incorrect initial values will always be flagged by a compile-time type error, and they are simple to fix once located. Consider this code fragment: \begin{example} (prog (foo) (declare (fixnum foo)) (setq foo ...) ...) \end{example} Here the variable \code{foo} is given an initial value of \false, but is declared to be a \code{fixnum}. Even if it is never read, the initial value of a variable must match the declared type. There are two ways to fix this problem. Change the declaration: \begin{example} (prog (foo) (declare (type (or fixnum null) foo)) (setq foo ...) ...) \end{example} or change the initial value: \begin{example} (prog ((foo 0)) (declare (fixnum foo)) (setq foo ...) ...) \end{example} It is generally preferable to change to a legal initial value rather than to weaken the declaration, but sometimes it is simpler to weaken the declaration than to try to make an initial value of the appropriate type. Another declaration problem occasionally encountered is incorrect declarations on \code{defmacro} arguments. This probably usually happens when a function is converted into a macro. Consider this macro: \begin{lisp} (defmacro my-1+ (x) (declare (fixnum x)) (the fixnum (1+ ,x))) \end{lisp} Although legal and well-defined \clisp, this meaning of this definition is almost certainly not what the writer intended. For example, this call is illegal: \begin{lisp} (my-1+ (+ 4 5)) \end{lisp} The call is illegal because the argument to the macro is \w{\code{(+ 4 5)}}, which is a \code{list}, not a \code{fixnum}. Because of macro semantics, it is hardly ever useful to declare the types of macro arguments. If you really want to assert something about the type of the result of evaluating a macro argument, then put a \code{the} in the expansion: \begin{lisp} (defmacro my-1+ (x) (the fixnum (1+ (the fixnum ,x)))) \end{lisp} In this case, it would be stylistically preferable to change this macro back to a function and declare it inline. Macros have no efficiency advantage over inline functions when using \Python. \xlref{inline-expansion}. Some more subtle problems are caused by incorrect declarations that can't be detected at compile time. Consider this code: \begin{example} (do ((pos 0 (position #\back{a} string :start (1+ pos)))) ((null pos)) (declare (fixnum pos)) ...) \end{example} Although \code{pos} is almost always a \code{fixnum}, it is \false{} at the end of the loop. If this example is compiled with full type checks (the default), then running it will signal a type error at the end of the loop. If compiled without type checks, the program will go into an infinite loop (or perhaps \code{position} will complain because \w{\code{(1+ nil)}} isn't a sensible start.) Why? Because if you compile without type checks, the compiler just quietly believes the type declaration. Since \code{pos} is always a \code{fixnum}, it is never \nil, so \w{\code{(null pos)}} is never true, and the loop exit test is optimized away. Such errors are sometimes flagged by unreachable code notes (\pxlref{dead-code-notes}), but it is still important to initially compile any system with full type checks, even if the system works fine when compiled using other compilers. In this case, the fix is to weaken the type declaration to \w{\code{(or fixnum null)}}.\footnote{Actually, this declaration is totally unnecessary in \Python, since it already knows \code{position} returns a non-negative \code{fixnum} or \false.} Note that there is usually little performance penalty for weakening a declaration in this way. Any numeric operations in the body can still assume the variable is a \code{fixnum}, since \false{} is not a legal numeric argument. Another possible fix would be to say: \begin{example} (do ((pos 0 (position #\back{a} string :start (1+ pos)))) ((null pos)) (let ((pos pos)) (declare (fixnum pos)) ...)) \end{example} This would be preferable in some circumstances, since it would allow a non-standard representation to be used for the local \code{pos} variable in the loop body (see section \ref{ND-variables}.) In summary, remember that \i{all} values that a variable \i{ever} has must be of the declared type, and that you should test using safe code initially. %% %%\node Compiler Policy, Open Coding and Inline Expansion, Getting Existing Programs to Run, The Compiler \section{Compiler Policy} \label{compiler-policy} \cpsubindex{policy}{compiler} \cindex{compiler policy} The policy is what tells the compiler \var{how} to compile a program. This is logically (and often textually) distinct from the program itself. Broad control of policy is provided by the \code{optimize} declaration; other declarations and variables control more specific aspects of compilation. %% \begin{comment} * The Optimize Declaration:: * The Optimize-Interface Declaration:: \end{comment} %%\node The Optimize Declaration, The Optimize-Interface Declaration, Compiler Policy, Compiler Policy \subsection{The Optimize Declaration} \label{optimize-declaration} \cindex{optimize declaration} \cpsubindex{declarations}{\code{optimize}} The \code{optimize} declaration recognizes six different \var{qualities}. The qualities are conceptually independent aspects of program performance. In reality, increasing one quality tends to have adverse effects on other qualities. The compiler compares the relative values of qualities when it needs to make a trade-off; i.e., if \code{speed} is greater than \code{safety}, then improve speed at the cost of safety. The default for all qualities (except \code{debug}) is \code{1}. Whenever qualities are equal, ties are broken according to a broad idea of what a good default environment is supposed to be. Generally this downplays \code{speed}, \code{compile-speed} and \code{space} in favor of \code{safety} and \code{debug}. Novice and casual users should stick to the default policy. Advanced users often want to improve speed and memory usage at the cost of safety and debuggability. If the value for a quality is \code{0} or \code{3}, then it may have a special interpretation. A value of \code{0} means totally unimportant'', and a \code{3} means ultimately important.'' These extreme optimization values enable heroic'' compilation strategies that are not always desirable and sometimes self-defeating. Specifying more than one quality as \code{3} is not desirable, since it doesn't tell the compiler which quality is most important. These are the optimization qualities: \begin{Lentry} \item[\code{speed}] \cindex{speed optimization quality}How fast the program should is run. \code{speed 3} enables some optimizations that hurt debuggability. \item[\code{compilation-speed}] \cindex{compilation-speed optimization quality}How fast the compiler should run. Note that increasing this above \code{safety} weakens type checking. \item[\code{space}] \cindex{space optimization quality}How much space the compiled code should take up. Inline expansion is mostly inhibited when \code{space} is greater than \code{speed}. A value of \code{0} enables promiscuous inline expansion. Wide use of a \code{0} value is not recommended, as it may waste so much space that run time is slowed. \xlref{inline-expansion} for a discussion of inline expansion. \item[\code{debug}] \cindex{debug optimization quality}How debuggable the program should be. The quality is treated differently from the other qualities: each value indicates a particular level of debugger information; it is not compared with the other qualities. \xlref{debugger-policy} for more details. \item[\code{safety}] \cindex{safety optimization quality}How much error checking should be done. If \code{speed}, \code{space} or \code{compilation-speed} is more important than \code{safety}, then type checking is weakened (\pxlref{weakened-type-checks}). If \code{safety} if \code{0}, then no run time error checking is done. In addition to suppressing type checks, \code{0} also suppresses argument count checking, unbound-symbol checking and array bounds checks. \item[\code{extensions:inhibit-warnings}] \cindex{inhibit-warnings optimization quality}This is a CMU extension that determines how little (or how much) diagnostic output should be printed during compilation. This quality is compared to other qualities to determine whether to print style notes and warnings concerning those qualities. If \code{speed} is greater than \code{inhibit-warnings}, then notes about how to improve speed will be printed, etc. The default value is \code{1}, so raising the value for any standard quality above its default enables notes for that quality. If \code{inhibit-warnings} is \code{3}, then all notes and most non-serious warnings are inhibited. This is useful with \code{declare} to suppress warnings about unavoidable problems. \end{Lentry} %%\node The Optimize-Interface Declaration, , The Optimize Declaration, Compiler Policy \subsection{The Optimize-Interface Declaration} \label{optimize-interface-declaration} \cindex{optimize-interface declaration} \cpsubindex{declarations}{\code{optimize-interface}} The \code{extensions:optimize-interface} declaration is identical in syntax to the \code{optimize} declaration, but it specifies the policy used during compilation of code the compiler automatically generates to check the number and type of arguments supplied to a function. It is useful to specify this policy separately, since even thoroughly debugged functions are vulnerable to being passed the wrong arguments. The \code{optimize-interface} declaration can specify that arguments should be checked even when the general \code{optimize} policy is unsafe. Note that this argument checking is the checking of user-supplied arguments to any functions defined within the scope of the declaration, \code{not} the checking of arguments to \llisp{} primitives that appear in those definitions. The idea behind this declaration is that it allows the definition of functions that appear fully safe to other callers, but that do no internal error checking. Of course, it is possible that arguments may be invalid in ways other than having incorrect type. Functions compiled unsafely must still protect themselves against things like user-supplied array indices that are out of bounds and improper lists. See also the \kwd{context-declarations} option to \macref{with-compilation-unit}. %% %%\node Open Coding and Inline Expansion, , Compiler Policy, The Compiler \section{Open Coding and Inline Expansion} \label{open-coding} \cindex{open-coding} \cindex{inline expansion} \cindex{static functions} Since \clisp{} forbids the redefinition of standard functions\footnote{See the proposed X3J13 lisp-symbol-redefinition'' cleanup.}, the compiler can have special knowledge of these standard functions embedded in it. This special knowledge is used in various ways (open coding, inline expansion, source transformation), but the implications to the user are basically the same: \begin{itemize} \item Attempts to redefine standard functions may be frustrated, since the function may never be called. Although it is technically illegal to redefine standard functions, users sometimes want to implicitly redefine these functions when they are debugging using the \code{trace} macro. Special-casing of standard functions can be inhibited using the \code{notinline} declaration. \item The compiler can have multiple alternate implementations of standard functions that implement different trade-offs of speed, space and safety. This selection is based on the compiler policy, \pxlref{compiler-policy}. \end{itemize} When a function call is \i{open coded}, inline code whose effect is equivalent to the function call is substituted for that function call. When a function call is \i{closed coded}, it is usually left as is, although it might be turned into a call to a different function with different arguments. As an example, if \code{nthcdr} were to be open coded, then \begin{lisp} (nthcdr 4 foobar) \end{lisp} might turn into \begin{lisp} (cdr (cdr (cdr (cdr foobar)))) \end{lisp} or even \begin{lisp} (do ((i 0 (1+ i)) (list foobar (cdr foobar))) ((= i 4) list)) \end{lisp} If \code{nth} is closed coded, then \begin{lisp} (nth x l) \end{lisp} might stay the same, or turn into something like: \begin{lisp} (car (nthcdr x l)) \end{lisp} In general, open coding sacrifices space for speed, but some functions (such as \code{car}) are so simple that they are always open-coded. Even when not open-coded, a call to a standard function may be transformed into a different function call (as in the last example) or compiled as \i{static call}. Static function call uses a more efficient calling convention that forbids redefinition. \hide{File:/afs/cs.cmu.edu/project/clisp/hackers/ram/docs/cmu-user/efficiency.ms} \hide{ -*- Dictionary: cmu-user -*- } %%\node Advanced Compiler Use and Efficiency Hints, UNIX Interface, The Compiler, Top \chapter{Advanced Compiler Use and Efficiency Hints} \begin{center} \b{By Robert MacLachlan} \end{center} \vspace{1 cm} \label{advanced-compiler} \begin{comment} * Advanced Compiler Introduction:: * More About Types in Python:: * Type Inference:: * Source Optimization:: * Tail Recursion:: * Local Call:: * Block Compilation:: * Inline Expansion:: * Byte Coded Compilation:: * Object Representation:: * Numbers:: * General Efficiency Hints:: * Efficiency Notes:: * Profiling:: \end{comment} %%\node Advanced Compiler Introduction, More About Types in Python, Advanced Compiler Use and Efficiency Hints, Advanced Compiler Use and Efficiency Hints \section{Advanced Compiler Introduction} In \cmucl, as is any language on any computer, the path to efficient code starts with good algorithms and sensible programming techniques, but to avoid inefficiency pitfalls, you need to know some of this implementation's quirks and features. This chapter is mostly a fairly long and detailed overview of what optimizations \python{} does. Although there are the usual negative suggestions of inefficient features to avoid, the main emphasis is on describing the things that programmers can count on being efficient. The optimizations described here can have the effect of speeding up existing programs written in conventional styles, but the potential for new programming styles that are clearer and less error-prone is at least as significant. For this reason, several sections end with a discussion of the implications of these optimizations for programming style. \begin{comment} * Types:: * Optimization:: * Function Call:: * Representation of Objects:: * Writing Efficient Code:: \end{comment} %%\node Types, Optimization, Advanced Compiler Introduction, Advanced Compiler Introduction \subsection{Types} Python's support for types is unusual in three major ways: \begin{itemize} \item Precise type checking encourages the specific use of type declarations as a form of run-time consistency checking. This speeds development by localizing type errors and giving more meaningful error messages. \xlref{precise-type-checks}. \python{} produces completely safe code; optimized type checking maintains reasonable efficiency on conventional hardware (\pxlref{type-check-optimization}.) \item Comprehensive support for the \clisp{} type system makes complex type specifiers useful. Using type specifiers such as \code{or} and \code{member} has both efficiency and robustness advantages. \xlref{advanced-type-stuff}. \item Type inference eliminates the need for some declarations, and also aids compile-time detection of type errors. Given detailed type declarations, type inference can often eliminate type checks and enable more efficient object representations and code sequences. Checking all types results in fewer type checks. See sections \ref{type-inference} and \ref{non-descriptor}. \end{itemize} %%\node Optimization, Function Call, Types, Advanced Compiler Introduction \subsection{Optimization} The main barrier to efficient Lisp programs is not that there is no efficient way to code the program in Lisp, but that it is difficult to arrive at that efficient coding. Common Lisp is a highly complex language, and usually has many semantically equivalent reasonable'' ways to code a given problem. It is desirable to make all of these equivalent solutions have comparable efficiency so that programmers don't have to waste time discovering the most efficient solution. Source level optimization increases the number of efficient ways to solve a problem. This effect is much larger than the increase in the efficiency of the best'' solution. Source level optimization transforms the original program into a more efficient (but equivalent) program. Although the optimizer isn't doing anything the programmer couldn't have done, this high-level optimization is important because: \begin{itemize} \item The programmer can code simply and directly, rather than obfuscating code to please the compiler. \item When presented with a choice of similar coding alternatives, the programmer can chose whichever happens to be most convenient, instead of worrying about which is most efficient. \end{itemize} Source level optimization eliminates the need for macros to optimize their expansion, and also increases the effectiveness of inline expansion. See sections \ref{source-optimization} and \ref{inline-expansion}. Efficient support for a safer programming style is the biggest advantage of source level optimization. Existing tuned programs typically won't benefit much from source optimization, since their source has already been optimized by hand. However, even tuned programs tend to run faster under \python{} because: \begin{itemize} \item Low level optimization and register allocation provides modest speedups in any program. \item Block compilation and inline expansion can reduce function call overhead, but may require some program restructuring. See sections \ref{inline-expansion}, \ref{local-call} and \ref{block-compilation}. \item Efficiency notes will point out important type declarations that are often missed even in highly tuned programs. \xlref{efficiency-notes}. \item Existing programs can be compiled safely without prohibitive speed penalty, although they would be faster and safer with added declarations. \xlref{type-check-optimization}. \item The context declaration mechanism allows both space and runtime of large systems to be reduced without sacrificing robustness by semi-automatically varying compilation policy without addition any \code{optimize} declarations to the source. \xlref{context-declarations}. \item Byte compilation can be used to dramatically reduce the size of code that is not speed-critical. \xlref{byte-compile} \end{itemize} %%\node Function Call, Representation of Objects, Optimization, Advanced Compiler Introduction \subsection{Function Call} The sort of symbolic programs generally written in \llisp{} often favor recursion over iteration, or have inner loops so complex that they involve multiple function calls. Such programs spend a larger fraction of their time doing function calls than is the norm in other languages; for this reason \llisp{} implementations strive to make the general (or full) function call as inexpensive as possible. \python{} goes beyond this by providing two good alternatives to full call: \begin{itemize} \item Local call resolves function references at compile time, allowing better calling sequences and optimization across function calls. \xlref{local-call}. \item Inline expansion totally eliminates call overhead and allows many context dependent optimizations. This provides a safe and efficient implementation of operations with function semantics, eliminating the need for error-prone macro definitions or manual case analysis. Although most \clisp{} implementations support inline expansion, it becomes a more powerful tool with \python{}'s source level optimization. See sections \ref{source-optimization} and \ref{inline-expansion}. \end{itemize} Generally, \python{} provides simple implementations for simple uses of function call, rather than having only a single calling convention. These features allow a more natural programming style: \begin{itemize} \item Proper tail recursion. \xlref{tail-recursion} \item Relatively efficient closures. \item A \code{funcall} that is as efficient as normal named call. \item Calls to local functions such as from \code{labels} are optimized: \begin{itemize} \item Control transfer is a direct jump. \item The closure environment is passed in registers rather than heap allocated. \item Keyword arguments and multiple values are implemented more efficiently. \end{itemize} \xlref{local-call}. \end{itemize} %%\node Representation of Objects, Writing Efficient Code, Function Call, Advanced Compiler Introduction \subsection{Representation of Objects} Sometimes traditional \llisp{} implementation techniques compare so poorly to the techniques used in other languages that \llisp{} can become an impractical language choice. Terrible inefficiencies appear in number-crunching programs, since \llisp{} numeric operations often involve number-consing and generic arithmetic. \python{} supports efficient natural representations for numbers (and some other types), and allows these efficient representations to be used in more contexts. \python{} also provides good efficiency notes that warn when a crucial declaration is missing. See section \ref{non-descriptor} for more about object representations and numeric types. Also \pxlref{efficiency-notes} about efficiency notes. %%\node Writing Efficient Code, , Representation of Objects, Advanced Compiler Introduction \subsection{Writing Efficient Code} \label{efficiency-overview} Writing efficient code that works is a complex and prolonged process. It is important not to get so involved in the pursuit of efficiency that you lose sight of what the original problem demands. Remember that: \begin{itemize} \item The program should be correct\dash{}it doesn't matter how quickly you get the wrong answer. \item Both the programmer and the user will make errors, so the program must be robust\dash{}it must detect errors in a way that allows easy correction. \item A small portion of the program will consume most of the resources, with the bulk of the code being virtually irrelevant to efficiency considerations. Even experienced programmers familiar with the problem area cannot reliably predict where these hot spots'' will be. \end{itemize} The best way to get efficient code that is still worth using, is to separate coding from tuning. During coding, you should: \begin{itemize} \item Use a coding style that aids correctness and robustness without being incompatible with efficiency. \item Choose appropriate data structures that allow efficient algorithms and object representations (\pxlref{object-representation}). Try to make interfaces abstract enough so that you can change to a different representation if profiling reveals a need. \item Whenever you make an assumption about a function argument or global data structure, add consistency assertions, either with type declarations or explicit uses of \code{assert}, \code{ecase}, etc. \end{itemize} During tuning, you should: \begin{itemize} \item Identify the hot spots in the program through profiling (section \ref{profiling}.) \item Identify inefficient constructs in the hot spot with efficiency notes, more profiling, or manual inspection of the source. See sections \ref{general-efficiency} and \ref{efficiency-notes}. \item Add declarations and consider the application of optimizations. See sections \ref{local-call}, \ref{inline-expansion} and \ref{non-descriptor}. \item If all else fails, consider algorithm or data structure changes. If you did a good job coding, changes will be easy to introduce. \end{itemize} %% %%\node More About Types in Python, Type Inference, Advanced Compiler Introduction, Advanced Compiler Use and Efficiency Hints \section{More About Types in Python} \label{advanced-type-stuff} \cpsubindex{types}{in python} This section goes into more detail describing what types and declarations are recognized by \python. The area where \python{} differs most radically from previous \llisp{} compilers is in its support for types: \begin{itemize} \item Precise type checking helps to find bugs at run time. \item Compile-time type checking helps to find bugs at compile time. \item Type inference minimizes the need for generic operations, and also increases the efficiency of run time type checking and the effectiveness of compile time type checking. \item Support for detailed types provides a wealth of opportunity for operation-specific type inference and optimization. \end{itemize} \begin{comment} * More Types Meaningful:: * Canonicalization:: * Member Types:: * Union Types:: * The Empty Type:: * Function Types:: * The Values Declaration:: * Structure Types:: * The Freeze-Type Declaration:: * Type Restrictions:: * Type Style Recommendations:: \end{comment} %%\node More Types Meaningful, Canonicalization, More About Types in Python, More About Types in Python \subsection{More Types Meaningful} \clisp{} has a very powerful type system, but conventional \llisp{} implementations typically only recognize the small set of types special in that implementation. In these systems, there is an unfortunate paradox: a declaration for a relatively general type like \code{fixnum} will be recognized by the compiler, but a highly specific declaration such as \code{\w{(integer 3 17)}} is totally ignored. This is obviously a problem, since the user has to know how to specify the type of an object in the way the compiler wants it. A very minimal (but rarely satisfied) criterion for type system support is that it be no worse to make a specific declaration than to make a general one. \python{} goes beyond this by exploiting a number of advantages obtained from detailed type information. Using more restrictive types in declarations allows the compiler to do better type inference and more compile-time type checking. Also, when type declarations are considered to be consistency assertions that should be verified (conditional on policy), then complex types are useful for making more detailed assertions. Python understands'' the list-style \code{or}, \code{member}, \code{function}, array and number type specifiers. Understanding means that: \begin{itemize} \item If the type contains more information than is used in a particular context, then the extra information is simply ignored, rather than derailing type inference. \item In many contexts, the extra information from these type specifier is used to good effect. In particular, type checking in \code{Python} is \var{precise}, so these complex types can be used in declarations to make interesting assertions about functions and data structures (\pxlref{precise-type-checks}.) More specific declarations also aid type inference and reduce the cost for type checking. \end{itemize} For related information, \pxlref{numeric-types} for numeric types, and section \ref{array-types} for array types. %%\node Canonicalization, Member Types, More Types Meaningful, More About Types in Python \subsection{Canonicalization} \cpsubindex{types}{equivalence} \cindex{canonicalization of types} \cindex{equivalence of types} When given a type specifier, \python{} will often rewrite it into a different (but equivalent) type. This is the mechanism that \python{} uses for detecting type equivalence. For example, in \python{}'s canonical representation, these types are equivalent: \begin{example} (or list (member :end)) \myequiv (or cons (member nil :end)) \end{example} This has two implications for the user: \begin{itemize} \item The standard symbol type specifiers for \code{atom}, \code{null}, \code{fixnum}, etc., are in no way magical. The \tindexed{null} type is actually defined to be \code{\w{(member nil)}}, \tindexed{list} is \code{\w{(or cons null)}}, and \tindexed{fixnum} is \code{\w{(signed-byte 30)}}. \item When the compiler prints out a type, it may not look like the type specifier that originally appeared in the program. This is generally not a problem, but it must be taken into consideration when reading compiler error messages. \end{itemize} %%\node Member Types, Union Types, Canonicalization, More About Types in Python \subsection{Member Types} \cindex{member types} The \tindexed{member} type specifier can be used to represent symbolic'' values, analogous to the enumerated types of Pascal. For example, the second value of \code{find-symbol} has this type: \begin{lisp} (member :internal :external :inherited nil) \end{lisp} Member types are very useful for expressing consistency constraints on data structures, for example: \begin{lisp} (defstruct ice-cream (flavor :vanilla :type (member :vanilla :chocolate :strawberry))) \end{lisp} Member types are also useful in type inference, as the number of members can sometimes be pared down to one, in which case the value is a known constant. %%\node Union Types, The Empty Type, Member Types, More About Types in Python \subsection{Union Types} \cindex{union (\code{or}) types} \cindex{or (union) types} The \tindexed{or} (union) type specifier is understood, and is meaningfully applied in many contexts. The use of \code{or} allows assertions to be made about types in dynamically typed programs. For example: \begin{lisp} (defstruct box (next nil :type (or box null)) (top :removed :type (or box-top (member :removed)))) \end{lisp} The type assertion on the \code{top} slot ensures that an error will be signaled when there is an attempt to store an illegal value (such as \kwd{rmoved}.) Although somewhat weak, these union type assertions provide a useful input into type inference, allowing the cost of type checking to be reduced. For example, this loop is safely compiled with no type checks: \begin{lisp} (defun find-box-with-top (box) (declare (type (or box null) box)) (do ((current box (box-next current))) ((null current)) (unless (eq (box-top current) :removed) (return current)))) \end{lisp} Union types are also useful in type inference for representing types that are partially constrained. For example, the result of this expression: \begin{lisp} (if foo (logior x y) (list x y)) \end{lisp} can be expressed as \code{\w{(or integer cons)}}. %%\node The Empty Type, Function Types, Union Types, More About Types in Python \subsection{The Empty Type} \label{empty-type} \cindex{NIL type} \cpsubindex{empty type}{the} \cpsubindex{errors}{result type of} The type \false{} is also called the empty type, since no object is of type \false{}. The union of no types, \code{(or)}, is also empty. \python{}'s interpretation of an expression whose type is \false{} is that the expression never yields any value, but rather fails to terminate, or is thrown out of. For example, the type of a call to \code{error} or a use of \code{return} is \false{}. When the type of an expression is empty, compile-time type warnings about its value are suppressed; presumably somebody else is signaling an error. If a function is declared to have return type \false{}, but does in fact return, then (in safe compilation policies) a \code{NIL Function returned}'' error will be signaled. See also the function \funref{required-argument}. %%\node Function Types, The Values Declaration, The Empty Type, More About Types in Python \subsection{Function Types} \label{function-types} \cpsubindex{function}{types} \cpsubindex{types}{function} \findexed{function} types are understood in the restrictive sense, specifying: \begin{itemize} \item The argument syntax that the function must be called with. This is information about what argument counts are acceptable, and which keyword arguments are recognized. In \python, warnings about argument syntax are a consequence of function type checking. \item The types of the argument values that the caller must pass. If the compiler can prove that some argument to a call is of a type disallowed by the called function's type, then it will give a compile-time type warning. In addition to being used for compile-time type checking, these type assertions are also used as output type assertions in code generation. For example, if \code{foo} is declared to have a \code{fixnum} argument, then the \code{1+} in \w{\code{(foo (1+ x))}} is compiled with knowledge that the result must be a fixnum. \item The types the values that will be bound to argument variables in the function's definition. Declaring a function's type with \code{ftype} implicitly declares the types of the arguments in the definition. \python{} checks for consistency between the definition and the \code{ftype} declaration. Because of precise type checking, an error will be signaled when a function is called with an argument of the wrong type. \item The type of return value(s) that the caller can expect. This information is a useful input to type inference. For example, if a function is declared to return a \code{fixnum}, then when a call to that function appears in an expression, the expression will be compiled with knowledge that the call will return a \code{fixnum}. \item The type of return value(s) that the definition must return. The result type in an \code{ftype} declaration is treated like an implicit \code{the} wrapped around the body of the definition. If the definition returns a value of the wrong type, an error will be signaled. If the compiler can prove that the function returns the wrong type, then it will give a compile-time warning. \end{itemize} This is consistent with the new interpretation of function types and the \code{ftype} declaration in the proposed X3J13 function-type-argument-type-semantics'' cleanup. Note also, that if you don't explicitly declare the type of a function using a global \code{ftype} declaration, then \python{} will compute a function type from the definition, providing a degree of inter-routine type inference, \pxlref{function-type-inference}. %%\node The Values Declaration, Structure Types, Function Types, More About Types in Python \subsection{The Values Declaration} \cindex{values declaration} \cmucl{} supports the \code{values} declaration as an extension to \clisp. The syntax is {\code{(values \var{type1} \var{type2}$\ldots$\var{typen})}}. This declaration is semantically equivalent to a \code{the} form wrapped around the body of the special form in which the \code{values} declaration appears. The advantage of \code{values} over \findexed{the} is purely syntactic\dash{}it doesn't introduce more indentation. For example: \begin{example} (defun foo (x) (declare (values single-float)) (ecase x (:this ...) (:that ...) (:the-other ...))) \end{example} is equivalent to: \begin{example} (defun foo (x) (the single-float (ecase x (:this ...) (:that ...) (:the-other ...)))) \end{example} and \begin{example} (defun floor (number &optional (divisor 1)) (declare (values integer real)) ...) \end{example} is equivalent to: \begin{example} (defun floor (number &optional (divisor 1)) (the (values integer real) ...)) \end{example} In addition to being recognized by \code{lambda} (and hence by \code{defun}), the \code{values} declaration is recognized by all the other special forms with bodies and declarations: \code{let}, \code{let*}, \code{labels} and \code{flet}. Macros with declarations usually splice the declarations into one of the above forms, so they will accept this declaration too, but the exact effect of a \code{values} declaration will depend on the macro. If you declare the types of all arguments to a function, and also declare the return value types with \code{values}, you have described the type of the function. \python{} will use this argument and result type information to derive a function type that will then be applied to calls of the function (\pxlref{function-types}.) This provides a way to declare the types of functions that is much less syntactically awkward than using the \code{ftype} declaration with a \code{function} type specifier. Although the \code{values} declaration is non-standard, it is relatively harmless to use it in otherwise portable code, since any warning in non-CMU implementations can be suppressed with the standard \code{declaration} proclamation. %%\node Structure Types, The Freeze-Type Declaration, The Values Declaration, More About Types in Python \subsection{Structure Types} \label{structure-types} \cindex{structure types} \cindex{defstruct types} \cpsubindex{types}{structure} Because of precise type checking, structure types are much better supported by Python than by conventional compilers: \begin{itemize} \item The structure argument to structure accessors is precisely checked\dash{}if you call \code{foo-a} on a \code{bar}, an error will be signaled. \item The types of slot values are precisely checked\dash{}if you pass the wrong type argument to a constructor or a slot setter, then an error will be signaled. \end{itemize} This error checking is tremendously useful for detecting bugs in programs that manipulate complex data structures. An additional advantage of checking structure types and enforcing slot types is that the compiler can safely believe slot type declarations. \python{} effectively moves the type checking from the slot access to the slot setter or constructor call. This is more efficient since caller of the setter or constructor often knows the type of the value, entirely eliminating the need to check the value's type. Consider this example: \begin{lisp} (defstruct coordinate (x nil :type single-float) (y nil :type single-float)) (defun make-it () (make-coordinate :x 1.0 :y 1.0)) (defun use-it (it) (declare (type coordinate it)) (sqrt (expt (coordinate-x it) 2) (expt (coordinate-y it) 2))) \end{lisp} \code{make-it} and \code{use-it} are compiled with no checking on the types of the float slots, yet \code{use-it} can use \code{single-float} arithmetic with perfect safety. Note that \code{make-coordinate} must still check the values of \code{x} and \code{y} unless the call is block compiled or inline expanded (\pxlref{local-call}.) But even without this advantage, it is almost always more efficient to check slot values on structure initialization, since slots are usually written once and read many times. %%\node The Freeze-Type Declaration, Type Restrictions, Structure Types, More About Types in Python \subsection{The Freeze-Type Declaration} \cindex{freeze-type declaration} \label{freeze-type} The \code{extensions:freeze-type} declaration is a CMU extension that enables more efficient compilation of user-defined types by asserting that the definition is not going to change. This declaration may only be used globally (with \code{declaim} or \code{proclaim}). Currently \code{freeze-type} only affects structure type testing done by \code{typep}, \code{typecase}, etc. Here is an example: \begin{lisp} (declaim (freeze-type foo bar)) \end{lisp} This asserts that the types \code{foo} and \code{bar} and their subtypes are not going to change. This allows more efficient type testing, since the compiler can open-code a test for all possible subtypes, rather than having to examine the type hierarchy at run-time. %%\node Type Restrictions, Type Style Recommendations, The Freeze-Type Declaration, More About Types in Python \subsection{Type Restrictions} \cpsubindex{types}{restrictions on} Avoid use of the \code{and}, \code{not} and \code{satisfies} types in declarations, since type inference has problems with them. When these types do appear in a declaration, they are still checked precisely, but the type information is of limited use to the compiler. \code{and} types are effective as long as the intersection can be canonicalized to a type that doesn't use \code{and}. For example: \begin{example} (and fixnum unsigned-byte) \end{example} is fine, since it is the same as: \begin{example} (integer 0 \var{most-positive-fixnum}) \end{example} but this type: \begin{example} (and symbol (not (member :end))) \end{example} will not be fully understood by type interference since the \code{and} can't be removed by canonicalization. Using any of these type specifiers in a type test with \code{typep} or \code{typecase} is fine, since as tests, these types can be translated into the \code{and} macro, the \code{not} function or a call to the satisfies predicate. %%\node Type Style Recommendations, , Type Restrictions, More About Types in Python \subsection{Type Style Recommendations} \cindex{style recommendations} Python provides good support for some currently unconventional ways of using the \clisp{} type system. With \python, it is desirable to make declarations as precise as possible, but type inference also makes some declarations unnecessary. Here are some general guidelines for maximum robustness and efficiency: \begin{itemize} \item Declare the types of all function arguments and structure slots as precisely as possible (while avoiding \code{not}, \code{and} and \code{satisfies}). Put these declarations in during initial coding so that type assertions can find bugs for you during debugging. \item Use the \tindexed{member} type specifier where there are a small number of possible symbol values, for example: \w{\code{(member :red :blue :green)}}. \item Use the \tindexed{or} type specifier in situations where the type is not certain, but there are only a few possibilities, for example: \w{\code{(or list vector)}}. \item Declare integer types with the tightest bounds that you can, such as \code{\w{(integer 3 7)}}. \item Define \findexed{deftype} or \findexed{defstruct} types before they are used. Definition after use is legal (producing no undefined type'' warnings), but type tests and structure operations will be compiled much less efficiently. \item Use the \code{extensions:freeze-type} declaration to speed up type testing for structure types which won't have new subtypes added later. \xlref{freeze-type} \item In addition to declaring the array element type and simpleness, also declare the dimensions if they are fixed, for example: \begin{example} (simple-array single-float (1024 1024)) \end{example} This bounds information allows array indexing for multi-dimensional arrays to be compiled much more efficiently, and may also allow array bounds checking to be done at compile time. \xlref{array-types}. \item Avoid use of the \findexed{the} declaration within expressions. Not only does it clutter the code, but it is also almost worthless under safe policies. If the need for an output type assertion is revealed by efficiency notes during tuning, then you can consider \code{the}, but it is preferable to constrain the argument types more, allowing the compiler to prove the desired result type. \item Don't bother declaring the type of \findexed{let} or other non-argument variables unless the type is non-obvious. If you declare function return types and structure slot types, then the type of a variable is often obvious both to the programmer and to the compiler. An important case where the type isn't obvious, and a declaration is appropriate, is when the value for a variable is pulled out of untyped structure (e.g., the result of \code{car}), or comes from some weakly typed function, such as \code{read}. \item Declarations are sometimes necessary for integer loop variables, since the compiler can't always prove that the value is of a good integer type. These declarations are best added during tuning, when an efficiency note indicates the need. \end{itemize} %% %%\node Type Inference, Source Optimization, More About Types in Python, Advanced Compiler Use and Efficiency Hints \section{Type Inference} \label{type-inference} \cindex{type inference} \cindex{inference of types} \cindex{derivation of types} Type inference is the process by which the compiler tries to figure out the types of expressions and variables, given an inevitable lack of complete type information. Although \python{} does much more type inference than most \llisp{} compilers, remember that the more precise and comprehensive type declarations are, the more type inference will be able to do. \begin{comment} * Variable Type Inference:: * Local Function Type Inference:: * Global Function Type Inference:: * Operation Specific Type Inference:: * Dynamic Type Inference:: * Type Check Optimization:: \end{comment} %%\node Variable Type Inference, Local Function Type Inference, Type Inference, Type Inference \subsection{Variable Type Inference} \label{variable-type-inference} The type of a variable is the union of the types of all the definitions. In the degenerate case of a let, the type of the variable is the type of the initial value. This inferred type is intersected with any declared type, and is then propagated to all the variable's references. The types of \findexed{multiple-value-bind} variables are similarly inferred from the types of the individual values of the values form. If multiple type declarations apply to a single variable, then all the declarations must be correct; it is as though all the types were intersected producing a single \tindexed{and} type specifier. In this example: \begin{example} (defmacro my-dotimes ((var count) &body body) (do ((,var 0 (1+ ,var))) ((>= ,var ,count)) (declare (type (integer 0 *) ,var)) ,@body)) (my-dotimes (i ...) (declare (fixnum i)) ...) \end{example} the two declarations for \code{i} are intersected, so \code{i} is known to be a non-negative fixnum. In practice, this type inference is limited to lets and local functions, since the compiler can't analyze all the calls to a global function. But type inference works well enough on local variables so that it is often unnecessary to declare the type of local variables. This is especially likely when function result types and structure slot types are declared. The main areas where type inference breaks down are: \begin{itemize} \item When the initial value of a variable is a untyped expression, such as \code{\w{(car x)}}, and \item When the type of one of the variable's definitions is a function of the variable's current value, as in: \code{(setq x (1+ x))} \end{itemize} %%\node Local Function Type Inference, Global Function Type Inference, Variable Type Inference, Type Inference \subsection{Local Function Type Inference} \cpsubindex{local call}{type inference} The types of arguments to local functions are inferred in the same was as any other local variable; the type is the union of the argument types across all the calls to the function, intersected with the declared type. If there are any assignments to the argument variables, the type of the assigned value is unioned in as well. The result type of a local function is computed in a special way that takes tail recursion (\pxlref{tail-recursion}) into consideration. The result type is the union of all possible return values that aren't tail-recursive calls. For example, \python{} will infer that the result type of this function is \code{integer}: \begin{lisp} (defun ! (n res) (declare (integer n res)) (if (zerop n) res (! (1- n) (* n res)))) \end{lisp} Although this is a rather obvious result, it becomes somewhat less trivial in the presence of mutual tail recursion of multiple functions. Local function result type inference interacts with the mechanisms for ensuring proper tail recursion mentioned in section \ref{local-call-return}. %%\node Global Function Type Inference, Operation Specific Type Inference, Local Function Type Inference, Type Inference \subsection{Global Function Type Inference} \label{function-type-inference} \cpsubindex{function}{type inference} As described in section \ref{function-types}, a global function type (\tindexed{ftype}) declaration places implicit type assertions on the call arguments, and also guarantees the type of the return value. So wherever a call to a declared function appears, there is no doubt as to the types of the arguments and return value. Furthermore, \python{} will infer a function type from the function's definition if there is no \code{ftype} declaration. Any type declarations on the argument variables are used as the argument types in the derived function type, and the compiler's best guess for the result type of the function is used as the result type in the derived function type. This method of deriving function types from the definition implicitly assumes that functions won't be redefined at run-time. Consider this example: \begin{lisp} (defun foo-p (x) (let ((res (and (consp x) (eq (car x) 'foo)))) (format t "It is ~:[not ~;~]foo." res))) (defun frob (it) (if (foo-p it) (setf (cadr it) 'yow!) (1+ it))) \end{lisp} Presumably, the programmer really meant to return \code{res} from \code{foo-p}, but he seems to have forgotten. When he tries to call do \code{\w{(frob (list 'foo nil))}}, \code{frob} will flame out when it tries to add to a \code{cons}. Realizing his error, he fixes \code{foo-p} and recompiles it. But when he retries his test case, he is baffled because the error is still there. What happened in this example is that \python{} proved that the result of \code{foo-p} is \code{null}, and then proceeded to optimize away the \code{setf} in \code{frob}. Fortunately, in this example, the error is detected at compile time due to notes about unreachable code (\pxlref{dead-code-notes}.) Still, some users may not want to worry about this sort of problem during incremental development, so there is a variable to control deriving function types. \begin{defvar}{extensions:}{derive-function-types} If true (the default), argument and result type information derived from compilation of \code{defun}s is used when compiling calls to that function. If false, only information from \code{ftype} proclamations will be used. \end{defvar} %%\node Operation Specific Type Inference, Dynamic Type Inference, Global Function Type Inference, Type Inference \subsection{Operation Specific Type Inference} \label{operation-type-inference} \cindex{operation specific type inference} \cindex{arithmetic type inference} \cpsubindex{numeric}{type inference} Many of the standard \clisp{} functions have special type inference procedures that determine the result type as a function of the argument types. For example, the result type of \code{aref} is the array element type. Here are some other examples of type inferences: \begin{lisp} (logand x #xFF) \result{} (unsigned-byte 8) (+ (the (integer 0 12) x) (the (integer 0 1) y)) \result{} (integer 0 13) (ash (the (unsigned-byte 16) x) -8) \result{} (unsigned-byte 8) \end{lisp} %%\node Dynamic Type Inference, Type Check Optimization, Operation Specific Type Inference, Type Inference \subsection{Dynamic Type Inference} \label{constraint-propagation} \cindex{dynamic type inference} \cindex{conditional type inference} \cpsubindex{type inference}{dynamic} Python uses flow analysis to infer types in dynamically typed programs. For example: \begin{example} (ecase x (list (length x)) ...) \end{example} Here, the compiler knows the argument to \code{length} is a list, because the call to \code{length} is only done when \code{x} is a list. The most significant efficiency effect of inference from assertions is usually in type check optimization. Dynamic type inference has two inputs: explicit conditionals and implicit or explicit type assertions. Flow analysis propagates these constraints on variable type to any code that can be executed only after passing though the constraint. Explicit type constraints come from \findexed{if}s where the test is either a lexical variable or a function of lexical variables and constants, where the function is either a type predicate, a numeric comparison or \code{eq}. If there is an \code{eq} (or \code{eql}) test, then the compiler will actually substitute one argument for the other in the true branch. For example: \begin{lisp} (when (eq x :yow!) (return x)) \end{lisp} becomes: \begin{lisp} (when (eq x :yow!) (return :yow!)) \end{lisp} This substitution is done when one argument is a constant, or one argument has better type information than the other. This transformation reveals opportunities for constant folding or type-specific optimizations. If the test is against a constant, then the compiler can prove that the variable is not that constant value in the false branch, or \w{\code{(not (member :yow!))}} in the example above. This can eliminate redundant tests, for example: \begin{example} (if (eq x nil) ... (if x a b)) \end{example} is transformed to this: \begin{example} (if (eq x nil) ... a) \end{example} Variables appearing as \code{if} tests are interpreted as \code{\w{(not (eq \var{var} nil))}} tests. The compiler also converts \code{=} into \code{eql} where possible. It is difficult to do inference directly on \code{=} since it does implicit coercions. When there is an explicit \code{$<$} or \code{$>$} test on \begin{changebar} numeric \end{changebar} variables, the compiler makes inferences about the ranges the variables can assume in the true and false branches. This is mainly useful when it proves that the values are small enough in magnitude to allow open-coding of arithmetic operations. For example, in many uses of \code{dotimes} with a \code{fixnum} repeat count, the compiler proves that fixnum arithmetic can be used. Implicit type assertions are quite common, especially if you declare function argument types. Dynamic inference from implicit type assertions sometimes helps to disambiguate programs to a useful degree, but is most noticeable when it detects a dynamic type error. For example: \begin{lisp} (defun foo (x) (+ (car x) x)) \end{lisp} results in this warning: \begin{example} In: DEFUN FOO (+ (CAR X) X) ==> X Warning: Result is a LIST, not a NUMBER. \end{example} Note that \llisp{}'s dynamic type checking semantics make dynamic type inference useful even in programs that aren't really dynamically typed, for example: \begin{lisp} (+ (car x) (length x)) \end{lisp} Here, \code{x} presumably always holds a list, but in the absence of a declaration the compiler cannot assume \code{x} is a list simply because list-specific operations are sometimes done on it. The compiler must consider the program to be dynamically typed until it proves otherwise. Dynamic type inference proves that the argument to \code{length} is always a list because the call to \code{length} is only done after the list-specific \code{car} operation. %%\node Type Check Optimization, , Dynamic Type Inference, Type Inference \subsection{Type Check Optimization} \label{type-check-optimization} \cpsubindex{type checking}{optimization} \cpsubindex{optimization}{type check} Python backs up its support for precise type checking by minimizing the cost of run-time type checking. This is done both through type inference and though optimizations of type checking itself. Type inference often allows the compiler to prove that a value is of the correct type, and thus no type check is necessary. For example: \begin{lisp} (defstruct foo a b c) (defstruct link (foo (required-argument) :type foo) (next nil :type (or link null))) (foo-a (link-foo x)) \end{lisp} Here, there is no need to check that the result of \code{link-foo} is a \code{foo}, since it always is. Even when some type checks are necessary, type inference can often reduce the number: \begin{example} (defun test (x) (let ((a (foo-a x)) (b (foo-b x)) (c (foo-c x))) ...)) \end{example} In this example, only one \w{\code{(foo-p x)}} check is needed. This applies to a lesser degree in list operations, such as: \begin{lisp} (if (eql (car x) 3) (cdr x) y) \end{lisp} Here, we only have to check that \code{x} is a list once. Since \python{} recognizes explicit type tests, code that explicitly protects itself against type errors has little introduced overhead due to implicit type checking. For example, this loop compiles with no implicit checks checks for \code{car} and \code{cdr}: \begin{lisp} (defun memq (e l) (do ((current l (cdr current))) ((atom current) nil) (when (eq (car current) e) (return current)))) \end{lisp} \cindex{complemented type checks} Python reduces the cost of checks that must be done through an optimization called \var{complementing}. A complemented check for \var{type} is simply a check that the value is not of the type \w{\code{(not \var{type})}}. This is only interesting when something is known about the actual type, in which case we can test for the complement of \w{\code{(and \var{known-type} (not \var{type}))}}, or the difference between the known type and the assertion. An example: \begin{lisp} (link-foo (link-next x)) \end{lisp} Here, we change the type check for \code{link-foo} from a test for \code{foo} to a test for: \begin{lisp} (not (and (or foo null) (not foo))) \end{lisp} or more simply \w{\code{(not null)}}. This is probably the most important use of complementing, since the situation is fairly common, and a \code{null} test is much cheaper than a structure type test. Here is a more complicated example that illustrates the combination of complementing with dynamic type inference: \begin{lisp} (defun find-a (a x) (declare (type (or link null) x)) (do ((current x (link-next current))) ((null current) nil) (let ((foo (link-foo current))) (when (eq (foo-a foo) a) (return foo))))) \end{lisp} This loop can be compiled with no type checks. The \code{link} test for \code{link-foo} and \code{link-next} is complemented to \w{\code{(not null)}}, and then deleted because of the explicit \code{null} test. As before, no check is necessary for \code{foo-a}, since the \code{link-foo} is always a \code{foo}. This sort of situation shows how precise type checking combined with precise declarations can actually result in reduced type checking. %% %%\node Source Optimization, Tail Recursion, Type Inference, Advanced Compiler Use and Efficiency Hints \section{Source Optimization} \label{source-optimization} \cindex{optimization} This section describes source-level transformations that \python{} does on programs in an attempt to make them more efficient. Although source-level optimizations can make existing programs more efficient, the biggest advantage of this sort of optimization is that it makes it easier to write efficient programs. If a clean, straightforward implementation is can be transformed into an efficient one, then there is no need for tricky and dangerous hand optimization. \begin{comment} * Let Optimization:: * Constant Folding:: * Unused Expression Elimination:: * Control Optimization:: * Unreachable Code Deletion:: * Multiple Values Optimization:: * Source to Source Transformation:: * Style Recommendations:: \end{comment} %%\node Let Optimization, Constant Folding, Source Optimization, Source Optimization \subsection{Let Optimization} \label{let-optimization} \cindex{let optimization} \cpsubindex{optimization}{let} The primary optimization of let variables is to delete them when they are unnecessary. Whenever the value of a let variable is a constant, a constant variable or a constant (local or non-notinline) function, the variable is deleted, and references to the variable are replaced with references to the constant expression. This is useful primarily in the expansion of macros or inline functions, where argument values are often constant in any given call, but are in general non-constant expressions that must be bound to preserve order of evaluation. Let variable optimization eliminates the need for macros to carefully avoid spurious bindings, and also makes inline functions just as efficient as macros. A particularly interesting class of constant is a local function. Substituting for lexical variables that are bound to a function can substantially improve the efficiency of functional programming styles, for example: \begin{lisp} (let ((a #'(lambda (x) (zow x)))) (funcall a 3)) \end{lisp} effectively transforms to: \begin{lisp} (zow 3) \end{lisp} This transformation is done even when the function is a closure, as in: \begin{lisp} (let ((a (let ((y (zug))) #'(lambda (x) (zow x y))))) (funcall a 3)) \end{lisp} becoming: \begin{lisp} (zow 3 (zug)) \end{lisp} A constant variable is a lexical variable that is never assigned to, always keeping its initial value. Whenever possible, avoid setting lexical variables\dash{}instead bind a new variable to the new value. Except for loop variables, it is almost always possible to avoid setting lexical variables. This form: \begin{example} (let ((x (f x))) ...) \end{example} is \var{more} efficient than this form: \begin{example} (setq x (f x)) ... \end{example} Setting variables makes the program more difficult to understand, both to the compiler and to the programmer. \python{} compiles assignments at least as efficiently as any other \llisp{} compiler, but most let optimizations are only done on constant variables. Constant variables with only a single use are also optimized away, even when the initial value is not constant.\footnote{The source transformation in this example doesn't represent the preservation of evaluation order implicit in the compiler's internal representation. Where necessary, the back end will reintroduce temporaries to preserve the semantics.} For example, this expansion of \code{incf}: \begin{lisp} (let ((#:g3 (+ x 1))) (setq x #:G3)) \end{lisp} becomes: \begin{lisp} (setq x (+ x 1)) \end{lisp} The type semantics of this transformation are more important than the elimination of the variable itself. Consider what happens when \code{x} is declared to be a \code{fixnum}; after the transformation, the compiler can compile the addition knowing that the result is a \code{fixnum}, whereas before the transformation the addition would have to allow for fixnum overflow. Another variable optimization deletes any variable that is never read. This causes the initial value and any assigned values to be unused, allowing those expressions to be deleted if they have no side-effects. Note that a let is actually a degenerate case of local call (\pxlref{let-calls}), and that let optimization can be done on calls that weren't created by a let. Also, local call allows an applicative style of iteration that is totally assignment free. %%\node Constant Folding, Unused Expression Elimination, Let Optimization, Source Optimization \subsection{Constant Folding} \cindex{constant folding} \cpsubindex{folding}{constant} Constant folding is an optimization that replaces a call of constant arguments with the constant result of that call. Constant folding is done on all standard functions for which it is legal. Inline expansion allows folding of any constant parts of the definition, and can be done even on functions that have side-effects. It is convenient to rely on constant folding when programming, as in this example: \begin{example} (defconstant limit 42) (defun foo () (... (1- limit) ...)) \end{example} Constant folding is also helpful when writing macros or inline functions, since it usually eliminates the need to write a macro that special-cases constant arguments. \cindex{constant-function declaration} Constant folding of a user defined function is enabled by the \code{extensions:constant-function} proclamation. In this example: \begin{example} (declaim (ext:constant-function myfun)) (defun myexp (x y) (declare (single-float x y)) (exp (* (log x) y))) ... (myexp 3.0 1.3) ... \end{example} The call to \code{myexp} is constant-folded to \code{4.1711674}. %%\node Unused Expression Elimination, Control Optimization, Constant Folding, Source Optimization \subsection{Unused Expression Elimination} \cindex{unused expression elimination} \cindex{dead code elimination} If the value of any expression is not used, and the expression has no side-effects, then it is deleted. As with constant folding, this optimization applies most often when cleaning up after inline expansion and other optimizations. Any function declared an \code{extensions:constant-function} is also subject to unused expression elimination. Note that \python{} will eliminate parts of unused expressions known to be side-effect free, even if there are other unknown parts. For example: \begin{lisp} (let ((a (list (foo) (bar)))) (if t (zow) (raz a))) \end{lisp} becomes: \begin{lisp} (progn (foo) (bar)) (zow) \end{lisp} %%\node Control Optimization, Unreachable Code Deletion, Unused Expression Elimination, Source Optimization \subsection{Control Optimization} \cindex{control optimization} \cpsubindex{optimization}{control} The most important optimization of control is recognizing when an \findexed{if} test is known at compile time, then deleting the \code{if}, the test expression, and the unreachable branch of the \code{if}. This can be considered a special case of constant folding, although the test doesn't have to be truly constant as long as it is definitely not \false. Note also, that type inference propagates the result of an \code{if} test to the true and false branches, \pxlref{constraint-propagation}. A related \code{if} optimization is this transformation:\footnote{Note that the code for \code{x} and \code{y} isn't actually replicated.} \begin{lisp} (if (if a b c) x y) \end{lisp} into: \begin{lisp} (if a (if b x y) (if c x y)) \end{lisp} The opportunity for this sort of optimization usually results from a conditional macro. For example: \begin{lisp} (if (not a) x y) \end{lisp} is actually implemented as this: \begin{lisp} (if (if a nil t) x y) \end{lisp} which is transformed to this: \begin{lisp} (if a (if nil x y) (if t x y)) \end{lisp} which is then optimized to this: \begin{lisp} (if a y x) \end{lisp} Note that due to \python{}'s internal representations, the \code{if}\dash{}\code{if} situation will be recognized even if other forms are wrapped around the inner \code{if}, like: \begin{example} (if (let ((g ...)) (loop ... (return (not g)) ...)) x y) \end{example} In \python, all the \clisp{} macros really are macros, written in terms of \code{if}, \code{block} and \code{tagbody}, so user-defined control macros can be just as efficient as the standard ones. \python{} emits basic blocks using a heuristic that minimizes the number of unconditional branches. The code in a \code{tagbody} will not be emitted in the order it appeared in the source, so there is no point in arranging the code to make control drop through to the target. %%\node Unreachable Code Deletion, Multiple Values Optimization, Control Optimization, Source Optimization \subsection{Unreachable Code Deletion} \label{dead-code-notes} \cindex{unreachable code deletion} \cindex{dead code elimination} Python will delete code whenever it can prove that the code can never be executed. Code becomes unreachable when: \begin{itemize} \item An \code{if} is optimized away, or \item There is an explicit unconditional control transfer such as \code{go} or \code{return-from}, or \item The last reference to a local function is deleted (or there never was any reference.) \end{itemize} When code that appeared in the original source is deleted, the compiler prints a note to indicate a possible problem (or at least unnecessary code.) For example: \begin{lisp} (defun foo () (if t (write-line "True.") (write-line "False."))) \end{lisp} will result in this note: \begin{example} In: DEFUN FOO (WRITE-LINE "False.") Note: Deleting unreachable code. \end{example} It is important to pay attention to unreachable code notes, since they often indicate a subtle type error. For example: \begin{example} (defstruct foo a b) (defun lose (x) (let ((a (foo-a x)) (b (if x (foo-b x) :none))) ...)) \end{example} results in this note: \begin{example} In: DEFUN LOSE (IF X (FOO-B X) :NONE) ==> :NONE Note: Deleting unreachable code. \end{example} The \kwd{none} is unreachable, because type inference knows that the argument to \code{foo-a} must be a \code{foo}, and thus can't be \false. Presumably the programmer forgot that \code{x} could be \false{} when he wrote the binding for \code{a}. Here is an example with an incorrect declaration: \begin{lisp} (defun count-a (string) (do ((pos 0 (position #\back{a} string :start (1+ pos))) (count 0 (1+ count))) ((null pos) count) (declare (fixnum pos)))) \end{lisp} This time our note is: \begin{example} In: DEFUN COUNT-A (DO ((POS 0 #) (COUNT 0 #)) ((NULL POS) COUNT) (DECLARE (FIXNUM POS))) --> BLOCK LET TAGBODY RETURN-FROM PROGN ==> COUNT Note: Deleting unreachable code. \end{example} The problem here is that \code{pos} can never be null since it is declared a \code{fixnum}. It takes some experience with unreachable code notes to be able to tell what they are trying to say. In non-obvious cases, the best thing to do is to call the function in a way that should cause the unreachable code to be executed. Either you will get a type error, or you will find that there truly is no way for the code to be executed. Not all unreachable code results in a note: \begin{itemize} \item A note is only given when the unreachable code textually appears in the original source. This prevents spurious notes due to the optimization of macros and inline functions, but sometimes also foregoes a note that would have been useful. \item Since accurate source information is not available for non-list forms, there is an element of heuristic in determining whether or not to give a note about an atom. Spurious notes may be given when a macro or inline function defines a variable that is also present in the calling function. Notes about \false{} and \true{} are never given, since it is too easy to confuse these constants in expanded code with ones in the original source. \item Notes are only given about code unreachable due to control flow. There is no note when an expression is deleted because its value is unused, since this is a common consequence of other optimizations. \end{itemize} Somewhat spurious unreachable code notes can also result when a macro inserts multiple copies of its arguments in different contexts, for example: \begin{lisp} (defmacro t-and-f (var form) (if ,var ,form ,form)) (defun foo (x) (t-and-f x (if x "True." "False."))) \end{lisp} results in these notes: \begin{example} In: DEFUN FOO (IF X "True." "False.") ==> "False." Note: Deleting unreachable code. ==> "True." Note: Deleting unreachable code. \end{example} It seems like it has deleted both branches of the \code{if}, but it has really deleted one branch in one copy, and the other branch in the other copy. Note that these messages are only spurious in not satisfying the intent of the rule that notes are only given when the deleted code appears in the original source; there is always \var{some} code being deleted when a unreachable code note is printed. %%\node Multiple Values Optimization, Source to Source Transformation, Unreachable Code Deletion, Source Optimization \subsection{Multiple Values Optimization} \cindex{multiple value optimization} \cpsubindex{optimization}{multiple value} Within a function, \python{} implements uses of multiple values particularly efficiently. Multiple values can be kept in arbitrary registers, so using multiple values doesn't imply stack manipulation and representation conversion. For example, this code: \begin{example} (let ((a (if x (foo x) u)) (b (if x (bar x) v))) ...) \end{example} is actually more efficient written this way: \begin{example} (multiple-value-bind (a b) (if x (values (foo x) (bar x)) (values u v)) ...) \end{example} Also, \pxlref{local-call-return} for information on how local call provides efficient support for multiple function return values. %%\node Source to Source Transformation, Style Recommendations, Multiple Values Optimization, Source Optimization \subsection{Source to Source Transformation} \cindex{source-to-source transformation} \cpsubindex{transformation}{source-to-source} The compiler implements a number of operation-specific optimizations as source-to-source transformations. You will often see unfamiliar code in error messages, for example: \begin{lisp} (defun my-zerop () (zerop x)) \end{lisp} gives this warning: \begin{example} In: DEFUN MY-ZEROP (ZEROP X) ==> (= X 0) Warning: Undefined variable: X \end{example} The original \code{zerop} has been transformed into a call to \code{=}. This transformation is indicated with the same \code{==$>$} used to mark macro and function inline expansion. Although it can be confusing, display of the transformed source is important, since warnings are given with respect to the transformed source. This a more obscure example: \begin{lisp} (defun foo (x) (logand 1 x)) \end{lisp} gives this efficiency note: \begin{example} In: DEFUN FOO (LOGAND 1 X) ==> (LOGAND C::Y C::X) Note: Forced to do static-function Two-arg-and (cost 53). Unable to do inline fixnum arithmetic (cost 1) because: The first argument is a INTEGER, not a FIXNUM. etc. \end{example} Here, the compiler commuted the call to \code{logand}, introducing temporaries. The note complains that the \var{first} argument is not a \code{fixnum}, when in the original call, it was the second argument. To make things more confusing, the compiler introduced temporaries called \code{c::x} and \code{c::y} that are bound to \code{y} and \code{1}, respectively. You will also notice source-to-source optimizations when efficiency notes are enabled (\pxlref{efficiency-notes}.) When the compiler is unable to do a transformation that might be possible if there was more information, then an efficiency note is printed. For example, \code{my-zerop} above will also give this efficiency note: \begin{example} In: DEFUN FOO (ZEROP X) ==> (= X 0) Note: Unable to optimize because: Operands might not be the same type, so can't open code. \end{example} %%\node Style Recommendations, , Source to Source Transformation, Source Optimization \subsection{Style Recommendations} \cindex{style recommendations} Source level optimization makes possible a clearer and more relaxed programming style: \begin{itemize} \item Don't use macros purely to avoid function call. If you want an inline function, write it as a function and declare it inline. It's clearer, less error-prone, and works just as well. \item Don't write macros that try to optimize'' their expansion in trivial ways such as avoiding binding variables for simple expressions. The compiler does these optimizations too, and is less likely to make a mistake. \item Make use of local functions (i.e., \code{labels} or \code{flet}) and tail-recursion in places where it is clearer. Local function call is faster than full call. \item Avoid setting local variables when possible. Binding a new \code{let} variable is at least as efficient as setting an existing variable, and is easier to understand, both for the compiler and the programmer. \item Instead of writing similar code over and over again so that it can be hand customized for each use, define a macro or inline function, and let the compiler do the work. \end{itemize} %% %%\node Tail Recursion, Local Call, Source Optimization, Advanced Compiler Use and Efficiency Hints \section{Tail Recursion} \label{tail-recursion} \cindex{tail recursion} \cindex{recursion} A call is tail-recursive if nothing has to be done after the the call returns, i.e. when the call returns, the returned value is immediately returned from the calling function. In this example, the recursive call to \code{myfun} is tail-recursive: \begin{lisp} (defun myfun (x) (if (oddp (random x)) (isqrt x) (myfun (1- x)))) \end{lisp} Tail recursion is interesting because it is form of recursion that can be implemented much more efficiently than general recursion. In general, a recursive call requires the compiler to allocate storage on the stack at run-time for every call that has not yet returned. This memory consumption makes recursion unacceptably inefficient for representing repetitive algorithms having large or unbounded size. Tail recursion is the special case of recursion that is semantically equivalent to the iteration constructs normally used to represent repetition in programs. Because tail recursion is equivalent to iteration, tail-recursive programs can be compiled as efficiently as iterative programs. So why would you want to write a program recursively when you can write it using a loop? Well, the main answer is that recursion is a more general mechanism, so it can express some solutions simply that are awkward to write as a loop. Some programmers also feel that recursion is a stylistically preferable way to write loops because it avoids assigning variables. For example, instead of writing: \begin{lisp} (defun fun1 (x) something-that-uses-x) (defun fun2 (y) something-that-uses-y) (do ((x something (fun2 (fun1 x)))) (nil)) \end{lisp} You can write: \begin{lisp} (defun fun1 (x) (fun2 something-that-uses-x)) (defun fun2 (y) (fun1 something-that-uses-y)) (fun1 something) \end{lisp} The tail-recursive definition is actually more efficient, in addition to being (arguably) clearer. As the number of functions and the complexity of their call graph increases, the simplicity of using recursion becomes compelling. Consider the advantages of writing a large finite-state machine with separate tail-recursive functions instead of using a single huge \code{prog}. It helps to understand how to use tail recursion if you think of a tail-recursive call as a \code{psetq} that assigns the argument values to the called function's variables, followed by a \code{go} to the start of the called function. This makes clear an inherent efficiency advantage of tail-recursive call: in addition to not having to allocate a stack frame, there is no need to prepare for the call to return (e.g., by computing a return PC.) Is there any disadvantage to tail recursion? Other than an increase in efficiency, the only way you can tell that a call has been compiled tail-recursively is if you use the debugger. Since a tail-recursive call has no stack frame, there is no way the debugger can print out the stack frame representing the call. The effect is that backtrace will not show some calls that would have been displayed in a non-tail-recursive implementation. In practice, this is not as bad as it sounds\dash{}in fact it isn't really clearly worse, just different. \xlref{debug-tail-recursion} for information about the debugger implications of tail recursion. In order to ensure that tail-recursion is preserved in arbitrarily complex calling patterns across separately compiled functions, the compiler must compile any call in a tail-recursive position as a tail-recursive call. This is done regardless of whether the program actually exhibits any sort of recursive calling pattern. In this example, the call to \code{fun2} will always be compiled as a tail-recursive call: \begin{lisp} (defun fun1 (x) (fun2 x)) \end{lisp} So tail recursion doesn't necessarily have anything to do with recursion as it is normally thought of. \xlref{local-tail-recursion} for more discussion of using tail recursion to implement loops. \begin{comment} * Tail Recursion Exceptions:: \end{comment} %%\node Tail Recursion Exceptions, , Tail Recursion, Tail Recursion \subsection{Tail Recursion Exceptions} Although \python{} is claimed to be properly'' tail-recursive, some might dispute this, since there are situations where tail recursion is inhibited: \begin{itemize} \item When the call is enclosed by a special binding, or \item When the call is enclosed by a \code{catch} or \code{unwind-protect}, or \item When the call is enclosed by a \code{block} or \code{tagbody} and the block name or \code{go} tag has been closed over. \end{itemize} These dynamic extent binding forms inhibit tail recursion because they allocate stack space to represent the binding. Shallow-binding implementations of dynamic scoping also require cleanup code to be evaluated when the scope is exited. %% %%\node Local Call, Block Compilation, Tail Recursion, Advanced Compiler Use and Efficiency Hints \section{Local Call} \label{local-call} \cindex{local call} \cpsubindex{call}{local} \cpsubindex{function call}{local} Python supports two kinds of function call: full call and local call. Full call is the standard calling convention; its late binding and generality make \llisp{} what it is, but create unavoidable overheads. When the compiler can compile the calling function and the called function simultaneously, it can use local call to avoid some of the overhead of full call. Local call is really a collection of compilation strategies. If some aspect of call overhead is not needed in a particular local call, then it can be omitted. In some cases, local call can be totally free. Local call provides two main advantages to the user: \begin{itemize} \item Local call makes the use of the lexical function binding forms \findexed{flet} and \findexed{labels} much more efficient. A local call is always faster than a full call, and in many cases is much faster. \item Local call is a natural approach to \i{block compilation}, a compilation technique that resolves function references at compile time. Block compilation speeds function call, but increases compilation times and prevents function redefinition. \end{itemize} \begin{comment} * Self-Recursive Calls:: * Let Calls:: * Closures:: * Local Tail Recursion:: * Return Values:: \end{comment} %%\node Self-Recursive Calls, Let Calls, Local Call, Local Call \subsection{Self-Recursive Calls} \cpsubindex{recursion}{self} Local call is used when a function defined by \code{defun} calls itself. For example: \begin{lisp} (defun fact (n) (if (zerop n) 1 (* n (fact (1- n))))) \end{lisp} This use of local call speeds recursion, but can also complicate debugging, since \findexed{trace} will only show the first call to \code{fact}, and not the recursive calls. This is because the recursive calls directly jump to the start of the function, and don't indirect through the \code{symbol-function}. Self-recursive local call is inhibited when the \kwd{block-compile} argument to \code{compile-file} is \false{} (\pxlref{compile-file-block}.) %%\node Let Calls, Closures, Self-Recursive Calls, Local Call \subsection{Let Calls} \label{let-calls} Because local call avoids unnecessary call overheads, the compiler internally uses local call to implement some macros and special forms that are not normally thought of as involving a function call. For example, this \code{let}: \begin{example} (let ((a (foo)) (b (bar))) ...) \end{example} is internally represented as though it was macroexpanded into: \begin{example} (funcall #'(lambda (a b) ...) (foo) (bar)) \end{example} This implementation is acceptable because the simple cases of local call (equivalent to a \code{let}) result in good code. This doesn't make \code{let} any more efficient, but does make local calls that are semantically the same as \code{let} much more efficient than full calls. For example, these definitions are all the same as far as the compiler is concerned: \begin{example} (defun foo () ...some other stuff... (let ((a something)) ...some stuff...)) (defun foo () (flet ((localfun (a) ...some stuff...)) ...some other stuff... (localfun something))) (defun foo () (let ((funvar #'(lambda (a) ...some stuff...))) ...some other stuff... (funcall funvar something))) \end{example} Although local call is most efficient when the function is called only once, a call doesn't have to be equivalent to a \code{let} to be more efficient than full call. All local calls avoid the overhead of argument count checking and keyword argument parsing, and there are a number of other advantages that apply in many common situations. \xlref{let-optimization} for a discussion of the optimizations done on let calls. %%\node Closures, Local Tail Recursion, Let Calls, Local Call \subsection{Closures} \cindex{closures} Local call allows for much more efficient use of closures, since the closure environment doesn't need to be allocated on the heap, or even stored in memory at all. In this example, there is no penalty for \code{localfun} referencing \code{a} and \code{b}: \begin{lisp} (defun foo (a b) (flet ((localfun (x) (1+ (* a b x)))) (if (= a b) (localfun (- x)) (localfun x)))) \end{lisp} In local call, the compiler effectively passes closed-over values as extra arguments, so there is no need for you to optimize'' local function use by explicitly passing in lexically visible values. Closures may also be subject to let optimization (\pxlref{let-optimization}.) Note: indirect value cells are currently always allocated on the heap when a variable is both assigned to (with \code{setq} or \code{setf}) and closed over, regardless of whether the closure is a local function or not. This is another reason to avoid setting variables when you don't have to. %%\node Local Tail Recursion, Return Values, Closures, Local Call \subsection{Local Tail Recursion} \label{local-tail-recursion} \cindex{tail recursion} \cpsubindex{recursion}{tail} Tail-recursive local calls are particularly efficient, since they are in effect an assignment plus a control transfer. Scheme programmers write loops with tail-recursive local calls, instead of using the imperative \code{go} and \code{setq}. This has not caught on in the \clisp{} community, since conventional \llisp{} compilers don't implement local call. In \python, users can choose to write loops such as: \begin{lisp} (defun ! (n) (labels ((loop (n total) (if (zerop n) total (loop (1- n) (* n total))))) (loop n 1))) \end{lisp} \begin{defmac}{extensions:}{iterate}{% \args{\var{name} (\mstar{(\var{var} \var{initial-value})}) \mstar{\var{declaration}} \mstar{\var{form}}}} This macro provides syntactic sugar for using \findexed{labels} to do iteration. It creates a local function \var{name} with the specified \var{var}s as its arguments and the \var{declaration}s and \var{form}s as its body. This function is then called with the \var{initial-values}, and the result of the call is return from the macro. Here is our factorial example rewritten using \code{iterate}: \begin{lisp} (defun ! (n) (iterate loop ((n n) (total 1)) (if (zerop n) total (loop (1- n) (* n total))))) \end{lisp} The main advantage of using \code{iterate} over \code{do} is that \code{iterate} naturally allows stepping to be done differently depending on conditionals in the body of the loop. \code{iterate} can also be used to implement algorithms that aren't really iterative by simply doing a non-tail call. For example, the standard recursive definition of factorial can be written like this: \begin{lisp} (iterate fact ((n n)) (if (zerop n) 1 (* n (fact (1- n))))) \end{lisp} \end{defmac} %%\node Return Values, , Local Tail Recursion, Local Call \subsection{Return Values} \label{local-call-return} \cpsubindex{return values}{local call} \cpsubindex{local call}{return values} One of the more subtle costs of full call comes from allowing arbitrary numbers of return values. This overhead can be avoided in local calls to functions that always return the same number of values. For efficiency reasons (as well as stylistic ones), you should write functions so that they always return the same number of values. This may require passing extra \false{} arguments to \code{values} in some cases, but the result is more efficient, not less so. When efficiency notes are enabled (\pxlref{efficiency-notes}), and the compiler wants to use known values return, but can't prove that the function always returns the same number of values, then it will print a note like this: \begin{example} In: DEFUN GRUE (DEFUN GRUE (X) (DECLARE (FIXNUM X)) (COND (# #) (# NIL) (T #))) Note: Return type not fixed values, so can't use known return convention: (VALUES (OR (INTEGER -536870912 -1) NULL) &REST T) \end{example} In order to implement proper tail recursion in the presence of known values return (\pxlref{tail-recursion}), the compiler sometimes must prove that multiple functions all return the same number of values. When this can't be proven, the compiler will print a note like this: \begin{example} In: DEFUN BLUE (DEFUN BLUE (X) (DECLARE (FIXNUM X)) (COND (# #) (# #) (# #) (T #))) Note: Return value count mismatch prevents known return from these functions: BLUE SNOO \end{example} \xlref{number-local-call} for the interaction between local call and the representation of numeric types. %% %%\node Block Compilation, Inline Expansion, Local Call, Advanced Compiler Use and Efficiency Hints \section{Block Compilation} \label{block-compilation} \cindex{block compilation} \cpsubindex{compilation}{block} Block compilation allows calls to global functions defined by \findexed{defun} to be compiled as local calls. The function call can be in a different top-level form than the \code{defun}, or even in a different file. In addition, block compilation allows the declaration of the \i{entry points} to the block compiled portion. An entry point is any function that may be called from outside of the block compilation. If a function is not an entry point, then it can be compiled more efficiently, since all calls are known at compile time. In particular, if a function is only called in one place, then it will be let converted. This effectively inline expands the function, but without the code duplication that results from defining the function normally and then declaring it inline. The main advantage of block compilation is that it it preserves efficiency in programs even when (for readability and syntactic convenience) they are broken up into many small functions. There is absolutely no overhead for calling a non-entry point function that is defined purely for modularity (i.e. called only in one place.) Block compilation also allows the use of non-descriptor arguments and return values in non-trivial programs (\pxlref{number-local-call}). \begin{comment} * Block Compilation Semantics:: * Block Compilation Declarations:: * Compiler Arguments:: * Practical Difficulties:: * Context Declarations:: * Context Declaration Example:: \end{comment} %%\node Block Compilation Semantics, Block Compilation Declarations, Block Compilation, Block Compilation \subsection{Block Compilation Semantics} The effect of block compilation can be envisioned as the compiler turning all the \code{defun}s in the block compilation into a single \code{labels} form: \begin{example} (declaim (start-block fun1 fun3)) (defun fun1 () ...) (defun fun2 () ... (fun1) ...) (defun fun3 (x) (if x (fun1) (fun2))) (declaim (end-block)) \end{example} becomes: \begin{example} (labels ((fun1 () ...) (fun2 () ... (fun1) ...) (fun3 (x) (if x (fun1) (fun2)))) (setf (fdefinition 'fun1) #'fun1) (setf (fdefinition 'fun3) #'fun3)) \end{example} Calls between the block compiled functions are local calls, so changing the global definition of \code{fun1} will have no effect on what \code{fun2} does; \code{fun2} will keep calling the old \code{fun1}. The entry points \code{fun1} and \code{fun3} are still installed in the \code{symbol-function} as the global definitions of the functions, so a full call to an entry point works just as before. However, \code{fun2} is not an entry point, so it is not globally defined. In addition, \code{fun2} is only called in one place, so it will be let converted. %%\node Block Compilation Declarations, Compiler Arguments, Block Compilation Semantics, Block Compilation \subsection{Block Compilation Declarations} \cpsubindex{declarations}{block compilation} \cindex{start-block declaration} \cindex{end-block declaration} The \code{extensions:start-block} and \code{extensions:end-block} declarations allow fine-grained control of block compilation. These declarations are only legal as a global declarations (\code{declaim} or \code{proclaim}). \noindent \vspace{1 em} The \code{start-block} declaration has this syntax: \begin{example} (start-block \mstar{\var{entry-point-name}}) \end{example} When processed by the compiler, this declaration marks the start of block compilation, and specifies the entry points to that block. If no entry points are specified, then \var{all} functions are made into entry points. If already block compiling, then the compiler ends the current block and starts a new one. \noindent \vspace{1 em} The \code{end-block} declaration has no arguments: \begin{lisp} (end-block) \end{lisp} The \code{end-block} declaration ends a block compilation unit without starting a new one. This is useful mainly when only a portion of a file is worth block compiling. %%\node Compiler Arguments, Practical Difficulties, Block Compilation Declarations, Block Compilation \subsection{Compiler Arguments} \label{compile-file-block} \cpsubindex{compile-file}{block compilation arguments} The \kwd{block-compile} and \kwd{entry-points} arguments to \code{extensions:compile-from-stream} and \funref{compile-file} provide overall control of block compilation, and allow block compilation without requiring modification of the program source. There are three possible values of the \kwd{block-compile} argument: \begin{Lentry} \item[\false{}] Do no compile-time resolution of global function names, not even for self-recursive calls. This inhibits any \code{start-block} declarations appearing in the file, allowing all functions to be incrementally redefined. \item[\true{}] Start compiling in block compilation mode. This is mainly useful for block compiling small files that contain no \code{start-block} declarations. See also the \kwd{entry-points} argument. \item[\kwd{specified}] Start compiling in form-at-a-time mode, but exploit \code{start-block} declarations and compile self-recursive calls as local calls. Normally \kwd{specified} is the default for this argument (see \varref{block-compile-default}.) \end{Lentry} The \kwd{entry-points} argument can be used in conjunction with \w{\kwd{block-compile} \true{}} to specify the entry-points to a block-compiled file. If not specified or \nil, all global functions will be compiled as entry points. When \kwd{block-compile} is not \true, this argument is ignored. \begin{defvar}{}{block-compile-default} This variable determines the default value for the \kwd{block-compile} argument to \code{compile-file} and \code{compile-from-stream}. The initial value of this variable is \kwd{specified}, but \false{} is sometimes useful for totally inhibiting block compilation. \end{defvar} %%\node Practical Difficulties, Context Declarations, Compiler Arguments, Block Compilation \subsection{Practical Difficulties} The main problem with block compilation is that the compiler uses large amounts of memory when it is block compiling. This places an upper limit on the amount of code that can be block compiled as a unit. To make best use of block compilation, it is necessary to locate the parts of the program containing many internal calls, and then add the appropriate \code{start-block} declarations. When writing new code, it is a good idea to put in block compilation declarations from the very beginning, since writing block declarations correctly requires accurate knowledge of the program's function call structure. If you want to initially develop code with full incremental redefinition, you can compile with \varref{block-compile-default} set to \false. Note if a \code{defun} appears in a non-null lexical environment, then calls to it cannot be block compiled. Unless files are very small, it is probably impractical to block compile multiple files as a unit by specifying a list of files to \code{compile-file}. Semi-inline expansion (\pxlref{semi-inline}) provides another way to extend block compilation across file boundaries. %% %%\node Context Declarations, Context Declaration Example, Practical Difficulties, Block Compilation \subsection{Context Declarations} \label{context-declarations} \cindex{context sensitive declarations} \cpsubindex{declarations}{context-sensitive} \cmucl{} has a context-sensitive declaration mechanism which is useful because it allows flexible control of the compilation policy in large systems without requiring changes to the source files. The primary use of this feature is to allow the exported interfaces of a system to be compiled more safely than the system internals. The context used is the name being defined and the kind of definition (function, macro, etc.) The \kwd{context-declarations} option to \macref{with-compilation-unit} has dynamic scope, affecting all compilation done during the evaluation of the body. The argument to this option should evaluate to a list of lists of the form: \begin{example} (\var{context-spec} \mplus{\var{declare-form}}) \end{example} In the indicated context, the specified declare forms are inserted at the head of each definition. The declare forms for all contexts that match are appended together, with earlier declarations getting precedence over later ones. A simple example: \begin{example} :context-declarations '((:external (declare (optimize (safety 2))))) \end{example} This will cause all functions that are named by external symbols to be compiled with \code{safety 2}. The full syntax of context specs is: \begin{Lentry} \item[\kwd{internal}, \kwd{external}] True if the symbol is internal (external) in its home package. \item[\kwd{uninterned}] True if the symbol has no home package. \item[\code{\w{(:package \mstar{\var{package-name}})}}] True if the symbol's home package is in any of the named packages (false if uninterned.) \item[\kwd{anonymous}] True if the function doesn't have any interesting name (not \code{defmacro}, \code{defun}, \code{labels} or \code{flet}). \item[\kwd{macro}, \kwd{function}] \kwd{macro} is a global (\code{defmacro}) macro. \kwd{function} is anything else. \item[\kwd{local}, \kwd{global}] \kwd{local} is a \code{labels} or \code{flet}. \kwd{global} is anything else. \item[\code{\w{(:or \mstar{\var{context-spec}})}}] True when any supplied \var{context-spec} is true. \item[\code{\w{(:and \mstar{\var{context-spec}})}}] True only when all supplied \var{context-spec}s are true. \item[\code{\w{(:not \mstar{\var{context-spec}})}}] True when \var{context-spec} is false. \item[\code{\w{(:member \mstar{\var{name}})}}] True when the defined name is one of these names (\code{equal} test.) \item[\code{\w{(:match \mstar{\var{pattern}})}}] True when any of the patterns is a substring of the name. The name is wrapped with \code{\$}'s, so \code{\$FOO}'' matches names beginning with \code{FOO}'', etc. \end{Lentry} %%\node Context Declaration Example, , Context Declarations, Block Compilation \subsection{Context Declaration Example} Here is a more complex example of \code{with-compilation-unit} options: \begin{example} :optimize '(optimize (speed 2) (space 2) (inhibit-warnings 2) (debug 1) (safety 0)) :optimize-interface '(optimize-interface (safety 1) (debug 1)) :context-declarations '(((:or :external (:and (:match "\%") (:match "SET"))) (declare (optimize-interface (safety 2)))) ((:or (:and :external :macro) (:match "\$PARSE-")) (declare (optimize (safety 2))))) \end{example} The \code{optimize} and \code{extensions:optimize-interface} declarations (\pxlref{optimize-declaration}) set up the global compilation policy. The bodies of functions are to be compiled completely unsafe (\code{safety 0}), but argument count and weakened argument type checking is to be done when a function is called (\code{speed 2 safety 1}). The first declaration specifies that all functions that are external or whose names contain both \code{\%}'' and \code{SET}'' are to be compiled compiled with completely safe interfaces (\code{safety 2}). The reason for this particular \kwd{match} rule is that \code{setf} inverse functions in this system tend to have both strings in their name somewhere. We want \code{setf} inverses to be safe because they are implicitly called by users even though their name is not exported. The second declaration makes external macros or functions whose names start with \code{PARSE-}'' have safe bodies (as well as interfaces). This is desirable because a syntax error in a macro may cause a type error inside the body. The \kwd{match} rule is used because macros often have auxiliary functions whose names begin with this string. This particular example is used to build part of the standard \cmucl{} system. Note however, that context declarations must be set up according to the needs and coding conventions of a particular system; different parts of \cmucl{} are compiled with different context declarations, and your system will probably need its own declarations. In particular, any use of the \kwd{match} option depends on naming conventions used in coding. %% %%\node Inline Expansion, Byte Coded Compilation, Block Compilation, Advanced Compiler Use and Efficiency Hints \section{Inline Expansion} \label{inline-expansion} \cindex{inline expansion} \cpsubindex{expansion}{inline} \cpsubindex{call}{inline} \cpsubindex{function call}{inline} \cpsubindex{optimization}{function call} Python can expand almost any function inline, including functions with keyword arguments. The only restrictions are that keyword argument keywords in the call must be constant, and that global function definitions (\code{defun}) must be done in a null lexical environment (not nested in a \code{let} or other binding form.) Local functions (\code{flet}) can be inline expanded in any environment. Combined with \python{}'s source-level optimization, inline expansion can be used for things that formerly required macros for efficient implementation. In \python, macros don't have any efficiency advantage, so they need only be used where a macro's syntactic flexibility is required. Inline expansion is a compiler optimization technique that reduces the overhead of a function call by simply not doing the call: instead, the compiler effectively rewrites the program to appear as though the definition of the called function was inserted at each call site. In \llisp, this is straightforwardly expressed by inserting the \code{lambda} corresponding to the original definition: \begin{lisp} (proclaim '(inline my-1+)) (defun my-1+ (x) (+ x 1)) (my-1+ someval) \result{} ((lambda (x) (+ x 1)) someval) \end{lisp} When the function expanded inline is large, the program after inline expansion may be substantially larger than the original program. If the program becomes too large, inline expansion hurts speed rather than helping it, since hardware resources such as physical memory and cache will be exhausted. Inline expansion is called for: \begin{itemize} \item When profiling has shown that a relatively simple function is called so often that a large amount of time is being wasted in the calling of that function (as opposed to running in that function.) If a function is complex, it will take a long time to run relative the time spent in call, so the speed advantage of inline expansion is diminished at the same time the space cost of inline expansion is increased. Of course, if a function is rarely called, then the overhead of calling it is also insignificant. \item With functions so simple that they take less space to inline expand than would be taken to call the function (such as \code{my-1+} above.) It would require intimate knowledge of the compiler to be certain when inline expansion would reduce space, but it is generally safe to inline expand functions whose definition is a single function call, or a few calls to simple \clisp{} functions. \end{itemize} In addition to this speed/space tradeoff from inline expansion's avoidance of the call, inline expansion can also reveal opportunities for optimization. \python{}'s extensive source-level optimization can make use of context information from the caller to tremendously simplify the code resulting from the inline expansion of a function. The main form of caller context is local information about the actual argument values: what the argument types are and whether the arguments are constant. Knowledge about argument types can eliminate run-time type tests (e.g., for generic arithmetic.) Constant arguments in a call provide opportunities for constant folding optimization after inline expansion. A hidden way that constant arguments are often supplied to functions is through the defaulting of unsupplied optional or keyword arguments. There can be a huge efficiency advantage to inline expanding functions that have complex keyword-based interfaces, such as this definition of the \code{member} function: \begin{lisp} (proclaim '(inline member)) (defun member (item list &key (key #'identity) (test #'eql testp) (test-not nil notp)) (do ((list list (cdr list))) ((null list) nil) (let ((car (car list))) (if (cond (testp (funcall test item (funcall key car))) (notp (not (funcall test-not item (funcall key car)))) (t (funcall test item (funcall key car)))) (return list))))) \end{lisp} After inline expansion, this call is simplified to the obvious code: \begin{lisp} (member a l :key #'foo-a :test #'char=) \result{} (do ((list list (cdr list))) ((null list) nil) (let ((car (car list))) (if (char= item (foo-a car)) (return list)))) \end{lisp} In this example, there could easily be more than an order of magnitude improvement in speed. In addition to eliminating the original call to \code{member}, inline expansion also allows the calls to \code{char=} and \code{foo-a} to be open-coded. We go from a loop with three tests and two calls to a loop with one test and no calls. \xlref{source-optimization} for more discussion of source level optimization. \begin{comment} * Inline Expansion Recording:: * Semi-Inline Expansion:: * The Maybe-Inline Declaration:: \end{comment} %%\node Inline Expansion Recording, Semi-Inline Expansion, Inline Expansion, Inline Expansion \subsection{Inline Expansion Recording} \cindex{recording of inline expansions} Inline expansion requires that the source for the inline expanded function to be available when calls to the function are compiled. The compiler doesn't remember the inline expansion for every function, since that would take an excessive about of space. Instead, the programmer must tell the compiler to record the inline expansion before the definition of the inline expanded function is compiled. This is done by globally declaring the function inline before the function is defined, by using the \code{inline} and \code{extensions:maybe-inline} (\pxlref{maybe-inline-declaration}) declarations. In addition to recording the inline expansion of inline functions at the time the function is compiled, \code{compile-file} also puts the inline expansion in the output file. When the output file is loaded, the inline expansion is made available for subsequent compilations; there is no need to compile the definition again to record the inline expansion. If a function is declared inline, but no expansion is recorded, then the compiler will give an efficiency note like: \begin{example} Note: MYFUN is declared inline, but has no expansion. \end{example} When you get this note, check that the \code{inline} declaration and the definition appear before the calls that are to be inline expanded. This note will also be given if the inline expansion for a \code{defun} could not be recorded because the \code{defun} was in a non-null lexical environment. %%\node Semi-Inline Expansion, The Maybe-Inline Declaration, Inline Expansion Recording, Inline Expansion \subsection{Semi-Inline Expansion} \label{semi-inline} Python supports \var{semi-inline} functions. Semi-inline expansion shares a single copy of a function across all the calls in a component by converting the inline expansion into a local function (\pxlref{local-call}.) This takes up less space when there are multiple calls, but also provides less opportunity for context dependent optimization. When there is only one call, the result is identical to normal inline expansion. Semi-inline expansion is done when the \code{space} optimization quality is \code{0}, and the function has been declared \code{extensions:maybe-inline}. This mechanism of inline expansion combined with local call also allows recursive functions to be inline expanded. If a recursive function is declared \code{inline}, calls will actually be compiled semi-inline. Although recursive functions are often so complex that there is little advantage to semi-inline expansion, it can still be useful in the same sort of cases where normal inline expansion is especially advantageous, i.e. functions where the calling context can help a lot. %%\node The Maybe-Inline Declaration, , Semi-Inline Expansion, Inline Expansion \subsection{The Maybe-Inline Declaration} \label{maybe-inline-declaration} \cindex{maybe-inline declaration} The \code{extensions:maybe-inline} declaration is a \cmucl{} extension. It is similar to \code{inline}, but indicates that inline expansion may sometimes be desirable, rather than saying that inline expansion should almost always be done. When used in a global declaration, \code{extensions:maybe-inline} causes the expansion for the named functions to be recorded, but the functions aren't actually inline expanded unless \code{space} is \code{0} or the function is eventually (perhaps locally) declared \code{inline}. Use of the \code{extensions:maybe-inline} declaration followed by the \code{defun} is preferable to the standard idiom of: \begin{lisp} (proclaim '(inline myfun)) (defun myfun () ...) (proclaim '(notinline myfun)) ;;; \i{Any calls to \code{myfun} here are not inline expanded.} (defun somefun () (declare (inline myfun)) ;; ;; \i{Calls to \code{myfun} here are inline expanded.} ...) \end{lisp} The problem with using \code{notinline} in this way is that in \clisp{} it does more than just suppress inline expansion, it also forbids the compiler to use any knowledge of \code{myfun} until a later \code{inline} declaration overrides the \code{notinline}. This prevents compiler warnings about incorrect calls to the function, and also prevents block compilation. The \code{extensions:maybe-inline} declaration is used like this: \begin{lisp} (proclaim '(extensions:maybe-inline myfun)) (defun myfun () ...) ;;; \i{Any calls to \code{myfun} here are not inline expanded.} (defun somefun () (declare (inline myfun)) ;; ;; \i{Calls to \code{myfun} here are inline expanded.} ...) (defun someotherfun () (declare (optimize (space 0))) ;; ;; \i{Calls to \code{myfun} here are expanded semi-inline.} ...) \end{lisp