#5673 transpose of a zero-column matrix loses dimensions and breaks double transpose

[Oleksandr Kulkov]

Transposing a matrix with rows but no columns currently loses its dimensions:

M: matrix([],[],[])$
transpose(M); /* matrix() */
transpose(transpose(M)); /* matrix(), rather than M */

Could transpose retain its noun form here, i.e. 'transpose(matrix([],[],[])), instead of collapsing to matrix()? This would represent the 0-by-3 transpose without inventing placeholder entries or changing the ordinary matrix layout.

The attached small patch to $transpose does that. Its existing handling of a transpose argument then restores the original matrix on the second transpose. Re-evaluation with ev(...,nouns) also preserves the noun form. The 0-by-0 case and ordinary nonempty transposes retain their results.

This is deliberately a first step toward preserving dimensions, not complete support for zero-row matrices. With the patch, multiplication involving this noun stays symbolic; matrixp is false, and matrix_size and rank still reject it. Also, zeromatrix(0,n) still loses n. Those operations need further work if this representation is accepted. Simply changing matrixp would be unsafe because many consumers directly access the row-list representation.

Related: #4685 concerns simplification of nested symbolic transpose expressions. This report concerns loss of dimensions when evaluating transpose on an explicit empty matrix.

The patch is against c24012415b91192b9c7f9303ca28f4391a890f01. A fresh build of that source with ECL 26.5.5 passes the extended rtest2: 332/332 ordinary cases, with two registered expected failures (334 total). Seven of the ten new focused cases fail before the change; all ten pass after it on the fresh build, installed Maxima 5.50.0, and Maxima 5.44.0. The broader current-build core/share run is still in progress at submission, so no complete full-suite pass is claimed.

AI assistance disclosure: the investigation, proposed patch, and tests were prepared with OpenAI Codex (GPT-6). Submitted as a proposed fix for maintainer review.

[Oleksandr Kulkov]

I have prepared a complete replacement for the earlier transpose-only patch. native-empty-matrices-complete.patch includes and supersedes native-empty-transpose.patch; please apply the new attachment on its own.

The earlier patch is insufficient: in particular, it regresses nullspace(matrix([],[],[])) through assumptions in addcol. The replacement fixes that and includes the proposed core, linearalgebra/eigen and LAPACK integration, documentation, and regression tests.

Representation and the matrix interface

The representation is still matrix() for 0x0, matrix([],[],[]) for 3x0, and 'transpose(matrix([],[],[])) for 0x3. Ordinary matrices retain their existing storage.

The patch introduces internal operations for row/column counts, row/column traversal, square-matrix validation, and construction from rows with an explicit column count. A constructor needs that last argument because an empty row list cannot determine the width. Affected algorithms use these operations in place of inspecting the expression layout directly; Maxima-level routines use matrix_size, row, col, or indexed access.

I think consistently using this interface is the better direction. It separates the algorithms from representation choices and allows more empty cases to follow from the general computation. The patch starts that conversion where needed for this support; it does not refactor every matrix consumer in Maxima.

For example, multiplication reuses the existing multiplication loop: an m-by-0 matrix times a 0-by-n matrix gives an m-by-n zero matrix because each entry is an empty sum. The empty determinant identity is implemented in the shared determinant kernels, so determinant/characteristic-polynomial/permanent front ends inherit it. spectral_rep, matrixexp and matrixfun require no new empty-case guards. Some routines still need mathematical base cases, such as Gram-Schmidt's initially empty family and full identity bases for empty SVDs.

I also tried a column-count field only on zero-row native matrices and a compact zeromatrix(0,n) expression. Neither produced a substantially smaller working patch. The metadata version needed additional equality, evaluation and printing hooks and still lost dimensions through expression reconstruction. The constructor expression exposed its dimensions to generic numeric transformations. The transpose form encodes shape with empty rows and has no scalar entries to transform.

Existing bugs and behaviour changes

These changes also address problems reproducible before either transpose patch. They should be considered explicitly in reviewing the larger scope:

• determinant, newdet, permanent, and charpoly reject matrix(); ncharpoly(matrix(),x) returns x instead of 1. Empty inversion, LU and Cholesky, and columnspace(matrix([],[],[])), are also rejected.
• Empty norm reductions can return minf or error instead of zero. gramschmidt([]) indexes past the end, and empty eigenvalue calculation fails.
• innerproduct([],[]) returns matrix([]) instead of scalar zero. coefmatrix([1],[]) and augcoefmatrix([1],[]) produce an extra column; circulant([]) produces a spurious row.
• LAPACK QR/SVD/eigen wrappers fail on existing matrix() or matrix([],[],[]) inputs. Leading dimensions must remain at least one, and workspace queries need storage even when there are no entries.
• Nonempty dgemm: with transpose flags and beta*C, the conversion uses the wrong dimensions for C. One attached rectangular example returns matrix([55,44],[104,96]) before the fix, instead of matrix([55,84],[104,136]).
• Nonempty zlange: real matrix input is passed as a real array to a routine expecting a complex array, causing a type error; it is now converted correctly.
• dgesv can accept a nonsquare A and return an answer. Square-input and RHS-shape checks are added; eigenvalue wrappers also validate square input before interpreting storage. identfor and ncharpoly now reject nonsquare matrices.
• locate_matrix_entry checks its fourth argument twice and does not validate that its fifth argument is an integer. Out-of-range row/col requests can return empty matrices, and ematrix(0,-1,x,1,1) accepts a negative width. These validation cases are corrected.

The attached empty-matrix-checks.zip includes the exact before/after inputs and results, including additional constructor, domain-validation and bounds changes. Extensions such as accepting zero-sized genmatrix, hilbert_matrix and krylov_matrix inputs are identified as extensions of the accepted domain rather than silent implementation details.

Scope and compatibility

The proposed support covers constructors, transpose, arithmetic and mapping, row/column operations, rank and spaces, norms, Kronecker products, LU/Cholesky/inverses, eigenvalue and matrix functions, and the nine public LAPACK matrix interfaces.

args, op, and length continue to describe the stored expression. Code assuming that args(M) always returns matrix rows must use matrix operations or handle the transpose form. Existing scalarmatrixp behaviour is retained; shape-sensitive tests bind it to false. The new representation does not make arbitrary third-party code that reads matrix internals automatically compatible. The amatrix fix remains separate in #5672.

Validation

Against c24012415b91192b9c7f9303ca28f4391a890f01, a fresh build with ECL 26.5.5 completes all 163/163 standard core/share test files, with 21,785 cases: 21,653 ordinary cases passed, with 132 registered expected failures and no unexpected failures. The cold run reached its 1,200-second limit after 146 completed files, having spent most of the time compiling numerical dependencies; the remaining 17 files completed in a second run on the same build. The additional attached runner passes 766/766 checks; its twelve targeted pre-existing-bug/control checks give eleven failures and one unchanged control on the original implementation. There are also 46/46 fresh-process before/after examples. The patch applies cleanly to the base commit; the attachment contains the per-file SHA-256 comparison for all 35 changed files. Patch SHA-256: f034b803176de679e1cde2d3be7cb037a59692d831ea6f7a1f505b8f04026ab5.

This complete patch has not been verified on other Lisp implementations or release branches. The earlier report's checks on Maxima 5.44/5.50 concerned the small transpose patch only.

The change inventory is 35 files, +686/-176 lines: implementation 22 files (+328/-173), tests nine files (+310/-3), and documentation four files (+48/-0). The extra validation inventory contains 109 function names: 101 distinct names from the matrix, linearalgebra and LAPACK manual chapters, plus eight additional public routines. 106 were exercised; the three excluded names are the vector-calculus routines scalefactors, vectorpotential, and vectorsimp. This is a defined coverage boundary, not a claim about all third-party packages.

If separate tickets would make review easier, the code can be split into core support, linearalgebra/eigen changes, and LAPACK fixes, with the independent pre-existing bugs separated further as appropriate. All proposed code and tests for the scope above are included in the attachments.

AI assistance disclosure: the investigation, proposed patch, comparison implementations, and tests were prepared with OpenAI Codex (GPT-6).