Papis Files

Dependency Changes

• Switched from pyparsing to Lark for query parsing.
• Removed the isbnlib dependency. The library itself has not been maintained in a while. The new ISBN functionality is built-in with no additional dependencies. The supported services are (no plugins are supported): goob, openl, and wiki.

Features

Major: SQLite database backend (#1007)

Papis added a new database backend based on the built-in sqlite3 database as an alternative to the papis backend (based on pickle) and the whoosh backend.

This new backend works similarly to the whoosh backend. To use it, you need to

1. Set database-backend = sqlite in your configuration file (globally or per-library).
2. Tinker with sqlite to add more fields you'd like to query.

From preliminary testing, this backend seems to be quite fast, both for searching and caching. A quick benchmark on a library with about 2000 documents (and a query that returns about 15), the default papis backend gives

> hyperfine --warmup 3 -- 'papis list --all QUERY'
Benchmark 1: papis list --all QUERY
  Time (mean ± σ):     322.8 ms ±  20.8 ms    [User: 290.3 ms, System: 31.0 ms]
  Range (min … max):   308.5 ms … 379.1 ms    10 runs

and the sqlite backend gives

Benchmark 1: papis list --all QUERY
  Time (mean ± σ):      97.4 ms ±  11.4 ms    [User: 81.1 ms, System: 15.3 ms]
  Range (min … max):    79.7 ms … 115.2 ms    27 runs

See the documentation for additional information and usage examples. In particular, regex support for this backend is rather limited.

Major: Improve query syntax (#1165)

Historically, the query syntax when using database-backend = papis has been limited to using strings and key-value pairs. The new version gets it closer to that of the other backends, by adding support for AND, OR and NOT boolean operators, as well as grouping using parentheses. This allows constructing complex queries for searching your libraries.

It generally looks as expected:

:::bash
# find documents with either one of the two authors
papis open 'author:einstein OR author:bohr'
# find papers referencing physics, but not Einstein
papis open 'physics NOT author:einstein'
# find Einstein's paper from some years
papis open 'author:einstein AND (year:1905 OR year:1915)'
# use regex to match the beginning of a title
papis open 'title:^Quantum'

Note that this new query syntax still uses the match-format configuration setting for free-form terms in the query (i.e. ones not using a key:value format).

Major: Improve mv command (#1104) and remove rename (#1196)

The mv command has been completely rewritten with many new features (see the docs). It now works similarly to the Unix mv command for moving document folders:

• if the target folder already exists, the document's folder is moved into it (e.g. papis mv --to existing-folder query places the document at existing-folder/<doc-folder>);
• if the target does not exist, the document is renamed to that name.

When no target folder is given, it defaults to the add-folder-name pattern, which means papis mv query now does everything that papis rename used to do. The new --to flag also accepts formatting patterns (e.g. papis mv --to "{doc[year]}" query). The command can now move documents between libraries by specifying a target library.

The new command performs pre-flight checks on all planned moves before touching any files. Issues such as duplicate target paths, nesting a document inside another document's folder, or missing source folders are warned about upfront. The user can abort, skip problematic moves, or fix collisions interactively. A new --batch flag allows skipping all interactive prompts.

Minor: papis update can rename files (1108)

The papis update command has been reworked quite extensively and has some new interesting functionality (see the docs). First, updating document files will also rename these files on disk. For example, running

:::bash
papis update --set files 0:my-new-file.pdf QUERY

will change the first file in the document to my-new-file.pdf and rename it on disk from its previous name. This also applies to notes in the document.

Second, we can also reset certain values to their defaults (as defined in the configuration file). For example, to update some document references according to ref-format, use

:::bash
papis update --all --reset ref QUERY

for a bulk operation. This can also be applied to author, files, and notes.

Other noteworthy features

• Vendored parts from isbnlib (#1102).
• Add a CSV exporter (#1172).
• Match directories to libraries if possible (#1179).
• Remove undocumented dirs option (only allow one dir per library) (#1182).
• Do not add "empty" values into the document (#1192).
• Move papis doctor checks to papis.doctor (#1200).
• Derive author_list from a flat author when creating a new document (#1202).
• Improve error reporting from git failures (#1204).
• Add a downloader for NBER working papers (#1218).

Bug Fixes

• Better matching initials in string-cleaner doctor check (#1134).
• The YAML exporter now starts each document with an explicit --- marker, so that papis export --format yaml --append produces a valid multi-document file (#1038).
• Detect article-number in more journals (#1152).
• Fix database caching with multiple libraries (#1195).
• Fix crash in papis add when a given file path contains spaces (the DOI importer no longer tries to resolve local paths on doi.org) (#1201).
• Skip local paths when matching DOIs (#1203).
• Fix extra separators when joining single-name authors (#1229).