AQFS version 0.4.0,
released under GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007
(https://www.gnu.org/licenses/gpl-3.0.txt)
* AQFS:
This is a fuse file system where metadata are saved in a SQLite database,
so that they can easily be queried.
Aqfs has:
integrated trashcan,
automatic versioning,
automatic mime type detection,
support for setting user mime type,
support for attachments,
support for tagging,
support for sql queries on metadata: attributes, extended attributes, fields
* License in brief:
I have written the license, basically you have the right
to use, copy, modify, and redistribute the work.
If you redistribute the work with your modifications,
you have to submit these to the original author and they can be included in the work.
If so these can become part of the work under this license.
You cannot use the work or derivative work in commercial products or in weapons.
* Install:
sudo apt install libfuse-dev
sudo apt install libmagic-dev
sudo apt install libsqlite3-dev
make
sudo make install
sudo mkdir /aqfs
sudo mkdir /aqfs/sck0
sudo mkfs.aqfs /aqfs/sck0
sudo aqfs /aqfs/sck0 /mnt or sudo mount -t aqfs /aqfs/sck0 /mnt
this will mount the AQFS file system going background (single threaded, using libfuse)
* Using a real device instead of a directory:
you can also use real devices instead of directories: aqfs command will perform a double mount,
so it's a kind of hack.
example:
aqfs /dev/sdc1 /mnt will:
mount /dev/sdc1 /aqfs/sdc1 (ext4)
mount /aqfs/sdc1 /mnt (aqfs)
* Attachment example:
$ ls -l
drwxr-xr-x 1 baron baron 4,0K lug 13 2018 release
drwxr-xr-x 1 baron baron 4,0K set 28 18:37 history
drwxr-xr-x 1 baron baron 4,0K feb 12 10:30 devel
drwxrwxr-x 1 baron baron 4,0K feb 16 09:08 documentation
-rw-rw-r-- 1 baron baron 1,3K feb 17 18:48 README
$ echo "https://sourceforge.net/projects/aqfs/files/README/download" > README^/link
$ ls -l README^
-rw-rw-r-- 1 baron baron 60 feb 17 19:53 link
$ cat README^/link
https://sourceforge.net/projects/aqfs/files/README/download
* Trashcan example:
$ rm README
$ sudo undelete README
$ ls
devel documentation history README release
* Automatic versioning example:
$ xattr -l README
aqfs.mime_type: text/plain
aqfs.comment:
aqfs.is_versioned: 0
aqfs.attachments.link: /aqfs/sda40CBE/LRU01QU1LO5CZNHJ4DSEMY5K2QKURRXZYED3LS3R3320410001000text/plain
$ xattr -w aqfs.is_versioned 1 README
$ xattr -l README
aqfs.mime_type: text/plain
aqfs.comment:
aqfs.is_versioned: 1
aqfs.attachments.link: /aqfs/sda40CBE/LRU01QU1LO5CZNHJ4DSEMY5K2QKURRXZYED3LS3R3320410001000text/plain
$ echo "byebye" > README
$ cat README
byebye
$ ls -la README^/
total 4
drwxrw-r-- 1 baron baron 7 feb 17 18:48 .
drwxr-xr-x 1 baron baron 4096 giu 5 2018 ..
-rw-rw-r-- 1 baron baron 4079 feb 17 20:21 .2019-02-17T20:21:26.0LBG1TOA
-rw-rw-r-- 1 baron baron 60 feb 17 19:53 link
$ head -3 README\^/.2019-02-17T20\:21\:26.0LBG1TOA
AQFS version 0.4.0,
released under GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007
(https://www.gnu.org/licenses/gpl-3.0.txt)
$ cp README\^/.2019-02-17T20\:21\:26.0LBG1TOA README
$ head -3 README
AQFS version 0.4.0,
released under GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007
(https://www.gnu.org/licenses/gpl-3.0.txt)
* Query example:
$ touch @my_images
$ echo "select @full_path from inodes where is_deleted=0 and @schema[image]" > @my_images^/query.sql
$ query @my_images
$ rm @my_images
* Galleries example:
$ mkdir @images_after_february
$ echo "select @full_path from inodes where is_deleted=0 and @schema[image] and btime > {2019-02-01}" > @images_after_february^/query.sql
$ ls -l @images_after_february
lrwxrwxrwx 1 root root 0 mar 23 2019 '123-2(25140).ICO' -> /MyWorkspace/123-2.ICO
lrwxrwxrwx 1 root root 0 mar 23 2019 '007(27486).ICO' -> /MyWorkspace/007.ICO
$ rmdir @images_after_february
A note on galleries:
- the query is executed by the file system itself, so it blocks other calls (we are single threaded)
while executing the query.
- if the user doesn't use the limit clause, "limit 8192" will be appended
* Tagging example:
$ xattr -w aqfs.schema.main.tags +perl+compression shannon.pl
$ xattr -w aqfs.schema.main.tags +compression madcow
$ query -t compression
/mnt/prova/partial_funpression/madcow
/mnt/prova/partial_funpression/shannon.pl
$ mkdir /mnt/prova/@compression
$ echo "select @full_path from inodes inner join main_details on main_details.fk_inode=inodes.inode where inodes.is_deleted=0 and main_details.value='compression'" > /mnt/prova/@compression\^/query.sql
$ ls -l /mnt/prova/@compression/
total 0
lrwxrwxrwx 1 root root 0 Jan 23 2018 'madcow(243)' -> /mnt/prova/partial_funpression/madcow
lrwxrwxrwx 1 root root 0 Jan 23 2018 'shannon(253).pl' -> /mnt/prova/partial_funpression/shannon.pl
* Schemas:
you can group data with schemas.
In this version there are main, image and audio schemas.
image is defined as follows:
$ cat /mnt/.schemas/image/rules
mime_type starts_with "image/"
$ ls -l /mnt/.schemas/image^
total 0
-rw-r--r-- 1 root root 0 ott 2 22:59 author:string
-rw-r--r-- 1 root root 0 ott 2 22:59 'concepts:string[]'
-rw-r--r-- 1 root root 0 ott 2 22:59 date_taken:datetime
-rw-r--r-- 1 root root 0 ott 2 22:59 height:integer
-rw-r--r-- 1 root root 0 ott 2 22:59 is_photo:integer
-rw-r--r-- 1 root root 0 ott 2 22:59 location:string
-rw-r--r-- 1 root root 0 ott 2 22:59 main_color:string
-rw-r--r-- 1 root root 0 ott 2 22:59 'objects:string[]'
-rw-r--r-- 1 root root 0 ott 2 22:59 orientation:string
-rw-r--r-- 1 root root 383 ott 2 22:59 'people:string[]'
-rw-r--r-- 1 root root 0 ott 2 22:59 width:integer
* Notes:
The documentation is ready in the manual folder. Some hints to read the source code:
main.cpp -> glue.cpp -> Aqfs.cpp (mapping c fuse calls to cpp methods of the object AQFS)
Superblock.cpp (it knows where is the catalog, the volume id, etc...)
Catalog.cpp (it manages the metadata, actually is a SQLite database)
JournaledCatalog.cpp (it makes AQFS faster by using a cache and a journal, dumping metadata onto the database
only when needed, performing bulk inserts)
Storage.cpp (it stores and manages the data)
* Example fstab:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
/usr/bin/aqfs#/dev/sda4 /MyWorkspace fuse rw,nodev,auto 0 0
* Changelog:
2021-01-04 (0.4.0) - no code changes, we have changed the license instead.
We have released the code under the GPL3 License
2020-12-20 (0.3.22) - added aqfs.identify setxattr to set the size and mime_type
- if the db has a zero lengt file, it takes the stat size
it's a trick to mitigate a problem that happens when the file
system is umounted and there are files in open write.
In this case the underlying file system records the right size,
but the catalog reports the old one (that usually is zero)
I have a chkdsk program that can detect and fix such cases
- I am thinking to relicense the software under another license,
It's obvious that with cloud computing nobody wants a software like this
but me, and maybe few other nerds. The license should be more open.
2020-03-25 (0.3.20) - changed the catalog physical layout,
+ description now is called reference
+ encoding now is called category
*** NOTE *** pass the script scripts/query-0.2.37-0.3.20.sql on your catalog
- query has -c flag now to lookup the entry also in the comment
- the logger is changed
- improved the manual
- managed the custom_mime_type: when it's set it is preserved
- fixed bug in link function
- AQFS can remove aqfs.is_versioned, aqfs.custom_mime_type, aqfs.category
- now we can identify the schemas by the special attribute category
2020-01-29 (0.2.37) - improved trashcan: now with no parameters shows the path
- improved the MANUAL to version 0.0.8
- added undelete to undelete subtree from trashcan (must be run as root)
- deltree to quicky delete subtree (must be run as root)
- reviewed trashcan (code review, no visible changes)
- added copyright for year 2020,
- improved query, added the -t switch to lookup files by tag
2019-08-29 (0.2.34) - added SHOWCASE.gif
- fixed a serious bug when deleting trashcan
- setAttribute datetime allows empty string
- added music brain tags to audio schema
- schema match extension lowercase
2019-06-09 (0.2.26) - @extension is case insensitive
- mtime problem should be fixed. Before this we have a problem
if one set mtime before close write and then closed the file
(like tar) we had to choose to whether to not set mtime
on close write (good for tar xvzf) or set it (good to be standard)
Now we say if someone set the mtime before close write keep it
otherwise set it on close.
BEHAVIOUR CHANGE
- the galleries do NOT use distinct queries anymore
BEHAVIOUR CHANGE
- aqfs.query now is called query and is changed a lot. query --help
will show the syntax
- added aqfs.trashcan.cpp that will compile to trashcan command
It's useful to manage the aqfs trashcan.
Before it was more limited and called aqfs.clean
- added audio schema. For newly created file system
it will be set this new schema.
For already created file system launch the script:
create_audio_schema.bash
you must have xattr command to make it work, check the code, it's very simple
- added Toolbox and Console used by aqfs progs to be consistent
2019-05-28 (0.2.16) - added @extension that can be used in queries
- the galleries have changed the link names from inode.name.extension
to name(inode).extension.
The browsing of galleries is a little faster now.
- The link syscall now clones even the automatic versions.
- Added the first version of the manual
2019-05-17 (0.2.10) - added use_ino, without this files could be corrupted.
THIS WAS A SERIOUS BUG
- improvements in performance while deleting the trashcan
- stronger code in Storage
- lowered the number of blocking calls
2019-03-07 (0.2.1) - aqfs.clean can clean the trashcan before a certain date
- now we set the fuse fsname as aqfs
- make install will create /aqfs and /sbin/mount.aqfs
so that we can use mount -t aqfs ...
2019-02-16 (0.2.0) - initial public version
