ToaruOS 2.3

This Release's Wallpaper

Two years ago, I moved from Tokyo to the mountains of western Chichibu. As with all wallpapers in ToaruOS for the last several years, this release's wallpaper is my own photo, taken out my window. It depicts a warbling white-eye.

This Release's Codename

ToaruOS release codenames come from the soundtrack to its namesake franchise. This release series is titled "Grow Slowly", after the first ending credits theme to A Certain Scientific Railgun S, performed by Yuka Iguchi.

What's new in ToaruOS 2.3?

Terminal Tabs

The terminal emulator now supports multiple tabs. Create tabs with Ctrl-Shift-T; swap with Alt-1 through Alt-9.

Manual Viewer

A full man stack has been added, combining a roff formatter and a more complete more, plus a bunch of manual pages.

Kernel Improvements

Tons of a new system calls like pread/pwrite, sigsuspend, sigqueue, fcntl; support for FD_CLOEXEC and FD_CLOFORK; TTY line control and hangups; and more.

And more!

• The latest Bim and Kuroko versions.
• Tons of new utilities and new options in existing utilities.

Changelog

Kernel

• On x86-64, rudimentary support for kernel load address randomization has been added in supported boot environments.
• On x86-64, the syscall ABI has been updated to use syscall/sysret.
• On aarch64, support for current QEMU virt machines has been implemented.
• On x86-64, the lapic timer is now the primary preemption source on the boot processor. The PIT is only used as a calibration source.
• The TTY layer now supports line control management, SIGTTOU/SIGTTIN, controlling sessions, VMIN, flushing on attribute changes, and sends SIGHUP to all session members when the master end closes.
• pread(), pwrite(), lchown(), getrusage() system calls have been added, with corresponding libc bindings.
• ptrace() now supports updating system call numbers, allowing debuggers and other tools to manipulate tracees.
• times() now correctly reports user time separately from system time.
• Various interfaces such as stat and readdir have been widened to 64 bits (with legacy ABI support for existing packages).
• The kernel command line parser has been rewritten to support quoted values.
• The network stack now supports multiple simultaneous ICMP sockets.
• sigsuspend() is implemented properly as a system call.
• Various missing permission checks have been added, thanks to issue reports from contributors.
• A real rename interface has been added to the VFS, with support in tmpfs.
• The access() system call now correctly reports whether the real user/group of a process has requested access permissions. An eaccess() system call was also added, alongside a libc binding, to support checking permissions for the effective user/group, which some GNU utilities like to have available.
• The reboot() system call now accepts a parameter. This parameter is currently unused, but may specify more detailed restart behavior in the future. A reboot() libc binding has also been added.
• The ABI for the sleep() system call has been adjusted to report remaining time correctly.
• /dev/random (and /dev/urandom, which remains an alias) is now seeded by the system time on startup. Additionally, an issue discovered by a contributor in which /dev/random did not return byte values of 255 has been resolved. The collection of random data has also been adjusted to use all of the bytes available from rand().
• The DSP audio interface now ensures that buffers are flushed when a client exits. This reduces the occurance of clicks and pops from clients that send data to the DSP and immediately exit.
• The mixer interface now reports sensible error codes.
• The getcwd() system call ABI has been adjusted to return error codes or bytes written (including a trailing NUL), rather than conforming to its userspace specification.
• Several system call ABIs have been adjusted to require buffer lengths instead of operating on strings of unknown lengths.
• The buffer size for pipes has been increased to 65536 bytes. This can be configured at boot with the pipesize= option.
• The kernel now terminates threads when a thread group leader exits.
• The VGA text mode device interface has been redesigned to allow for emulation of VGA text mode on a framebuffer.
• O_CLOEXEC and O_CLOFORK have been implemented (previously, the kernel would always keep file descriptors 0, 1, and 2, and always close other file descriptors on exec).
• pipe2 and dup3 system calls have been added.

libc

• The DNS parser in gethostbyname() has been improved.
• Bounds checks have been added to strftime.
• Added getdelim, and getline, and many utilities have been adjusted to use them.
• Added scandir, telldir, rewinddir, and seekdir.
• Added ftruncate, fchmod, and fchown.
• Added popen, pclose.
• Previously stubbed functions rename, truncate, fcntl, and getppid now have real implementations.
• ferror now indicates error states (mostly).
• abort now behaves as specified, making it more useful in dbg; assert() now calls abort(), as well.
• New POSIX functions sig2str and str2sig have been added (now used in kill, killall, strace, and dbg).
• usleep() now handles error conditions from its underlying sleep() system call correctly.
• The asprintf family correctly returns the number of bytes written (excluding the trailing NUL). Previously, it would erroneously return 0.
• The execvp family no longer attempts to run preflight checks when searching $PATH; it instead attempts to perform the underlying system call on each potential candidate; if no candidate succeeded and EACCES was seen while attempting a candidate, that error will persist.
• perror now correctly skips an argument which is NULL or an empty string.
• getcwd has been adjusted to comply with GNU extensions.
• %o has been added to the printf family.
• Some issues in realpath() related to a current directory or target path of "/" have been resolved.
• getopt no longer outright rejects any option characters as invalid - any single-byte value is allowed.
• getopt_long now supports unanmbiguous prefix matches and detects unexpected arguments to no_argument long options.
• getopt and getopt_long permutate their arguments, unless the first character in optstring is +, such that all non-option arguments are eventually after all option arguments, following the GNU convention.

Terminal

• terminal and terminal-vga now supports multiple tabs.
• Create new tabs with Ctrl-Shift-T, or through the context or File menu.
• Switch tabs with Alt-1 through Alt-9.
• Close tabs by terminating their sessions (or using the "Close tab" context menu entry in Terminal).
• Tabs have independent font configuration (scaling, bitmap, bold emulation); each new tab will inherit the configuration of the tab that was active when it was opened.
• The first tab starts with the command specified in the arguments to terminal; subsequent tabs start with the user's shell.
  • Currently, new tabs start in the same working directory as terminal; support may be added in the future to start new tabs in the same working directory as the tab that was open when they were created.
  • With the -l (--login) option, all new tabs run login-loop.
• ESC c is now supported for a full reset, and reset emits it.
• CSI 2 J now correctly saves lines up to the cursor position to scrollback and does not move the cursor.
• The TTY layer has support for ECHOCTL.
• Controlling sessions are now set by the terminal.
• An issue has been resolved wherein keyboard events were sent to the terminal window when it was not focused.
• The terminal now extends the background colors of cells on the bottom and right edges to fill the remainder of the screen.
• When using the bitmap font, the terminal can emulate bold text by double-striking glyphs. This feature is optional and can be enabled from the "View" menu. While this can lead to a lack of glyph separation in some characters, the effect remains highly readable and can aid in tools like man that use bold text to differentiate certain content.
• The terminal no longer sends fatal signals to its children on exit, as the kernel TTY layer handles this task now.
• Behavior in alternate screen mode has been changed as follows:
• Scrollback is not affected or accessible.
• When ^[[1007h is enabled, the mouse wheel sends arrow up and arrow down key sequences without the need to enable mouse reporting.
• When text selection starts by double-clicking a word, the whole word is maintained as part of the selection.
• A new "Terminal" menu has been added.
• The "Terminal state" submenu shows the state of various modes (alt screen, mouse, paste bracketing, etc.); the menu entries can also be used to manually change these states.
• The "Send signal" submenu allows signals to be sent to the foreground process of the terminal.
• These submenus are also available from the context menu.
• "Reset", "Clear", and "Clear scrollback" are also provided in the "Terminal" menu.
• A new option to copy text with escape sequences, which includes escape sequences to recreate the colors and display effects of selected terminal cells, is available from the "Edit" menu.
• terminal-vga can now operate against an emulated VGA text mode device when booted with start=--vga emulvga and an appropriate vid= configuration. By default, this emulated text mode will be large enough to fill the screen; a smaller size can be specified with an optional argument to emulvga, eg., emulvga=80x25.
• terminal-vga also now supports scrollback.
• terminal now reads a JSON configuration file from ~/.terminal.json on startup.
• terminal can now be configured to make an audible beep on the BEL character.

Applications

• The latest upstream versions of Kuroko, Bim, and the nyancat utility are included.
• Kuroko is upgraded from v1.4.0 to v1.5.0rc2. Kuroko has received significant development in this period, and the changes are too numerous to list in these release notes.
• Bim is updated to 3.2, with many bug fixes, improved syntax highlighters, greater support for Kuroko scripting, and a brand new command line tab completion system.
• For contributors and users building from git sources, Bim is now included as a submodule.
• strace has additional support for more system calls, including all of the new ones in this release, as well as new syscall groups to use with -e trace.
• New standard utilities have been added: rmdir, uniq, cmp, zcat, realpath, id, nohup, cksum.
• New modes have been added in grep, and fgrep is now implemented as an alternate mode for grep.
• -F: Operate as fgrep with patterns being treated as plain strings.
• -l: List matching files instead of printing matching content.
• -n: Prefix matches with line numbers.
• -s: Suppress errors messages.
• -x: Match whole lines
• -A, -B, -C: Print context surrounding matches.
• --color is now opt-in, like it is on other greps.
• Many utilities have been improved to be closer to POSIX.
• ps, top, pstree, killall, and pidof have all been rewritten around a shared libtoaru_procfs, and each tool has received various improvements:
• pstree acquired many options from its namesake, including -C options to color entries (and ours has many special options like -C cpu to color processes by CPU usage), as well as -a to display full command lines, -p and -g to display PIDs and process group IDs, -h to hilight the ancestry of the current process, -H to highlight the ancestry of a requested PID, -T to hide threads, and the ability to display the tree rooted in a particular PID rather than init.
• top received some coloring improvements and now supports the display of threads (after years of having a disabled "TID" column!), and also has a log mode so it can be redirected to a file.
• kill received POSIX options like -l.
• pidof gets several options from its namesake, including -o to omit a pid from results (use multiple -o options to omit multiple pids; use -o %PPID to omit pidof's parent process), and support for multiple process name arguments.
• dirname and basename support additional options.
• more has received several changes, including scrollback, alternate screen support, and searching, to make it usable for displaying manual pages.
• A rudimentary roff has been added, along with a man and some basic manual pages. More manual pages will arrive in future point releases and full coverage of all library functions is planned.
• esh received some improvements:
• An issue wherein esh would fail to exit child processes if they encountered errors when binding redirects has been fixed.
• Tab completions are now shown in columns.
• An option is available to enable coloring of some completions (currently, this only makes directories in path completions blue).
• An option is available to enable hints for some completions (labels for built-in commands, and the sections of manual pages). This options is enabled in the local user's .eshrc file.
• Syntax highlighting can now be disabled, if desired, with set -o no-syntax-color.
• Rudimentary support for backtick sequences has been added.
• The beep utility now has a PCM backend for use on platforms without PC speaker support (and is now used by some other applications).
• The which utility now correctly checks that the calling user has execute permissions when searching $PATH, which more accurately represents the new behavior in execvp.
• The color-picker utility now has a --print option to print the selected color to standard output on exit.
• The qemu-fwcfg utility now works on aarch64 platforms.
• ls received many new features:
• ls now formats columns in the same direction as other OSes; the old behavior can be restored with the -x option.
• Support was added for considering wide characters when formatting columns.
• Long forms were added for some existing options.
• The -A (--almost-all) option was added.
• The -F and -p options were added to use textual indicators for various special file types.
• The -f option was added to retain the original order of directory entries from the filesystem.
• The -i (--inode) option was added to display inode numbers next to file names.
• The -s (--size) option was added to display block counts next to file names.
• The -x and -C options were added to control column display.
• tar now accepts old V7-format archives, handles traditional argument formatting, and correctly handles filtering by arguments.
• gunzip performs its own output buffering, working around an issue where our libc only supports line buffering.
• Test apps are no longer part of the default build (but their sources remain available in the live environment); some test utilities have been removed entirely, and some have been renamed.
• stty now accepts an -f option and operates on standard input by default.
• cal now supports more options, including displaying specific months, multiple months, and whole years.
• drawlines can be resized and now has decorations.
• polygons has been ported to the libtoaru_text contour library and now has decorations.
• ping supports a -c option.
• crc32 now supports a suite of options and is used to implement the cksum utility.
• fetch fetches the content of error pages by default, or fails early and returns a non-zero exit code with -f; the progress bar has been redesigned.

Libraries

• libtoaru_procfs: A new library has been added to centralize code from several applications that iterated over process entries in /proc.
• libtoaru_json: Serialization support has been added, alongside new functions to simplify generating JSON.
• libtoaru_auth: Improvements have been made to the management of group identity.
• libtoaru_hashmap: New iteration utilities have been added.
• libtoaru_text: New functions allow the edges of a contour to be queried, and for a point to be checked for overlap with a shape.
• libtoaru_graphics: Image loading has been redesigned to use a magic identification approach rather than relying on file extensions.
• libtoaru_menu:
• A new "Toggle" menu entry type has been added, and menu entries can now be disabled.
• Menu bars can now have "tab" entries.
• libtoaru_rline: New bindings have been added and ^C cancels reverse-search.

UI

• The panel now correctly reports the focused window when it has menus opened.
• A new check-image utility is added, which can determine if the graphics library is capable of loading an image and report its size, suitable for use from the shell.
• A yutani-screenshot utility has been added for platforms on which a Print Screen key may not be accessible.
• A new submenu has been added to the decoration context menu that allows windows to be tiled in various positions.
• This allows window tiling to be accessed when the keyboard shortcuts in the compositor are unavailable.
• Double clicking window decorations now maximizes/unmaximizes windows.

What is ToaruOS?

ToaruOS is a hobbyist, educational operating system for x86-64 and AArch64 systems, focused primarily on use in virtual machines. It provides a Unix-like environment, complete with a graphical desktop interface, shared libraries, feature-rich terminal emulator, and support for running, GCC, Quake, and several other ports. The core of ToaruOS, provided by the CD images in this release, is built completely from scratch. The bootloader, kernel, drivers, C standard library, and userspace applications are all original software created by the authors, as are the graphical assets.

Who wrote ToaruOS?

ToaruOS is primarily written by a single maintainer, with several contributions from others. A complete list of contributors is available from AUTHORS.

Running ToaruOS

It is recommended that you run ToaruOS in a virtual machine / emulator, for maximum compatibility. ToaruOS's driver support is limited, and running on real "bare metal", while possible, does not provide the most complete experience of the OS's capabilities except on very particular hardware. ToaruOS is regularly tested in VirtualBox, QEMU, and VMWare Player, and can be successfully booted (with poor performance) in Bochs. ToaruOS is intended to run from a live CD, though it is possible to install to a hard disk. Additional details on running ToaruOS in different virtual machines is available from the README.

Release Files

• image.iso is the standard x86-64 PC build of ToaruOS, built by the Github Actions CI workflow. It uses ToaruOS's native bootloaders and should work in most virtual machines using BIOS.
• kernel8.img is a payload suitable for use on a Raspberry Pi 400 (and possibly 4 and 5). Note: ToaruOS lacks drivers for much of the Pi's hardware, and while these images generally boot to a graphical desktop, it is not usable.
• aarch64-virt.zip contains three files: misaka-kernel, ramdisk.igz, and bootstub; these can be used with QEMU's virt machine type by using bootstub as a kernel and supplying the ramdisk and actual kernel through fw_cfg options: -fw_cfg name=opt/org.toaruos.initrd,file=ramdisk.igz -fw_cfg name=opt/org.toaruos.kernel,file=misaka-kernel; command line options may be passed normally through -append.

x86-64

Most development occurs in QEMU, so QEMU is the recommended environment.

Linux with KVM

:::sh
qemu-system-x86_64 -M q35 -m 3G -smp 4 -enable-kvm \
    -no-reboot -serial mon:stdio -device AC97 -name "ToaruOS x86_64" -cdrom image.iso

Booting ToaruOS with Grub

Extract the files kernel and ramdisk.igz from the CD image using a tool such as File Roller, place them in a directory accessible to Grub, and boot the OS using the following commands:

multiboot2 /path/to/kernel vid=auto start=live-session root="/dev/ram0"
module2 /path/to/ramdisk.igz
set gfxpayload=keep

Adjust these commands as needed to set a desired video mode or use VGA text mode where available.

AArch64 Builds

Running the AArch64 builds under QEMU requires a bit more configuration.

On macOS with HVF

:::sh
qemu-system-aarch64 -M virt -m 4G -smp 4 -cpu host -accel hvf -no-reboot \
    -serial mon:stdio -device ramfb -device virtio-tablet-pci \
    -device virtio-keyboard-pci   -device AC97 -d guest_errors \
    -net user -netdev hubport,id=u1,hubid=0, -device e1000e,netdev=u1 \
    -name "ToaruOS aarch64" -kernel bootstub  \
    -append "root=/dev/ram0 migrate start=live-session ramfb vid=preset" \
    -fw_cfg name=opt/org.toaruos.initrd,file=ramdisk.igz \
    -fw_cfg name=opt/org.toaruos.kernel,file=misaka-kernel

On Linux (RPi, etc.) with KVM

:::sh
qemu-system-aarch64 -M virt -m 4G -smp 4 -cpu host -accel kvm -no-reboot \
    -serial mon:stdio -device ramfb -device virtio-tablet-pci \
    -device virtio-keyboard-pci   -device AC97 -d guest_errors \
    -net user -netdev hubport,id=u1,hubid=0, -device e1000e,netdev=u1 \
    -name "ToaruOS aarch64" -kernel bootstub  \
    -append "root=/dev/ram0 migrate start=live-session ramfb vid=preset" \
    -fw_cfg name=opt/org.toaruos.initrd,file=ramdisk.igz \
    -fw_cfg name=opt/org.toaruos.kernel,file=misaka-kernel

Without Hardware Virtualization

:::sh
qemu-system-aarch64 -M virt -m 4G -smp 4 -cpu cortex-a72 -no-reboot \
    -serial mon:stdio -device ramfb -device virtio-tablet-pci \
    -device virtio-keyboard-pci   -device AC97 -d guest_errors \
    -net user -netdev hubport,id=u1,hubid=0, -device e1000e,netdev=u1 \
    -name "ToaruOS aarch64" -kernel bootstub  \
    -append "root=/dev/ram0 migrate start=live-session ramfb vid=preset" \
    -fw_cfg name=opt/org.toaruos.initrd,file=ramdisk.igz \
    -fw_cfg name=opt/org.toaruos.kernel,file=misaka-kernel

Future

The next major release of ToaruOS is expected to be labeled 3.0. The following projects are currently in development and targeted for this next release:

• A completely new virtual file system, modeled after the one in Linux, which aims to resolve several issues with how ToaruOS's existing virtual file system fails to provide the abstractions expected of a POSIX-compliant operating system. Some features expected to come alongside this new VFS include:
• A full suite of *at() system calls.
• An initial implementation of an mmap system call that can map files.
• Possible support for advisory file locks.
• A new package manager, which will restore some functionality from the old "legacy" version of msk. Packages will once again be single files with embedded manifest data.
• A new Help Browser utility, which will provide a graphical interface for viewing manual pages.

Known Issues

The usual boilerplate: There are many known security issues with ToaruOS. You should not use ToaruOS in a production environment - it is a hobby project, not a production operating system. If you find security issues in ToaruOS and would like to responsibly report them, please file a regular issue report here on GitHub.

Known defects in this release

• Several things related to signals have not reached conformance with POSIX specifications:
• Handling of SIGCHLD should not be relied upon to provide accurate counting of terminated children.
• SIGCHLD is not sent for trace events (nor is SIGTRAP generally sent to traced processes in most cases).
• WCONTINUED is not supported (but SIGCHLD is sent to parents of continued processes).
• "Stop" and "continue" signals do not clear pending instances of their inverses.
• Some kernel-issued signals may still report SI_USER when used with SA_SIGINFO. A future point release may fix these cases.
• Alternate signal stacks are still not supported.
• Several things related to file descriptors and C streams remain incomplete:
• O_APPEND continues to not be a persistent flag on file descriptions; Misaka does not properly differentiate the "file descriptor", "file description", and "file" abstractions in POSIX - improvements to this situation should arrive with the new VFS in 3.0.
• stdio streams still have many bugs related to seeking and offsets and do not use any locking; more extensive work will be done on the stdio implementation for the next major release.
• Advisory locks are not available.
• more does not handle screen resizing.