prex-devel

[Prex-devel] Compressed ROM file system

[David G. <dg...@co...>]

So I appear to be terminally stuck with my vfork() issue --- I think 
there's at least one thing wrong that I'm doing plus at least one thing 
wrong that Prex is doing. The next stage is to try and duplicate the 
issue on GBA and right now I don't have the energy to build the wacky 
gcc variant it needs.

However: in the process of working on the Cybiko, I ran out of ROM space 
(it's only got 256kB). To work round this I implemented a compressed ROM 
file system using zlib's puff lightweight inflator. This compresses each 
512 byte block individually so the compression ratio's not brilliant, 
but it's still achieving about 2:1 and supports true random access. puff 
is about 2kB of code and is under a pretty much identical license to Prex.

So is this useful to anybody? Shall I try and clean up the patch, and if 
so, what should I do with it (the Prex mailing list doesn't seem to 
accept patches)?

One complication is that it needs a custom program to create the 
compressed file system image, and right now the Prex build system 
doesn't support these --- I have a hack that builds and runs it as 
required. This needs to be cleaned up.

-- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ life←{ ↑1 ⍵∨.^3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵ }
│ --- Conway's Game Of Life, in one line of APL

Re: [Prex-devel] Compressed ROM file system

[Andrew D. <and...@gm...>]

On Tuesday, June 15, 2010, David Given <dg...@co...> wrote:
> So I appear to be terminally stuck with my vfork() issue --- I think
> there's at least one thing wrong that I'm doing plus at least one thing
> wrong that Prex is doing. The next stage is to try and duplicate the
> issue on GBA and right now I don't have the energy to build the wacky
> gcc variant it needs.
>

Hi david,
Has much changed in this area of Prex since 0.8.1? Maybe there its an
issue in 0.9?

> However: in the process of working on the Cybiko, I ran out of ROM space
> (it's only got 256kB). To work round this I implemented a compressed ROM
> file system using zlib's puff lightweight inflator. This compresses each
> 512 byte block individually so the compression ratio's not brilliant,
> but it's still achieving about 2:1 and supports true random access. puff
> is about 2kB of code and is under a pretty much identical license to Prex.
>
> So is this useful to anybody? Shall I try and clean up the patch, and if
> so, what should I do with it (the Prex mailing list doesn't seem to
> accept patches)?
>

This sounds interesting. The mailing list only accepts patches as
inline text. I can make a 0.9 branch available on github if that
helps?

Andrew

> One complication is that it needs a custom program to create the
> compressed file system image, and right now the Prex build system
> doesn't support these --- I have a hack that builds and runs it as
> required. This needs to be cleaned up.
>
> --
> ┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
> │
> │ life←{ ↑1 ⍵∨.^3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵ }
> │ --- Conway's Game Of Life, in one line of APL
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultima
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> Prex-devel mailing list
> Pre...@li...
> https://lists.sourceforge.net/lists/listinfo/prex-devel
>

Re: [Prex-devel] Compressed ROM file system

[David G. <dg...@co...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14/06/10 20:17, Andrew Dennison wrote:
[...]
> Has much changed in this area of Prex since 0.8.1? Maybe there its an
> issue in 0.9?

I don't know --- I never got this far with 0.8.1. I'm reasonably sure
that there's some stack corruption in the no-MMU vfork implementation,
but until (if and when) I can demonstrate it in a vanilla Prex without
my hacks I don't want to commit myself.

The only other non-MMU port is the GBA one, right? It doesn't build with
a modern arm-elf, it does build with a modern arm-eabi but then won't
boot, so I'm now building the ancient gcc 3.4.3 that' actually mentioned
in the instructions.

[...]
> This sounds interesting. The mailing list only accepts patches as
> inline text. I can make a 0.9 branch available on github if that
> helps?

Provided it's not too much work, that'd be really helpful.

The code is straightforward, but the build system tinkering may not be
- --- the Prex build process generally doesn't support custom build stages
when generating the OS image. I'll see how clean I can make it.

- -- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ "Money is a sign of poverty." --- Iain Banks
│
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFMFp7af9E0noFvlzgRAhwNAKCkS/ynrwWEdM160WPB0G2smUbXTQCgsaAf
nPEWGxvM0gqGwO9xvC+mf1o=
=tMvr
-----END PGP SIGNATURE-----

Re: [Prex-devel] Compressed ROM file system

[Andrew D. <and...@gm...>]

On Tue, Jun 15, 2010 at 7:27 AM, David Given <dg...@co...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 14/06/10 20:17, Andrew Dennison wrote:
> [...]
>> Has much changed in this area of Prex since 0.8.1? Maybe there its an
>> issue in 0.9?
>
> I don't know --- I never got this far with 0.8.1. I'm reasonably sure
> that there's some stack corruption in the no-MMU vfork implementation,
> but until (if and when) I can demonstrate it in a vanilla Prex without
> my hacks I don't want to commit myself.
>
> The only other non-MMU port is the GBA one, right? It doesn't build with
> a modern arm-elf, it does build with a modern arm-eabi but then won't
> boot, so I'm now building the ancient gcc 3.4.3 that' actually mentioned
> in the instructions.

Arm integrator and i386 both have non-mmu ports. I've run all four
options (arm integrator and i386 both with and without mmu) using
qemu.

>
> [...]
>> This sounds interesting. The mailing list only accepts patches as
>> inline text. I can make a 0.9 branch available on github if that
>> helps?
>
> Provided it's not too much work, that'd be really helpful.

Done. See the prex branch, there is a prex_090 tag as well.

Re: [Prex-devel] Compressed ROM file system

[David G. <dg...@co...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 15/06/10 00:07, Andrew Dennison wrote:
[...]
> Arm integrator and i386 both have non-mmu ports. I've run all four
> options (arm integrator and i386 both with and without mmu) using
> qemu.

Thank you for pointing me at that --- I'd seen pc-nommu but couldn't
make it build; I only now discovered the --profile option to ./configure.

I think I've figured out my vfork problem --- I wasn't correctly
updating context->uregs on system calls. There's a bit of very subtle
code to do with the location of context->uregs and the kernel stack
which means that it's not obvious where this happens. My understanding is:

- - context->uregs is located at the top of the kernel stack.

- - when a system call occurs, the stack point is reset to the top of the
kernel stack.

- - the first thing the system call entry code does is to push the
registers onto the kernel stack, so populating the context->uregs
structure with the current values for this thread.

- - now the stack pointer is immediately *below* context->uregs, and we're
ready to run the kernel code.

This explains a number of oddities which I was having trouble figuring out.

I haven't fixed my H8300S port yet, but I intend to comment it copiously...

- -- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ "Money is a sign of poverty." --- Iain Banks
│
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFMGAaLf9E0noFvlzgRAuFXAKCcOpnrZ91XnFMzeE1o68fFz38ncQCgjMdV
ZDGfuV2QG7mEwc20oygif4U=
=7nUe
-----END PGP SIGNATURE-----

Re: [Prex-devel] Compressed ROM file system

[Kohsuke O. <ko...@us...>]

Hi,

(6/15/2010 1:24 AM) David Given wrote:
> However: in the process of working on the Cybiko, I ran out of ROM space
> (it's only got 256kB). To work round this I implemented a compressed ROM
> file system using zlib's puff lightweight inflator. This compresses each
> 512 byte block individually so the compression ratio's not brilliant,
> but it's still achieving about 2:1 and supports true random access. puff
> is about 2kB of code and is under a pretty much identical license to Prex.

This sounds interesting. When I tried to support the direct execution of 
compressed executable files in an exec server, the standard inflate() 
routine in zlib required additional 30KB of code. So, I decided to drop 
this feature before..

> One complication is that it needs a custom program to create the
> compressed file system image, and right now the Prex build system
> doesn't support these

Good point. The custom build tools depending on the host OS will bring 
complications, and it may break the source code portability. This is the 
reason why I hesitate to add a compressed file system like cramfs. An 
archive file system in Prex was designed to make a file system image by 
using a general tools like "ar".

Thanks.
- Kohsuke

Re: [Prex-devel] Compressed ROM file system

[David G. <dg...@co...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 15/06/10 01:16, Kohsuke Ohtani wrote:
[...]
> This sounds interesting. When I tried to support the direct execution of 
> compressed executable files in an exec server, the standard inflate() 
> routine in zlib required additional 30KB of code. So, I decided to drop 
> this feature before..

Okay, I'm still really hazy on how git works, but the patch ought to be
here:

http://github.com/davidgiven/prex/commit/e6a1f3b63804afe5fa86c00fc085b885d52626e1

I've tested it with x86-nommu and it appears to work fine. Comments, please.

Do you use any kind of version control system internally? If so, what's
the easiest way of getting changes to you?

[...]
> Good point. The custom build tools depending on the host OS will bring 
> complications, and it may break the source code portability. This is the 
> reason why I hesitate to add a compressed file system like cramfs. An 
> archive file system in Prex was designed to make a file system image by 
> using a general tools like "ar".

Right now the build system is building the mkcromdisk tool with a simple:

    cc -o $@ $< -lz

...which ought to work on pretty much all Unixoids. The zlib dependency
is a pain but it's the easiest way of doing it.

- -- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ "Money is a sign of poverty." --- Iain Banks
│
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFMF946f9E0noFvlzgRAokYAJ4g9wFkgjFK5m9gpP7MVrm+HtMGTQCg2DpI
GpdjRQJBLwp8YTvC3u4+iGQ=
=aKKZ
-----END PGP SIGNATURE-----

Re: [Prex-devel] Compressed ROM file system

[Andrew D. <and...@gm...>]

On Wed, Jun 16, 2010 at 6:10 AM, David Given <dg...@co...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 15/06/10 01:16, Kohsuke Ohtani wrote:
> [...]
>> This sounds interesting. When I tried to support the direct execution of
>> compressed executable files in an exec server, the standard inflate()
>> routine in zlib required additional 30KB of code. So, I decided to drop
>> this feature before..
>
> Okay, I'm still really hazy on how git works, but the patch ought to be
> here:
>
> http://github.com/davidgiven/prex/commit/e6a1f3b63804afe5fa86c00fc085b885d52626e1
>
> I've tested it with x86-nommu and it appears to work fine. Comments, please.

Hi David,

Interesting approach - I had assumed you were implementing this as a
filesystem instead of a compressed block device. Doing this as a block
driver is certainly less invasive.

Andrew