64-bit math support for large disks

BSzili
2014-02-26
2014-05-27
1 2 3 .. 14 > >> (Page 1 of 14)
  • BSzili

    BSzili - 2014-02-26

    Since 4GB+ disks are the norm, and most amiga file systems support volumes larger than 4GB, I think it's important to add 64-bit math support before 5.90 is released. I want it to be 68k compatible, so I plan to add a few new functions namely 64-bit aware versions of DivideToString and BytesToString, and some 64-bit math functions. I think these should be added to dopus5.library instead of creating a separate one, so modules could use the new functions too without any hassle.
    To store the 64-bit integer a new type has to be added, called DO_QUAD or something like that. This type could be defined to the native 64-bit types on NG OS' (QUAD, int64) and for 68k it would be a struct like this:

    typedef struct
    {
        ULONG high;
        ULONG low;
    } DO_QUAD;
    

    To do the 64-bit math (multiplying to 32-bit ints which results in a 64-bit one, etc.) we could reuse Olaf's code from SendRawDisk: http://aminet.net/package/comm/tcp/SendRawDisk
    It's written in portable C, so we wouldn't have to muck with assembler and utility.library. On non-68k systems GCC's native math library could be used. What do you think?

     
  • xenic

    xenic - 2014-03-04

    @BSzili
    Sounds like a good idea. What does a lister show when you select a directory with more than 4GB (like a dir full of pictures/music etc.) and press the "GetSizes" button? Does it show the right number of bytes?

     
  • BSzili

    BSzili - 2014-03-04

    @xenic
    That would show the wrong value, I haven't updated the getsizes function yet.

     
  • kas1e

    kas1e - 2014-03-05

    @BSzili
    Cool :) Is there list of what we need to fix so whole dopus will show correct size everywhere ? I know only about:
    1). RMB on icon, and Information or Disk Info
    2). GetSize
    3). Lister's at top

    What else ? Or that all ?

     
    Last edit: kas1e 2014-03-05
  • BSzili

    BSzili - 2014-03-05

    @kas1e
    Searching for id_BytesPerBlock should give you a rough idea.

     
  • xenic

    xenic - 2014-03-05

    @BSzili
    I don't see how we could test your 64 bit math support for OS3 unless the OS3 crash is fixed first. I would suggest that after you complete your 64 bit update, you do not commit it to the repository until the current crash is fixed. However, you're doing the coding so do what you think is best.

    The 64bit math and byte size displays in OS3 Dopus4 were accomplished by using a Libnix2 that was ported from MOS. I wasn't the programmer for the latter OS3 versions of Dopus4 but I obtained a copy of the ported Libnix2 so I could compile OS3 Dopus4 myself. The OS3 Dopus4 code (which is in the same SVN repository now as OS4 Dopus4) appears to use "long long" variables and math so I thought you might be interrested in the possibility of using Libnix2 to compile OS3 Dopus5. I don't know if my GeekGadgets libnix2 will work with a cross compiler but here is a simple test program that I compiled and it prints the correct result in a shell:

    // Compile = gcc mathtest.c -noixemul -lm -o mathtest
    // Requires libnix2

    #include <proto/dos.h>
    #include <stdio.h>

    int main(int argc, char **argv)
    {
    long long big = 8111111111;
    long med = 123456;
    long long result = 0;

    result = big + med;

    printf("Number = %qd\n", result);
    return(RETURN_OK);
    }

    // console output is "Number = 8111234567"

     
  • BSzili

    BSzili - 2014-03-06

    I don't know how you could test it, but it runs in UAE just fine. Apart from the slow startup, it works on at least one AGA amiga too.
    Anyway, this is interesting! I didn't know you can use the GCC math library do to 64-bit math on 68k too. I designed the new 64-bit DivideToString, BytesToString and ItoaU to be independent from the underlying 64-bit implementation, so it wouldn't take much time to make them use the GCC math library. For the time being I'll keep using Olsen's code for prototyping.

    edit: Ok, I missed the LibNIX2 part. We could add different 64-bit backends, and use LibNIX2 if it's faster.

     
    Last edit: BSzili 2014-03-06
  • xenic

    xenic - 2014-03-06

    @BSzili
    I mentioned the possiblilty of using libnix2 as a backup plan in case you run into problems with your custom functions. I think that getting my libnix to work for you and the overnight compile could be problematic. The search paths that are compiled into my libnix2 might cause conflicts with a cross-compiler. It was compiled specifically for a GeekGadgets installation. I think you should continue with your custom functions and just keep libnix2 in mind if there are problems.

    There are also some libnix2 sources at https://sourceforge.net/projects/libnix/files/ that appear to have some 64 bit functions for math & string conversion. It might be worth looking at if you have any problem with Olsen's code.

     
  • BSzili

    BSzili - 2014-03-07

    Here's a quick status update. The lister title, the gauge and the devicelist now show the correct size of 4G+ volumes. The Total/Selected bytes in the lister status bar (topmost) are also 64-bit values. This almost takes care of the main executable, the only thing left is the de_Size field of the DirEntry structure. This is required for GetSizes to work correctly on directories containing more than 4G worth of files, and also paves the way for the 64-bit DOS support in future releases for files larger 4GB on MOS and OS4.
    Fixing diskinfo.module and the others should be easy after this. The biggest hurdle will be moving the new functions into the library, and regenerating all the headers and glue. For this reason I think we should stick to the main executable only for 5.90. I've broken enough things before the release :)

     
  • BSzili

    BSzili - 2014-03-08

    Since I didn't want to leave checksizes/getsizes non 64-bit aware, I had to change quite a few structures to add 64-bit fields for the file sizes. I don't use 64-64 division or multiplication, and I intend to keep it that way, but I'm starting to worry there will be some noticeable performance penalty on slower 020 systems. I'm not sure if it would be worth having m68k builds without the 64-bit support. I'll do some benchmarks.

     
  • BSzili

    BSzili - 2014-03-08

    @xenic
    It has become clear that using the custom 64-bit math routines will unnecessarily complicate the code in the future, when I plan to add support for 64-bit DOS functions (Examine64/ExamineObject). If you were talking about this libnix2, then I don't see why it shouldn't work with the cross compiler. The host and target OS are two separate things after all. After the current 68k issues have been fixed, we need to test if linking against libnix2 breaks anything.

    edit: I tried libnix2 with my cygwin cross-compiler, and it works, and correctly shows the result of two 64-bit interers multiplied together. The only catch is I had to add a few symbols, which should be in libstubs.a:

    char __intuitionname[]="intuition.library";
    char __dosname[]="dos.library";
    char __mathieeedoubbasname[]="mathieeedoubbas.library";
    char __mathieeedoubtransname[]="mathieeedoubtrans.library";
    char __utilityname[]="utility.library";
    
     
    Last edit: BSzili 2014-03-08
  • xenic

    xenic - 2014-03-08

    @BSzili
    There are a number of libstubs.a files included in Libnix2 such as:
    lib/libm020/libnix/libstubs.a
    lib/libb/libnix/libstubs.a
    lib/libnix/libstubs.a

    I think that only one of them is broken and it's "lib/libnix/libstubs.a" which is twice the size of the others. I was able to compile by replacing that libsubs.a file with the same file from Libnix 1.2 and didn't seem to have any problems when compiling OS3 Dopus4. I have only done one test compile of Dopus5 with Libnix2 and it seemed to work. I had to add a "LIB:" assignment and create a directory link from GG:68k-amigaos/lib/libnix to GG:m68k-amigaos/lib/libnix to solve some problems in the default search path.

    We may need for kas1e to test it with his overnight cross compiler too.

     
  • BSzili

    BSzili - 2014-03-08

    Thanks for the tip, meanwhile I've found this discussion too:
    https://groups.yahoo.com/neo/groups/opus4/conversations/topics/7166
    Replacing libnix/libstubs.a with the 1.2 one fixed the problem, now I can compile your example on Windows without any issues, and I get the correct result. If will work on kas1e's server too, then I think this is the way to go, much cleaner than littering the code with custom math functions for the sake of one platform.

     
  • xenic

    xenic - 2014-03-10

    @BSzilli

    If will work on kas1e's server too, then I think this is the way to go, much cleaner than littering the code with custom math functions for the sake of one platform.

    Yes. Dopus5 is complicated enough without adding more code if we don't need to.

     
  • kas1e

    kas1e - 2014-03-10

    @All
    Build os3 version with new libnix of rev917 with adding -DUSE_64BIT to os3 makefile and results is : its all actually works as intended. I.e. in device list of listers i have for my work partion: 99% full, 23mb free, 10.3g in use. In the lister's title values also right ones now. Kewl !

     
  • kas1e

    kas1e - 2014-03-10

    And also tested OS4 build by adding -DUSE_64BIT to os4's makefile: works too as expected.

     
  • BSzili

    BSzili - 2014-03-10

    @kas1e
    Excellent. I used pointers to pass the 64-bit integers, so there shouldn't be any problem to move the new functions into the library. I can regenerate the headers for AROS/MOS/OS3 but I have no clue about idltool.

     
  • kas1e

    kas1e - 2014-03-11

    @BSZili

    I can regenerate the headers for AROS/MOS/OS3 but I have no clue about idltool.

    It's easy too, check this out:
    http://www.os4coding.net/blog/kas1e/how-build-ppc-stubs-68k-libraries

    But in brief:

    generate sfd:
    shell:> fd2pragma dopus5_lib.fd clib dopus5_protos.h special 112
    
    generate xml:
    shell:> fdtrans dopus5_lib.sfd -x
    
    generate proto/interface/inline/:
    shell:> idltool -p -i -n dopus5.xml 
    

    Through there can be some moments in includes : some of them we adapt manually. For example you can see in include/inline4/dopus5.h there is commented out some "autogenerated ones", and replaced with new own ones (at end of file). The same done for 68k/mos (and i assume you do it for aros too?). Also do be honest i do not remember right now which switch of fd2pragma was used to generate 68k includes: as only one of them works. But i assume will be better for 68k includes just manually add those new functions then.

    Also there will be needs to regenerate aos4_68k_to_ppc_vectors.c, add new functions to aos4_ppc_libstubs.c, as well as update morphos_stubs.c too. While aos4_ppc_libstubs.c and morphos_stubs.c its just some copy+paste with some replace, aos4_68k_to_ppc_vectors.c need to regenerate.

    But all that i can do later. In general, you can just do it for 68k for now, and i then will adapt it for os4/mos.

     
    Last edit: kas1e 2014-03-11
  • BSzili

    BSzili - 2014-03-11

    @kas1e
    Neat, then I'll probably just merge the new functions after I generated the glues. Now that I remember I had to hack the AROS headers by hand too, to change the variadic inlines a little.
    I'm not in favor of changing the LVO's unless it's necessary. It causes no harm to have a couple of dummy entries, and changing the offsets can cause trouble if someone decides to use some SAS/C modules on 68k.

     
  • BSzili

    BSzili - 2014-03-11

    @kas1e
    I committed my changes. In the end I didn't update the following files:

    Include/inline4/dopus5.h
    Include/interfaces/dopus5.h
    Library/aos4_68k_to_ppc_vectors.c
    

    I'll leave those to you.

     
  • kas1e

    kas1e - 2014-03-11

    @BSzili
    Ok few mins

    ps. aos4_ppc_libstubs.c also a bit wrong, will fix now too

     
    Last edit: kas1e 2014-03-11
  • kas1e

    kas1e - 2014-03-11

    @BSZili
    Done, updated all stuff, as well as rebuild everything from scratch with new changes for os4: all works as expected. Doh, we quite lucky that go through all of this without much pain :)

     
  • BSzili

    BSzili - 2014-03-11

    @kas1e
    The pain has just begun, the OS3 version crashes for me.

     
  • kas1e

    kas1e - 2014-03-11

    @BSzili
    I only test os4 whole recompile and it was ok. Maybe you also need whole recompile of everything (make os3 clean; make os3) ? But if that not help, and as os4 (and aros?) version works, i assume it can be something about os3 includes. If you will not find roots today, i can track it down tomorrow when will back at work.

     
1 2 3 .. 14 > >> (Page 1 of 14)

Log in to post a comment.