module updates

xenic
2014-05-30
2014-06-25
  • xenic
    xenic
    2014-05-30

    The "64-bit math support for large disks" topic is getting too big with 14 pages, so I'm starting this topic for the remainder of the 64 bit etc. module updates.

     
    Last edit: xenic 2014-05-30
  • xenic
    xenic
    2014-05-30

    @kas1e
    I think I've finished the join.module update. The joined or splitted files now show in the destination lister without clicking on the lister. The progress window is fixed to show more accurate progress.
    OS4 can now join and split 64bit files. The MOS version should also work with 64bit when lock64, seek64 etc. are added in the appropriate places by MOS programmer. It won't do any good to add -DUSE_64BIT to the other platforms makefiles until they are 64bit enabled so they can stay as 32bit.

    EDIT: The module will only work for files up to about 1 terabyte in size; not the full 64 bit filesize. I can't even test with a 1 terabyte file anyway.

    Here is the OS4 64bit join.module for you to test. If you don't find any bugs I'll commit the code.

     
    Last edit: xenic 2014-05-30
    Attachments
  • xenic
    xenic
    2014-05-31

    @kas1e
    I've never used the DOpus5 FTP recently, so I set up some buttons with FTP commands for some testing. I'm not sure it needs a 64bit update so I need to know where it needs an update (lister display, copying or what). AmigaDOS Examine() isn't used for FTP display; the sizes are provided by the FTP server as far as I know.

    I couldn't connect to aminet.net and had to connect to a mirror site. What FTP address do you use to connect to main Aminet site? Do you know an FTP site (address) that has 2GB+ files so I can see how Dopus5 FTP displays them?

     
  • kas1e
    kas1e
    2014-06-01

    @Xenic
    Give me plz few more days to test join.module (i will be at home tomorrow, so can do all the tests)

    As for ftp.module, it is lister display of files more that 2gb , and probably copy of them/progress bar, but that need to test, maybe only lister display need to fix (at least, that what i myself note when login to ftps which contain files which is bigger that 2gb).

    For tests use : ftp://ftp.free.fr/mirrors/ftp.centos.org/6/isos/x86_64/
    And there is a file CentOS-6.5-x86_64-bin-DVD1.iso , which are 4,2 gb of size, and in ftp.module lister it showns as 2gb size currently. Then, after (if) you will be able to fix show in lister fine, then try to copy that file to see how progress bar will looks like. Probably that only 2 places we need to worry.

    As i say before i already trying to change all Examine and ExNext in ftp.module on 64bit equalents as we do everywhere without luck. But, i assume its hooks which need to be changes (those dc_blablba in libraries/dopus.h, which contain Fibs as arguments).

     
  • xenic
    xenic
    2014-06-01

    @kas1e
    Since the MOS version of join.module should work the same way as the OS4 version, I went ahead and changed it to use Seek64() and SetFileSize64() so the MOS version should work with USE_64BIT. You'll have to test the MOS version after we're sure the OS4 version is O.K. and I commit the new join.module code. I also added a new define to the join.module to prevent OS3 & AROS versions from being compiled for 64bit. That is necessary because the Seek() function can't handle a QUAD argument.

    I'll see if we can fix the filesize display for the FTP lister. The progress bar is an issue for the copy command too. If you copy a 2GB+ file between regular listers, the progress display stops and then restarts; making it look like 2 files were copied. I added some scaling to the join.module to fix the progress display but we need to see if we can add a TAG for 64bit sizes in the SetProgressWindowTags() function and add the scaling there instead (or change the math routiones). Then we won't need to add scaling everywhere the progress window is used and I can remove it from the join.module.

     
  • xenic
    xenic
    2014-06-02

    @kas1e
    I didn't realize that it wouldn't be practical for me to test any 64bit changes to the ftp.module because of my Internet connection speed. It would take up to 3 hours to perform an FTP transfer of a 2GB+ file. I'm not going to make any changes that I can't test. Whoever updates the FTP module to 64 bit needs to have an FTP server on a local network with high-speed ethernet in order to test large file transfers. I'll leave any updates to the ftp.module to someone else.

    When i get time, I'll see if I can improve the progress window display.

     
  • kas1e
    kas1e
    2014-06-03

    @Xenic
    You can just easy setup local ftpd server on os4, and connect to it from the same os4, and speed will be good. There is very easy to setup os4 ftpd:
    http://os4depot.net/share/network/server/ftp/a-ftp_server.lha

    As for join.module: i test it , and it seems works fine, so commit it imho.

     
    Last edit: kas1e 2014-06-03
  • xenic
    xenic
    2014-06-03

    @kas1e
    I commited the join.module changes but you need to test the MOS version. I can't compile or test MOS version so my changes for MOS were just educated guesses. I'm going to work on fixing the progress display window before I look at ftp.module so I won't need to add scaling factors everywhere the progress window is used.

    I downloaded the ftpd server and checked the included source code. It's using a long variable for tracking how many bytes of a file have been trasnferred; which will overflow for 64 bit filesizes and produce undefined results. If it works for 64bit transfers, it would just be "by coincidence". I suggest you test a-ftp_server.lha (or ask the author) to see if it is 64bit aware and can transfer 64 bit files. I don't really want to install a program I don't need unless it will be of use for testing.

     
  • kas1e
    kas1e
    2014-06-04

    @Xenic
    Checked commits, and imho we should use USE_64BIT everywhere , without any DO_64BIT as it make code looks different (while we use USE_64BIT everywhere else).

    I see that in join.h you add that :

    // Using QUAD args with OS3/AROS Seek() & SetFileSize() can corrupt files.
    #undef DO_64BIT
    #ifdef USE_64BIT
    #if defined(__amigaos4__) || defined(__MORPHOS__)
    #define DO_64BIT
    #endif
    #endif
    

    But that imho not need it at all. As for os3/aros we didn't change makefiles to do USE_64BIT, and when os3/aros developers will worry about, they will deal with. For us its imho enough to not have that block in join.h and use pure USE_64BIT everywhere.

    In other words, let's use pure USE_64BIT everywhere.

    As for aftpd server: you probably download wrong archive. Archive on which i point out, its ftpd server done on hollywood/mui-royale, and it didn't have any source code inside. You just unpack it and run. And as its hollywood, then probably it uses all os4 fucntions which mean 4gb+ should be ok.

     
    Last edit: kas1e 2014-06-04
  • xenic
    xenic
    2014-06-04

    @kas1e
    I added that definition as an additional protective measure in case someone who doesn't read and analyze the code decides to add -DUSE_64BIT to the AROS or OS3 makefile. I should have added a warning but can do it today. It could be something like this:

    #warning Compiling this module as 64 bit for some platforms could damage a user's disks.
    #warning Do not change these definitions unless you know what you are doing.

    Th only option is to change the #ifdefs in the code from:
    #ifdef DO_64BIT
    to
    #if defined(USE_64BIT) && !defined(__amigaos3__) && !defined(__AROS__)

    If you want me to change it that way just let me know.

    The reason we need to be extra cautious and prevent the code from being compiled for 64 bit on AROS and OS3 is because they use the Seek() command which only accepts a "long" argument for the file position. If you compile for 64 bit and type cast a 64 bit number like 4294967297 to a "long" (int32), the value of the "long" argument will be 1 !! That means Seek() could move back to position 1 in the file and begin copying the same data it previously copied. The result could be an infinite copy loop or filling a user's hard disk with millions of splitted files.

    Do you want me to eliminate the DO_64BIT definitions in join.h and change the conditional statements in the join.c code??

     
  • tomsmart1
    tomsmart1
    2014-06-05

    @xenic
    If i remember right than is no problem with Seek() in OS3 it seek form the current position + or -2GB max so for big file >2GB you have to split the seek. For example for 3GB position in the file you need one Seek() at 2GB and then 1GB. I think the new dospakets for OS3 and SFS works this way to handle files >2GB.

    For be 100% sure ask Joerg Strohmayer.

     
  • xenic
    xenic
    2014-06-05

    @tomsmart1
    I was a little harsh so nuff said.

     
    Last edit: xenic 2014-06-06
  • tomsmart1
    tomsmart1
    2014-06-05

    @xenic

    Sorry for pi**ed you off that was not my intention, i only want to help.

     
  • kas1e
    kas1e
    2014-06-06

    @Xenic
    Oki, let's it be like it now then, with comments you also added lately its all fine enough imho.

     
  • xenic
    xenic
    2014-06-06

    @kas1e
    O.K. I installed A-FTP_Server in ram: to see if it works. I activated the Tooltypes so I would see the local and external addresses and set it for anonymous connection. I created the directory "Data:_FTP" and entered "Data:_FTP" as the path for anonymous connection. I activated the server and tried to connect to the "local" address shown in the A-FTP_Server GUI. Both AmiFTP and Dopus5 FTP failed to connect. Also, A-FTP_Server doesn't show any external address even though I have it activated with the Tooltypes. Since I see your name in the A-FTP_Server readme, I assume you have been able to successfully connect to the server.

    I am using a DHCP connection through a wireless adapter to my router. I've never changed any RoadShow settings, so my setup should be a standard setup. Can you tell me how you managed to make a local connection between Dopus5 FTP and A-FTP_Server. I don't know much about network stuff so I am probably doing something wrong. If I don't get the server working I don't have any reasonable way to test large file transfers with Dopus5 FTP.

     
  • kas1e
    kas1e
    2014-06-11

    @Xenic

    Can you tell me how you managed to make a local connection between Dopus5 FTP and A-FTP_Server. I don't know much about network stuff so I am probably doing something wrong.

    I do test it now, and indeed, dopus5 ftp connection didn't works. It just fail to connect when trying to do "Querying FTP server type..." , and in the log of a-ftp server i see:

    Client : USER annonymous
    Server: 230 Hi anonymous!
    Client: SYST
    Server: 215 UNIX Type: L8

    And then disconnected. Aftp server only bring window about "error triggered by function: senddata" after that.

    Then, i try windows commander on my winxp. And it works. In the log then i have:

    Client : USER annonymous
    Server: 230 Hi anonymous!
    Client: SYST
    Server: 215 UNIX Type: L8
    Client: FEAT
    Server: 211-Features
    Client: CLNT Total Commander (UTF-8)
    Server: 200 Don't care
    Client: OPTS UTF8 ON
    Server: 200 UTF8 mode enabled

    and so on and so on.

    I.e. when we connect from dopus5, we just fail after line #4.

    I also tried pure "ftp" unix command from my linux , and it also works, but have some bugs. I.e. when i connect, and just try to do "ls" , then i have "510 Port is disabled", and then again error window about "Error triggered by Function: SendData". But if i do "pas on" before connect to the a-ftp server in the ftp shell command, then it works as well.

    I of course tried in dopus5 ftp custom settings (Miscellaneous/Passive Transfer) set it to ON, but that didn't help. Maybe just that option didn't works for us in dopus5 currently..

    I send that all info to author of a-ftp server and ask him to fix it, so if he around he will fix it and we can test it with.

    But in meantime i will try to find out some other local ftpd server which support 4gb+ files and works with dopus5.

     
    Last edit: kas1e 2014-06-11
  • kas1e
    kas1e
    2014-06-12

    @Xenic

    Severin report he have new crash in dopus_progressbar when moving files. Here is his report:

    Got a new bug though, I'm getting a DSI in dopus_progressbar when moving files.
    
    I previously updated on the 5th June so this bug was introduced after that.
    
    Crash log for task "dopus_progressbar"
    Generated by GrimReaper 53.19
    Crash occured in module dopus5.library at address 0x7FA56C18
    Type of crash: DSI (Data Storage Interrupt) exception
    Alert number: 0x80000003
    
    Register dump:
    GPR (General Purpose Registers):
    0: 7FA56C0C 57938F20 73AC7475 55BBDF18 57938F08 55BBDF18 55BBDF18 02A78644 
    8: 00000001 00000002 00000001 55BBDF18 59953559 7574AE74 FFFFFFFF 57541C70 
    16: 02B20000 DF983C00 02B20000 57A4DDA0 00000000 00000000 00000000 5AF10368 
    24: 08000008 00000001 08000007 5AF1039C 5AF10000 55BBDF18 00000000 578DE750 
    
    FPR (Floating Point Registers, NaN = Not a Number):
    0: nan -7.60054e+306 -7.60054e+306 -7.60054e+306 
    4: -7.60054e+306 -7.60054e+306 -7.60054e+306 -7.60054e+306 
    8: 1 255 216 31 
    12: 47 258 0 0 
    16: 0 0 0 0 
    20: 0 0 0 0 
    24: 0 0 0 0 
    28: 0 0 0 0 
    
    FPSCR (Floating Point Status and Control Register): 0x82004000
    
    SPRs (Special Purpose Registers):
    Machine State (msr) : 0x0200F030
    Condition (cr) : 0x538B1D70
    Instruction Pointer (ip) : 0x7FA56C18
    Xtended Exception (xer) : 0x0000FFFF
    Count (ctr) : 0x5587AA64
    Link (lr) : 0x5B75C610
    DSI Status (dsisr) : 0x538B1D90
    Data Address (dar) : 0x02021140
    
    680x0 emulated registers:
    DATA: 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
    ADDR: 6FFB8700 93D7AA00 00000000 00000000 00000000 00000000 00000000 57938300 
    FPU0: 0 0 0 0 
    FPU4: 0 0 0 0 
    
    Symbol info:
    Instruction pointer 0x7FA56C18 belongs to module "dopus5.library" (PowerPC) 
    Symbol: progress_set + 0x130 in section 1 offset 0x00039BF4
    
    Stack trace:
    [progress_win.c:914] progress_set()+0x130 (section 1 @ 0x39BF4)
    [progress_win.c:912] progress_set()+0x124 (section 1 @ 0x39BE8)
    native kernel module dos.library.kmod+0x00024cb8
    native kernel module kernel+0x00042530
    native kernel module kernel+0x000425b0
    
    PPC disassembly:
    7fa56c10: 4182003c beq- 0x7FA56C4C
    7fa56c14: 81230004 lwz r9,4(r3)
    *7fa56c18: 80090000 lwz r0,0(r9)
    7fa56c1c: 81290004 lwz r9,4(r9)
    7fa56c20: 2f800000 cmpwi cr7,r0,0
    
    System information:
    
    CPU 
    Model: P.A. Semi PWRficient PA6T-1682M VB1 
    CPU speed: 1800 MHz 
    FSB speed: 900 MHz 
    Extensions: altivec 
    
    Machine 
    Machine name: AmigaOne X1000 
    Memory: 2097152 KB
    
     
  • xenic
    xenic
    2014-06-12

    @kas1e
    Committed revision 1064 which should fix the crash with the "Move" command. Have Severin test it to be sure. I think the next overnight compile will contain the fixes.

     
  • xenic
    xenic
    2014-06-12

    @kas1e
    The code to fix the move command progress display was awkward and I started wondering why I've been passing addresses for QUAD values instead of the values themselves as the arguments for functions. Does passing 64 bit values as arguments fail on AROS or MOS?

     
  • kas1e
    kas1e
    2014-06-14

    @Xenic
    Severin confirmed that progress bar fixed.

    As for aros/mos , i think as currently we only two who worry about, we can make things as we think is better to, and then, when any developer will worry about, we will think how to make it better if any problems will occurs.

    As for ftp-module : i recieved a version of aftp server which now works in passive mode and dopus5 can connect to. I can upload it for you , if you wish to have a look at ftp.module 64bit fixes stuff

     
  • xenic
    xenic
    2014-06-14

    @kasle
    O.K. Just post a link to the working aftp server and I will see if I can do anything with the Dopus5 FTP module. The FTP code is somewhat confusing so it may take a while.

     
  • kas1e
    kas1e
    2014-06-16

    @Xenic
    There is new binary: http://kas1e.mikendezign.com/temp/A-FTP_Server.lha

    I.e. download version from os4depot, and just replace binary from that link. Then you will be able to connect via dopus5 with enabled passive mode in custom prefs of ftp module

     
  • xenic
    xenic
    2014-06-18

    @kas1e
    I was able to get the server working well enough to do some testing. It looks like updating the ftp.module for 64bit will be a lot more complicated than adding the new 64bit Examine... function calls. Originally I thought ftp.module was controlling listers with callbacks but it looks like most of the lister interaction is done with ARexx commands. I don't know why it was done that way but it means that the program ARexx commands would need to be updated to accept 64bit sizes before ftp.module can be updated. After the program Arexx commands are updated; a lot of code, variables and structures will need to be changed in ftp.module. In addition, all the ftp.module progress window variables and calculations need to be changed.

    For me it would be a time consuming project even if I could figure out how to do it. The bottom line is that there is nothing I can do to update the ftp.module. Like BSzili, I think I've done all I can do with Dopus5.

     
  • kas1e
    kas1e
    2014-06-20

    @Xenic
    Understood. Yeah, i tired with dopus5 as well for now. I think that maybe we can fix that crash in lister's colours: https://sourceforge.net/p/dopus5allamigas/bugs/36/

    then we can release 5.91 and child out till (if) motivation will back ?

    Probably crash will autofixes if we will put back way of how we use assembler replacements before (maybe there was some typo when you put asm parts inside of .c files from our previous asm.c file) ?

     
  • xenic
    xenic
    2014-06-21

    @kas1e
    O.K. I'll take a look at that bug since I can reproduce it.

     
  • xenic
    xenic
    2014-06-21

    @kas1e
    Even though the ShowPaletteBox() function is a library function, it is only called from one place in the Configopus code and nowhere else in Dopus5. I eliminated the assembler code in OS3 ShowPaletteBox() by changing the function to an internal function call in ConfigOpus. I named the "real" function as ShowPalBox() and made L_ShowPaletteBox() a dummy funcion that just returns zero "0". It works and I'll commit the change after I test some more.

    The side effect of this change is that in OS3 ShowPaletteBox() cannot be called from the program or other modules. That will only be a problem in the future if someone needs to add a function call to ShowPaletteBox() in the program or modules. Technically that means we still need to change L_ShowPaletteBox() and other functions to pass some arguments in a structure or with Tags. I left the #warning in place so we remember that it eventually needs to be changed.

     
  • kas1e
    kas1e
    2014-06-22

    @Xenic
    Ok, i will start to prepare 5.91 release archives tomorrow.
    Also will create a BZ for 5.92 about "we need to change other of those asm functions to pass arguments in a sturcture or with Tags", so that will be not forgotten.

    Btw, i found some interesting moment (not related to the modules, but to avoid new thread). Check plz: https://sourceforge.net/p/dopus5allamigas/featurerequests/14/

    What is interesting, is that i test our current os3 version on winaue/os3.9, and "esc" works for closing requesters ! It didn't on os4 (and probably mos, need to check), and maybe it is something again related to those signed/unsigned chars .. Will do more tests, but that find that same code works on os3 already make me just wander.

     
  • xenic
    xenic
    2014-06-22

    @kas1e
    The difference could be due to some custom setting in UAE. Regardless, I'm done now.

     
  • kas1e
    kas1e
    2014-06-23

    @Xenic
    Oki, will release 5.91 then.

     
  • xenic
    xenic
    2014-06-25

    @kas1e
    O.K.