HSCP Files

HSCOPY  v.0.8.1  2010-06-15  by F.Mizutani,RCCS

What is HSCOPY
  HSCOPY (Hybrid secure copy, High speed copy) is developed to transmit the many files
  at high speed on the long distance and wideband infrastructure.  It has achieved the
  fast transfer by changing the data transfer part of scp into the UDP.

What is different from hscp
  hscp is same to scp. hscp is transfered by files. hscopy is using tar command in 
  both side.  hscopy connects pipe local-remote.  Cipher and compress are possible
  to use together.
  hscopy needs some commands, bash, tar, gzip,gunzip(optional) and openssl(optional).
  If you use cipher and compress, hscopy (these commands image) is as follows,
    tar cO file | gzip -1 -c | openssl rc4 -e | UDT | openssl -d | gunzip -c | tar x

How to use
  hscopy [-1246AafgkMNnqsTtVvXxY] [-b bind_address]
         [-D [bind_address:]port] [-e escape_char] [-F configfile] [-i identity_file]
         [-L  [bind_address:]port:host:hostport] [-m mac_spec]
         [-O ctl_cmd] [-o option] [-p port] [-S ctl_path] [-w tunnel:tunnel]
         [-RPZC] [-l limit] [-I display_mode] [-z level] [-c cipher_spec]
         [[user@]host1:]SOURCE_FILEorSOURCE_DIRECTORY [[user@]host2:]DIRECORY

  Alpha version is able to use only one sourceFile or sourceDirectory. 
  A basic usage is the same as ssh.
  The switch enhanced from ssh is as follows.

  Add switch
   -P      Print remote shell variable. Default prints REMOTEPIPESTATUS, REMOTEEXITCODE.
           Append printed variables by -G switch. This outputs to stderr.
   -C      Use openssl for cipher. C switch use rc4 by cipher spec.
   -c rc4  Use openssl. c switch needs cipher spec.  See opensll.
   -Z      Use gzip for compress. Z switch is fastest. Same gzip -1.
   -z 1    Use gzip. z switch needs level parameter by the numerical value of 1-9. 
   -R　　  It executes it with Rsh instead of ssh.
   -I 0　  The display (Information) mode is set by the numerical value of 0-3.
      0    Similar to scp.
      1    The display of the transmission rate etc. changes line and is displayed 
           every second. 
      2    The same display as mode 1 is displayed without changing line. 
      3    When the transmission ends, only the result of the transmission rate etc. 
           is displayed.
      4    Nothing (default)
  Masked ssh switch [-TtCc]
  Useful scp switch is as follows.
   -l 1000 Limit of upper transfer rate. (kbit/s)  Example means 1Mbps
           Supported prefix.  ex. 500M 0.5G  Examples mean 500Mbps
   -6      Use IPv6 to connect.

  ex. Use print remote shell variables.
    hscopy -P user@host:file targetDirectory
      REMOTEEXITCODE=(0)
      REMOTEPIPESTATUS=(0)

----- Under construction
Platform (same HSCP)
  The build or the operation verification is settled with the following environments now.

  * AIX5.3　     There is an example of the trouble for AIX-AIX, when large size file
                 transmission. (Ajusted)
  * SunOS 5.10   The server operation is not confirmed, yet. (Ajusted)
  * Linux x86_64 It has been confirmed in AMD system, too.
  * Linux x86_32 There are a transfer results of over 2GB file.
  * Linux IA64   It has confirmed in SGI Altix4700, Fujitsu PRIMEQUEST.
  * Cygwin1.7.5  It doesn't check in the previous version. (Ajusted')
  * MacOSX　     High speed transfer is not confirmed. (There are results of 500Mbps)
                 (Ajusted)
    It succeeded in Linux IA32 edition hscp if it was on Windows(Vista32)+andLinux.

  (Ajusted)  : We had to decrease UDPRecvBufSize in hscp.conf (ex.768000) .
  (Ajusted') : We had to increase UDPSendBufSize in hscp.conf (ex.12288000) .

Archive
  Each archive has binary program.
  If you don't want to build, please read 'How to install' next.

How to build
1.  The build is done with the static link excluding a standard library.
    The dependence libraries are openssh-5.5p1, boostC++ 1.43.0 and udt4.6s.
    Udt4.6s is a special library for hssh that adds the correction to original udt4.6.
    Udt4.6s is modified udt.h, core.h, core.cpp, and ccc.cpp in src.
    It is necessary to do the build with os=UNIX for SunOS.
2.  Create directory for hssh.
3.  Store hssh-x.x.x.tar, openssh-5.5p1.tar.gz, boost_1_43_0.tar.gz 
    and udt4.6s.tgz in it.
4.  Execute ”make libudt.a" in udt4ps/src. (or "gmake libudt.a").
    for AIX and Cygwin, mask pthread_join() in cleanup() at api.cpp.
5.  Execute "./configure" and "make" in openssh-5.5p1.
    -lz is not found:  yum install zlib-devel.x86_64 (or ia32)
    -lcrypto is not found:  yum install openssl-devel.x86_64 (or ia32)
6.  No need build boost_1_43_0 
7.  Execute "make" in hssh directory. (or "gmake").
    When the error goes out by the signal function, two lines are added to 
    openssh-5.5p1/openbsd-compat/bsd-misc.h.
      /* wrapper for signal interface */
      #ifndef __cplusplus /* this line */
      typedef void (*mysig_t)(int);
      mysig_t mysignal(int sig, mysig_t act);
      #endif /* this line */
    MacOSX,AIX are different add point like this.
      /* wrapper for signal interface */
      #ifndef __cplusplus /* this line */
      typedef void (*mysig_t)(int);
      mysig_t mysignal(int sig, mysig_t act);

      #define signal(a,b) mysignal(a,b)
      #endif /* this line */
8.  Successful build version for openssh, 5.4p1, 5.5p1.
9.  Successful build version for UDT, 4.6s.
10. Successful build version for boost, 1_42_0, 1_43_0.

How to install (same HSCP)
  For use both the server and the client
    1. HSCOPY is put on /usr/local/bin (It is necessary to set path).
    2. File mode and owner of HSCOPY are changed as well as scp.
    3. "hscp.conf" is preserved in /usr/local/etc.
    4. File mode of "hscp.conf" is set 644 and owner is set to equally to scp.

  For only use the client （when you install it in the system）
    1. Installs them as well as 1-4 on.
    2. "ServerAccept" parameter of "hscp.conf" is set 0 or mask.

  For only use the client （When it is not possible to install it in the system）
    1. PATH is passed by arranging HSCOPY suitably ($HOME/bin etc.).
    2. "hscp.conf" is put on $HOME/etc.
       If "hscp.conf" is not found or that cannot start HSCOPY from sshd, it doesn't 
       start as a server.
       Definition file "hscp.conf" gives priority and reads $HOME/etc.

How to setup (same HSCP)
  Setup for client
    * Set the following values of "hscp.conf" appropriately.
    * Please set it with "SshPath" parameter when path is not "/usr/bin/ssh" of ssh
      (nether error).
        /usr/bin/ssh: No such file or directory
    * Please set it with "RshPath" parameter when path is not "/usr/bin/rsh" of rsh
      (nether error).
        /usr/bin/rsh: No such file or directory
    * When the error below is displayed when executing it, it is necessary to set 
      "UDPRecvBufSize" paramter small.
        connect: Connection setup failure: unable to create/configure UDP socket: 
        No buffer space available.
      However, it leads to deterioration in the transmission rate if the buffer is small.
      The reference value is equivalent with size ( MaxPktSize(1500) * 8192 = 12288000 )
      of the buffer of UDT.
      There is an example that cannot be used so far if it doesn't drop to about 768000
      with AIX, SunOS, and MacOSX.
    * Client port number is assigned automatic.
      If you want to fix to range, please set bellow parameters.
      UDPClientStartPort: The first number of UDP port that stands by client
      UDPClientEndPort:   The last number of UDP port that stands by client
      The maximum range is limited to 256.

Setup for server (same HSCP)
    * Set the following parameters of "/usr/local/etc/hscp.conf" appropriately.
      UDPStartPort: The first number of UDP port that stands by server
      UDPEndPort:   The last number of UDP port that stands by server
      18001-18010 is set in default. The maximum range is limited to 256.
    * Firewall should be able to be passed UDP port within the range of the above-
      mentioned and the setting.
    * Not only the machine but also that when operating it on the site is necessary for 
      setting firewall.

License
  Copyright (c) 2010, Fumiyasu Mizutani, Fumitsuna Teshima, Masataka Sawa, 
  Kensuke Iwahashi, Takakazu Nagaya, Shigeki Naitoh, Jun-ichi Matsuo All rights reserved.
 
Redistribution and use in source and binary forms, with or without modification, are 
permitted provided that the following conditions are met:

  * Redistributions of source code must retain the above copyright notice, this list of
    conditions and the following disclaimer.
  * Redistributions in binary form must reproduce the above copyright notice, this 
    list of conditions and the following disclaimer in the documentation and/or other 
    materials provided with the distribution.
  * Neither the name of the "NINS (National Institutes of Natural Sciences), 
    IMS (Institute for Molecular Science)" nor the names of its contributors may be used
    to endorse or promote products derived from this software without specific prior 
    written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY 
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL 
THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT 
OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

FAQ
  cf. HSCP

People
  HSCOPY is developing by Technical staff of NINS IMS.
  It develops as RCCS service
 
  Members,
    section chief　Fumiyasu Mizutani (contact: mizutani(at)ims.ac.jp)
    unit 1　Fumitsuna Teshima, Masataka Sawa 
    unit 2　Kensuke Iwahashi, Takakazu Nagaya 
    unit 3　Shigeki Naitoh, Jun-ichi Matsuo
 
  Special thanks
    We wish to express my gratitude for RCCS helped when this program was developed.
    We thanks for Hironori Kogawa (Hitachi and Ltd.) that did the improvement that uses 
    UDT for scp.
    We are thankful to the developers of OpenSSH, UDT and BoostC++, they are the base of
    hssh.
    And, We are thankful to all open source developers.

  Acknowledge
    We acknowledge many users.
    Sergey Cherepanov gives a lot of reports and comments always.