Re: [xSocket-develop] [discusstion] applying xSocket to implement an effective and robust file serv
Status: Inactive
Brought to you by:
grro
Menu
▾
▴
Re: [xSocket-develop] [discusstion] applying xSocket to implement an effective and robust file server
|
From: beijing w. <wel...@gm...> - 2008-11-24 03:45:34
|
2008/11/22 Gregor Roth <gre...@go...>
> Hi,
>
> by using the pool, the retrieved connection should always be closed after
> using it. The close method returns the connection into the pool. If an error
> is occurred by using the connection, the connection should be destroyed by
> calling the pool's destroy method to avoid that the connection will be
> returned into the pool (the pool internal always checks if the returned
> connection is healthy, however it is a good behaviour to destroy the
> connection if it is known that an error is occurred).
oh, yes, it did InetSocketAddress(host, port) every time, when reaching the
max active connection number, it will retrieve the connection in the pool.
why not reuse the connection for each connection, anyway, it's not a general
client solution, the client which need transfer several files in a very
short time need to do that, and other client such as chat, also need to
reuse the connection.
>
> Regarding to interoperable communication I do have no specific hints for
> you
you have helped me a lot here.
>
>
> Gregor
>
>
> 2008/11/19 beijing welcome <wel...@gm...>
>
> Gregor, thanks you so much for your great help again!
>>
>> two minor basic questions about client implements.
>>
>> 1. about BlockingConnectionPool
>>
>> client {
>> private final BlockingConnectionPool pool = new
>> BlockingConnectionPool();
>> public void *read*(filename) {
>> try {
>> con = pool.getBlockingConnection(host, port);
>> // do something...
>> } catch () {
>> // the TCP connection is still kept, i mena when this function was
>> called for many times in for loop,
>> // it still keep the same connection, not waste resource here. right?
>> pool.destroy(con);
>> }
>> }
>>
>> void main() {
>> for (int i = 0; i < size; i++) {
>> client.*read*(files[i]);
>> }
>> }
>> }
>>
>>
>> 2. if the client is implemented by C++ or other language, any tricky
>> issues for the connection?
>>
>>
>>
>> 2008/11/19 Gregor Roth <gre...@go...>
>>
>> Hi,
>>>
>>> I'm not sure if I understand your code. The DataHander is used to send
>>> the file to the client. On the other side the DataHandler will be used by
>>> handling the data command (which is used to upload data from the client to
>>> the server?)
>>>
>>> I added a simple FileServer into the test trunk of xSocket. See
>>> http://xsocket.svn.sourceforge.net/viewvc/xsocket/xsocket/core/trunk/src/test/java/org/xsocket/connection/SimpleFileServer.javaand
>>> http://xsocket.svn.sourceforge.net/viewvc/xsocket/xsocket/core/trunk/src/test/java/org/xsocket/connection/SimpleFileServerClient.java
>>>
>>> May be it helps you to write your server
>>>
>>> Gregor
>>>
>>>
>>>
>>> 2008/11/18 beijing welcome <wel...@gm...>
>>>
>>> hi, Gregor,
>>>> many thanks for your suggestion, i tried to do.
>>>>
>>>> the code below were copied from SimpleSmtpClient.java and
>>>> SimpleSmtpServer.java, for some reason, it didn't work.
>>>> i'm in trouble in the filechannel parts, which were comments in *bold*below.
>>>> could you please point me out what's the error?
>>>>
>>>>
>>>> ========== SimplePicClient.java ============
>>>> package picserver;
>>>>
>>>> import java.io.File;
>>>> import java.io.FileInputStream;
>>>> import java.io.IOException;
>>>> import java.io.RandomAccessFile;
>>>> import java.nio.BufferUnderflowException;
>>>> import java.nio.channels.FileChannel;
>>>> import java.util.logging.ConsoleHandler;
>>>> import java.util.logging.Level;
>>>> import java.util.logging.Logger;
>>>>
>>>>
>>>> import org.xsocket.LogFormatter;
>>>> import org.xsocket.MaxReadSizeExceededException;
>>>> import org.xsocket.connection.BlockingConnection;
>>>> import org.xsocket.connection.IBlockingConnection;
>>>>
>>>>
>>>> public final class SimplePicClient {
>>>>
>>>> private static final Logger LOG =
>>>> Logger.getLogger(SimplePicClient.class.getName());
>>>>
>>>>
>>>> private String host = null;
>>>> private int port = -1;
>>>>
>>>> public static void main(String[] args) throws Exception {
>>>>
>>>> Logger logger =
>>>> Logger.getLogger(SimplePicClient.class.getName());
>>>> logger.setLevel(Level.FINE);
>>>>
>>>> ConsoleHandler ch = new ConsoleHandler();
>>>> ch.setLevel(Level.FINE);
>>>> ch.setFormatter(new LogFormatter());
>>>> logger.addHandler(ch);
>>>>
>>>> String host = "127.0.0.1";
>>>> int port = 9527;
>>>>
>>>> SimplePicClient smtpClient = new SimplePicClient(host, port);
>>>> smtpClient.send();
>>>> }
>>>>
>>>>
>>>>
>>>> public SimplePicClient(String host, int port) throws IOException {
>>>> this.host = host;
>>>> this.port = port;
>>>> }
>>>>
>>>>
>>>>
>>>> public void send() throws IOException {
>>>>
>>>> IBlockingConnection con = new BlockingConnection(host, port);
>>>>
>>>> // read greeting
>>>> readResponse(con);
>>>>
>>>> sendCmd(con, "Helo you");
>>>> readResponse(con);
>>>>
>>>> sendCmd(con, "Data");
>>>> readFileResponse(con);
>>>> con.write("\r\n.\r\n");
>>>>
>>>>
>>>> sendCmd(con, "Quit");
>>>> readResponse(con);
>>>>
>>>> con.close();
>>>> }
>>>>
>>>>
>>>> private void sendCmd(IBlockingConnection con, String cmd) throws
>>>> IOException {
>>>> LOG.fine("sending " + cmd);
>>>> con.write(cmd + "\r\n");
>>>> }
>>>>
>>>> private String readResponse(IBlockingConnection con) throws
>>>> IOException {
>>>> String response = con.readStringByDelimiter("\r\n");
>>>> LOG.fine("receiving " + response);
>>>>
>>>> return response;
>>>> }
>>>>
>>>> * private String readFileResponse(IBlockingConnection con)
>>>> throws IOException, BufferUnderflowException,
>>>> MaxReadSizeExceededException {
>>>>
>>>> int length = con.readInt();
>>>> // the output is:
>>>> // 24:54:06,359 10 FINE [SimplePicClient#readFileResponse] length:
>>>> 1164866661
>>>> LOG.fine("length: " + length);
>>>> FileChannel fc = new RandomAccessFile(
>>>> "D:\\1\\tmp.txt", "rw").getChannel();
>>>> fc.transferTo(0, length, con);
>>>> fc.close();
>>>> return "Done";
>>>> }*
>>>> }
>>>>
>>>> ============== SimplePicServer.java =================
>>>> package picserver;
>>>>
>>>> import java.io.File;
>>>> import java.io.IOException;
>>>> import java.io.RandomAccessFile;
>>>> import java.nio.BufferUnderflowException;
>>>> import java.nio.channels.FileChannel;
>>>>
>>>> import org.xsocket.Execution;
>>>> import org.xsocket.MaxReadSizeExceededException;
>>>> import org.xsocket.connection.IConnectHandler;
>>>> import org.xsocket.connection.IDataHandler;
>>>> import org.xsocket.connection.INonBlockingConnection;
>>>> import org.xsocket.connection.IServer;
>>>> import org.xsocket.connection.Server;
>>>> import org.xsocket.connection.ConnectionUtils;
>>>> import org.xsocket.connection.IConnection.FlushMode;
>>>>
>>>>
>>>> public final class SimplePicServer extends Server {
>>>>
>>>>
>>>> public static void main(String[] args) throws Exception {
>>>> int port = 9527;
>>>> String outDir = "c://tmp";
>>>> if (!new File(outDir).exists()) {
>>>> System.out.println(outDir + " does not exists. creating
>>>> directory");
>>>> new File(outDir).mkdirs();
>>>> }
>>>>
>>>> System.out.println("writing mails to directory " + outDir);
>>>>
>>>> IServer server = new SimplePicServer(port, outDir);
>>>> server.setFlushMode(FlushMode.ASYNC);
>>>> // server.start();
>>>> ConnectionUtils.registerMBean(server);
>>>> server.run();
>>>> }
>>>>
>>>>
>>>> public SimplePicServer(int port, String outDir) throws IOException {
>>>> super(port, new SimplePicHandler(outDir));
>>>> }
>>>>
>>>> @Execution(Execution.NONTHREADED)
>>>> private static final class SimplePicHandler
>>>> implements IConnectHandler, IDataHandler {
>>>>
>>>> private String msgFileDir = null;
>>>> private int countReceivedMessages = 0;
>>>>
>>>> public SimplePicHandler(String msgFileDir) {
>>>> this.msgFileDir = msgFileDir;
>>>> }
>>>>
>>>>
>>>>
>>>> public boolean onConnect(INonBlockingConnection connection)
>>>> throws IOException, BufferUnderflowException,
>>>> MaxReadSizeExceededException {
>>>> connection.write("220 ready \r\n");
>>>>
>>>> return true;
>>>> }
>>>>
>>>>
>>>> public boolean onData(INonBlockingConnection connection)
>>>> throws IOException, BufferUnderflowException,
>>>> MaxReadSizeExceededException {
>>>>
>>>> String cmd =
>>>> connection.readStringByDelimiter("\r\n").toUpperCase();
>>>> System.out.println(cmd);
>>>> if (cmd.startsWith("HELO")) {
>>>> connection.write("250 Service\r\n");
>>>> System.out.println("when HELO");
>>>> } else if(cmd.equals("DATA")) {
>>>> File msgFile = new File("D:\\1\\mail.txt");
>>>> connection.setHandler(
>>>> new DataHandler(msgFile, this));
>>>> connection.write("Enter message, ending with
>>>> \".\"\r\n");
>>>>
>>>> System.out.println("when DATA");
>>>>
>>>> } else if (cmd.startsWith("QUIT")) {
>>>> connection.write("221 service closing connection\r\n");
>>>> connection.close();
>>>> System.out.println("when QUIT");
>>>>
>>>> } else {
>>>> connection.write("500 Unrecognized command\r\n");
>>>> System.out.println("when else");
>>>>
>>>> }
>>>>
>>>> return true;
>>>> }
>>>>
>>>>
>>>> String getMessageFileDirectory() {
>>>> return msgFileDir;
>>>> }
>>>>
>>>> void setMessageFileDirectory(String msgFileDir) {
>>>> this.msgFileDir = msgFileDir;
>>>> }
>>>>
>>>> int getCountReceivedMessages() {
>>>> return countReceivedMessages;
>>>> }
>>>>
>>>> void incCountReceiveMessages() {
>>>> countReceivedMessages++;
>>>> }
>>>> }
>>>>
>>>>
>>>> @Execution(Execution.MULTITHREADED)
>>>> private static final class DataHandler implements IDataHandler {
>>>>
>>>> private SimplePicHandler simplePicHandler;
>>>> private File file;
>>>> private FileChannel fc;
>>>>
>>>> public DataHandler(File file, SimplePicHandler simplePicHandler)
>>>> {
>>>> this.file = file;
>>>> this.simplePicHandler = simplePicHandler;
>>>> }
>>>>
>>>> * public boolean onData(INonBlockingConnection connection)
>>>> throws IOException, BufferUnderflowException,
>>>> MaxReadSizeExceededException {
>>>>
>>>> if (fc == null) {
>>>> fc = new RandomAccessFile(file, "r").getChannel();
>>>> }
>>>> int length = (int)fc.size();
>>>> System.out.println("length: " + length);
>>>> connection.write(length);
>>>> connection.transferFrom(fc);
>>>> fc.close();
>>>> fc = null;
>>>> // it's not correct, the handler here failed to go back to
>>>> SimplePicHandler
>>>> // the following command are all dealed by DataHandler, which should be
>>>> dealed by SimplePicHandler.
>>>> connection.setHandler(simplePicHandler);
>>>> connection.write("250 OK\r\n");
>>>> return true;
>>>> }*
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> 2008/11/18 Gregor Roth <gre...@go...>
>>>>
>>>> Hi,
>>>>>
>>>>> in most cases it is better to use the multithreaded mode (multithreaded
>>>>> is the default mode). Running the service in a non-threaded mode is a
>>>>> performance optimization. However, the non-threaded mode will work only
>>>>> work, if the handler doesn't perform I/O operations such as file or network
>>>>> operations.
>>>>>
>>>>> To avoid outstanding threads while receiving file data, you could read
>>>>> the network data and write it to a file in a non-blocking mode. For instance
>>>>> see the Data Handler of the SimpleSmtpServer ->
>>>>> http://xsocket.svn.sourceforge.net/viewvc/xsocket/xsocket/core/trunk/src/test/java/org/xsocket/connection/SimpleSmtpServer.java
>>>>>
>>>>> To send a file a sync mode has to be used. For instance see the
>>>>> UploadHandler2 of the SimpleFileChannelPerformanceTest >
>>>>> http://xsocket.svn.sourceforge.net/viewvc/xsocket/xsocket/core/trunk/src/test/java/org/xsocket/connection/SimpleFileChannelPerformanceTest.java
>>>>> With Java 1.7 you will have the option to access the file in an async
>>>>> mode. In this case you can use a non-blocking, async approach similar to the
>>>>> file upload scenario.
>>>>>
>>>>> If the server reaches the limit by serving many clients, a transport
>>>>> level load balancing solution will be used to spread the request over a farm
>>>>> of servers. For a more detailed explanation see link
>>>>> http://www.javaworld.com/javaworld/jw-10-2008/jw-10-load-balancing-1.html
>>>>>
>>>>> Gregor
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2008/11/17 beijing welcome <wel...@gm...>
>>>>>
>>>>>> hi,
>>>>>> a discussion for applying xSocket to implement an effective and robust
>>>>>> file server.
>>>>>>
>>>>>> a TCP server stores lots of small pictures (from 1k to 5k for each
>>>>>> size).
>>>>>> a client downloads pictures from server frequently, but seldom do
>>>>>> uploading job.
>>>>>>
>>>>>> as far as i know for xSocket, the server can use
>>>>>> SERVER: transferFrom a FileChannel to BlockingConnection in on
>>>>>> IDataHandler.onData()
>>>>>> CLIENT: can use transferTo a FileChannel from BlockingConnection
>>>>>>
>>>>>> the communication between client and server has two steps:
>>>>>> 1. client sends a query to server, server send back the pictures list
>>>>>> which need to download.
>>>>>> 2. client begins to download the pictures from server.
>>>>>>
>>>>>> if the client need to download more than 10 pictures in a seconds,
>>>>>> then it seems reasonable to use multi thread.
>>>>>>
>>>>>> the question is how to design in server side.
>>>>>> 1) if use multi thread, how to write the IDataHandler.onData()? if
>>>>>> it's BlockingConnection, will it block other threads for this connection? if
>>>>>> NonBlockingConnection, need to write to bytebuffer.duplicate()?
>>>>>>
>>>>>> 2) if many clients connect to the same server (e.g. more than 10k),
>>>>>> the multi thread will rise the connection number? how to minimize the load
>>>>>> of server? if use proxy to do load balance, any other problem?
>>>>>>
>>>>>>
>>>>>>
>>>>>> what's your opinion?
>>>>>>
>>>>>> any comments are appreciated.
>>>>>>
>>>>>>
>>>>>> -------------------------------------------------------------------------
>>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>>>>> challenge
>>>>>> Build the coolest Linux based applications with Moblin SDK & win great
>>>>>> prizes
>>>>>> Grand prize is a trip for two to an Open Source event anywhere in the
>>>>>> world
>>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>> _______________________________________________
>>>>>> xSocket-develop mailing list
>>>>>> xSo...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/xsocket-develop
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> |
