[Erlyaws-list] Yaws2 architecture proposal

[Jonathan G. <de...@sa...>]

Claes and others --

Thanks for letting me share my ideas on how yaws2 could be put together. 

Like I mentioned, we've shied away from version 1 of yaws because the
architecture and coupling of components just seems too heavyweight and
unpredictable for our (admittedly very low-level) needs. Coupled with the
unpredictable development and release cycle for yaws as a whole, that has made
us uncomfortable with building our systems on top of yaws today.

What follows is inspired by our desire to see a yaws2 that allows users to build
up only the functionality they require, but still provides a nice single
entrypoint for those wanting to run a standalone server and build sophisticated
websites, without requiring them to do the work of putting together all of the
low-level components by themselves.

I hope people will find this as a decent place to have some discussion, and if
there are sufficient replies that the text of the thread makes it difficult to
read the emerging plan, I can just assimilate feedback into the original copy
and start a new thread for another iteration.


jon


------------------------- Begin Yaws2 Proposal ---------------------------------





The following is a sketch of how yaws today could be refactored and repackaged
to support cleaner separation of functionalities in yaws 2. It lays yaws out as
a collection of applications, all but one of them library applications, with the
final one being an actual supervisor application (what today most people would
have to say is "yaws").

Readers who are interested in seeing the progression from base http protocol and
socket handling to a fully-loaded standalone webserver should read from the top
down. Readers who want to start with the familiar standalone webserver concept
and see how it's composed of smaller library applications may prefer to read
this from the bottom back to the top.



--------------------------------------------------------------------------------
Yaws Core (library application)
--------------------------------------------------------------------------------

This would be the core http functionality at the heart of all yaws applications,
whether standalone or embedded. The libraries in this application are meant to
have zero knowledge of their enclosing environment, lifecycle mechanism (whether
supervised or totally unmanaged), logging facilities, web-development toolkits
being used (if any), etc. 

These libraries only know how to receive connections, manage the basics of the
HTTP protocol, and send logging events to somewhere to be handled appropriately.
This application will be a suitable building block for both web-based
applications (that ultimately are sending html to a user) as well as low-level
messaging backbones that use HTTP as a transport mechanism (but have no need for
anything related to xhtml, cookies, sessions, soap, authentication, ssl, etc)

    - proto_http.erl

        This is the real workhorse module, and will export a huge number of
        side-effect free utility functions. Contains a majority of the utility
        type and http protocol functions found currently across yaws.erl and
        yaws_api.erl. This includes functions for formatting http dates,
        strings, browser recognition and special casing, cookie setting and
        lookup, base64, compression, http auth, status code setting, connection
        pipelining/closing, redirect, header accumulation, etc.

    - gen_yaws_server.erl

        This module implements the gen_server behaviour. Holds a yaws_conf
        record (see yaws_config.hrl below) as its state (passed to init/1), and
        operates by receiving requests and passing them to the request handler
        (specified in the yaws_conf). This module will make heavy use of
        proto_http.

    - gen_yaws_responder

        A simple behaviour with the single callback out/1. There will only be
        one of these per gen_yaws_server process, but downstream applications
        that wish to combine multiple routing facilities and responders will do
        so by creating one base implementation of gen_yaws_responder which holds
        some lookup and switching facility to call a number of other responders
        known to it. 
        
        For example, the standalone yaws application (see below) would create a
        top-level responder which, in its out/1 method, looks at the path and
        picks the appropriate sub-responder (json rpc, .yaws compilation, an
        appmod, etc) to do the actual work.

        Creative authors downstream will likely make it possible to plug various
        responders together into useful pipelines. For example, one can envision
        a DOS-attack filtering responder which wraps another arbitrary
        responder, refusing the request if it is deemed part of an attack.
        Similarl utilites might include those that check for session
        authentication before propagating a request futher, etc. In any event,
        these will be built outside of this core library application by more
        specific yaws instance owners.

    - gen_yaws_logger

        Logging/debugging behavior with callbacks for access/info/error/debug
        types of logging. Different applications using yaws will supply the
        actual handling code by implementing this behavior.
    
    - yaws_config.hrl

        Contains a record definition for a yaws_conf record, specifying
        everything that a gen_yaws process would need to operate, including:

            - Listen Address(es): 
                [ { nossl, Listen } | { ssl, Listen, SSLOpts }, ... ]
            - Module implementing gen_yaws_logger
            - Other HTTP Protocol/Socket configuration
            - Responder: { gen_yaws_responder, Mod } | { anonymous, Fun }



--------------------------------------------------------------------------------
Yaws Ehtml (library application)
--------------------------------------------------------------------------------

Simple library application combining the .yaws template compiler and the very
useful ehtml term library. Downstream applications may use both of these
together, or simply include this application solely to make use of the ehtml
libraries, even without templating.

    - ehtml.erl
        Handles the translation of ehtml terms to xhtml.

    - template_ehtml.erl

        This would implement the 'gen_yaws_responder' behaviour. Basically this
        is what is today called yaws_compile.erl



--------------------------------------------------------------------------------
Yaws Sessions (library application)
--------------------------------------------------------------------------------

A framework and standard set of implementations of session management, for those
server installations that use sessions.

    - gen_yaws_session_server.erl
        The basic callback module for a session server.

    - yaws_session_server_ets.erl
    - yaws_session_server.dets.erl
    - yaws_session_server.mnesia.erl
    - yaws_session_server_conf.hrl

        Specific session server implementations that depend only on the otp
        distro (others using SQL dbs and such should be outside of the central
        yaws packaging):



--------------------------------------------------------------------------------
Yaws RPC (library applications)
--------------------------------------------------------------------------------
    
Now some RPC library applications. It's not clear whether the various RPC
schemes have enough in common to warrant a common base set of behaviours (the
first of these, Yaws RPC, is just that), or whether they're all different enough
that they would share no common toolkit. 

What follows is meant to be illustrative, since I really have only limited
exposure to the fine details of the various RPC mechanisms, but just wanted to
provide at least a placeholder for the RPC folks to be more specific:

    - gen_yaws_encoder.erl

    - gen_yaws_rpc.erl 
        Implements gen_responder

- Yaws SOAP (library application)

    - yaws_soap_encoding.erl
    - yaws_soap_rpc.erl

- Yaws HAXE (library application)

    - yaws_haxe_encoding.erl
    - yaws_haxe_rpc.erl

- Yaws JSON (library application)

    - yaws_json_encoding.erl
    - yaws_json_rpc.erl



--------------------------------------------------------------------------------
Yaws Standalone (supervisor application)
--------------------------------------------------------------------------------

This is the supervisor tree application that is the runtime application known as
"yaws" today.  It will depend on and use all/most of the library applications
above and bundle them together into something that people can run as a
standalone server if they want, with multiple sconfs, virtual servers,
templating systems, web development toolkits, rpc mechanisms preinstalled, etc.

In many cases, it should be possible that the yaws 2 standalone application
below can still process yaws.conf files from the current yaws 1 release. If not,
we can envision providing some level of backwards compatibility, or migration
tools to rewrite yaws.conf to yaws 2 type conf files, or to warn about
incompatibilities.

    - yaws_config_reader.erl

        Parses config files in various formats (yaws version 1, XML, erlang
        terms, etc) and produces a yaws_conf record specified in
        yaws_config.hrl, as well as other custom configuration specific to the
        standalone server (virtual server confs, appmod setup, templating
        directories and options, etc)

    - yaws_standalone_responder

        An implementation of gen_yaws_responder that acts as the request router
        between incoming requests and the appropriate other responders, such as
        for the various RPC requests, custom appmods, the ehtml template
        compiler and responder (for .yaws files).

    - yaws_logger_default

        Default implementation of gen_yaws_logger, suitable for most common
        applications.

    - yaws_sup.erl 
        The top level supervisor for the standalone yaws

    - yaws_ctl.erl

        Provides mechanisms to interact with the running yaws application to
        update confs or tune other runtime behaviour