great. thank you very much. I misunderstood webmachine. I thought that it could be used by itself to make a server, and did not require a full scale web server underneath. I thought that mochiweb is just a library of useful pieces that could be used to make a server. Definitely I do not want to develop yet another web server (pun intended).

Konstantin



2013/9/11 Steve Vinoski <vinoski@ieee.org>
Webmachine isn't a web server itself, but rather a web framework that sits on top of a web server. The default webmachine sits on top of the mochiweb web server, but I have a branch for it that works on top of Yaws instead:

https://github.com/vinoski/webmachine/tree/yaws

I hope to merge this into webmachine proper sometime before the end of the year.

Webmachine works via a dispatch map, where you tell it what resources to invoke for each given URL path your app supports. Yaws supports something similar where you specify

dispatchmod = name_of_your_dispatch_module

in the server portion of your config, and your dispatch module implements a dispatch/1 function that takes an #arg{} record, same as an appmod out/1 function. Your dispatch function should return done if it handles the request, closed if it handles the request and also closes the socket, or continue if it declines to handle the request and it wants Yaws to continue dispatching thru its normal path.

What Yaws gives you is a full service web server, with lots of features, flexibility, and proven production worthiness. But that doesn't mean it's heavy, or slow, or that it scales poorly. If you don't think you need the features Yaws provides, then by all means you're welcome to use something that provides less, but be aware that if your app succeeds it's likely to need more features as time goes on and you're likely to wind up needing a number of these features anyway.

--steve



On Wed, Sep 11, 2013 at 2:05 PM, akonsu <akonsu@gmail.com> wrote:
Thank you, this has been a tremendous help.

My app needs to respond with only current data that twitter is streaming to it. No historical data. I also think that I will need multiple web servers behind a load balancer that my clients will connect to. So it looks like I will need a separate process that consumes twitter data and that will then broadcast the filtered data to the connected clients.

I am still trying to decide on the technology. Maybe I do not need Yaws and Webmachine will suffice? I do not need templating etc that is provided by yaws because this is just a service that I am writing and it won't have any UI. I would appreciate an advice or two in this respect...

Konstantin


2013/9/10 Steve Vinoski <vinoski@ieee.org>



On Tue, Sep 10, 2013 at 10:56 PM, akonsu <akonsu@gmail.com> wrote:
Hello, I am totally new to yaws, and I am looking for an advice on how to implement my application.

I need to connect to the twitter's streaming API, and read the tweets non-stop from it. With a specific set of OAuth credentials, I can make only one connection to twitter because this is how their streaming API works.

I need to process the incoming stream of tweets and provide the resulting stream to the clients that should connect to my application over HTTP.

So I think I can write a custom application and start yaws in embedded mode from it. Or maybe writing a yapp is a better approach? But I am not even sure a yapp would work, because I need to start my twitter consumer process when the web server starts.

Part of your application will connect to Twitter and consume the stream. You could either have your own application to do this and have it embed Yaws, or you could use a yapp that encapsulates your Twitter app together with something to service your web clients, or you could write an appmod to service your web clients and use the runmod feature of Yaws to start the Twitter portion of your app. The latter two approaches are very similar, and the first approach, embedding, isn't difficult at all. It really depends on what your web tier looks like vs. the Twitter client app, i.e., will you need to have a number of hosts serving the web clients? If the web tier is separate from your Twitter client, then I'd use separate apps and just use a stand-alone (non-embedded) Yaws in the web tier. Will they connect directly to your Twitter client app to get the data it's streaming from Twitter, or will that app put the data into a datastore that your web tier reads from, or some other approach entirely? There are a variety of solutions here, and they're partly dependent on whether your web clients are supposed to be able to get historical data or whether they just get whatever happens to be streaming at the time they're connected.

To handle your web clients I suggest using a Yaws appmod. It's essentially just a module that you configure into Yaws at a URL path:


Your Twitter client portion of your app will presumably somehow buffer the data it consumes, and your appmod will work with the Twitter client to retrieve that data and send it back to the web clients. Your web clients could access Yaws using long polling, or websockets, or even Server-Sent Events (SSE) I would guess. Here's a link about general streaming with Yaws:


And here are links about websockets and SSE:


Here's some general good advice for anyone writing web apps in Erlang: http://steve.vinoski.net/pdf/IC-Erlang_Web_Process_Bottlenecks.pdf

And this is a little older but it still might help you get a better feel for developing apps with Yaws: http://www.infoq.com/articles/vinoski-erlang-rest

Any further questions, just send them here to this list.

--steve