On Tue, Sep 10, 2013 at 10:56 PM, akonsu <akonsu@gmail.com> wrote:
Hello, I am totally new to yaws, and I am looking for an advice on how to implement my application.

I need to connect to the twitter's streaming API, and read the tweets non-stop from it. With a specific set of OAuth credentials, I can make only one connection to twitter because this is how their streaming API works.

I need to process the incoming stream of tweets and provide the resulting stream to the clients that should connect to my application over HTTP.

So I think I can write a custom application and start yaws in embedded mode from it. Or maybe writing a yapp is a better approach? But I am not even sure a yapp would work, because I need to start my twitter consumer process when the web server starts.

Part of your application will connect to Twitter and consume the stream. You could either have your own application to do this and have it embed Yaws, or you could use a yapp that encapsulates your Twitter app together with something to service your web clients, or you could write an appmod to service your web clients and use the runmod feature of Yaws to start the Twitter portion of your app. The latter two approaches are very similar, and the first approach, embedding, isn't difficult at all. It really depends on what your web tier looks like vs. the Twitter client app, i.e., will you need to have a number of hosts serving the web clients? If the web tier is separate from your Twitter client, then I'd use separate apps and just use a stand-alone (non-embedded) Yaws in the web tier. Will they connect directly to your Twitter client app to get the data it's streaming from Twitter, or will that app put the data into a datastore that your web tier reads from, or some other approach entirely? There are a variety of solutions here, and they're partly dependent on whether your web clients are supposed to be able to get historical data or whether they just get whatever happens to be streaming at the time they're connected.

To handle your web clients I suggest using a Yaws appmod. It's essentially just a module that you configure into Yaws at a URL path:

http://yaws.hyber.org/appmods.yaws

Your Twitter client portion of your app will presumably somehow buffer the data it consumes, and your appmod will work with the Twitter client to retrieve that data and send it back to the web clients. Your web clients could access Yaws using long polling, or websockets, or even Server-Sent Events (SSE) I would guess. Here's a link about general streaming with Yaws:

http://yaws.hyber.org/stream.yaws

And here are links about websockets and SSE:

Websockets: http://yaws.hyber.org/websockets.yaws
Server-Sent Events: http://yaws.hyber.org/server_sent_events.yaws and see also http://steve.vinoski.net/pdf/IC-SSE_with_Yaws.pdf

Here's some general good advice for anyone writing web apps in Erlang: http://steve.vinoski.net/pdf/IC-Erlang_Web_Process_Bottlenecks.pdf

And this is a little older but it still might help you get a better feel for developing apps with Yaws: http://www.infoq.com/articles/vinoski-erlang-rest

Any further questions, just send them here to this list.

--steve