Re: [Erlyaws-list] streaming from external source

[Steve V. <vi...@ie...>]

Webmachine isn't a web server itself, but rather a web framework that sits
on top of a web server. The default webmachine sits on top of the mochiweb
web server, but I have a branch for it that works on top of Yaws instead:

https://github.com/vinoski/webmachine/tree/yaws

I hope to merge this into webmachine proper sometime before the end of the
year.

Webmachine works via a dispatch map, where you tell it what resources to
invoke for each given URL path your app supports. Yaws supports something
similar where you specify

dispatchmod = name_of_your_dispatch_module

in the server portion of your config, and your dispatch module implements a
dispatch/1 function that takes an #arg{} record, same as an appmod out/1
function. Your dispatch function should return done if it handles the
request, closed if it handles the request and also closes the socket, or
continue if it declines to handle the request and it wants Yaws to continue
dispatching thru its normal path.

What Yaws gives you is a full service web server, with lots of features,
flexibility, and proven production worthiness. But that doesn't mean it's
heavy, or slow, or that it scales poorly. If you don't think you need the
features Yaws provides, then by all means you're welcome to use something
that provides less, but be aware that if your app succeeds it's likely to
need more features as time goes on and you're likely to wind up needing a
number of these features anyway.

--steve


On Wed, Sep 11, 2013 at 2:05 PM, akonsu <ak...@gm...> wrote:

> Thank you, this has been a tremendous help.
>
> My app needs to respond with only current data that twitter is streaming
> to it. No historical data. I also think that I will need multiple web
> servers behind a load balancer that my clients will connect to. So it looks
> like I will need a separate process that consumes twitter data and that
> will then broadcast the filtered data to the connected clients.
>
> I am still trying to decide on the technology. Maybe I do not need Yaws
> and Webmachine will suffice? I do not need templating etc that is provided
> by yaws because this is just a service that I am writing and it won't have
> any UI. I would appreciate an advice or two in this respect...
>
> Konstantin
>
>
> 2013/9/10 Steve Vinoski <vi...@ie...>
>
>>
>>
>>
>> On Tue, Sep 10, 2013 at 10:56 PM, akonsu <ak...@gm...> wrote:
>>
>>> Hello, I am totally new to yaws, and I am looking for an advice on how
>>> to implement my application.
>>>
>>> I need to connect to the twitter's streaming API, and read the tweets
>>> non-stop from it. With a specific set of OAuth credentials, I can make only
>>> one connection to twitter because this is how their streaming API works.
>>>
>>> I need to process the incoming stream of tweets and provide the
>>> resulting stream to the clients that should connect to my application over
>>> HTTP.
>>>
>>> So I think I can write a custom application and start yaws in embedded
>>> mode from it. Or maybe writing a yapp is a better approach? But I am not
>>> even sure a yapp would work, because I need to start my twitter consumer
>>> process when the web server starts.
>>>
>>
>> Part of your application will connect to Twitter and consume the stream.
>> You could either have your own application to do this and have it embed
>> Yaws, or you could use a yapp that encapsulates your Twitter app together
>> with something to service your web clients, or you could write an appmod to
>> service your web clients and use the runmod feature of Yaws to start the
>> Twitter portion of your app. The latter two approaches are very similar,
>> and the first approach, embedding, isn't difficult at all. It really
>> depends on what your web tier looks like vs. the Twitter client app, i.e.,
>> will you need to have a number of hosts serving the web clients? If the web
>> tier is separate from your Twitter client, then I'd use separate apps and
>> just use a stand-alone (non-embedded) Yaws in the web tier. Will they
>> connect directly to your Twitter client app to get the data it's streaming
>> from Twitter, or will that app put the data into a datastore that your web
>> tier reads from, or some other approach entirely? There are a variety of
>> solutions here, and they're partly dependent on whether your web clients
>> are supposed to be able to get historical data or whether they just get
>> whatever happens to be streaming at the time they're connected.
>>
>> To handle your web clients I suggest using a Yaws appmod. It's
>> essentially just a module that you configure into Yaws at a URL path:
>>
>> http://yaws.hyber.org/appmods.yaws
>>
>> Your Twitter client portion of your app will presumably somehow buffer
>> the data it consumes, and your appmod will work with the Twitter client to
>> retrieve that data and send it back to the web clients. Your web clients
>> could access Yaws using long polling, or websockets, or even Server-Sent
>> Events (SSE) I would guess. Here's a link about general streaming with Yaws:
>>
>> http://yaws.hyber.org/stream.yaws
>>
>> And here are links about websockets and SSE:
>>
>> Websockets: http://yaws.hyber.org/websockets.yaws
>> Server-Sent Events: http://yaws.hyber.org/server_sent_events.yaws and
>> see also http://steve.vinoski.net/pdf/IC-SSE_with_Yaws.pdf
>>
>> Here's some general good advice for anyone writing web apps in Erlang:
>> http://steve.vinoski.net/pdf/IC-Erlang_Web_Process_Bottlenecks.pdf
>>
>> And this is a little older but it still might help you get a better feel
>> for developing apps with Yaws:
>> http://www.infoq.com/articles/vinoski-erlang-rest
>>
>> Any further questions, just send them here to this list.
>>
>> --steve
>>
>
>