yaws

after some research i found out, that longer uploads are since several
versions not possible after an change, to avoid dos attacks with the file
upload.
Is there anything planed, to fix this, that also longer uploads are
possible?
Or could someone show me the place, where this change were, to see, if i
could make an fix by myself?

Daniel Fahlke wrote:
> after some research i found out, that longer uploads are since several 
> versions not possible after an change, to avoid dos attacks with the 
> file upload.

Which research, I don't recall this. Please remind me.

/klacke

*search trough the logs of the last days*

First this Bugfix.
It was in the News (Sat Dec 11 2004 Version 1.50 released)

"Form post parameter was still always managed as atoms. This is a backwards
incompatible change. It broke the wiki aswell as the upload example in the
Yaws docs.However, the change is sound since it was easy to DOS a yaws
server by sending file upload posts with new atoms. Eventually the atom
table would overflow. However it does break code !!! (mikl) "

couldnt find now the place, that directed me to this bugfix, but here a
report of this bug, from 2006
http://osdir.com/ml/web.server.yaws.general/2006-07/msg00035.html

Daniel Fahlke wrote:
> *search trough the logs of the last days*
> 
> First this Bugfix.
> It was in the News (Sat Dec 11 2004 Version 1.50 released)
> 
> "Form post parameter was still always managed as atoms. This is a 
> backwards incompatible change. It broke the wiki aswell as the upload 
> example in the Yaws docs.However, the change is sound since it was easy 
> to DOS a yaws server by sending file upload posts with new atoms. 
> Eventually the atom table would overflow. However it does break code !!! 


Ok, looong ago.

That fix didn't have anything todo whatsoever with the size of the
uploaded file.  The problem was that the POST parse code did list_to_atom/1
while parsing. This meant that a malicious user could POST a long series
of wacko POST requests where each POST request potentially created
a set of new e.g. random atoms - that way filling up the erl atom table
which is of fixed size - thus eventually making erl die.


/klacke

2010/3/2 Claes Wikström <klacke@...>

> Daniel Fahlke wrote:
>
>> *search trough the logs of the last days*
>>
>> First this Bugfix.
>> It was in the News (Sat Dec 11 2004 Version 1.50 released)
>>
>> "Form post parameter was still always managed as atoms. This is a
>> backwards incompatible change. It broke the wiki aswell as the upload
>> example in the Yaws docs.However, the change is sound since it was easy to
>> DOS a yaws server by sending file upload posts with new atoms. Eventually
>> the atom table would overflow. However it does break code !!!
>>
>
>
> Ok, looong ago.
>
> That fix didn't have anything todo whatsoever with the size of the
> uploaded file.  The problem was that the POST parse code did list_to_atom/1
> while parsing. This meant that a malicious user could POST a long series
> of wacko POST requests where each POST request potentially created
> a set of new e.g. random atoms - that way filling up the erl atom table
> which is of fixed size - thus eventually making erl die.
>
>
> /klacke
>

than i was wrong, but why and since when does the upload no bigger( or only
longer?) uploads? And where is the cause for that?
I would help with that, if i know, where to search =(

I use yaws to upload large files (up to 2 gb) and it works fine (1.6x ->
1.80 versions), so I guess not a server is an issue. It also behaves well on
low bandwidth networks where upload time is quite long.

Make sure that in multipart request you do not try send more than 2 gb. All
browsers silently crash in this case. Here is example issue report for
Mozilla https://bugzilla.mozilla.org/show_bug.cgi?id=383446

On 2 March 2010 09:55, Daniel Fahlke <flyingmana@...> wrote:

> 2010/3/2 Claes Wikström <klacke@...>
>
> Daniel Fahlke wrote:
>>
>>> *search trough the logs of the last days*
>>>
>>> First this Bugfix.
>>> It was in the News (Sat Dec 11 2004 Version 1.50 released)
>>>
>>> "Form post parameter was still always managed as atoms. This is a
>>> backwards incompatible change. It broke the wiki aswell as the upload
>>> example in the Yaws docs.However, the change is sound since it was easy to
>>> DOS a yaws server by sending file upload posts with new atoms. Eventually
>>> the atom table would overflow. However it does break code !!!
>>>
>>
>>
>> Ok, looong ago.
>>
>> That fix didn't have anything todo whatsoever with the size of the
>> uploaded file.  The problem was that the POST parse code did
>> list_to_atom/1
>> while parsing. This meant that a malicious user could POST a long series
>> of wacko POST requests where each POST request potentially created
>> a set of new e.g. random atoms - that way filling up the erl atom table
>> which is of fixed size - thus eventually making erl die.
>>
>>
>> /klacke
>>
>
> than i was wrong, but why and since when does the upload no bigger( or only
> longer?) uploads? And where is the cause for that?
> I would help with that, if i know, where to search =(
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Erlyaws-list mailing list
> Erlyaws-list@...
> https://lists.sourceforge.net/lists/listinfo/erlyaws-list
>
>

2010/3/2 Michal Zajda <zajdamichal@...>

> I use yaws to upload large files (up to 2 gb) and it works fine (1.6x ->
> 1.80 versions), so I guess not a server is an issue. It also behaves well on
> low bandwidth networks where upload time is quite long.
>
> Make sure that in multipart request you do not try send more than 2 gb. All
> browsers silently crash in this case. Here is example issue report for
> Mozilla https://bugzilla.mozilla.org/show_bug.cgi?id=383446
>
>
Was that on a local machine?

for example http://yaws.hyber.org/upload0.yaws
i tried an 4,9 MB file and get an "The connection was reset" after some
time.
I use Yaws version 1.87

after some tries the biggest file i could upload was 3,1 MB big.

>
> Was that on a local machine?
>
>
tested on local and remote machines.



> for example http://yaws.hyber.org/upload0.yaws
> i tried an 4,9 MB file and get an "The connection was reset" after some
> time.
> I use Yaws version 1.87
>
> after some tries the biggest file i could upload was 3,1 MB big.
>
>
well, the limit 3.~ MB may be set to avoid DOS. So use only your own
environment to test uploads.





>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Erlyaws-list mailing list
> Erlyaws-list@...
> https://lists.sourceforge.net/lists/listinfo/erlyaws-list
>
>

Daniel Fahlke wrote:

> 
> for example http://yaws.hyber.org/upload0.yaws
> i tried an 4,9 MB file and get an "The connection was reset" after some 
> time.
> I use Yaws version 1.87
> 
> after some tries the biggest file i could upload was 3,1 MB big.

Well now that is a bug report I can relate to, I can reproduce.
I'll have a look at it ASAP.

My first look at the code triggered a giggle though, the Yaws upload test
code actually stores all files in

-define(DIR, "/tmp/YawsTestUploads/").

So, I see:

# ls -lat /tmp/YawsTestUploads/

-rw-r--r-- 1 root root 3051969 Mar  2 12:14 Secret_of_Mana_Theme_-_Anime_Techno_Remix.mp3
-rw-r--r-- 1 root root 3367947 Mar  2 12:13 Princess_Mononoke_Theme_Song-Full_Length_English_Mix_version.mp3
-rw-r--r-- 1 root root 3018533 Mar  2 12:11 Eisblume_-_Leben_ist_sch_n.mp3
-rw-r--r-- 1 root root 2843408 Mar  2 12:11 Ashley_Tisdale_-_Its_Alright__OK_w_lyrics.mp3
......

:-)

You might want do something about that music taste :-)


/klacke

2010/3/2 Claes Wikström <klacke@...>

> Daniel Fahlke wrote:
>
>
>> for example http://yaws.hyber.org/upload0.yaws
>> i tried an 4,9 MB file and get an "The connection was reset" after some
>> time.
>> I use Yaws version 1.87
>>
>> after some tries the biggest file i could upload was 3,1 MB big.
>>
>
> Well now that is a bug report I can relate to, I can reproduce.
> I'll have a look at it ASAP.
>
> My first look at the code triggered a giggle though, the Yaws upload test
> code actually stores all files in
>
> -define(DIR, "/tmp/YawsTestUploads/").
>
> So, I see:
>
> # ls -lat /tmp/YawsTestUploads/
>
> -rw-r--r-- 1 root root 3051969 Mar  2 12:14
> Secret_of_Mana_Theme_-_Anime_Techno_Remix.mp3
> -rw-r--r-- 1 root root 3367947 Mar  2 12:13
> Princess_Mononoke_Theme_Song-Full_Length_English_Mix_version.mp3
> -rw-r--r-- 1 root root 3018533 Mar  2 12:11 Eisblume_-_Leben_ist_sch_n.mp3
> -rw-r--r-- 1 root root 2843408 Mar  2 12:11
> Ashley_Tisdale_-_Its_Alright__OK_w_lyrics.mp3
> ......
>
> :-)
>
> You might want do something about that music taste :-)
>
>
> /klacke
>
>
> hehe, not all musik was for me(anyway,I have no music taste ), but thats
not the point^^
was simpler to test with this music-files, because there size was in the
right range.

That there must an tmp dir dont suprise me, there is most  a tmp dir for
fileuploads^^
maybe you should mention, that there will be a temporary copy.

And thank you for your time

Daniel Fahlke wrote:

> And thank you for your time
> 


I actually got pretty worried here, there is a config
parameter

       partial_post_size = Integer
               When  a  yaws  file  receives  large  POSTs,  the amount of data
               received in each chunk is determined by the this parameter.  The
               deafult value is 10240.

I found two errors in the yaws code,

The man page is wrong, there is an additional value, which erronously
was the default value - namely nolimit

So I've just changed the default value and corrected the man page. Your
remedy here is to explicitly set the partial_post_size to e.g 10240 in
your yaws.conf file and you'll be all set.

Thanks for reporting

/klacke

heh, this is it. I had it in my yaws.conf (partial_post_size=65536), thats
way I could not observe the bug.

2010/3/2 Claes Wikström <klacke@...>

> Daniel Fahlke wrote:
>
> > And thank you for your time
> >
>
>
> I actually got pretty worried here, there is a config
> parameter
>
>       partial_post_size = Integer
>               When  a  yaws  file  receives  large  POSTs,  the amount of
> data
>               received in each chunk is determined by the this parameter.
>  The
>               deafult value is 10240.
>
> I found two errors in the yaws code,
>
> The man page is wrong, there is an additional value, which erronously
> was the default value - namely nolimit
>
> So I've just changed the default value and corrected the man page. Your
> remedy here is to explicitly set the partial_post_size to e.g 10240 in
> your yaws.conf file and you'll be all set.
>
> Thanks for reporting
>
> /klacke
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Erlyaws-list mailing list
> Erlyaws-list@...
> https://lists.sourceforge.net/lists/listinfo/erlyaws-list
>

On Mon, Mar 1, 2010 at 6:09 AM, fruity <fruity@...> wrote:

> Hello to all,
>
> Sorry me for I have sinned. I've used the latest yaws in production
> environment and...
>

When you say "latest yaws" do you mean the latest released version, or the
latest version of the sources from github?


> anytime anyone open my django powered website for the first time they see:
>
> CGI failure: {send_to_application_server_failed,timeout}
>
> then if the page is refreshed it works. For me that means that everytime
> somebody come to the website the first time they get prompted with this
> error...
>

Unfortunately it looks like the code has a bug where it uses
"send_to_application_server_failed" to describe errors from both sending and
receiving, which means we can't completely nail down the cause here, but
note that I recently changed the send timeout from 1 second to 10 seconds in
the github repo. If you're using the released version, perhaps you're still
using the 1 second timeout.

What's really needed here is a configurable timeout, of course.


> I will rollback to apache for now and keep yaws only on my testing
> stations.
>

I'm sure we can get it working for you if you'd like.

--steve

On Tue, Feb 23, 2010 at 7:53 PM, Hank Knight <hknight555@...> wrote:

> I want to return a plain text 404 result.
>
> This sends status 200 not 404:
>
> <erl>
> out(Arg) ->
>        {header, "HTTP/1.1 404 Not Found"},
>        {status, "404"},
>        {content, "text/plain", ""}.
> </erl>
> Not Found
>

You don't need the first line, status takes an integer, not a list/string,
and you're missing grouping your tuples into a list. You want:

<erl>
out(_) ->
       [{status, 404},
        {content, "text/plain", ""}].
</erl>

--steve

On Tue, Feb 23, 2010 at 7:57 AM, Hank Knight <hknight555@...> wrote:

> The following Yaws code returns results to the web.  Instead, I want
> it to output the results to a text file named "output.txt"
>
> <erl>
> out(A) ->
>    Hdrs = A#arg.headers,
>    Other = Hdrs#headers.other,
>    {ehtml,
>       [
>        {item, [],
>         ["Host: ",
>          {span, [], Hdrs#headers.host}]},
>        {item, [],
>         ["IP Address: ",
>          {span, [], f("~p", [inet:peername(A#arg.clisock)])}]},
>        {item, [],
>         ["Your user agent is: ",
>          {span, [], Hdrs#headers.user_agent}]},
>        {item, [],
>         ["Accept-Charset: ",
>          {span, [],
>           f("~p", [yaws_api:parse_query(A)]) }]}
>        ]}.
> </erl>
>

I think the code below is what you want, adjust the output file pathname to
taste.

--steve

<erl>
out(A) ->
    Hdrs = A#arg.headers,
    Other = Hdrs#headers.other,
    Reply = [{item, [],
              ["Host: ",
               {span, [], Hdrs#headers.host}]},
             {item, [],
              ["IP Address: ",
               {span, [], f("~p", [inet:peername(A#arg.clisock)])}]},
             {item, [],
              ["Your user agent is: ",
               {span, [], Hdrs#headers.user_agent}]},
             {item, [],
              ["Accept-Charset: ",
               {span, [],
                f("~p", [yaws_api:parse_query(A)])}]}
            ],
    Html = yaws_api:ehtml_expand(Reply),
    ok = file:write_file("/tmp/output.txt", Html),
    {ehtml, Reply}.
</erl>

On Sun, Feb 21, 2010 at 9:03 PM, Hank Knight <hknight555@...> wrote:
> I need to analyse web traffic that passes through a Yaws server.  I do
> not want to use Google Analytics or similar third-party solutions that
> are hosted elsewhere.
>
> Piwik has the functionality I need however it is written in PHP and
> uses a MySQL database with many ugly joins that interfere with
> performance.

Should have given a look at Piwik before answering :(

So no, not a solution like this one I'm aware of

-- 
Olivier

Hank Knight wrote:
> I need to analyse web traffic that passes through a Yaws server.  I do
> not want to use Google Analytics or similar third-party solutions that
> are hosted elsewhere.

There are many many tools out there that work by just reading the logs.
Yaws logs in what is known as Common Log Format which I guess means
apache log format.
I've been using webalyzer earlier on the yaws logs.

/klacke