From: Michael F. <mic...@tr...> - 2010-10-28 23:33:38
|
I am trying to send the contents of a file using the {content, MimeType, Content} return value. I am having the problem that it is adding an extra byte to the end of the data. The extra byte is hex 0A, which is a linefeed character. Anyone else encounter this? I am using the {content, MimeType, Content} method to return the file contents because I normally authenticate before sending the file data. But for testing I have reduced it down to just reading and sending the file, and I still have get the same problem. Here is the contents of my test file: <erl> out(Arg) -> F = "/home/michaelf/testfile.gz", {ok, Bin} = file:read_file(F), io:format("DEBUG size(Bin): ~p~n",[size(Bin)]), %debug %% {content, yaws_api:mime_type(F), Bin}; {content, "application/x-gzip", Bin}. </erl> The testfile.gz is exactly 100 bytes, and the debug statement confirms that Bin is size 100. But using curl to request the data with something like "curl -v http://localhost:8080/content_test.yaws" I get header values stating "Content-Length: 101", and if I download with firefox I get a file that is 101 bytes. I am running on Ubuntu 10.04 amd64 with yaws 1.84-2 from the repository. I have looked at the changelogs for up to 1.89, but did not see anything that looked like it might address this problem. Should I be using a different method to return the file data? maybe streaming? File sizes will probably never be any bigger than 10MB, so I didn't think streaming should be necessary. Thanks for any help with this, Michael -- Michael Foley TracMap NZ Ltd |
From: Steve V. <vi...@ie...> - 2010-10-29 00:18:48
|
On Thu, Oct 28, 2010 at 7:10 PM, Michael Foley <mic...@tr...> wrote: > I am trying to send the contents of a file using the {content, MimeType, > Content} return value. I am having the problem that it is adding an extra > byte to the end of the data. The extra byte is hex 0A, which is a linefeed > character. Anyone else encounter this? I don't see it with the latest bits from github. > I am using the {content, MimeType, Content} method to return the file > contents because I normally authenticate before sending the file data. But > for testing I have reduced it down to just reading and sending the file, and > I still have get the same problem. > Here is the contents of my test file: > <erl> > out(Arg) -> > F = "/home/michaelf/testfile.gz", > {ok, Bin} = file:read_file(F), > io:format("DEBUG size(Bin): ~p~n",[size(Bin)]), %debug > %% {content, yaws_api:mime_type(F), Bin}; > {content, "application/x-gzip", Bin}. > </erl> > The testfile.gz is exactly 100 bytes, and the debug statement confirms that > Bin is size 100. But using curl to request the data with something like > "curl -v http://localhost:8080/content_test.yaws" I get header values > stating "Content-Length: 101", and if I download with firefox I get a file > that is 101 bytes. > I am running on Ubuntu 10.04 amd64 with yaws 1.84-2 from the repository. I > have looked at the changelogs for up to 1.89, but did not see anything that > looked like it might address this problem. I looked through the log also and don't see anything that might have affected this. > Should I be using a different method to return the file data? maybe > streaming? File sizes will probably never be any bigger than 10MB, so I > didn't think streaming should be necessary. If the file to be returned resides under your docroot, you can instead let yaws send it back to you just like any normal file: out(Arg) -> {page, "/path/to/file"}. where the "/path/to/file" is the path portion of the file's URI, which except for the leading slash is the same as the filesystem path relative to docroot. Alternatively you could make it a relative path if you prefer. With the latest yaws bits from github, your approach and this approach both return the same content with the correct size. --steve |
From: Steve V. <vi...@ie...> - 2010-10-29 04:39:04
|
On Thu, Oct 28, 2010 at 8:18 PM, Steve Vinoski <vi...@ie...> wrote: > On Thu, Oct 28, 2010 at 7:10 PM, Michael Foley <mic...@tr...> wrote: >> I am trying to send the contents of a file using the {content, MimeType, >> Content} return value. I am having the problem that it is adding an extra >> byte to the end of the data. The extra byte is hex 0A, which is a linefeed >> character. Anyone else encounter this? > > I don't see it with the latest bits from github. Correction -- I do see the same problem. I wrote my attempt using a regular out/1 function and compiled it into a beam file, whereas Michael, you're using a .yaws file, which dawned on me when you sent me your code offline. I'm currently debugging where the extra data is coming from and will hopefully have it fixed shortly. thanks, --steve |
From: Steve V. <vi...@ie...> - 2010-10-29 04:55:23
|
On Fri, Oct 29, 2010 at 12:38 AM, Steve Vinoski <vi...@ie...> wrote: > On Thu, Oct 28, 2010 at 8:18 PM, Steve Vinoski <vi...@ie...> wrote: >> On Thu, Oct 28, 2010 at 7:10 PM, Michael Foley <mic...@tr...> wrote: >>> I am trying to send the contents of a file using the {content, MimeType, >>> Content} return value. I am having the problem that it is adding an extra >>> byte to the end of the data. The extra byte is hex 0A, which is a linefeed >>> character. Anyone else encounter this? >> >> I don't see it with the latest bits from github. > > Correction -- I do see the same problem. I wrote my attempt using a > regular out/1 function and compiled it into a beam file, whereas > Michael, you're using a .yaws file, which dawned on me when you sent > me your code offline. I'm currently debugging where the extra data is > coming from and will hopefully have it fixed shortly. OK, what's occurring is that since this is a .yaws file, the contents of that file are delivered, minus the stuff between <erl>...</erl> tags. In your case, your out/1 function first accumulates content to be delivered via the {content, ...} construct, but when Yaws strips the erl tags out of your .yaws file, it finds a "\n" at the end of the file, following the </erl> tag, so that's added to the content to be delivered as well. If you save your file minus that final newline, it'll work. Alternatively, you might consider moving this to a regular appmod and compiling it to a beam file, since that way your function determines exactly what's returned via the {content, ...} construct. --steve |
From: Michael F. <mic...@tr...> - 2010-11-18 23:24:11
|
On 29/10/10 17:55, Steve Vinoski wrote: > On Fri, Oct 29, 2010 at 12:38 AM, Steve Vinoski<vi...@ie...> wrote: >> On Thu, Oct 28, 2010 at 8:18 PM, Steve Vinoski<vi...@ie...> wrote: >>> On Thu, Oct 28, 2010 at 7:10 PM, Michael Foley<mic...@tr...> wrote: >>>> I am trying to send the contents of a file using the {content, MimeType, >>>> Content} return value. I am having the problem that it is adding an extra >>>> byte to the end of the data. The extra byte is hex 0A, which is a linefeed >>>> character. Anyone else encounter this? >>> I don't see it with the latest bits from github. >> Correction -- I do see the same problem. I wrote my attempt using a >> regular out/1 function and compiled it into a beam file, whereas >> Michael, you're using a .yaws file, which dawned on me when you sent >> me your code offline. I'm currently debugging where the extra data is >> coming from and will hopefully have it fixed shortly. > OK, what's occurring is that since this is a .yaws file, the contents > of that file are delivered, minus the stuff between<erl>...</erl> > tags. In your case, your out/1 function first accumulates content to > be delivered via the {content, ...} construct, but when Yaws strips > the erl tags out of your .yaws file, it finds a "\n" at the end of the > file, following the</erl> tag, so that's added to the content to be > delivered as well. > > If you save your file minus that final newline, it'll work. > Alternatively, you might consider moving this to a regular appmod and > compiling it to a beam file, since that way your function determines > exactly what's returned via the {content, ...} construct. > > --steve Just wanted to finally get around to responding to the list with more details about the solution that Steve was so helpful with. Contents of a file can be shipped from yaws in several ways. First, shipping the data straight from a .yaws file can be done in 3 ways: 1) return {content, MimeType, Content}. But NOT recommended since you have to configure your text editor to not append an extra newline after the last </erl> tag as Steve explained above. In emacs this is done by setting the variable "require-final-newline" to nil. I wouldn't recommend this because it depends to much on always configuring your editor correctly. 2) return {page, Filename}. Will return the contents of Filename. This only works if the data is a file and that file is under your docroot. 3) return {streamcontent, "application/octet-stream", <<>>}, and send the file from another process. This method of using streaming chunked data does not send extra newlines or other content at the end of the file after the last </erl> tag. Refer to http://yaws.hyber.org/stream.yaws for more information on streaming data. Second, as Steve suggested, the data can be shipped from an appmod module using any of these methods. This seems to be the most sensible choice for most webapps. It is the method I have decided to go with, but I thought the rest of the above information might be useful to someone else. And I just wanted to say thanks again for the quick help Steve gave me. Michael -- Michael Foley TracMap NZ Ltd |
From: Carsten S. <ca...@co...> - 2011-01-15 12:55:18
|
Am 29.10.10 06:55, schrieb Steve Vinoski: > On Fri, Oct 29, 2010 at 12:38 AM, Steve Vinoski <vi...@ie...> wrote: >> On Thu, Oct 28, 2010 at 8:18 PM, Steve Vinoski <vi...@ie...> wrote: >>> On Thu, Oct 28, 2010 at 7:10 PM, Michael Foley <mic...@tr...> wrote: >>>> I am trying to send the contents of a file using the {content, MimeType, >>>> Content} return value. I am having the problem that it is adding an extra >>>> byte to the end of the data. The extra byte is hex 0A, which is a linefeed >>>> character. Anyone else encounter this? >>> >>> I don't see it with the latest bits from github. >> >> Correction -- I do see the same problem. I wrote my attempt using a >> regular out/1 function and compiled it into a beam file, whereas >> Michael, you're using a .yaws file, which dawned on me when you sent >> me your code offline. I'm currently debugging where the extra data is >> coming from and will hopefully have it fixed shortly. > > OK, what's occurring is that since this is a .yaws file, the contents > of that file are delivered, minus the stuff between <erl>...</erl> > tags. In your case, your out/1 function first accumulates content to > be delivered via the {content, ...} construct, but when Yaws strips > the erl tags out of your .yaws file, it finds a "\n" at the end of the > file, following the </erl> tag, so that's added to the content to be > delivered as well. At least in my modified version version of 1.49 this is taken care of and this kind of white space is deleted. The code revolves around a function optimize_spec/2. I am not sure if I never contributed the code or of it got lost in the meantime, I suspect the latter. I probably should have documented it better to point out why it is desirable. Best Carsten -- Carsten Schultz (2:38, 33:47) http://carsten.codimi.de/ PGP/GPG key on the pgp.net key servers, fingerprint on my home page. |
From: Carsten S. <ca...@co...> - 2011-01-15 15:59:58
|
Am 15.01.11 13:38, schrieb Carsten Schultz: > Am 29.10.10 06:55, schrieb Steve Vinoski: >> On Fri, Oct 29, 2010 at 12:38 AM, Steve Vinoski <vi...@ie...> wrote: >>> On Thu, Oct 28, 2010 at 8:18 PM, Steve Vinoski <vi...@ie...> wrote: >>>> On Thu, Oct 28, 2010 at 7:10 PM, Michael Foley <mic...@tr...> wrote: >>>>> I am trying to send the contents of a file using the {content, MimeType, >>>>> Content} return value. I am having the problem that it is adding an extra >>>>> byte to the end of the data. The extra byte is hex 0A, which is a linefeed >>>>> character. Anyone else encounter this? >>>> >>>> I don't see it with the latest bits from github. >>> >>> Correction -- I do see the same problem. I wrote my attempt using a >>> regular out/1 function and compiled it into a beam file, whereas >>> Michael, you're using a .yaws file, which dawned on me when you sent >>> me your code offline. I'm currently debugging where the extra data is >>> coming from and will hopefully have it fixed shortly. >> >> OK, what's occurring is that since this is a .yaws file, the contents >> of that file are delivered, minus the stuff between <erl>...</erl> >> tags. In your case, your out/1 function first accumulates content to >> be delivered via the {content, ...} construct, but when Yaws strips >> the erl tags out of your .yaws file, it finds a "\n" at the end of the >> file, following the </erl> tag, so that's added to the content to be >> delivered as well. > > At least in my modified version version of 1.49 this is taken care of > and this kind of white space is deleted. The code revolves around a > function optimize_spec/2. I am not sure if I never contributed the code > or of it got lost in the meantime, I suspect the latter. I probably > should have documented it better to point out why it is desirable. > It seems that I never submitted these changes, I do not remember if there was any reason for it. Anyway, here is (the relevant part of) a diff against (I think) 1.49: *** yaws_server.erl 6 Oct 2004 19:42:36 -0000 1.256 --- yaws_server.erl 15 Jan 2011 15:55:42 -0000 *************** *** 1412,1419 **** {ok, [{errors, Errs}| Spec]} = yaws_compile:compile_file(UT#urltype.fullpath), ?Debug("Spec for file ~s is:~n~p~n",[UT#urltype.fullpath, Spec]), ! ets:insert(SC#sconf.ets, {Key, spec, Mtime, Spec, Errs}), ! deliver_dyn_file(CliSock, Spec, ARG, UT, N) end. --- 1412,1421 ---- {ok, [{errors, Errs}| Spec]} = yaws_compile:compile_file(UT#urltype.fullpath), ?Debug("Spec for file ~s is:~n~p~n",[UT#urltype.fullpath, Spec]), ! OptSpec = optimize_spec(UT, Spec), ! ?Debug("Optimized spec is:~n~p~n",[OptSpec]), ! ets:insert(SC#sconf.ets, {Key, spec, Mtime, OptSpec, Errs}), ! deliver_dyn_file(CliSock, OptSpec, ARG, UT, N) end. *************** *** 1526,1531 **** --- 1528,1535 ---- %% do the header and continue + deliver_dyn_file(CliSock, Specs=[{mod, _, _, _, _, _}], ARG, UT, N) -> + deliver_dyn_file(CliSock, undefined, {bin, undefined}, Specs, ARG, UT, N); deliver_dyn_file(CliSock, Specs, ARG, UT, N) -> Fd = ut_open(UT), Bin = ut_read(Fd), *************** *** 1696,1705 **** --- 1700,1759 ---- done_or_continue(). + %% optimize yaws specification + + optimize_spec(UT, Spec) -> + FD = ut_open(UT), + Bin = ut_read(FD), + Spec1 = del_trailing_blanks(Bin, FD, Spec), + ut_close(FD), + case lists:any(fun({data, _}) -> + true; + ({error, _, _}) -> + true; + (_) -> false end, Spec1) of + true -> + Spec1; + false -> + lists:filter(fun({mod, _, _, _, _, _}) -> + true; + (_) -> false end, Spec1) + end. + del_trailing_blanks(_Bin, _FD, []) -> + []; + del_trailing_blanks(Bin, FD, [X={data, N}|T]) -> + {Bin1, Bin2} = skip_data(Bin, FD, N), + case lists:all(fun($\s) -> true; + ($\r) -> true; + ($\t) -> true; + ($\n) -> true; + (_) -> false end, + binary_to_list(to_binary(Bin1))) of + true -> + case del_trailing_blanks(Bin2, FD, T) of + [] -> + []; + L -> [X|L] + end; + false -> + [X|del_trailing_blanks(Bin2, FD, T)] + end; + del_trailing_blanks(Bin, FD, [X={skip, N}|T]) -> + {_Bin1, Bin2} = skip_data(Bin, FD, N), + [X|del_trailing_blanks(Bin2, FD, T)]; + del_trailing_blanks(Bin, FD, [X={mod, _, _, N, _, _}|T]) -> + {_Bin1, Bin2} = skip_data(Bin, FD, N), + [X|del_trailing_blanks(Bin2, FD, T)]; + del_trailing_blanks(_Bin, _FD, L) -> + L. + + %% what about trailers ?? + skip_data(undefined, _, _) -> + {undefined, undefined}; skip_data(List, Fd, Sz) when list(List) -> skip_data(list_to_binary(List), Fd, Sz); skip_data(Bin, Fd, Sz) when binary(Bin) -> *************** *** 2596,2602 **** clear_ets(E) -> ! ets:match_delete(E, {'_', spec, '_', '_', '_'}), ets:match_delete(E, {{url, '_'}, '_', '_'}), ets:match_delete(E, {{urlc, '_'}, '_', '_'}), ets:insert(E, {num_files, 0}), --- 2650,2656 ---- clear_ets(E) -> ! %%ets:match_delete(E, {'_', spec, '_', '_', '_'}), ets:match_delete(E, {{url, '_'}, '_', '_'}), ets:match_delete(E, {{urlc, '_'}, '_', '_'}), ets:insert(E, {num_files, 0}), I am not sure if the last section is related. I did this change, because I like using .yaws files for almost everything. Maybe it is useful to someone. Best Carsten -- Carsten Schultz (2:38, 33:47) http://carsten.codimi.de/ PGP/GPG key on the pgp.net key servers, fingerprint on my home page. |
From: Steve V. <vi...@ie...> - 2011-01-17 20:22:22
|
On Sat, Jan 15, 2011 at 10:59 AM, Carsten Schultz <ca...@co...> wrote: >> >> At least in my modified version version of 1.49 this is taken care of >> and this kind of white space is deleted. The code revolves around a >> function optimize_spec/2. I am not sure if I never contributed the code >> or of it got lost in the meantime, I suspect the latter. I probably >> should have documented it better to point out why it is desirable. >> > > It seems that I never submitted these changes, I do not remember if > there was any reason for it. Anyway, here is (the relevant part of) a > diff against (I think) 1.49: Hi Carsten, I too believe that deleting whitespace from around <erl></erl> is a perfectly viable solution. I'll take look at your patch and see what might be incorporated. thanks, --steve |