cppcms-users — Disussions of users of CppCMS

[Cppcms-users] access JSON array through path

[miniME <min...@go...>]

Hey,

is it possible to specify the path to a specific array value?
(see the "cppcms::json::value::get" method)

consider the following JSON:

{
    "path": {
         "some_array": [
             "val1",
             "val2",
              [
                     "nested_array1_val1",
                     "nested_array1_val2",
                     {
                           "deeply_nested" : [
                                   "foo"
                            ]
                     }
              ]
         ]
     }
}

I would expect to use the path including array values as following:

"path.some_array[0]"  --> "val1"
"path.some_array[1]"  --> "val2"
"path.some_array[2]"  --> array

even nested:
"path.some_array[2][1]"  --> "nested_array1_val2"
"path.some_array[2][2].deeply_nested[0]"  --> "foo"

It is not documented and doesn't work like this. *Is there some easy way 
to achive this?*

Re: [Cppcms-users] CppCMS 1.0.5 Released

[Artyom B. <art...@ya...>]

http_timeouts_test_write frequently fails - on *BSD almost always due to some glitches in tcp/ip stack. For example in FreeBSD if you write to socket lots of data and get at somepoint that it would block because client does not read, than on BSD the socketbecomes writeable after 5 sec and allows you to push some more data even whennothing had changed.
So timeouts test fail quite oftern. AFAIR Mac OS X is prone to these errors as well.
It does not cause a problem it just makes problem to test to complete successfully.
About test_locale_codepage, AFAIR it is broken/missing shift-jis support fromlibc++ (it fails in std locale backed) - so you'll have some problemswith non-utf-8 Japanese or Chinese locales in some cases,but the default backend should be posix if you don't build with icu(which is your case)
So in general I don't think there are any real problems with these tests failing. 
Artyom Beilis
--------------
CppCMS - C++ Web Framework:   http://cppcms.com/
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
 
      From: Vizcayno <viz...@gm...>
 To: cpp...@li... 
 Sent: Tuesday, November 4, 2014 2:59 AM
 Subject: Re: [Cppcms-users] CppCMS 1.0.5 Released
   
Thanks for the new version, some hours ago I finished the installation of this 
one on an macbook pro with OSX10.10 (Yosemite) and the c++ compiler:
Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.0.0
Thread model: posix

The only objects that I downloaded from internet were cmake and pcre, the
remaining objects in the mandatory requirements are included in the 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/
SDKs/Mac OSX10.10.sdk

After building cppcms and executing the “make test” the next failed tests were
generated: 
23:http_timeouts_test_write
75:test_locale_codepage

However, after installation of cppcms, I could compile, link and execute the
first example program “hello” without problems.

My question is if the two failed tests are important (see details below) and
how I can fix them?
My best regards.
Viz
 
=============================================================
Description of error 23 is:
23/88 Testing: http_timeouts_test_write
23/88 Test: http_timeouts_test_write
Command: "/Users/viz/cppcms/build/http_timeouts_test" "-c" 
"/Users/viz/cppcms/tests/proto_test.js" "--service-api=http"
"--service-port=8080" "--service- ip=127.0.0.1" "--http-timeout=5"
"--logging-level=debug" "--test-write=true" "--test- exec=/usr/bin/python
/Users/viz/cppcms/tests/http_timeouts_test.py write"
Directory: /Users/viz/cppcms/build
"http_timeouts_test_write" start time: Nov 03 17:31 BOT
Output:
----------------------------------------------------------
2014-11-03 17:31:05; cppcms_http, info: GET /async/long (http_api.cpp:251)
  File "/Users/viz/cppcms/tests/http_timeouts_test.py", line 72, in <module>
    test_unfinished_read('GET /async/long HTTP/1.0',0,0)
  File "/Users/viz/cppcms/tests/http_timeouts_test.py", line 65, in
test_unfinished_read
    test(n < read_size * 16 or n < 1000000)
  File "/Users/viz/cppcms/tests/http_timeouts_test.py", line 16, in test
    traceback.print_stack()
Write to the client timeout
Tesing GET /async/long HTTP/1.0 with 0 reads
SO_RCVBUF=408300
Error
Statistics:
 sync_bad_count =0
 async_bad_count =0
 count_timeouts =0
 above 15 =0
 below 10 =0
 eof =0
Failed
<end of output>
Test time =  14.21 sec
----------------------------------------------------------
Test Failed.
"http_timeouts_test_write" end time: Nov 03 17:31 BOT
"http_timeouts_test_write" time elapsed: 00:00:14


=============================================================
Description of error 75 is:
75/88 Testing: test_locale_codepage
75/88 Test: test_locale_codepage
Command: "/Users/viz/cppcms/build/booster/test_locale_codepage"
Directory: /Users/viz/cppcms/build/booster
"test_locale_codepage" start time: Nov 03 17:34 BOT
Output:
----------------------------------------------------------
- Testing correct invalid bytes skipping
-- ISO-8859-8
-- cp932
Testing for backend std
no iso locales availible, passing
Testing wide I/O
  wchar_t
    UTF-8
    Testing codepoints above 0xFFFF
      Single U+2008A
      Single U+2008A withing text
      U+2008A x 1000
    Shift-JIS
Error in line:67 read_file<Char>(f1) == cmp
Error in line:76 read_file<char>(testo) == file
Testing charset to/from UTF conversion functions
  char
  wchar_t
Testing utf_to_utf
  char<-char
  char<-wchar
  wchar<-char
  wchar<-wchar
Testing for backend posix
Testing wide I/O
  wchar_t
    UTF-8
    Testing codepoints above 0xFFFF
      Single U+2008A
      Single U+2008A withing text
      U+2008A x 1000
    Shift-JIS
Testing charset to/from UTF conversion functions
  char
  wchar_t
Testing utf_to_utf
  char<-char
  char<-wchar
  wchar<-char
  wchar<-wchar

Passed 362 tests
Failed 2 tests
  99.5% of tests completed sucsessefully
<end of output>
Test time =  0.04 sec
----------------------------------------------------------
Test Failed.
"test_locale_codepage" end time: Nov 03 17:34 BOT
"test_locale_codepage" time elapsed: 00:00:00
----------------------------------------------------------




------------------------------------------------------------------------------
_______________________________________________
Cppcms-users mailing list
Cpp...@li...
https://lists.sourceforge.net/lists/listinfo/cppcms-users


   

Re: [Cppcms-users] CppCMS 1.0.5 Released

[Vizcayno <viz...@gm...>]

Thanks for the new version, some hours ago I finished the installation of this 
one on an macbook pro with OSX10.10 (Yosemite) and the c++ compiler:
Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.0.0
Thread model: posix

The only objects that I downloaded from internet were cmake and pcre, the
remaining objects in the mandatory requirements are included in the 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/
SDKs/Mac OSX10.10.sdk

After building cppcms and executing the “make test” the next failed tests were
generated: 
23:http_timeouts_test_write
75:test_locale_codepage

However, after installation of cppcms, I could compile, link and execute the
first example program “hello” without problems.

My question is if the two failed tests are important (see details below) and
how I can fix them?
My best regards.
Viz
 
=============================================================
Description of error 23 is:
23/88 Testing: http_timeouts_test_write
23/88 Test: http_timeouts_test_write
Command: "/Users/viz/cppcms/build/http_timeouts_test" "-c" 
"/Users/viz/cppcms/tests/proto_test.js" "--service-api=http"
"--service-port=8080" "--service- ip=127.0.0.1" "--http-timeout=5"
"--logging-level=debug" "--test-write=true" "--test- exec=/usr/bin/python
/Users/viz/cppcms/tests/http_timeouts_test.py write"
Directory: /Users/viz/cppcms/build
"http_timeouts_test_write" start time: Nov 03 17:31 BOT
Output:
----------------------------------------------------------
2014-11-03 17:31:05; cppcms_http, info: GET /async/long (http_api.cpp:251)
  File "/Users/viz/cppcms/tests/http_timeouts_test.py", line 72, in <module>
    test_unfinished_read('GET /async/long HTTP/1.0',0,0)
  File "/Users/viz/cppcms/tests/http_timeouts_test.py", line 65, in
test_unfinished_read
    test(n < read_size * 16 or n < 1000000)
  File "/Users/viz/cppcms/tests/http_timeouts_test.py", line 16, in test
    traceback.print_stack()
Write to the client timeout
Tesing GET /async/long HTTP/1.0 with 0 reads
SO_RCVBUF=408300
Error
Statistics:
 sync_bad_count =0
 async_bad_count =0
 count_timeouts =0
 above 15 =0
 below 10 =0
 eof =0
Failed
<end of output>
Test time =  14.21 sec
----------------------------------------------------------
Test Failed.
"http_timeouts_test_write" end time: Nov 03 17:31 BOT
"http_timeouts_test_write" time elapsed: 00:00:14


=============================================================
Description of error 75 is:
75/88 Testing: test_locale_codepage
75/88 Test: test_locale_codepage
Command: "/Users/viz/cppcms/build/booster/test_locale_codepage"
Directory: /Users/viz/cppcms/build/booster
"test_locale_codepage" start time: Nov 03 17:34 BOT
Output:
----------------------------------------------------------
- Testing correct invalid bytes skipping
-- ISO-8859-8
-- cp932
Testing for backend std
no iso locales availible, passing
Testing wide I/O
  wchar_t
    UTF-8
    Testing codepoints above 0xFFFF
      Single U+2008A
      Single U+2008A withing text
      U+2008A x 1000
    Shift-JIS
Error in line:67 read_file<Char>(f1) == cmp
Error in line:76 read_file<char>(testo) == file
Testing charset to/from UTF conversion functions
  char
  wchar_t
Testing utf_to_utf
  char<-char
  char<-wchar
  wchar<-char
  wchar<-wchar
Testing for backend posix
Testing wide I/O
  wchar_t
    UTF-8
    Testing codepoints above 0xFFFF
      Single U+2008A
      Single U+2008A withing text
      U+2008A x 1000
    Shift-JIS
Testing charset to/from UTF conversion functions
  char
  wchar_t
Testing utf_to_utf
  char<-char
  char<-wchar
  wchar<-char
  wchar<-wchar

Passed 362 tests
Failed 2 tests
  99.5% of tests completed sucsessefully
<end of output>
Test time =   0.04 sec
----------------------------------------------------------
Test Failed.
"test_locale_codepage" end time: Nov 03 17:34 BOT
"test_locale_codepage" time elapsed: 00:00:00
----------------------------------------------------------

[Cppcms-users] Debian & Ubuntu Repositories for 1.0.5

[Artyom B. <art...@ya...>]

FYI,
Debian & Ubuntu Repositories for 1.0.5 are ready Artyom Beilis
--------------
CppCMS - C++ Web Framework:   http://cppcms.com/
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/

[Cppcms-users] CppCMS 1.0.5 Released

[Artyom B. <art...@ya...>]

Hello CppCMS Users,
CppCMS 1.0.5 was released and available for download.
Bug Fixes:
   
   - Fixed 121, 98 - bug caused invalid year formatting/parsing by icu backend - fixed incorrect use of year of the week instead year
   - Fixed 122 - memory leak in Win32 threading library
   - Fixed 105 - string_key.h has a bad operator '!='
   - Fixed 119 - bad html formatting.
   - Fixed 106 - IPv6 support on Winows
   - Fixed 129 - cppcms_make_key - invalid option name
   - Fixed 97 - impossible to use upper case in namespace in <% include %>
   - Fixed 84 - 64K fd limit
   - Fixed 108 - test_locale_boundary & booster_locale_formatting failure
   - Fixed various issues libc++/clang support
   - Significantly improved multipart parsing closing f.r. 27
   - Removed reuse_address socket option in Windows
Minor Security Improvements:   
   - Issue 117: possibility of Timing Attack Vulnerability
Platform Support:   
   - NetBSD is supported platform
   - FreeBSD added support of POSIX locale & clang/libc++

RPM repositories at Open Build Service are already ready, Debian packages for Ubuntu and Debian would be published soon. 

Artyom Beilis
--------------
CppCMS - C++ Web Framework:   http://cppcms.com/
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/

Re: [Cppcms-users] Getting ready to 1.0.5

[Artyom B. <art...@ya...>]

> 

>>  BUT... I discovered a huge issue with TLS implementation
>>  in the pthreads library that requires workaround.
>> 
>>  If you run pthread_key_create more than 255 times
>>  (even if you call pthread_key_delete and release the keys)
>>  the key creation.
>>  256  is the total number of TLS keys allowed at NetBSD. 
>> 
>> 
>>  However pthread_key_delete does not "free" the spot - it
>>  caused all ICU tests that use TLS for optimization
>>  fail and it may cause other problems that aren't
>>  visible right now.
> 
> That one has been addressed after NetBSD 6 has been branches. The core
> of the issue is that pthread_key_destroy was added as an afterthought
> with some non-trivial implications for how it behaves on existing
> threads.

Destructors are not called if data still exists, use of the key
afterwards leads to undefined behavior.

It is very well defined practically defining the implementation.

  http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_delete.html

All implementation I'm aware of follow this spec.



> Please avoid the term TLS when talking about pthread_key -- it
> is normally used for __thread and the related C++11 feature.
> 

AFAIK Thread local storage is still called TLS regardless implementaion: 
using API like POSIX  pthread_key_create or WinAPI TlsAlloc or
using language based features.

> Joerg
> 

In any case it implies some limits on use of CppCMS in NetBSD other than that
I don't see any other problems. I commited latest fixes so you are welcome to
take a look on them. 



> ------------------------------------------------------------------------------
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 

Re: [Cppcms-users] Getting ready to 1.0.5

[Joerg S. <jo...@br...>]

On Sun, Oct 26, 2014 at 01:25:45PM +0000, Artyom Beilis wrote:
> BUT... I discovered a huge issue with TLS implementation
> in the pthreads library that requires workaround.
> 
> If you run pthread_key_create more than 255 times
> (even if you call pthread_key_delete and release the keys)
> the key creation.
> 256  is the total number of TLS keys allowed at NetBSD. 
> 
> 
> However pthread_key_delete does not "free" the spot - it
> caused all ICU tests that use TLS for optimization
> fail and it may cause other problems that aren't
> visible right now.

That one has been addressed after NetBSD 6 has been branches. The core
of the issue is that pthread_key_destroy was added as an afterthought
with some non-trivial implications for how it behaves on existing
threads. Please avoid the term TLS when talking about pthread_key -- it
is normally used for __thread and the related C++11 feature.

Joerg

Re: [Cppcms-users] Getting ready to 1.0.5

[Artyom B. <art...@ya...>]

Ok...

II installed NetBSD 6.1.5 on VBox and run the tests.


1st I didn't see clang but rather gcc is used systemwise
so I tested with gcc (not really big difference I think)

I did several fixed and adaptations for NetBSD and no
problem with internal_forwarder_test and proto_* test werevisible


BUT... I discovered a huge issue with TLS implementation
in the pthreads library that requires workaround.

If you run pthread_key_create more than 255 times
(even if you call pthread_key_delete and release the keys)
the key creation.
256  is the total number of TLS keys allowed at NetBSD. 


However pthread_key_delete does not "free" the spot - it
caused all ICU tests that use TLS for optimization
fail and it may cause other problems that aren't
visible right now.

Bottom line - TLS implementation requires workaround or
fix in standard library for NetBSD to be used safely with
CppCMS.


I'll commit various fixes for NetBSD for 1.0.5 but I need
to think whether to implement TLS workaround because
it is a clear pthreads library bug that MUST be solved
by NetBSD developers.


Also service_test fix does not work as it looks like 
under NetBSD killall behaves differently.


Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:  http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/

Re: [Cppcms-users] Getting ready to 1.0.5

[Joerg S. <jo...@br...>]

On Wed, Oct 22, 2014 at 06:03:03AM -0700, Artyom Beilis wrote:
>         19 - internal_forwarder_test (OTHER_FAULT)
>         46 - proto_test_async_fastcgi (Failed)

This two are spurious failures. Small patch to avoid the bash depency in
test/ is attached.

Joerg

Re: [Cppcms-users] Getting ready to 1.0.5

[Artyom B. <art...@ya...>]

Ok...

Now I can work

Ok it is a bug I fixed it in: cs2274
         3 - storage_test (Failed)

Check if the prort 8081 is avalible?
        19 - internal_forwarder_test (OTHER_FAULT)

I have same problem on FreeBSD... it may be actually some OS problem
I don't think it is a bug
        23 - http_timeouts_test_write (Failed) -

interesting something to do with realpath and symlinks need to debug... 
it fails on symlink handling
Can you add debug outputs to file src/internal_file_server.cpp
around lines 192 - pelase print what realpath returns

        24 - file_server_test (Failed)
        25 - file_server_with_listing_test (Failed)

No bash... Check if possible to change to /bin/sh and make it work.

        34 - cppcms_service_test (Failed)

Ok looks like problem to look into... but it isn't trivial - lets see if other get solved
        46 - proto_test_async_fastcgi (Failed)

booster/lib/thread/src/pthread.cpp lines 155 & 175 - try to change to that it would handle it like on Darwin
if works give a patch
        59 - test_thread_thread (Timeout)


Locale: try to install ICU :-)

Ok probably due to iconv... not really sure
        75 - test_locale_codepage (Failed)

this is due to iconv problematic work try to build with icu but in general it works fine
        76 - test_locale_message (Failed)

Need to see what libc++ uses for localization in general not a big problem.
        89 - test_locale_std_formatting (Failed)
        90 - test_locale_std_collate (Failed)
 
Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:   http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/


>________________________________
> From: Joerg Sonnenberger <jo...@br...>
>To: Artyom Beilis <art...@ya...>; cpp...@li... 
>Sent: Wednesday, October 22, 2014 2:43 PM
>Subject: Re: [Cppcms-users] Getting ready to 1.0.5
> 
>
>On Wed, Oct 22, 2014 at 11:21:49AM +0000, Artyom Beilis wrote:
>> No I didn't commit cppcms_run changes yet.
>> Locale tests... they are just mostly tricky. If you use ICU most probably you wouldn't have any issues in reality.
>> Can you send the full Testing/LatestTest.log under build directory?
>> The rest are much more disturbing. There may be some problems with old libc & pthreads model or some non-standard stuff.
>
>Attached is the log.
>
>> May I just ask from curiosity why NetBSD?
>> According to the changelog it looks like very outdated system. Only recently implemented TLS. Artyom Beilis
>
>So does Windows. TLS in any form has been standardised only very
>
>recently...
>
>Joerg
>
>
>

Re: [Cppcms-users] Getting ready to 1.0.5

[Joerg S. <jo...@br...>]

On Wed, Oct 22, 2014 at 11:21:49AM +0000, Artyom Beilis wrote:
> No I didn't commit cppcms_run changes yet.
> Locale tests... they are just mostly tricky. If you use ICU most probably you wouldn't have any issues in reality.
> Can you send the full Testing/LatestTest.log under build directory?
> The rest are much more disturbing. There may be some problems with old libc & pthreads model or some non-standard stuff.

Attached is the log.

> May I just ask from curiosity why NetBSD?
> According to the changelog it looks like very outdated system. Only recently implemented TLS. Artyom Beilis

So does Windows. TLS in any form has been standardised only very
recently...

Joerg

Re: [Cppcms-users] Getting ready to 1.0.5

[Artyom B. <art...@ya...>]

No I didn't commit cppcms_run changes yet.
Locale tests... they are just mostly tricky. If you use ICU most probably you wouldn't have any issues in reality.
Can you send the full Testing/LatestTest.log under build directory?
The rest are much more disturbing. There may be some problems with old libc & pthreads model or some non-standard stuff.
May I just ask from curiosity why NetBSD?
According to the changelog it looks like very outdated system. Only recently implemented TLS. Artyom Beilis
--------------
CppCMS - C++ Web Framework:   http://cppcms.com/
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
 
      From: Joerg Sonnenberger <jo...@br...>
 To: cpp...@li... 
 Sent: Wednesday, October 22, 2014 12:31 PM
 Subject: Re: [Cppcms-users] Getting ready to 1.0.5
   
On Sun, Oct 19, 2014 at 01:32:13AM -0700, Artyom Beilis wrote:


> - Net BSD with clang and libc++

The current list of failures is:

wing tests FAILED:
          3 - storage_test (Failed)
        19 - internal_forwarder_test (OTHER_FAULT)
        23 - http_timeouts_test_write (Failed)
        24 - file_server_test (Failed)
        25 - file_server_with_listing_test (Failed)
        34 - cppcms_service_test (Failed)
        46 - proto_test_async_fastcgi (Failed)
        59 - test_thread_thread (Timeout)
        75 - test_locale_codepage (Failed)
        76 - test_locale_message (Failed)
        89 - test_locale_std_formatting (Failed)
        90 - test_locale_std_collate (Failed)

3 is the lack of process shared mutex. Might be better to mark it as
skipped?

19 is somewhat nasty as it hang -- possibly related to redirecting
output as it worked earlier when executed normally.

23: output from running manually:

build# ./http_timeouts_test -c tests/proto_test.js "--service-api=http"
"--service-port=8080" "--service-ip=127.0.0.1" "--http-timeout=5"
"--logging-level=debug" --test-write=true
--test-exec="/usr/pkg/bin/python2.7 tests/http_timeouts_test.py write"
Write to the client timeout
Tesing GET /async/long HTTP/1.0 with 0 reads
SO_RCVBUF=32768
2014-10-22 10:09:31; cppcms_http, info: GET /async/long
(http_api.cpp:251)
Error
  File "tests/http_timeouts_test.py", line 72, in <module>
    test_unfinished_read('GET /async/long HTTP/1.0',0,0)
  File "tests/http_timeouts_test.py", line 65, in test_unfinished_read
    test(n < read_size * 16 or n < 1000000)
  File "tests/http_timeouts_test.py", line 16, in test
    traceback.print_stack()
Statistics:
 sync_bad_count =0
 async_bad_count =0
 count_timeouts =0
 above 15 =0
 below 10 =0
 eof =0
Failed

24: Failed to open filename:tests/file_server_test.js
25: same

34: not sure how to run?
46: not sure how to run?
59: hangs in "Test recursive_shared_mutex recursive shared lock"
75: charset translation fails, haven't checked why
76: not checked
89: 10% of test cases fail, likely minor output differences
90: expected, missing libc support in NetBSD

The bashism patches for bin/cppcms_run are still present?

Joerg

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Cppcms-users mailing list
Cpp...@li...
https://lists.sourceforge.net/lists/listinfo/cppcms-users


   

Re: [Cppcms-users] Getting ready to 1.0.5

[Joerg S. <jo...@br...>]

On Sun, Oct 19, 2014 at 01:32:13AM -0700, Artyom Beilis wrote:
> - Net BSD with clang and libc++

The current list of failures is:

wing tests FAILED:
          3 - storage_test (Failed)
         19 - internal_forwarder_test (OTHER_FAULT)
         23 - http_timeouts_test_write (Failed)
         24 - file_server_test (Failed)
         25 - file_server_with_listing_test (Failed)
         34 - cppcms_service_test (Failed)
         46 - proto_test_async_fastcgi (Failed)
         59 - test_thread_thread (Timeout)
         75 - test_locale_codepage (Failed)
         76 - test_locale_message (Failed)
         89 - test_locale_std_formatting (Failed)
         90 - test_locale_std_collate (Failed)

3 is the lack of process shared mutex. Might be better to mark it as
skipped?

19 is somewhat nasty as it hang -- possibly related to redirecting
output as it worked earlier when executed normally.

23: output from running manually:

build# ./http_timeouts_test -c tests/proto_test.js "--service-api=http"
"--service-port=8080" "--service-ip=127.0.0.1" "--http-timeout=5"
"--logging-level=debug" --test-write=true
--test-exec="/usr/pkg/bin/python2.7 tests/http_timeouts_test.py write"
Write to the client timeout
Tesing GET /async/long HTTP/1.0 with 0 reads
SO_RCVBUF=32768
2014-10-22 10:09:31; cppcms_http, info: GET /async/long
(http_api.cpp:251)
Error
  File "tests/http_timeouts_test.py", line 72, in <module>
    test_unfinished_read('GET /async/long HTTP/1.0',0,0)
  File "tests/http_timeouts_test.py", line 65, in test_unfinished_read
    test(n < read_size * 16 or n < 1000000)
  File "tests/http_timeouts_test.py", line 16, in test
    traceback.print_stack()
Statistics:
 sync_bad_count =0
 async_bad_count =0
 count_timeouts =0
 above 15 =0
 below 10 =0
 eof =0
Failed

24: Failed to open filename:tests/file_server_test.js
25: same

34: not sure how to run?
46: not sure how to run?
59: hangs in "Test recursive_shared_mutex recursive shared lock"
75: charset translation fails, haven't checked why
76: not checked
89: 10% of test cases fail, likely minor output differences
90: expected, missing libc support in NetBSD

The bashism patches for bin/cppcms_run are still present?

Joerg

Re: [Cppcms-users] render sub-template without parents

[Lee E. <le...@el...>]

take a look at the generated c++ code generate from a template - you will
see how a template can inherit from another, and override it
i am not free to take a look myself right now, i will try to give a more
complete answer later

On Mon, Oct 20, 2014 at 3:11 PM, bibermann <fab...@gm...> wrote:

> Lee Elenbaas <lee@...> writes:
>
> >
> >
> > Template hirarchi translates to class hirarchi, with methods
> > You can call the news method from two templates, one as part of a full
> page and another as partial
> > בתאריך 15 באוק 2014 17:45, "bibermann" <fabianvss-
> Re5...@pu...> כתב:Hello,
> > I'm going to use CppCMS for an Ajax-driven site and am wondering whether
> it's
> > possible to only return the HTML for a sub-template.
> > Considering the example given
> inhttp://cppcms.com/wikipp/en/page/cppcms_1x_tut_templates_inheritance
> > where we have the following template hierarchie:
> >      master
> >    /    |   \
> > news  page  intro
> > I want to be able to both render the news normally and as part of an ajax
> > request to update the news item without reloading the full page, i.e.
> > returning page_content() of this template.
> > Can I achieve this functionality with CppCMS or do I need to create two
> > version of "news"?
> > Thanks in advance,
> > bibermann
> >
> --------------------------------------------------------------------------
> ----
> > Comprehensive Server Monitoring with Site24x7.
> > Monitor 10 servers for $9/Month.
> > Get alerted through email, SMS, voice calls or mobile push notifications.
> > Take corrective actions from your mobile device.http://p.sf.net/sfu/Zoho
> > _______________________________________________
> > Cppcms-users mailing listCppcms-users-
> 5NW...@pu...://
> lists.sourceforge.net/lists
> /listinfo/cppcms-users
> >
> >
> >
> >
> >
> --------------------------------------------------------------------------
> ----
> > Comprehensive Server Monitoring with Site24x7.
> > Monitor 10 servers for $9/Month.
> > Get alerted through email, SMS, voice calls or mobile push notifications.
> > Take corrective actions from your mobile device.
> > http://p.sf.net/sfu/Zoho
> >
> >
> --------------------------------------------------------------------------
> ----
> > Comprehensive Server Monitoring with Site24x7.
> > Monitor 10 servers for $9/Month.
> > Get alerted through email, SMS, voice calls or mobile push notifications.
> > Take corrective actions from your mobile device.
> > http://p.sf.net/sfu/Zoho
>
>
> Sorry, I did not understand. Which function do you mean? I think I cannot
> access the News instance from within the Code.
>
> If this does not work I would linke to somehow include a separate template
> (like #include but at template-compile time - I didn't found a possibility
> to do this).
>
> Regards,
> bibermann
>
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

2014-10-21 8:40 GMT+03:00 Artyom Beilis <art...@ya...>:
> Thanks...
>
> Can you take a look on v1.0_fixes branch of the cppcms
>
> This is the fix I had written: https://sourceforge.net/p/cppcms/code/2271/

I've tried only this patch applied on 1.0.2 (I could setup my tests
faster this way).

>
> It improved 600MB file upload from 30-40s to 4-5s. And it is twice faster than PHP
> (that has multipart parsing embedded into the C code of PHP)

It improved my previous ~20s upload to ~5s. The nodejs app still beats
this with ~3s but I guess that's acceptable since it could depend on
build options. I'm still building cppcms with MSVC 2005.

Thanks for the fix.

Regards,
Cristian

>
> So I assume it is almost as efficient as it can get.
>
> Take a look and if it is ok I'll close the ticket. It would get
> released in 1.0.5
>
> Artyom Beilis
> --------------
> CppCMS - C++ Web Framework:   http://cppcms.com/
> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>
>
> ----- Original Message -----
>> From: Cristian Oneț <one...@gm...>
>> To: cppcms-users <cpp...@li...>
>> Cc:
>> Sent: Monday, October 20, 2014 6:41 PM
>> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
>>
>> 2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>:
>>>  1st of all please open a ticket...
>>
>> Done.
>>
>> https://sourceforge.net/p/cppcms/feature-requests/27/
>>
>> Regards,
>> Cristian
>>
>>>
>>>  I got the problem... it is indeed in multipart_parser or more accuratly
>>>  in the way I write to the buffered iostream...
>>>
>>>  My assumption that the iostream optimization is very good was actually
>> wrong
>>>  and there is some terrible bottle neck.
>>>
>>>  I'll get this fixed for 1.0.5.
>>>
>>>  Artyom Beilis
>>>  --------------
>>>  CppCMS - C++ Web Framework:  http://cppcms.com/
>>>  CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>>
>>>
>>>  ----- Original Message -----
>>>>  From: Cristian Oneț <one...@gm...>
>>>>  To: cppcms-users <cpp...@li...>
>>>>  Cc:
>>>>  Sent: Monday, October 20, 2014 3:54 PM
>>>>  Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
>>>>
>>>>  2014-10-20 16:10 GMT+03:00 Cristian Oneț
>> <one...@gm...>:
>>>>>   2014-10-20 14:51 GMT+03:00 Artyom Beilis
>> <art...@ya...>:
>>>>>>   1st of all for the iostreams there is a huge difference
>> between debug
>>>>  and
>>>>>>   release mode additionally all the callback
>>>>>>   code of http code depends on optimization.
>>>>>
>>>>>   Well the difference is 20 something seconds in release vs 1 minute
>> and
>>>>>   some seconds in debug (~270 MB file), the CPU is still hugged
>> during
>>>>>   those 20 seconds, I'll try the 'out <<
>> in.rdbuf()'
>>>>  version, just need
>>>>>   to rebuild cppcms.
>>>>
>>>>  It didn't made any difference (still 20-30 secs) so the cause must
>> be
>>>>  somewhere else like in multipart_parser (it's just a hunch),
>> I'll let
>>>>  you look at this issue :) and stop making assumptions.
>>>>
>>>>  Thanks,
>>>>  Cristian
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>   Also you are right out << in.rdbuf() should be better...
>> probably
>>>>  one of
>>>>>>   these parts of code I written too fast without thinking...
>>>>>>
>>>>>>   However I doubt that it is the problem.
>>>>>>
>>>>>>   How   http::file works when we receive an input data we write
>> it to the
>>>>>>   memory at first place but after certain threshold it
>>>>>>   is written directly to the file instead of memory.  So this
>> part of
>>>>  code has
>>>>>>   only small use.
>>>>>
>>>>>   I'm aware of that.
>>>>>
>>>>>>
>>>>>>   Also I suggest when you actually "save" the file use
>>>>  http::file::save_to as
>>>>>>   it would rather move the file on the disk and not copy it.
>>>>>>
>>>>>
>>>>>   I use it.
>>>>>
>>>>>>   If you still have problems with uploading in release more it
>> would be a
>>>>  good
>>>>>>   idea for me to profile the code. If so I need to know
>>>>>>   what version of cppcms you use cppcms 1.0.x or trunk (there
>> were some
>>>>>>   changes in that part)
>>>>>
>>>>>   I use cppcms-1.0.2. This is the test code, using the following
>> settings:
>>>>>
>>>>>   "security.multipart_form_data_limit" -> 300000; //
>> ~300MB
>>>>  multipart limit
>>>>>   "security.uploads_path" ->
>>>>  "d:\\tmp\\uploads"
>>>>>
>>>>>   Code (note that it's a test, I know about the security
>> issues):
>>>>>
>>>>>   class uploader: public cppcms::application {
>>>>>   public:
>>>>>     uploader(cppcms::service &s) :
>>>>>         cppcms::application(s)
>>>>>         {
>>>>>         }
>>>>>
>>>>>         void main(std::string /*unused*/)
>>>>>         {
>>>>>           if (request().request_method() == "POST")
>>>>>           {
>>>>>             const cppcms::http::request::files_type &files =
>>>>  request().files();
>>>>>             for (int i = 0; i < files.size(); ++i)
>>>>>             {
>>>>>
>>>>
>> files[i]->save_to("D:\\tmp\\uploads\\" +
>>>>  files[i]->filename());
>>>>>             }
>>>>>           }
>>>>>           else
>>>>>           {
>>>>>             response().out() << "<form
>> action='/upload'
>>>>>   enctype='multipart/form-data'
>> method='post'><input
>>>>  type='text'
>>>>>   name='title'><br><input type='file'
>>>>  name='upload'
>>>>>   multiple='multiple'><br><input
>> type='submit'
>>>>  value='Upload'></form>";
>>>>>           }
>>>>>         }
>>>>>   };
>>>>>
>>>>>   Regards,
>>>>>   Cristian
>>>>>
>>>>>>
>>>>>>
>>>>>>   Artyom Beilis
>>>>>>   --------------
>>>>>>   CppCMS - C++ Web Framework:  http://cppcms.com/
>>>>>>   CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>>>>>
>>>>>>   ________________________________
>>>>>>   From: Cristian Oneț <one...@gm...>
>>>>>>   To: cppcms-users <cpp...@li...>
>>>>>>   Sent: Monday, October 20, 2014 1:06 PM
>>>>>>   Subject: [Cppcms-users] Slow http:file copy_stream for big
>> files
>>>>>>
>>>>>>   Hi,
>>>>>>
>>>>>>   I'm using a simplified version (no widgets::file only
>>>>  request::files)
>>>>>>   of the uploader [1] to implement an upload service. What I
>> observed is
>>>>>>   that uploading large files takes a lot of CPU and takes way
>> more than
>>>>>>   a similar nodejs service [2] (1 min vs 1 s for the same file -
>> 300
>>>>>>   MB). I have to admit that this was cppcms compiled in debug
>> against a
>>>>>>   release nodejs but the difference is too great.
>>>>>>
>>>>>>   Intrigued by this finding I've took a glimpse at how the
>> file is
>>>>  saved
>>>>>>   to the temporary location using the input stream and found
>> [3].
>>>>>>   Reading from the input stream in 1024 byte chunks then writing
>> that to
>>>>>>   the output does not seem such a great idea and could explain
>> the high
>>>>>>   CPU usage and the overall slowness.
>>>>>>
>>>>>>   Shouldn't this perform better?
>>>>>>
>>>>>>   out << in.rdbuf();
>>>>>>
>>>>>>   I promise I'll get back with the results of running a
>> release
>>>>  build.
>>>>>>
>>>>>>   Regards,
>>>>>>   Cristian
>>>>>>
>>>>>>   [1]
>>>>  http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>>>>>>   [2] https://github.com/felixge/node-formidable
>>>>>>   [3]
>>>>>>
>>>>
>> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>>>>>
>>>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>>>   Comprehensive Server Monitoring with Site24x7.
>>>>>>   Monitor 10 servers for $9/Month.
>>>>>>   Get alerted through email, SMS, voice calls or mobile push
>>>>  notifications.
>>>>>>   Take corrective actions from your mobile device.
>>>>>>   http://p.sf.net/sfu/Zoho
>>>>>>   _______________________________________________
>>>>>>   Cppcms-users mailing list
>>>>>>   Cpp...@li...
>>>>>>   https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>>>   Comprehensive Server Monitoring with Site24x7.
>>>>>>   Monitor 10 servers for $9/Month.
>>>>>>   Get alerted through email, SMS, voice calls or mobile push
>>>>  notifications.
>>>>>>   Take corrective actions from your mobile device.
>>>>>>   http://p.sf.net/sfu/Zoho
>>>>>>   _______________________________________________
>>>>>>   Cppcms-users mailing list
>>>>>>   Cpp...@li...
>>>>>>   https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>>>>>>
>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>  Comprehensive Server Monitoring with Site24x7.
>>>>  Monitor 10 servers for $9/Month.
>>>>  Get alerted through email, SMS, voice calls or mobile push
>> notifications.
>>>>  Take corrective actions from your mobile device.
>>>>  http://p.sf.net/sfu/Zoho
>>>>  _______________________________________________
>>>>  Cppcms-users mailing list
>>>>  Cpp...@li...
>>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>
>>>
>>>
>> ------------------------------------------------------------------------------
>>>  Comprehensive Server Monitoring with Site24x7.
>>>  Monitor 10 servers for $9/Month.
>>>  Get alerted through email, SMS, voice calls or mobile push notifications.
>>>  Take corrective actions from your mobile device.
>>>  http://p.sf.net/sfu/Zoho
>>>  _______________________________________________
>>>  Cppcms-users mailing list
>>>  Cpp...@li...
>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Artyom B. <art...@ya...>]

Thanks...

Can you take a look on v1.0_fixes branch of the cppcms

This is the fix I had written: https://sourceforge.net/p/cppcms/code/2271/

It improved 600MB file upload from 30-40s to 4-5s. And it is twice faster than PHP
(that has multipart parsing embedded into the C code of PHP)

So I assume it is almost as efficient as it can get.
 
Take a look and if it is ok I'll close the ticket. It would get
released in 1.0.5

Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:   http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/


----- Original Message -----
> From: Cristian Oneț <one...@gm...>
> To: cppcms-users <cpp...@li...>
> Cc: 
> Sent: Monday, October 20, 2014 6:41 PM
> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
> 
> 2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>:
>>  1st of all please open a ticket...
> 
> Done.
> 
> https://sourceforge.net/p/cppcms/feature-requests/27/
> 
> Regards,
> Cristian
> 
>> 
>>  I got the problem... it is indeed in multipart_parser or more accuratly
>>  in the way I write to the buffered iostream...
>> 
>>  My assumption that the iostream optimization is very good was actually 
> wrong
>>  and there is some terrible bottle neck.
>> 
>>  I'll get this fixed for 1.0.5.
>> 
>>  Artyom Beilis
>>  --------------
>>  CppCMS - C++ Web Framework:  http://cppcms.com/
>>  CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>> 
>> 
>>  ----- Original Message -----
>>>  From: Cristian Oneț <one...@gm...>
>>>  To: cppcms-users <cpp...@li...>
>>>  Cc:
>>>  Sent: Monday, October 20, 2014 3:54 PM
>>>  Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
>>> 
>>>  2014-10-20 16:10 GMT+03:00 Cristian Oneț 
> <one...@gm...>:
>>>>   2014-10-20 14:51 GMT+03:00 Artyom Beilis 
> <art...@ya...>:
>>>>>   1st of all for the iostreams there is a huge difference 
> between debug
>>>  and
>>>>>   release mode additionally all the callback
>>>>>   code of http code depends on optimization.
>>>> 
>>>>   Well the difference is 20 something seconds in release vs 1 minute 
> and
>>>>   some seconds in debug (~270 MB file), the CPU is still hugged 
> during
>>>>   those 20 seconds, I'll try the 'out << 
> in.rdbuf()'
>>>  version, just need
>>>>   to rebuild cppcms.
>>> 
>>>  It didn't made any difference (still 20-30 secs) so the cause must 
> be
>>>  somewhere else like in multipart_parser (it's just a hunch), 
> I'll let
>>>  you look at this issue :) and stop making assumptions.
>>> 
>>>  Thanks,
>>>  Cristian
>>> 
>>> 
>>>> 
>>>>> 
>>>>>   Also you are right out << in.rdbuf() should be better... 
> probably
>>>  one of
>>>>>   these parts of code I written too fast without thinking...
>>>>> 
>>>>>   However I doubt that it is the problem.
>>>>> 
>>>>>   How   http::file works when we receive an input data we write 
> it to the
>>>>>   memory at first place but after certain threshold it
>>>>>   is written directly to the file instead of memory.  So this 
> part of
>>>  code has
>>>>>   only small use.
>>>> 
>>>>   I'm aware of that.
>>>> 
>>>>> 
>>>>>   Also I suggest when you actually "save" the file use
>>>  http::file::save_to as
>>>>>   it would rather move the file on the disk and not copy it.
>>>>> 
>>>> 
>>>>   I use it.
>>>> 
>>>>>   If you still have problems with uploading in release more it 
> would be a
>>>  good
>>>>>   idea for me to profile the code. If so I need to know
>>>>>   what version of cppcms you use cppcms 1.0.x or trunk (there 
> were some
>>>>>   changes in that part)
>>>> 
>>>>   I use cppcms-1.0.2. This is the test code, using the following 
> settings:
>>>> 
>>>>   "security.multipart_form_data_limit" -> 300000; // 
> ~300MB
>>>  multipart limit
>>>>   "security.uploads_path" ->
>>>  "d:\\tmp\\uploads"
>>>> 
>>>>   Code (note that it's a test, I know about the security 
> issues):
>>>> 
>>>>   class uploader: public cppcms::application {
>>>>   public:
>>>>     uploader(cppcms::service &s) :
>>>>         cppcms::application(s)
>>>>         {
>>>>         }
>>>> 
>>>>         void main(std::string /*unused*/)
>>>>         {
>>>>           if (request().request_method() == "POST")
>>>>           {
>>>>             const cppcms::http::request::files_type &files =
>>>  request().files();
>>>>             for (int i = 0; i < files.size(); ++i)
>>>>             {
>>>> 
>>> 
> files[i]->save_to("D:\\tmp\\uploads\\" +
>>>  files[i]->filename());
>>>>             }
>>>>           }
>>>>           else
>>>>           {
>>>>             response().out() << "<form 
> action='/upload'
>>>>   enctype='multipart/form-data' 
> method='post'><input
>>>  type='text'
>>>>   name='title'><br><input type='file'
>>>  name='upload'
>>>>   multiple='multiple'><br><input 
> type='submit'
>>>  value='Upload'></form>";
>>>>           }
>>>>         }
>>>>   };
>>>> 
>>>>   Regards,
>>>>   Cristian
>>>> 
>>>>> 
>>>>> 
>>>>>   Artyom Beilis
>>>>>   --------------
>>>>>   CppCMS - C++ Web Framework:  http://cppcms.com/
>>>>>   CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>>>> 
>>>>>   ________________________________
>>>>>   From: Cristian Oneț <one...@gm...>
>>>>>   To: cppcms-users <cpp...@li...>
>>>>>   Sent: Monday, October 20, 2014 1:06 PM
>>>>>   Subject: [Cppcms-users] Slow http:file copy_stream for big 
> files
>>>>> 
>>>>>   Hi,
>>>>> 
>>>>>   I'm using a simplified version (no widgets::file only
>>>  request::files)
>>>>>   of the uploader [1] to implement an upload service. What I 
> observed is
>>>>>   that uploading large files takes a lot of CPU and takes way 
> more than
>>>>>   a similar nodejs service [2] (1 min vs 1 s for the same file - 
> 300
>>>>>   MB). I have to admit that this was cppcms compiled in debug 
> against a
>>>>>   release nodejs but the difference is too great.
>>>>> 
>>>>>   Intrigued by this finding I've took a glimpse at how the 
> file is
>>>  saved
>>>>>   to the temporary location using the input stream and found 
> [3].
>>>>>   Reading from the input stream in 1024 byte chunks then writing 
> that to
>>>>>   the output does not seem such a great idea and could explain 
> the high
>>>>>   CPU usage and the overall slowness.
>>>>> 
>>>>>   Shouldn't this perform better?
>>>>> 
>>>>>   out << in.rdbuf();
>>>>> 
>>>>>   I promise I'll get back with the results of running a 
> release
>>>  build.
>>>>> 
>>>>>   Regards,
>>>>>   Cristian
>>>>> 
>>>>>   [1]
>>>  http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>>>>>   [2] https://github.com/felixge/node-formidable
>>>>>   [3]
>>>>> 
>>> 
> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>>>> 
>>>>> 
>>> 
> ------------------------------------------------------------------------------
>>>>>   Comprehensive Server Monitoring with Site24x7.
>>>>>   Monitor 10 servers for $9/Month.
>>>>>   Get alerted through email, SMS, voice calls or mobile push
>>>  notifications.
>>>>>   Take corrective actions from your mobile device.
>>>>>   http://p.sf.net/sfu/Zoho
>>>>>   _______________________________________________
>>>>>   Cppcms-users mailing list
>>>>>   Cpp...@li...
>>>>>   https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>> 
> ------------------------------------------------------------------------------
>>>>>   Comprehensive Server Monitoring with Site24x7.
>>>>>   Monitor 10 servers for $9/Month.
>>>>>   Get alerted through email, SMS, voice calls or mobile push
>>>  notifications.
>>>>>   Take corrective actions from your mobile device.
>>>>>   http://p.sf.net/sfu/Zoho
>>>>>   _______________________________________________
>>>>>   Cppcms-users mailing list
>>>>>   Cpp...@li...
>>>>>   https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 
>>>>> 
>>> 
>>> 
> ------------------------------------------------------------------------------
>>>  Comprehensive Server Monitoring with Site24x7.
>>>  Monitor 10 servers for $9/Month.
>>>  Get alerted through email, SMS, voice calls or mobile push 
> notifications.
>>>  Take corrective actions from your mobile device.
>>>  http://p.sf.net/sfu/Zoho
>>>  _______________________________________________
>>>  Cppcms-users mailing list
>>>  Cpp...@li...
>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>> 
>> 
>> 
> ------------------------------------------------------------------------------
>>  Comprehensive Server Monitoring with Site24x7.
>>  Monitor 10 servers for $9/Month.
>>  Get alerted through email, SMS, voice calls or mobile push notifications.
>>  Take corrective actions from your mobile device.
>>  http://p.sf.net/sfu/Zoho
>>  _______________________________________________
>>  Cppcms-users mailing list
>>  Cpp...@li...
>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>:
> 1st of all please open a ticket...

Done.

https://sourceforge.net/p/cppcms/feature-requests/27/

Regards,
Cristian

>
> I got the problem... it is indeed in multipart_parser or more accuratly
> in the way I write to the buffered iostream...
>
> My assumption that the iostream optimization is very good was actually wrong
> and there is some terrible bottle neck.
>
> I'll get this fixed for 1.0.5.
>
> Artyom Beilis
> --------------
> CppCMS - C++ Web Framework:   http://cppcms.com/
> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>
>
> ----- Original Message -----
>> From: Cristian Oneț <one...@gm...>
>> To: cppcms-users <cpp...@li...>
>> Cc:
>> Sent: Monday, October 20, 2014 3:54 PM
>> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
>>
>> 2014-10-20 16:10 GMT+03:00 Cristian Oneț <one...@gm...>:
>>>  2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
>>>>  1st of all for the iostreams there is a huge difference between debug
>> and
>>>>  release mode additionally all the callback
>>>>  code of http code depends on optimization.
>>>
>>>  Well the difference is 20 something seconds in release vs 1 minute and
>>>  some seconds in debug (~270 MB file), the CPU is still hugged during
>>>  those 20 seconds, I'll try the 'out << in.rdbuf()'
>> version, just need
>>>  to rebuild cppcms.
>>
>> It didn't made any difference (still 20-30 secs) so the cause must be
>> somewhere else like in multipart_parser (it's just a hunch), I'll let
>> you look at this issue :) and stop making assumptions.
>>
>> Thanks,
>> Cristian
>>
>>
>>>
>>>>
>>>>  Also you are right out << in.rdbuf() should be better... probably
>> one of
>>>>  these parts of code I written too fast without thinking...
>>>>
>>>>  However I doubt that it is the problem.
>>>>
>>>>  How   http::file works when we receive an input data we write it to the
>>>>  memory at first place but after certain threshold it
>>>>  is written directly to the file instead of memory.  So this part of
>> code has
>>>>  only small use.
>>>
>>>  I'm aware of that.
>>>
>>>>
>>>>  Also I suggest when you actually "save" the file use
>> http::file::save_to as
>>>>  it would rather move the file on the disk and not copy it.
>>>>
>>>
>>>  I use it.
>>>
>>>>  If you still have problems with uploading in release more it would be a
>> good
>>>>  idea for me to profile the code. If so I need to know
>>>>  what version of cppcms you use cppcms 1.0.x or trunk (there were some
>>>>  changes in that part)
>>>
>>>  I use cppcms-1.0.2. This is the test code, using the following settings:
>>>
>>>  "security.multipart_form_data_limit" -> 300000; // ~300MB
>> multipart limit
>>>  "security.uploads_path" ->
>> "d:\\tmp\\uploads"
>>>
>>>  Code (note that it's a test, I know about the security issues):
>>>
>>>  class uploader: public cppcms::application {
>>>  public:
>>>    uploader(cppcms::service &s) :
>>>        cppcms::application(s)
>>>        {
>>>        }
>>>
>>>        void main(std::string /*unused*/)
>>>        {
>>>          if (request().request_method() == "POST")
>>>          {
>>>            const cppcms::http::request::files_type &files =
>> request().files();
>>>            for (int i = 0; i < files.size(); ++i)
>>>            {
>>>
>> files[i]->save_to("D:\\tmp\\uploads\\" +
>> files[i]->filename());
>>>            }
>>>          }
>>>          else
>>>          {
>>>            response().out() << "<form action='/upload'
>>>  enctype='multipart/form-data' method='post'><input
>> type='text'
>>>  name='title'><br><input type='file'
>> name='upload'
>>>  multiple='multiple'><br><input type='submit'
>> value='Upload'></form>";
>>>          }
>>>        }
>>>  };
>>>
>>>  Regards,
>>>  Cristian
>>>
>>>>
>>>>
>>>>  Artyom Beilis
>>>>  --------------
>>>>  CppCMS - C++ Web Framework:  http://cppcms.com/
>>>>  CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>>>
>>>>  ________________________________
>>>>  From: Cristian Oneț <one...@gm...>
>>>>  To: cppcms-users <cpp...@li...>
>>>>  Sent: Monday, October 20, 2014 1:06 PM
>>>>  Subject: [Cppcms-users] Slow http:file copy_stream for big files
>>>>
>>>>  Hi,
>>>>
>>>>  I'm using a simplified version (no widgets::file only
>> request::files)
>>>>  of the uploader [1] to implement an upload service. What I observed is
>>>>  that uploading large files takes a lot of CPU and takes way more than
>>>>  a similar nodejs service [2] (1 min vs 1 s for the same file - 300
>>>>  MB). I have to admit that this was cppcms compiled in debug against a
>>>>  release nodejs but the difference is too great.
>>>>
>>>>  Intrigued by this finding I've took a glimpse at how the file is
>> saved
>>>>  to the temporary location using the input stream and found [3].
>>>>  Reading from the input stream in 1024 byte chunks then writing that to
>>>>  the output does not seem such a great idea and could explain the high
>>>>  CPU usage and the overall slowness.
>>>>
>>>>  Shouldn't this perform better?
>>>>
>>>>  out << in.rdbuf();
>>>>
>>>>  I promise I'll get back with the results of running a release
>> build.
>>>>
>>>>  Regards,
>>>>  Cristian
>>>>
>>>>  [1]
>> http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>>>>  [2] https://github.com/felixge/node-formidable
>>>>  [3]
>>>>
>> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>  Comprehensive Server Monitoring with Site24x7.
>>>>  Monitor 10 servers for $9/Month.
>>>>  Get alerted through email, SMS, voice calls or mobile push
>> notifications.
>>>>  Take corrective actions from your mobile device.
>>>>  http://p.sf.net/sfu/Zoho
>>>>  _______________________________________________
>>>>  Cppcms-users mailing list
>>>>  Cpp...@li...
>>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>
>>>>
>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>  Comprehensive Server Monitoring with Site24x7.
>>>>  Monitor 10 servers for $9/Month.
>>>>  Get alerted through email, SMS, voice calls or mobile push
>> notifications.
>>>>  Take corrective actions from your mobile device.
>>>>  http://p.sf.net/sfu/Zoho
>>>>  _______________________________________________
>>>>  Cppcms-users mailing list
>>>>  Cpp...@li...
>>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Artyom B. <art...@ya...>]

1st of all please open a ticket...

I got the problem... it is indeed in multipart_parser or more accuratly
in the way I write to the buffered iostream...

My assumption that the iostream optimization is very good was actually wrong
and there is some terrible bottle neck.

I'll get this fixed for 1.0.5.
 
Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:   http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/


----- Original Message -----
> From: Cristian Oneț <one...@gm...>
> To: cppcms-users <cpp...@li...>
> Cc: 
> Sent: Monday, October 20, 2014 3:54 PM
> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
> 
> 2014-10-20 16:10 GMT+03:00 Cristian Oneț <one...@gm...>:
>>  2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
>>>  1st of all for the iostreams there is a huge difference between debug 
> and
>>>  release mode additionally all the callback
>>>  code of http code depends on optimization.
>> 
>>  Well the difference is 20 something seconds in release vs 1 minute and
>>  some seconds in debug (~270 MB file), the CPU is still hugged during
>>  those 20 seconds, I'll try the 'out << in.rdbuf()' 
> version, just need
>>  to rebuild cppcms.
> 
> It didn't made any difference (still 20-30 secs) so the cause must be
> somewhere else like in multipart_parser (it's just a hunch), I'll let
> you look at this issue :) and stop making assumptions.
> 
> Thanks,
> Cristian
> 
> 
>> 
>>> 
>>>  Also you are right out << in.rdbuf() should be better... probably 
> one of
>>>  these parts of code I written too fast without thinking...
>>> 
>>>  However I doubt that it is the problem.
>>> 
>>>  How   http::file works when we receive an input data we write it to the
>>>  memory at first place but after certain threshold it
>>>  is written directly to the file instead of memory.  So this part of 
> code has
>>>  only small use.
>> 
>>  I'm aware of that.
>> 
>>> 
>>>  Also I suggest when you actually "save" the file use 
> http::file::save_to as
>>>  it would rather move the file on the disk and not copy it.
>>> 
>> 
>>  I use it.
>> 
>>>  If you still have problems with uploading in release more it would be a 
> good
>>>  idea for me to profile the code. If so I need to know
>>>  what version of cppcms you use cppcms 1.0.x or trunk (there were some
>>>  changes in that part)
>> 
>>  I use cppcms-1.0.2. This is the test code, using the following settings:
>> 
>>  "security.multipart_form_data_limit" -> 300000; // ~300MB 
> multipart limit
>>  "security.uploads_path" -> 
> "d:\\tmp\\uploads"
>> 
>>  Code (note that it's a test, I know about the security issues):
>> 
>>  class uploader: public cppcms::application {
>>  public:
>>    uploader(cppcms::service &s) :
>>        cppcms::application(s)
>>        {
>>        }
>> 
>>        void main(std::string /*unused*/)
>>        {
>>          if (request().request_method() == "POST")
>>          {
>>            const cppcms::http::request::files_type &files = 
> request().files();
>>            for (int i = 0; i < files.size(); ++i)
>>            {
>>             
> files[i]->save_to("D:\\tmp\\uploads\\" + 
> files[i]->filename());
>>            }
>>          }
>>          else
>>          {
>>            response().out() << "<form action='/upload'
>>  enctype='multipart/form-data' method='post'><input 
> type='text'
>>  name='title'><br><input type='file' 
> name='upload'
>>  multiple='multiple'><br><input type='submit' 
> value='Upload'></form>";
>>          }
>>        }
>>  };
>> 
>>  Regards,
>>  Cristian
>> 
>>> 
>>> 
>>>  Artyom Beilis
>>>  --------------
>>>  CppCMS - C++ Web Framework:  http://cppcms.com/
>>>  CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>> 
>>>  ________________________________
>>>  From: Cristian Oneț <one...@gm...>
>>>  To: cppcms-users <cpp...@li...>
>>>  Sent: Monday, October 20, 2014 1:06 PM
>>>  Subject: [Cppcms-users] Slow http:file copy_stream for big files
>>> 
>>>  Hi,
>>> 
>>>  I'm using a simplified version (no widgets::file only 
> request::files)
>>>  of the uploader [1] to implement an upload service. What I observed is
>>>  that uploading large files takes a lot of CPU and takes way more than
>>>  a similar nodejs service [2] (1 min vs 1 s for the same file - 300
>>>  MB). I have to admit that this was cppcms compiled in debug against a
>>>  release nodejs but the difference is too great.
>>> 
>>>  Intrigued by this finding I've took a glimpse at how the file is 
> saved
>>>  to the temporary location using the input stream and found [3].
>>>  Reading from the input stream in 1024 byte chunks then writing that to
>>>  the output does not seem such a great idea and could explain the high
>>>  CPU usage and the overall slowness.
>>> 
>>>  Shouldn't this perform better?
>>> 
>>>  out << in.rdbuf();
>>> 
>>>  I promise I'll get back with the results of running a release 
> build.
>>> 
>>>  Regards,
>>>  Cristian
>>> 
>>>  [1] 
> http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>>>  [2] https://github.com/felixge/node-formidable
>>>  [3]
>>> 
> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>> 
>>> 
> ------------------------------------------------------------------------------
>>>  Comprehensive Server Monitoring with Site24x7.
>>>  Monitor 10 servers for $9/Month.
>>>  Get alerted through email, SMS, voice calls or mobile push 
> notifications.
>>>  Take corrective actions from your mobile device.
>>>  http://p.sf.net/sfu/Zoho
>>>  _______________________________________________
>>>  Cppcms-users mailing list
>>>  Cpp...@li...
>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>> 
>>> 
>>> 
>>> 
> ------------------------------------------------------------------------------
>>>  Comprehensive Server Monitoring with Site24x7.
>>>  Monitor 10 servers for $9/Month.
>>>  Get alerted through email, SMS, voice calls or mobile push 
> notifications.
>>>  Take corrective actions from your mobile device.
>>>  http://p.sf.net/sfu/Zoho
>>>  _______________________________________________
>>>  Cppcms-users mailing list
>>>  Cpp...@li...
>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>> 
> 
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

2014-10-20 16:10 GMT+03:00 Cristian Oneț <one...@gm...>:
> 2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
>> 1st of all for the iostreams there is a huge difference between debug and
>> release mode additionally all the callback
>> code of http code depends on optimization.
>
> Well the difference is 20 something seconds in release vs 1 minute and
> some seconds in debug (~270 MB file), the CPU is still hugged during
> those 20 seconds, I'll try the 'out << in.rdbuf()' version, just need
> to rebuild cppcms.

It didn't made any difference (still 20-30 secs) so the cause must be
somewhere else like in multipart_parser (it's just a hunch), I'll let
you look at this issue :) and stop making assumptions.

Thanks,
Cristian

>
>>
>> Also you are right out << in.rdbuf() should be better... probably one of
>> these parts of code I written too fast without thinking...
>>
>> However I doubt that it is the problem.
>>
>> How   http::file works when we receive an input data we write it to the
>> memory at first place but after certain threshold it
>> is written directly to the file instead of memory.  So this part of code has
>> only small use.
>
> I'm aware of that.
>
>>
>> Also I suggest when you actually "save" the file use http::file::save_to as
>> it would rather move the file on the disk and not copy it.
>>
>
> I use it.
>
>> If you still have problems with uploading in release more it would be a good
>> idea for me to profile the code. If so I need to know
>> what version of cppcms you use cppcms 1.0.x or trunk (there were some
>> changes in that part)
>
> I use cppcms-1.0.2. This is the test code, using the following settings:
>
> "security.multipart_form_data_limit" -> 300000; // ~300MB multipart limit
> "security.uploads_path" -> "d:\\tmp\\uploads"
>
> Code (note that it's a test, I know about the security issues):
>
> class uploader: public cppcms::application {
> public:
>   uploader(cppcms::service &s) :
>       cppcms::application(s)
>       {
>       }
>
>       void main(std::string /*unused*/)
>       {
>         if (request().request_method() == "POST")
>         {
>           const cppcms::http::request::files_type &files = request().files();
>           for (int i = 0; i < files.size(); ++i)
>           {
>             files[i]->save_to("D:\\tmp\\uploads\\" + files[i]->filename());
>           }
>         }
>         else
>         {
>           response().out() << "<form action='/upload'
> enctype='multipart/form-data' method='post'><input type='text'
> name='title'><br><input type='file' name='upload'
> multiple='multiple'><br><input type='submit' value='Upload'></form>";
>         }
>       }
> };
>
> Regards,
> Cristian
>
>>
>>
>> Artyom Beilis
>> --------------
>> CppCMS - C++ Web Framework:   http://cppcms.com/
>> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>
>> ________________________________
>> From: Cristian Oneț <one...@gm...>
>> To: cppcms-users <cpp...@li...>
>> Sent: Monday, October 20, 2014 1:06 PM
>> Subject: [Cppcms-users] Slow http:file copy_stream for big files
>>
>> Hi,
>>
>> I'm using a simplified version (no widgets::file only request::files)
>> of the uploader [1] to implement an upload service. What I observed is
>> that uploading large files takes a lot of CPU and takes way more than
>> a similar nodejs service [2] (1 min vs 1 s for the same file - 300
>> MB). I have to admit that this was cppcms compiled in debug against a
>> release nodejs but the difference is too great.
>>
>> Intrigued by this finding I've took a glimpse at how the file is saved
>> to the temporary location using the input stream and found [3].
>> Reading from the input stream in 1024 byte chunks then writing that to
>> the output does not seem such a great idea and could explain the high
>> CPU usage and the overall slowness.
>>
>> Shouldn't this perform better?
>>
>> out << in.rdbuf();
>>
>> I promise I'll get back with the results of running a release build.
>>
>> Regards,
>> Cristian
>>
>> [1] http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>> [2] https://github.com/felixge/node-formidable
>> [3]
>> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
> 1st of all for the iostreams there is a huge difference between debug and
> release mode additionally all the callback
> code of http code depends on optimization.

Well the difference is 20 something seconds in release vs 1 minute and
some seconds in debug (~270 MB file), the CPU is still hugged during
those 20 seconds, I'll try the 'out << in.rdbuf()' version, just need
to rebuild cppcms.

>
> Also you are right out << in.rdbuf() should be better... probably one of
> these parts of code I written too fast without thinking...
>
> However I doubt that it is the problem.
>
> How   http::file works when we receive an input data we write it to the
> memory at first place but after certain threshold it
> is written directly to the file instead of memory.  So this part of code has
> only small use.

I'm aware of that.

>
> Also I suggest when you actually "save" the file use http::file::save_to as
> it would rather move the file on the disk and not copy it.
>

I use it.

> If you still have problems with uploading in release more it would be a good
> idea for me to profile the code. If so I need to know
> what version of cppcms you use cppcms 1.0.x or trunk (there were some
> changes in that part)

I use cppcms-1.0.2. This is the test code, using the following settings:

"security.multipart_form_data_limit" -> 300000; // ~300MB multipart limit
"security.uploads_path" -> "d:\\tmp\\uploads"

Code (note that it's a test, I know about the security issues):

class uploader: public cppcms::application {
public:
  uploader(cppcms::service &s) :
      cppcms::application(s)
      {
      }

      void main(std::string /*unused*/)
      {
        if (request().request_method() == "POST")
        {
          const cppcms::http::request::files_type &files = request().files();
          for (int i = 0; i < files.size(); ++i)
          {
            files[i]->save_to("D:\\tmp\\uploads\\" + files[i]->filename());
          }
        }
        else
        {
          response().out() << "<form action='/upload'
enctype='multipart/form-data' method='post'><input type='text'
name='title'><br><input type='file' name='upload'
multiple='multiple'><br><input type='submit' value='Upload'></form>";
        }
      }
};

Regards,
Cristian

>
>
> Artyom Beilis
> --------------
> CppCMS - C++ Web Framework:   http://cppcms.com/
> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>
> ________________________________
> From: Cristian Oneț <one...@gm...>
> To: cppcms-users <cpp...@li...>
> Sent: Monday, October 20, 2014 1:06 PM
> Subject: [Cppcms-users] Slow http:file copy_stream for big files
>
> Hi,
>
> I'm using a simplified version (no widgets::file only request::files)
> of the uploader [1] to implement an upload service. What I observed is
> that uploading large files takes a lot of CPU and takes way more than
> a similar nodejs service [2] (1 min vs 1 s for the same file - 300
> MB). I have to admit that this was cppcms compiled in debug against a
> release nodejs but the difference is too great.
>
> Intrigued by this finding I've took a glimpse at how the file is saved
> to the temporary location using the input stream and found [3].
> Reading from the input stream in 1024 byte chunks then writing that to
> the output does not seem such a great idea and could explain the high
> CPU usage and the overall slowness.
>
> Shouldn't this perform better?
>
> out << in.rdbuf();
>
> I promise I'll get back with the results of running a release build.
>
> Regards,
> Cristian
>
> [1] http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
> [2] https://github.com/felixge/node-formidable
> [3]
> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>

Re: [Cppcms-users] render sub-template without parents

[Artyom B. <art...@ya...>]

Think this

you have a template that generates some interesting content


<% view some_view uses page_content %>
<% template page_content() %>
...
<% end template %>
<% end view %>

Lets derive a new template

<% view page_content_only uses page_content extends some_view %>

<% template render() %>
<% include page_content() %>
<% end template %>
<% end view %>


Now you can render "page_content_only" view that would create what you need.

{
   ...
   render("page_content_only",c);
   ...
}

Thinks of classes and inheritance. 

You can also use "helpers" concept and generate a separate view that only
generates a page_content and use it in various places:

http://cppcms.com/wikipp/en/page/cppcms_1x_templates_comm#Views.Helpers
 


Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:   http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/


----- Original Message -----
> From: bibermann <fab...@gm...>
> To: cpp...@li...
> Cc: 
> Sent: Monday, October 20, 2014 2:11 PM
> Subject: Re: [Cppcms-users] render sub-template without parents
> 
> Lee Elenbaas <lee@...> writes:
> 
>> 
>> 
>>  Template hirarchi translates to class hirarchi, with methods
>>  You can call the news method from two templates, one as part of a full 
> page and another as partial
>>  בתאריך 15 באוק 2014 17:45, "bibermann" <fabianvss-
> Re5...@pu...> כתב:Hello,
>>  I'm going to use CppCMS for an Ajax-driven site and am wondering 
> whether 
> it's
>>  possible to only return the HTML for a sub-template.
>>  Considering the example given 
> inhttp://cppcms.com/wikipp/en/page/cppcms_1x_tut_templates_inheritance
>>  where we have the following template hierarchie:
>>       master
>>     /    |   \
>>  news  page  intro
>>  I want to be able to both render the news normally and as part of an ajax
>>  request to update the news item without reloading the full page, i.e.
>>  returning page_content() of this template.
>>  Can I achieve this functionality with CppCMS or do I need to create two
>>  version of "news"?
>>  Thanks in advance,
>>  bibermann
>>  --------------------------------------------------------------------------
> ----
>>  Comprehensive Server Monitoring with Site24x7.
>>  Monitor 10 servers for $9/Month.
>>  Get alerted through email, SMS, voice calls or mobile push notifications.
>>  Take corrective actions from your mobile device.http://p.sf.net/sfu/Zoho
>>  _______________________________________________
>>  Cppcms-users mailing listCppcms-users-
> 5NW...@pu...://lists.sourceforge.net/lists
> /listinfo/cppcms-users
>> 
>> 
>> 
>> 
>>  --------------------------------------------------------------------------
> ----
>>  Comprehensive Server Monitoring with Site24x7.
>>  Monitor 10 servers for $9/Month.
>>  Get alerted through email, SMS, voice calls or mobile push notifications.
>>  Take corrective actions from your mobile device.
>>  http://p.sf.net/sfu/Zoho
>> 
>>  --------------------------------------------------------------------------
> ----
>>  Comprehensive Server Monitoring with Site24x7.
>>  Monitor 10 servers for $9/Month.
>>  Get alerted through email, SMS, voice calls or mobile push notifications.
>>  Take corrective actions from your mobile device.
>>  http://p.sf.net/sfu/Zoho
> 
> 
> Sorry, I did not understand. Which function do you mean? I think I cannot 
> access the News instance from within the Code.
> 
> If this does not work I would linke to somehow include a separate template 
> (like #include but at template-compile time - I didn't found a possibility 
> to do this).
> 
> Regards,
> bibermann
> 
> 
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 

Re: [Cppcms-users] render sub-template without parents

[bibermann <fab...@gm...>]

Lee Elenbaas <lee@...> writes:

> 
> 
> Template hirarchi translates to class hirarchi, with methods
> You can call the news method from two templates, one as part of a full 
page and another as partial
> בתאריך 15 באוק 2014 17:45, "bibermann" <fabianvss-
Re5...@pu...> כתב:Hello,
> I'm going to use CppCMS for an Ajax-driven site and am wondering whether 
it's
> possible to only return the HTML for a sub-template.
> Considering the example given 
inhttp://cppcms.com/wikipp/en/page/cppcms_1x_tut_templates_inheritance
> where we have the following template hierarchie:
>      master
>    /    |   \
> news  page  intro
> I want to be able to both render the news normally and as part of an ajax
> request to update the news item without reloading the full page, i.e.
> returning page_content() of this template.
> Can I achieve this functionality with CppCMS or do I need to create two
> version of "news"?
> Thanks in advance,
> bibermann
> --------------------------------------------------------------------------
----
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing listCppcms-users-
5NW...@pu...://lists.sourceforge.net/lists
/listinfo/cppcms-users
> 
> 
> 
> 
> --------------------------------------------------------------------------
----
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> 
> --------------------------------------------------------------------------
----
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho


Sorry, I did not understand. Which function do you mean? I think I cannot 
access the News instance from within the Code.

If this does not work I would linke to somehow include a separate template 
(like #include but at template-compile time - I didn't found a possibility 
to do this).

Regards,
bibermann

Re: [Cppcms-users] Slow http:file copy_stream for big files

[Artyom B. <art...@ya...>]

1st of all for the iostreams there is a huge difference between debug and release mode additionally all the callback
code of http code depends on optimization.

Also you are right out << in.rdbuf() should be better... probably one of these parts of code I written too fast without thinking...

However I doubt that it is the problem.

How   http::file works when we receive an input data we write it to the memory at first place but after certain threshold it

is written directly to the file instead of memory.  So this part of code has only small use.

Also I suggest when you actually "save" the file use http::file::save_to as it would rather move the file on the disk and not copy it.

If you still have problems with uploading in release more it would be a good idea for me to profile the code. If so I need to know
what version of cppcms you use cppcms 1.0.x or trunk (there were some changes in that part)

 
Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:   http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/


>________________________________
> From: Cristian Oneț <one...@gm...>
>To: cppcms-users <cpp...@li...> 
>Sent: Monday, October 20, 2014 1:06 PM
>Subject: [Cppcms-users] Slow http:file copy_stream for big files
> 
>
>Hi,
>
>I'm using a simplified version (no widgets::file only request::files)
>of the uploader [1] to implement an upload service. What I observed is
>that uploading large files takes a lot of CPU and takes way more than
>a similar nodejs service [2] (1 min vs 1 s for the same file - 300
>MB). I have to admit that this was cppcms compiled in debug against a
>release nodejs but the difference is too great.
>
>Intrigued by this finding I've took a glimpse at how the file is saved
>to the temporary location using the input stream and found [3].
>Reading from the input stream in 1024 byte chunks then writing that to
>the output does not seem such a great idea and could explain the high
>CPU usage and the overall slowness.
>
>Shouldn't this perform better?
>
>out << in.rdbuf();
>
>I promise I'll get back with the results of running a release build.
>
>Regards,
>Cristian
>
>[1] http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>[2] https://github.com/felixge/node-formidable
>[3] http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>
>------------------------------------------------------------------------------
>Comprehensive Server Monitoring with Site24x7.
>Monitor 10 servers for $9/Month.
>Get alerted through email, SMS, voice calls or mobile push notifications.
>Take corrective actions from your mobile device.
>http://p.sf.net/sfu/Zoho
>_______________________________________________
>Cppcms-users mailing list
>Cpp...@li...
>https://lists.sourceforge.net/lists/listinfo/cppcms-users
>
>
>

[Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

Hi,

I'm using a simplified version (no widgets::file only request::files)
of the uploader [1] to implement an upload service. What I observed is
that uploading large files takes a lot of CPU and takes way more than
a similar nodejs service [2] (1 min vs 1 s for the same file - 300
MB). I have to admit that this was cppcms compiled in debug against a
release nodejs but the difference is too great.

Intrigued by this finding I've took a glimpse at how the file is saved
to the temporary location using the input stream and found [3].
Reading from the input stream in 1024 byte chunks then writing that to
the output does not seem such a great idea and could explain the high
CPU usage and the overall slowness.

Shouldn't this perform better?

out << in.rdbuf();

I promise I'll get back with the results of running a release build.

Regards,
Cristian

[1] http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
[2] https://github.com/felixge/node-formidable
[3] http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78

Re: [Cppcms-users] Getting ready to 1.0.5

[Joerg S. <jo...@br...>]

On Sun, Oct 19, 2014 at 01:32:13AM -0700, Artyom Beilis wrote:
> I'm getting ready to release 1.0.5 it includes many fixes to support
> different compilers. Especially clang and libc++ and some bug fixes.

Can you wait a few days? I'd like to get everything tested and hopefully
write a regression test for the escape option...

Joerg