cppcms-users

[Cppcms-users] Many opened TCP files

[CN <cn...@fa...>]

/var/log/syslog:

Sep 28 06:01:51 host1 cppcms-app: cppcms: Caught exception [Failed to
read /dev/urandom]#0120x4302ad:
booster::runtime_error::runtime_error(std::string const&) + 0x5d in
/usr/local/bin/cppcms-app#0120x7fa3e52ea5e4:
cppcms::urandom_device::generate(void*, unsigned int) + 0xa4 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e52ed961:
cppcms::session_interface::generate_csrf_token() + 0x31 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e52f2e37:
cppcms::session_interface::save() + 0x257 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e52694a3:
cppcms::http::response::write_http_headers(std::ostream&) + 0x23 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e526b1f2:
cppcms::http::response::out() + 0x542 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e527d024:
cppcms::application::render(std::string, cppcms::base_content&) + 0x44
in /usr/local/lib/libcppcms.so.1#0120x454a40: user::login() + 0x690 in
/usr/local/bin/cppcms-app#0120x456ae0: booster::function<void
()>::callable_impl<void, cppcms::url_dispatcher::binder0<user> >::call()
+ 0x30 in /usr/local/bin/cppcms-app#0120x7fa3e5289f61: ??? + 0xe5289f61
in /usr/local/lib/libcppcms.so.1#0120x7fa3e52828cd:
cppcms::url_dispatcher::dispatch(std::string) + 0xdd in
/usr/local/lib/libcppcms.so.1#0120x7fa3e524503b:
cppcms::application::main(std::string) + 0x2b in
/usr/local/lib/libcppcms.so.1#0120x7fa3e528257c: ??? + 0xe528257c in
/usr/local/lib/libcppcms.so.1#0120x7fa3e52828cd:
cppcms::url_dispatcher::dispatch(std::string) + 0xdd in
/usr/local/lib/libcppcms.so.1#0120x42be40:
multi_thread::main(std::string) + 0x470 in
/usr/local/bin/cppcms-app#0120x7fa3e5275824:
cppcms::http::context::dispatch(booster::intrusive_ptr<cppcms::application>
const&, std::string const&, bool) + 0xa4 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e527678e:
cppcms::http::context::dispatch(booster::shared_ptr<cppcms::application_specific_pool>
const&, booster::shared_ptr<cppcms::http::context> const&, std::string
const&) + 0x7e in /usr/local/lib/libcppcms.so.1#0120x7fa3e52786c3:
cppcms::impl::thread_pool::worker() + 0xc3 in
/usr/local/lib/libcppcms.so.1#0120x7fa3e566d52a: booster_thread_func +
0x1a in /usr/local/lib/libbooster.so.0#0120x7fa3e36e60a4: ??? +
0xe36e60a4 in /lib/x86_64-linux-gnu/libpthread.so.0#0120x7fa3e39e387d:
clone + 0x6d in /lib/x86_64-linux-gnu/libc.so.6#012
(http_context.cpp:336)

Sep 28 04:46:15 host1 cppcms-app: cppcms: Caught exception
[cppdb::posgresql: failed to connect: could not create socket: Too many
open files#012]#012 (http_context.cpp:336)
Sep 28 04:46:15 host1 cppcms-app: cppcms: Caught exception
[cppdb::posgresql: failed to connect: could not create socket: Too many
open files#012]#012 (http_context.cpp:336)
Sep 28 04:46:15 host1 cppcms-app: cppcms: Caught exception
[cppdb::posgresql: failed to connect: could not create socket: Too many
open files#012]#012 (http_context.cpp:336)

--------------------

lsof | grep 104.16.51.3 | wc
yields 5916 records like these:

cppcms-app      27567 27591        root 1013u     IPv4        
1052388506      0t0        TCP host1:39384->104.16.54.3:https
(CLOSE_WAIT)
cppcms-app      27567 27591        root 1014u     IPv4        
1052388558      0t0        TCP host1:39390->104.16.54.3:https
(CLOSE_WAIT)
cppcms-app      27567 27591        root 1015u     IPv4        
1052388584      0t0        TCP host1:39394->104.16.54.3:https
(CLOSE_WAIT)
cppcms-app      27567 27591        root 1016u     IPv4        
1052388586      0t0        TCP host1:39400->104.16.54.3:https
(CLOSE_WAIT)

IP 104.16.54.3 is one of CloudFlare's host.

--------------------

Process# 27567 comes from the running CppCMS program:
./cppcms-app -c my-config.js

Is it possible that CppCMS does not cleanly close files?

Best Regards,
CN

-- 
http://www.fastmail.com - Access your email from home and the web

Re: [Cppcms-users] Many opened TCP files

[Artyom B. <art...@gm...>]

It should.

Several questions:

1. Who is the client - are these users or PROXY server?
2. What is the protocol? it appears as https in the list - is it
HTTP/FastCGI or SGI
3. Post your my-confg.js - do you use http timeouts, what are they?
4. What version of CppCMS do you use
5. Can you connect with strace to the server and see how typical
connection behave - i.e. see what system calls related to a socket
executed.

Artyom

>
> Process# 27567 comes from the running CppCMS program:
> ./cppcms-app -c my-config.js
>
> Is it possible that CppCMS does not cleanly close files?
>
> Best Regards,
> CN
>
> --
> http://www.fastmail.com - Access your email from home and the web
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users

Re: [Cppcms-users] Many opened TCP files

[CN <cn...@fa...>]

On Wed, Oct 5, 2016, at 04:37 PM, Artyom Beilis wrote:
> It should.
> 
> Several questions:
> 
> 1. Who is the client - are these users or PROXY server?

CppCMS program runs in my host, which is behind CloudFlare's PROXY
network:

- Clients (browsers?) send http or https requests to my host.
- CloudFlare sends these requests to my host on behalf of these clients.
- These requests arrive at my host. Httpd converts http to https.
- CppCMS program responds to CloudFlare.
- CloudFlare responds results to clients using https.

> 2. What is the protocol? it appears as https in the list - is it
> HTTP/FastCGI or SGI

HTTPS/FastCGI

> 3. Post your my-confg.js - do you use http timeouts, what are they?

{
	"service":{
		"api":"fastcgi",
		"socket":"/tmp/home.sock",
		"disable_global_exit_handling":true
	},
	"session":{
		"location":"client",
		"expire":"browser",
		"timeout":72000,
		"cookies":{
			"prefix":"home"
		},
		"client":{
			"cbc":"aes",
			"cbc_key":"my cbc key",
			"hmac":"sha1",
			"hmac_key":"my hmac key"
		}
	},
	"daemon":{
		"enable":false,
		"lock":"/var/run/home.pid",
		"user":"user2",
		"group":"group2"
	},
	"http":{
		"script_names":["/home"]
	},
	"cache":{
		"backend":"thread_shared", 
		"limit":100
	},
	"security":{
		"csrf":{"enable":true}
	},
	"logging":{
		"level":"debug",
		"stderr":false,
		"syslog":{
			"enable":true,
			"id":"home"
		}
	}
}

Parameter "http.timeout" is not specified in configuration file. I
assume it defaults to 30 seconds according to
http://cppcms.com/wikipp/en/page/cppcms_1x_config#http.timeout

Last time when there was huge number of CLOSE_WAIT, those open
connections appeared to remain open regardless of the 30-second timeout.
They seemed to be not automatically closed by the time out mechanism.

> 4. What version of CppCMS do you use

version 1.2

> 5. Can you connect with strace to the server and see how typical
> connection behave - i.e. see what system calls related to a socket
> executed.

After I re-started CppCMS, I can no longer reproduce this problem -
"lsof" now prints only one CLOSE_WAIT entry.  I know this is not
professional, but I will postpone trying strace until when the number of
CLOSE_WAIT surges again.

Best Regards,
CN

-- 
http://www.fastmail.com - Does exactly what it says on the tin

Re: [Cppcms-users] Many opened TCP files

[Artyom B. <art...@gm...>]

Another reason I can see is that your async app does not handle
timeouts/disconnects properly and keeps connection alive.

Artyom

On Wed, Oct 5, 2016 at 11:37 AM, Artyom Beilis <art...@gm...> wrote:
> It should.
>
> Several questions:
>
> 1. Who is the client - are these users or PROXY server?
> 2. What is the protocol? it appears as https in the list - is it
> HTTP/FastCGI or SGI
> 3. Post your my-confg.js - do you use http timeouts, what are they?
> 4. What version of CppCMS do you use
> 5. Can you connect with strace to the server and see how typical
> connection behave - i.e. see what system calls related to a socket
> executed.
>
> Artyom
>
>>
>> Process# 27567 comes from the running CppCMS program:
>> ./cppcms-app -c my-config.js
>>
>> Is it possible that CppCMS does not cleanly close files?
>>
>> Best Regards,
>> CN
>>
>> --
>> http://www.fastmail.com - Access your email from home and the web
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users

Re: [Cppcms-users] Many opened TCP files

[CN <cn...@fa...>]

On Wed, Oct 5, 2016, at 04:39 PM, Artyom Beilis wrote:
> Another reason I can see is that your async app does not handle
> timeouts/disconnects properly and keeps connection alive.

This program does not mount any asynchronous application.

But your enlightenment does lead me to suspect my synchronous
applications.  Some methods in these synchronous applications do not
return until they acquire outside resources like Google re-captcha. 
Probably this the cause of the large number of CLOSE_WAIT.

On the other hand, even the delayed (or never) responses from outside
resources might be the cause to blame, I still have no idea why
http.timeout mechanism did not proactively close the long idle open
connections.

Best Regards,
CN

-- 
http://www.fastmail.com - Access your email from home and the web

Re: [Cppcms-users] Many opened TCP files

[Artyom B. <art...@gm...>]

Hi,

>From what I see

1. The open connections you see aren't ones created by the cppcms framework
itself. The communication with the server actually uses Unix domain sockets
rather than tcp/ip.
2. HTTP. Timeout affects only HTTP connections, not fastcgi as the sever is
already handles them.
3. Indeed check your client.

Artyom Beilis

בתאריך 7 באוק׳ 2016 08:59,‏ "CN" <cn...@fa...> כתב:
>
> On Wed, Oct 5, 2016, at 04:39 PM, Artyom Beilis wrote:
> > Another reason I can see is that your async app does not handle
> > timeouts/disconnects properly and keeps connection alive.
>
> This program does not mount any asynchronous application.
>
> But your enlightenment does lead me to suspect my synchronous
> applications.  Some methods in these synchronous applications do not
> return until they acquire outside resources like Google re-captcha.
> Probably this the cause of the large number of CLOSE_WAIT.
>
> On the other hand, even the delayed (or never) responses from outside
> resources might be the cause to blame, I still have no idea why
> http.timeout mechanism did not proactively close the long idle open
> connections.
>
> Best Regards,
> CN
>
> --
> http://www.fastmail.com - Access your email from home and the web
>
>
>
------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users

Re: [Cppcms-users] Many opened TCP files

[CN <cn...@fa...>]

Many thanks for your kind help!

Now I remember I configured the proxy (my sole client) to always send
only https requests to my host.

Is there any chance that the following scenario could possibly happen?

Lighttpd somehow did not provide services - maybe temporarily brought
down by myself.  As a result, all requests from proxy rushed to CppCMS
process.  Because the parameter "service.api" is configured as
"fastcgi", CppCMS process tried to ignore all http requests.  However,
because the http server embed in CppCMS did not recognize https, it
therefore did not completely close the https connections with proxy.

Best Regards,
CN

On Sat, Oct 8, 2016, at 01:12 AM, Artyom Beilis wrote:
> Hi,
> From what I see
> 1. The open connections you see aren't ones created by the cppcms
>    framework itself. The communication with the server actually uses
>    Unix domain sockets rather than tcp/ip.
>  2. HTTP. Timeout affects only HTTP connections, not fastcgi as the
>     sever is already handles them.
>  3. Indeed check your client.

-- 
http://www.fastmail.com - Does exactly what it says on the tin

Re: [Cppcms-users] Many opened TCP files

[Artyom B. <art...@gm...>]

> Many thanks for your kind help!
>
> Now I remember I configured the proxy (my sole client) to always send only
> https requests to my host.
>
> Is there any chance that the following scenario could possibly happen?
>
> Lighttpd somehow did not provide services - maybe temporarily brought down
> by myself.  As a result, all requests from proxy rushed to CppCMS process.
> Because the parameter "service.api" is configured as "fastcgi", CppCMS
> process tried to ignore all http requests.  However, because the http server
> embed in CppCMS did not recognize https, it therefore did not completely
> close the https connections with proxy.

See, quote from the configuration file"

    "service":{
                "api":"fastcgi",
                "socket":"/tmp/home.sock",
                "disable_global_exit_handling":true
    }

It means that CppCMS:

1. Brings up ONLY FastCGI module and HTTP is turned off
2. It works over UNIX domain sockets, not TCP/IP (that https uses),
basically CppCMS does not open or close ANY tcp/ip connection so it
must be your's :-)


Artyom Beilis