#1759 Messages like "Cannot open template file /etc/privoxy/templates/connect-failed: Too many open files"

[withoutname]

Messages like this appear from time to time:

01:27:27.041 7f968b7fe6c0 Error: Cannot open template file /etc/privoxy/templates/connect-failed: Too many open files
01:27:27.041 7f97b043d6c0 Error: Cannot open template file /etc/privoxy/templates/cgi-style.css: Too many open files
01:27:28.048 7f97af43b6c0 Error: Cannot open template file /etc/privoxy/templates/cgi-style.css: Too many open files
01:27:28.051 7f97af43b6c0 Error: Cannot open template file /etc/privoxy/templates/connect-failed: Too many open files
01:27:29.012 7f97957fa6c0 Error: Cannot open template file /etc/privoxy/templates/connect-failed: Too many open files
01:27:29.041 7f972cff96c0 Error: Cannot open template file /etc/privoxy/templates/cgi-style.css: Too many open files

At the same time, there is an increase in open sockets to approximately 8898 for the privoxy process. I gradually increased the value of "Max open files" through the parameter for the privoxy unit "LimitNOFILE" and reached 10240, i.e. "LimitNOFILE=10240". But it seems that after this other problems begin. After reaching this limit, privoxy begins to load the processor more than usual, 100-200% and above for a long time of 1-4 hours. When "LimitNOFILE=8192" the processor load also increases, a message is observed, as above in the log, after which the processor load drops to normal, 1-10%. The value "LimitNOFILE=8900" is actually the maximum for privoxy. I'm using the latest privoxy snapshot.

[Fabian Keil]

How many users does your Privoxy instance have?

The number of open sockets can sometimes be reduced by decreasing the keep-alive-timeout and socket-timeout values. Did you increase them from the defaults?

[withoutname]

From 1 to 3 users. The same messages appear at night, when in fact users are left with only one device - NAS.
Part of the config:

keep-alive-timeout 60
default-server-timeout 5
socket-timeout 30
max-client-connections 8192

It looks like this.

[withoutname]

The number of open sockets.

[Fabian Keil]

Note that most client connections need two file descriptors to be served so with 8192 client connections may need ~16384 file descriptors.

It's unclear to me why the processor load should significantly increase when the limit is reached, though.

Also 8192 connections seem to be a lot for up to three clients.

In ticket #1762 you wrote that you are Squid in "transparent mode".

Is it possible that some of Privoxy's outgoing connections are intercepted again?

Enabling logging would probably help to diagnose this.

[withoutname]

So far this is what I see. But this coincides when for some reason Privoxy increases (or something forces it to do so) the number of threads (first picture). At this moment, the messages that were reported in my initial message appear in the log. Regarding Squid, I don’t understand what you mean, but I’ll describe the config. Privoxy is an upstream proxy for Squid, the Squid config is literally as in the documentation from Privoxy. Ports 80 and 443 are redirected to Squid via iptables.
Privoxy:

...
listen-address 127.0.0.1:8118
listen-address [::1]:8118
...

Squid:

...
http_port 3128
http_port 3129 intercept
https_port 3130 intercept ssl-bump connection-auth=off options=ALL tls-cert=/etc/squid/squidCA.pem

...
cache_peer 127.0.0.1 parent 8118 0 no-query no-digest
cache_peer_access 127.0.0.1 deny globalIPv6
cache_peer [::1] parent 8118 0 no-query no-digest
cache_peer_access [::1] deny !globalIPv6
...

iptables:

...
ip(6)tables -t nat -A PREROUTING -i br0 -p tcp -m multiport --dports 80,2710,6969 -j REDIRECT --to-ports 3129
ip(6)tables -t nat -A PREROUTING -i br0 -p tcp -m tcp --dport 443-j REDIRECT --to-ports 3130
...

br0 - local interface

[withoutname]

When you suggested that the problem might be related to Squid, you prompted me to analyze this situation. In the end, I found out that the problem was related to the "accept-intercepted-requests 1" option. I tried setting it to 0 and so far this problem has not occurred. But the message "Error: ::1's request: 'GET /squid-internal-dynamic/netdb HTTP/1.1' is invalid. Privoxy isn't configured to accept intercepted requests" appeared in the log. There are also others that are similar (I changed the contents of the requests):

Error: 127.0.0.1's request: 'GET /ann?pk=abcdeа HTTP/1.1' is invalid. Privoxy isn't configured to accept intercepted requests.
Error: 127.0.0.1's request: 'GET /qwerty/announce?info_hash=abcdef HTTP/1.1' is invalid. Privoxy isn't configured to accept intercepted requests.

In general it works, but I don’t know whether these requests are going through or not

[Fabian Keil]

Setting "accept-intercepted-requests 0" in your setup seems like the right thing to do as you are redirecting intercepted connections into Squid and not into Privoxy.

With "accept-intercepted-requests 1" one has to be careful and prevent Privoxy from connecting to itself. Quoting the documentation:

"Make sure that Privoxy's own requests aren't redirected as well. Additionally take care that Privoxy can't intentionally connect to itself, otherwise you could run into redirection loops if Privoxy's listening port is reachable by the outside or an attacker has access to the pages you visit."
https://www.privoxy.org/user-manual/config.html#ACCEPT-INTERCEPTED-REQUESTS

If Squid is the only client that is supposed to use Privoxy there should be no need to have Privoxy listen on both {::1] and 127.0.0.1 so you could simplify your setup by having Privoxy only listen on one address.

The requests that result in "... is invalid. Privoxy isn't configured to accept intercepted requests." are rejected by Privoxy so you may want to investigate why they reach
Privoxy in the first place.

[withoutname]

OK I understood.