[Galene] Websocket close 1006 (abnormal closure): unexpected EOF

[Galene] Websocket close 1006 (abnormal closure): unexpected EOF

[Jeroen van Veen]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

Hi,

I've been testing with Galene a bit, but noticed that I got disconnected automatically after a while.
It seems that mywebsocket connection is killed:

2021/01/05 11:24:22 client: websocket: close 1006 (abnormal closure): unexpected EOF

Another client had no disconnect issues at all. I'm not sure if this is a Gorilla websocket issue(https://stackoverflow.com/questions/61108552/go-websocket-error-close-1006-abnormal-closure-unexpected-eof), an unstable wifi connection at my side or whether Nginx may be an issue. Anyone?

My nginx config:

server {
listen 443 ssl;
server_name galene.domain.app;
index index.html;

access_log /var/log/nginx/galene.domain.app.access.log;

error_log /var/log/nginx/galene.domain.app.error.log;

location /ws {

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $host;

proxy_pass https://127.0.0.1:8443;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection "Upgrade";

}

location / {

proxy_pass https://127.0.0.1:8443;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection "Upgrade";

}

ssl_certificate /etc/letsencrypt/live/galene.domain.app/fullchain.pem; # managed by Certbot

ssl_certificate_key /etc/letsencrypt/live/galene.domain.app/privkey.pem; # managed by Certbot

}

[-- Attachment #2: Type: text/html, Size: 5084 bytes --]

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Juliusz Chroboczek]

> I've been testing with Galene a bit, but noticed that I got disconnected
> automatically after a while.

Nginx timing out the websocket, perhaps?  Could you please try with the
following patch applied?

diff --git a/rtpconn/webclient.go b/rtpconn/webclient.go
index b651f57..577efba 100644
--- a/rtpconn/webclient.go
+++ b/rtpconn/webclient.go
@@ -905,7 +905,7 @@ func clientLoop(c *webClient, ws *websocket.Conn) error {
 			if time.Since(readTime) > 90*time.Second {
 				return errors.New("client is dead")
 			}
-			if time.Since(readTime) > 60*time.Second {
+			if time.Since(readTime) > 20*time.Second {
 				err := c.write(clientMessage{
 					Type: "ping",
 				})

-- Juliusz

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Toke Høiland-Jørgensen]

Juliusz Chroboczek <jch@irif.fr> writes:

>> I've been testing with Galene a bit, but noticed that I got disconnected
>> automatically after a while.
>
> Nginx timing out the websocket, perhaps?  Could you please try with the
> following patch applied?

Yeah, the default timeout in nginx is 60s:

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

so if Galene actually waits that long it is likely that it trips the
timeout...

I just normally put

proxy_read_timeout 1200s;
proxy_send_timeout 1200s;

in my nginx config (though I'm not proxying Galene)...

-Toke

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Jeroen van Veen]

Thanks for the help and info! I tested both the patch and the proxy timeout config
change. The patch fixes the issue with the default nginx timeout and the config does
the same using the existing code.

cheers,

Jeroen


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Op dinsdag, januari 5, 2021 3:52 PM, Toke Høiland-Jørgensen <toke@toke.dk> schreef:

> Juliusz Chroboczek jch@irif.fr writes:
>
> > > I've been testing with Galene a bit, but noticed that I got disconnected
> > > automatically after a while.
> >
> > Nginx timing out the websocket, perhaps? Could you please try with the
> > following patch applied?
>
> Yeah, the default timeout in nginx is 60s:
>
> https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
>
> so if Galene actually waits that long it is likely that it trips the
> timeout...
>
> I just normally put
>
> proxy_read_timeout 1200s;
> proxy_send_timeout 1200s;
>
> in my nginx config (though I'm not proxying Galene)...
>
> -Toke
>
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Michael Ströder]

On 1/5/21 3:52 PM, Toke Høiland-Jørgensen wrote:
> Juliusz Chroboczek <jch@irif.fr> writes:
> 
>>> I've been testing with Galene a bit, but noticed that I got disconnected
>>> automatically after a while.
>>
>> Nginx timing out the websocket, perhaps?  Could you please try with the
>> following patch applied?
> 
> Yeah, the default timeout in nginx is 60s:
> https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
> 
> so if Galene actually waits that long it is likely that it trips the
> timeout...
> 
> I just normally put
> 
> proxy_read_timeout 1200s;
> proxy_send_timeout 1200s;

20min? Really? Even 120s would be quite long.

Ciao, Michael.

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Toke Høiland-Jørgensen]

Michael Ströder <michael@stroeder.com> writes:

> On 1/5/21 3:52 PM, Toke Høiland-Jørgensen wrote:
>> Juliusz Chroboczek <jch@irif.fr> writes:
>> 
>>>> I've been testing with Galene a bit, but noticed that I got disconnected
>>>> automatically after a while.
>>>
>>> Nginx timing out the websocket, perhaps?  Could you please try with the
>>> following patch applied?
>> 
>> Yeah, the default timeout in nginx is 60s:
>> https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
>> 
>> so if Galene actually waits that long it is likely that it trips the
>> timeout...
>> 
>> I just normally put
>> 
>> proxy_read_timeout 1200s;
>> proxy_send_timeout 1200s;
>
> 20min? Really? Even 120s would be quite long.

Yeah, won't necessarily recommend this for an internet-facing
deployment. But this is mostly for services I run for myself internally
(and I'm not proxying Galene at all), and the server has plenty of
resources to spare to keep a few sockets open :)

-Toke

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Juliusz Chroboczek]

>> I just normally put
>> 
>> proxy_read_timeout 1200s;
>> proxy_send_timeout 1200s;

> 20min? Really? Even 120s would be quite long.

I've reduced the keepalive timeout to be 45 to 55s (depending on how the
timer gets scheduled), even though I rather agree with Toke.  Scratch
that, Toke is not radical enoough: the default timeout for WebSockets
should be 2 hours 4 minutes, for consistency with RFC 5382 REQ-5.

WebSocket is layered over TCP.  TCP indicates that a connection is closed
by sending explicit FIN segments in both directions: both the client and
the server explicitly close the connection.

The reverse proxy should assume that the server performs its own timing
out of idle clients, and only close the client connection when the server
has closed its side.  If the server crashes, the OS will close the
connection.  If the server's OS crashes, then the client will timeout the
connection.  The only case when the timeout is needed is when both the
server and the client have left the network without closing their
connections -- and in this rare case, a large timeout is good enough.

The tradition to use small timeouts comes from Apache, which used to use
up a whole process for every connection, even when idle.  There is no
reason I can see to do the same with nginx, which is only limited by the
number of available file descriptors.

-- Juliusz

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Michael Ströder]

On 1/8/21 2:19 PM, Juliusz Chroboczek wrote:
>>> I just normally put
>>> proxy_read_timeout 1200s;
>>> proxy_send_timeout 1200s;
> 
>> 20min? Really? Even 120s would be quite long.
> 
> I've reduced the keepalive timeout to be 45 to 55s (depending on how the
> timer gets scheduled), even though I rather agree with Toke.

Yeah, I've installed that right now.

> Scratch
> that, Toke is not radical enoough: the default timeout for WebSockets
> should be 2 hours 4 minutes, for consistency with RFC 5382 REQ-5.

Noted.

> The tradition to use small timeouts comes from Apache, which used to use
> up a whole process for every connection, even when idle.

This should not be an issue with Apache's MPM event model anymore. Right?

Ciao, Michael.

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Juliusz Chroboczek]

>> The tradition to use small timeouts comes from Apache, which used to use
>> up a whole process for every connection, even when idle.

> This should not be an issue with Apache's MPM event model anymore. Right?

I'm not sure how Apache's reverse proxy module is implemented.  Probably
well, but I haven't checked.

(I didn't mean my previous message as a dig at Apache, there are good
reasons to prefer simplicity of implementation and robustness to
performance.  With nginx and IIS, a single buggy module will bring your
server down, while at least some configurations of Apache will happily
chug along after the PHP interpreter has crashed.

I did mean it as a dig at people who use the same timeouts for HTTP and
WebSockets.  This practice has a real cost in terms of mobile battery usage.)

-- Juliusz

[Galene] Re: Websocket close 1006 (abnormal closure): unexpected EOF

[Michael Ströder]

On 1/8/21 3:58 PM, Juliusz Chroboczek wrote:
>>> The tradition to use small timeouts comes from Apache, which used to use
>>> up a whole process for every connection, even when idle.
> 
>> This should not be an issue with Apache's MPM event model anymore. Right?
> 
> I'm not sure how Apache's reverse proxy module is implemented.  Probably
> well, but I haven't checked.
> 
> (I didn't mean my previous message as a dig at Apache,

I did not read any negative attitude in your previous message.

> I did mean it as a dig at people who use the same timeouts for HTTP and
> WebSockets.  This practice has a real cost in terms of mobile battery usage.)

Noted. These hints are highly appreciated.

Ciao, Michael.