[Galene] Galene Turn

[Galene] Galene Turn

[Fabrice Rouillier]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

Hi, 

I did post an « Issue » on gitlab but I realize this was not the right place, sorry.

Let me re-post.

Thanks a lot for this project, Galène is working like a charm and is so simple to install / customize / use.

I have one problem to submit :

Server : in a linux 20.04 virtual box - behind a NAT (Freebox) and a reverse proxy (Traefik) launched with the command :

/home/galene/galene -insecure -turn EXTERNAL_IP:1194

Works like a charm excepted when the turn is solicited, for example from a client using IPV4 and firefox from a 4G connexion.

the /relay-test command on the client side drives to an Error (timeout) which tends to prove that the turn is not working properly
the "Relay test" on the server siide is successfull


So I conclude that the turn can no be accessed from outside

I am certainly missing something in the configuration but I do not find ….


All the best,

Fabrice.


[-- Attachment #2: Type: text/html, Size: 5002 bytes --]

[Galene] Re: Galene Turn

[Alexandre IOOSS]

On 4/9/21 4:37 PM, Fabrice Rouillier wrote:
> Server : in a linux 20.04 virtual box - behind a NAT (Freebox) and
> a reverse proxy (Traefik) launched with the command :
> 

Hello Fabrice,

In the NAT table, did you forward the external 1194 port to your server
1194 port in UDP and TCP?

On Galène web client, you can try to open the JavaScript console and
type `serverConnection.rtcConfiguration.iceServers[0].urls` to check
that the URLs here matches your public IP address.

Best,

-- 
Alexandre
Trying to help

[Galene] Re: Galene Turn

[Fabrice Rouillier]

Hi Alexandre,

> In the NAT table, did you forward the external 1194 port to your server
> 1194 port in UDP and TCP?

I am using the DMZ

Freebox -> [DMZ] -> Server -> [Traefik] -> Galene


> On Galène web client, you can try to open the JavaScript console and
> type `serverConnection.rtcConfiguration.iceServers[0].urls` to check
> that the URLs here matches your public IP address.
> 

In the Chrome console, they do.

In the firefox console … the instruction does not exist.


Best


Fabrice

[Galene] Re: Galene Turn

[Fabrice Rouillier]

>> On Galène web client, you can try to open the JavaScript console and
>> type `serverConnection.rtcConfiguration.iceServers[0].urls` to check
>> that the URLs here matches your public IP address.
>> 


Sorry, match also in firefox.


Best,

Fabrice



> 
> In the Chrome console, they do.
> 
> In the firefox console … the instruction does not exist.
> 
> 
> Best
> 
> 
> Fabrice
> 
> 
> 

[Galene] Re: Galene Turn

[Alexandre IOOSS]

On 4/9/21 5:32 PM, Fabrice Rouillier wrote:
> I am using the DMZ
> 
> Freebox -> [DMZ] -> Server -> [Traefik] -> Galene

I don't think the fact that you are using a reverse proxy (Traefik) is
causing the problem. If the DMZ is doing what I think, you should not
even need a TURN if all UDP port from 10000 to 65535 are open.

Are you running Galène in a container, or directly on the machine?
Does your server have a firewall that would restrict 1194 in TCP/UDP?

I don't really now how Galène TURN server works, but maybe because you
are passing your public IP address, it's going to try to listen on a
interface with this public address which you don't have (because of DMZ).

Last idea that might help to find the issue: you can try to scan for
open ports on your public IP using `nmap`. You should see "openvpn" service.

```
$ sudo nmap -sS -1194 PUBLIC_IP
Starting Nmap 7.80 ( https://nmap.org ) at 2021-04-09 19:05 CEST
Nmap scan report for PUBLIC_IP
Host is up (0.014s latency).

PORT     STATE SERVICE
1194/tcp open  openvpn

Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
```

Best,

-- 
Alexandre

[Galene] Re: Galene Turn

[Fabrice Rouillier]

First : I have founded a solution using a second server (still in a VM behind my rooter) implementing coturn, so I have now a working solution without any additional material.

But I am interesting in using the builtin turn so I will pay time to test and try to understand.

> Are you running Galène in a container, or directly on the machine?

Directly on the machine.

> Does your server have a firewall that would restrict 1194 in TCP/UDP?
> 
No.

> Last idea that might help to find the issue: you can try to scan for
> open ports on your public IP using `nmap`. You should see "openvpn" service.
> 
Good idea, I will try.

Best,

Fabrice.

> ```
> $ sudo nmap -sS -1194 PUBLIC_IP
> Starting Nmap 7.80 ( https://nmap.org ) at 2021-04-09 19:05 CEST
> Nmap scan report for PUBLIC_IP
> Host is up (0.014s latency).
> 
> PORT     STATE SERVICE
> 1194/tcp open  openvpn
> 
> Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
> ```
> 
> Best,
> 
> -- 
> Alexandre

[Galene] Re: Galene Turn

[Fabrice Rouillier]

A complement due to my last try : the coturn server  and the  Galene server can co-exist on the same machine and everything works fine.

So one should concentrate on the -turn  EXTERNAL_IP:1194 that does not work properly with such a specific configuration.
 
Cheers,

Fabrice.

> Le 9 avr. 2021 à 19:36, Fabrice Rouillier <fabrice@rouillier.fr> a écrit :
> 
> First : I have founded a solution using a second server (still in a VM behind my rooter) implementing coturn, so I have now a working solution without any additional material.
> 
> But I am interesting in using the builtin turn so I will pay time to test and try to understand.
> 
>> Are you running Galène in a container, or directly on the machine?
> 
> Directly on the machine.
> 
>> Does your server have a firewall that would restrict 1194 in TCP/UDP?
>> 
> No.
> 
>> Last idea that might help to find the issue: you can try to scan for
>> open ports on your public IP using `nmap`. You should see "openvpn" service.
>> 
> Good idea, I will try.
> 
> Best,
> 
> Fabrice.
> 
>> ```
>> $ sudo nmap -sS -1194 PUBLIC_IP
>> Starting Nmap 7.80 ( https://nmap.org ) at 2021-04-09 19:05 CEST
>> Nmap scan report for PUBLIC_IP
>> Host is up (0.014s latency).
>> 
>> PORT     STATE SERVICE
>> 1194/tcp open  openvpn
>> 
>> Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
>> ```
>> 
>> Best,
>> 
>> -- 
>> Alexandre
> 

[Galene] Re: Galene Turn

[Juliusz Chroboczek]

> A complement due to my last try : the coturn server and the Galene
> server can co-exist on the same machine and everything works fine.  So
> one should concentrate on the -turn EXTERNAL_IP:1194 that does not work
> properly with such a specific configuration.

The problem is probably that your NAT doesn't implement hairpinning, so
Galène and the TURN server cannot communicate.  Coturn is probably
connecting over IPv6, which the built-in server doesn't handle.

Could you please check Galène's log for mentions of a relay test?  If the
server-side relay test (the one in Galène's log) fails, that's probably
indicative of a problem with hairpinning.

Fabrice — if your instance of Galène is behind NAT, then I strongly
recommend running an instance of coturn on a host that is not behind NAT.
Twiddling with NAT rules is fragile and error-prone, I doubt you'll be
able to run a robust service in that way.

-- Juliusz

[Galene] Re: Galene Turn

[Fabrice Rouillier]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

Hi Juliusz,

Thanks for your help.
See below for a simple setting that works with our external coturn.

> The problem is probably that your NAT doesn't implement hairpinning, so
> Galène and the TURN server cannot communicate.
>  Coturn is probably
> connecting over IPv6, which the built-in server doesn't handle.

I have deactivated the IPV6 everywhere and double checked (inspecting about:websocket <about:websocket>) in Firefox that it does not use IPV6.

> Could you please check Galène's log for mentions of a relay test?  If the
> server-side relay test (the one in Galène's log) fails, that's probably
> indicative of a problem with hairpinning.

Server side : the relay-test do not fails.

Client side : 
   - using the builtin turn, the relay test fails , 
   - using coturn (on the same server) it works.

> Fabrice — if your instance of Galène is behind NAT, then I strongly
> recommend running an instance of coturn on a host that is not behind NAT.

The following works : 

- port forwarding from my router (Freebox) to a Linux 20.04  VM (virtual box) : 

   49152-65535 (UDP/TCP)
   3478 (UDP/TCP) 
   8443 (UDP/TCP) 

- coturn on the VM with the following configuration

  listening-port=3478
  fingerprint
  lt-cred-mech
  user=<TURN USER>:<TURN PASSWD>
  server-name=<TURN SERVER NAME>
  realm=<DOMAINE>

- galene on the VM launched with the option -turn auto 
  and data/ice-servers.json with the following contents

[
    {
        "Urls": [
            "turn:<TURN SERVER NAME>:3478",
            « turn:<TURN SERVER NAME>:3478?transport=tcp"
        ],
        "username": "<TURN USER>",
        "credential": "<TURN PASSWD>"
    }
]


[-- Attachment #2: Type: text/html, Size: 5541 bytes --]

[Galene] Installation Script

[Fabrice Rouillier]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

Hi,

I have implemented and tested a script to install Galene and a TURN that works behind a Router, even a domestic one.

You can doowload it here : https://gitlab.inria.fr/rouillie/visio/-/tree/master/galene

It works only for Ubuntu 20.04 or Debian buster for the moment, mainly because I am lazy :-) but it opens the door to an very easy deployment on a cloud or simply at home.

Cheers,

Fabrice
-------------------------
Fabrice Rouillier
fabrice@rouillier.fr

Bureau virtuel : https://www.rouillier <https://www.rouillier/>.fr/visio/fabrice





> Le 10 avr. 2021 à 16:41, Fabrice Rouillier <fabrice@rouillier.fr> a écrit :
> 
> Hi Juliusz,
> 
> Thanks for your help.
> See below for a simple setting that works with our external coturn.
> 
>> The problem is probably that your NAT doesn't implement hairpinning, so
>> Galène and the TURN server cannot communicate.
>>  Coturn is probably
>> connecting over IPv6, which the built-in server doesn't handle.
> 
> I have deactivated the IPV6 everywhere and double checked (inspecting about:websocket <about:websocket>) in Firefox that it does not use IPV6.
> 
>> Could you please check Galène's log for mentions of a relay test?  If the
>> server-side relay test (the one in Galène's log) fails, that's probably
>> indicative of a problem with hairpinning.
> 
> Server side : the relay-test do not fails.
> 
> Client side : 
>    - using the builtin turn, the relay test fails , 
>    - using coturn (on the same server) it works.
> 
>> Fabrice — if your instance of Galène is behind NAT, then I strongly
>> recommend running an instance of coturn on a host that is not behind NAT.
> 
> The following works : 
> 
> - port forwarding from my router (Freebox) to a Linux 20.04  VM (virtual box) : 
> 
>    49152-65535 (UDP/TCP)
>    3478 (UDP/TCP) 
>    8443 (UDP/TCP) 
> 
> - coturn on the VM with the following configuration
> 
>   listening-port=3478
>   fingerprint
>   lt-cred-mech
>   user=<TURN USER>:<TURN PASSWD>
>   server-name=<TURN SERVER NAME>
>   realm=<DOMAINE>
> 
> - galene on the VM launched with the option -turn auto 
>   and data/ice-servers.json with the following contents
> 
> [
>     {
>         "Urls": [
>             "turn:<TURN SERVER NAME>:3478",
>             « turn:<TURN SERVER NAME>:3478?transport=tcp"
>         ],
>         "username": "<TURN USER>",
>         "credential": "<TURN PASSWD>"
>     }
> ]
> 


[-- Attachment #2: Type: text/html, Size: 9548 bytes --]

[Galene] Re: Installation Script

[Rémy Dernat]

[-- Attachment #1: Type: text/plain, Size: 6450 bytes --]

Hi Fabrice,

Thanks. That could be helpful.

Nevertheless, I have done a couple of things that could also be useful 
for your script. If you want, feel free to pick things/change your 
script accordingly.

First, there is a nginx configuration available from Yunohost package 
here : 
https://github.com/YunoHost-Apps/galene_ynh/blob/testing/conf/nginx.conf

This configuration file was very useful to me. I am using it for 
proxying Galène on HTTPS. Here is a resulting configuration using it 
in/etc/nginx/sites-enabled/galene :

    server {
         listen 443 ssl default_server;
         listen [::]:443 ssl default_server;
         server_name galene.domain.tld visio.domain.tld;
         ssl_certificate
    /etc/letsencrypt/live/galene.domain.tld/fullchain.pem;
         ssl_certificate_key
    /etc/letsencrypt/live/galene.domain.tld/privkey.pem;

         location / {

           # Force usage of https
           if ($scheme = http) {
             rewrite ^ https://$server_name$request_uri? permanent;
           }

           proxy_pass        https://127.0.0.1:8443;
           proxy_redirect    off;
           proxy_set_header  Host $host;
           proxy_set_header  X-Real-IP $remote_addr;
           proxy_set_header  X-Forwarded-Proto $scheme;
           proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header  X-Forwarded-Host $server_name;

           # WebSocket support
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection "upgrade";
         }
    }

To avoid any issue with let's encrypt renew crons and scripts (that 
would need another copy in ~galene/data), I added the acl package to 
allow galene user to read /etc/letsencrypt/ (instead of just copying the 
certs in /home/galene/data/:

    apt-get install acl

    setfacl -R -m u:galene:r /etc/letsencrypt

    setfacl -m u:galene:x /etc/letsencrypt

    setfacl -m u:galene:x /etc/letsencrypt/live

    setfacl -m u:galene:x /etc/letsencrypt/live/galene.domain.tld

    setfacl -m u:galene:x /etc/letsencrypt/archive/galene.domain.tld

    setfacl -m u:galene:x /etc/letsencrypt/archive

    ln -s /etc/letsencrypt/live/galene.domain.tld/chain.pem
    ~galene/data/chain.pem

    ln -s /etc/letsencrypt/live/galene.domain.tld/privkey.pem
    ~galene/data/key.pem

    ln -s /etc/letsencrypt/live/galene.domain.tld/privkey.pem
    ~galene/data/privkey.pem

    ln -s /etc/letsencrypt/live/galene.domain.tld/cert.pem
    ~galene/data/cert.pem

    ln -s /etc/letsencrypt/live/galene.domain.tld/fullchain.pem
    ~galene/data/fullchain.pem


Then, I also added a configuration file to source in 
/etc/default/galene, containing (after creating the /var/log directories) :

    ARGS=" -memprofile /var/log/galene/mem.log -cpuprofile
    /var/log/galene/cpu.log"

But you could add your turn configuration or whatever option...

Modification to the systemd script is very simple; just change:

    ExecStart=/home/galene/galene

to:

    EnvironmentFile=/etc/default/galene
    ExecStart=/home/galene/galene $ARGS


By the way, if anyone knows how to use these {cpu,mem} profile files ...?

Thanks,

Best regards,

Rémy

Le 12/04/2021 à 09:10, Fabrice Rouillier a écrit :
> Hi,
>
> I have implemented and tested a script to install Galene and a TURN 
> that works behind a Router, even a domestic one.
>
> You can doowload it here : 
> https://gitlab.inria.fr/rouillie/visio/-/tree/master/galene 
> <https://gitlab.inria.fr/rouillie/visio/-/tree/master/galene>
>
> It works only for Ubuntu 20.04 or Debian buster for the moment, mainly 
> because I am lazy :-) but it opens the door to an very easy deployment 
> on a cloud or simply at home.
>
> Cheers,
>
> Fabrice
> -------------------------
> Fabrice Rouillier
> fabrice@rouillier.fr <mailto:fabrice@rouillier.fr>
>
> Bureau virtuel : https://www.rouillier 
> <https://www.rouillier>.fr/visio/fabrice
>
>
>
>
>
>> Le 10 avr. 2021 à 16:41, Fabrice Rouillier <fabrice@rouillier.fr 
>> <mailto:fabrice@rouillier.fr>> a écrit :
>>
>> Hi Juliusz,
>>
>> Thanks for your help.
>> See below for a simple setting that works with our external coturn.
>>
>>> The problem is probably that your NAT doesn't implement hairpinning, so
>>> Galène and the TURN server cannot communicate.
>>>  Coturn is probably
>>> connecting over IPv6, which the built-in server doesn't handle.
>>
>> I have deactivated the IPV6 everywhere and double checked (inspecting 
>> about:websocket <about:websocket>) in Firefox that it does not use IPV6.
>>
>>> Could you please check Galène's log for mentions of a relay test? 
>>>  If the
>>> server-side relay test (the one in Galène's log) fails, that's probably
>>> indicative of a problem with hairpinning.
>>
>> Server side : the relay-test do not fails.
>>
>> Client side :
>>    - using the builtin turn, the relay test fails ,
>>    - using coturn (on the same server) it works.
>>
>>> Fabrice — if your instance of Galène is behind NAT, then I strongly
>>> recommend running an instance of coturn on a host that is not behind 
>>> NAT.
>>
>> The following works :
>>
>> - port forwarding from my router (Freebox) to a Linux 20.04  VM 
>> (virtual box) :
>>
>>    49152-65535 (UDP/TCP)
>>    3478 (UDP/TCP)
>>    8443 (UDP/TCP)
>>
>> - coturn on the VM with the following configuration
>>
>>   listening-port=3478
>>   fingerprint
>>   lt-cred-mech
>>   user=<TURN USER>:<TURN PASSWD>
>>   server-name=<TURN SERVER NAME>
>>   realm=<DOMAINE>
>>
>> - galene on the VM launched with the option -turn auto
>>   and data/ice-servers.json with the following contents
>>
>> [
>>     {
>>         "Urls": [
>> "turn:<TURN SERVER NAME>:3478",
>> « turn:<TURN SERVER NAME>:3478?transport=tcp"
>>         ],
>>         "username": "<TURN USER>",
>> "credential": "<TURN PASSWD>"
>>     }
>> ]
>>
>
>
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org

-- 
Rémy Dernat
Chef de projet SI
IR CNRS - ISI / ISEM


[-- Attachment #2: Type: text/html, Size: 20844 bytes --]

[Galene] Re: Installation Script

[Antonin Décimo]

> Then, I also added a configuration file to source in
> /etc/default/galene, containing (after creating the /var/log
> directories) :
>
> ARGS=" -memprofile /var/log/galene/mem.log -cpuprofile
> /var/log/galene/cpu.log"

I'm not sure if it's really useful to benchmark Galène, unless you
really want to use the profiler data to try to further optimize
Galène.

> By the way, if anyone knows how to use these {cpu,mem} profile files
> ...?

Try:
- https://golang.org/doc/diagnostics#profiling for an introduction;
- https://blog.golang.org/pprof for the complete tutorial.

-- Antonin

[Galene] Re: Installation Script

[Rémy Dernat]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

Hi,

Ok; Thanks Antonin. Indeed, profiling the binary is not very useful in a 
prod environment, except if there is a memory leak somewhere... I will 
keep using basic monitoring tools to benchmark IOs/CPU/memory/network.

Rémy

Le 12/04/2021 à 10:07, Antonin Décimo a écrit :
>> Then, I also added a configuration file to source in
>> /etc/default/galene, containing (after creating the /var/log
>> directories) :
>>
>> ARGS=" -memprofile /var/log/galene/mem.log -cpuprofile
>> /var/log/galene/cpu.log"
> I'm not sure if it's really useful to benchmark Galène, unless you
> really want to use the profiler data to try to further optimize
> Galène.
>
>> By the way, if anyone knows how to use these {cpu,mem} profile files
>> ...?
> Try:
> - https://golang.org/doc/diagnostics#profiling for an introduction;
> - https://blog.golang.org/pprof for the complete tutorial.
>
> -- Antonin


[-- Attachment #2: Type: text/html, Size: 1800 bytes --]

[Galene] Re: Installation Script

[Juliusz Chroboczek]

>> ARGS=" -memprofile /var/log/galene/mem.log -cpuprofile
>> /var/log/galene/cpu.log"

> I'm not sure if it's really useful to benchmark Galène, unless you
> really want to use the profiler data to try to further optimize Galène.

I agree.

-- Juliusz

[Galene] Re: Installation Script

[Dernat Rémy]

[-- Attachment #1.1: Type: text/plain, Size: 1166 bytes --]

Hi,

However, thinking about the logs, I do not know if this is already 
possible, but it would be useful for a galène administrator to get 
additional metadata about galene events. My "stats" page is almost empty 
(with only the group names; maybe it is because the server is not really 
active for now...).

Nginx already provides some interesting stuff (if it is used as a proxy, 
or any other proxy tool), but IMHO, it lacks information about who is 
entering in which room, the total amount of time for an active room, the 
maximum/average number of people/audience...

Would it be possible (if it is not already possible) to add these kind 
of data/output in a log flat file ?

Best regards,

Rémy

Le 12/04/2021 à 13:31, Juliusz Chroboczek a écrit :
>>> ARGS=" -memprofile /var/log/galene/mem.log -cpuprofile
>>> /var/log/galene/cpu.log"
>> I'm not sure if it's really useful to benchmark Galène, unless you
>> really want to use the profiler data to try to further optimize Galène.
> I agree.
>
> -- Juliusz
>
-- 
Dernat Rémy
Chef de projet SI, CNRS
Infrastructure des Systèmes d'Information ISI
ISEM Montpellier


[-- Attachment #1.2: Type: text/html, Size: 2051 bytes --]

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 3623 bytes --]

[Galene] Statistics [was: Installation Script]

[Juliusz Chroboczek]

> Would it be possible (if it is not already possible) to add these kind of
> data/output in a log flat file ?

Similar things have been discussed before, see for example here:

  https://github.com/jech/galene/pull/23

The problem is that nobody seems to have a clear idea what statistics need
to be provided, and how to provide statistics without sacrificing user
privacy -- I administer three instances of Galène right now, and I don't
want to know who is having a discussion with whom, it's none of my business.

What is more, Galène is designed to scale well on multicore systems, and
the patches to add statistics tend to introduce additional points of
contention.  (Since Go doesn't provide either CPU-local or thread-local
data, you'll need to implement sharding at the application level.)

So if you have a clear idea of what statistics are (1) useful to the
administrator, (2) don't impair user privacy, (3) are cheap to compute,
please outline your design, and we'll think together about how to
implement it without creating a point of contention.

-- Juliusz

[Galene] Re: Statistics [was: Installation Script]

[Dernat Rémy]

[-- Attachment #1.1: Type: text/plain, Size: 2577 bytes --]

Juliusz,

Le 12/04/2021 à 16:42, Juliusz Chroboczek a écrit :
>> Would it be possible (if it is not already possible) to add these kind of
>> data/output in a log flat file ?
> Similar things have been discussed before, see for example here:
>
>    https://github.com/jech/galene/pull/23
>
> The problem is that nobody seems to have a clear idea what statistics need
> to be provided, and how to provide statistics without sacrificing user
> privacy -- I administer three instances of Galène right now, and I don't
> want to know who is having a discussion with whom, it's none of my business.
>
> What is more, Galène is designed to scale well on multicore systems, and
> the patches to add statistics tend to introduce additional points of
> contention.  (Since Go doesn't provide either CPU-local or thread-local
> data, you'll need to implement sharding at the application level.)
>
> So if you have a clear idea of what statistics are (1) useful to the
> administrator, (2) don't impair user privacy, (3) are cheap to compute,
> please outline your design, and we'll think together about how to
> implement it without creating a point of contention.

(1) The way I see it, useful stats would be to add something that allow 
an admin to check the hardware/VM load compare to the number of rooms*users.

eg: %%TIMESTAMP%% 4 rooms with a total of 130 users with a mean 
resolution ... and bandwidth ...

A data with %users{Mic On/Webcam On} would also be useful, as well as % 
of drop packets.

Those metrics would help an admin to plan an upgrade for the Galène 
service, locate any bottleneck/problem, scale it at a wider range 
(linear progression, or ...?), get an alert if a treshold is reached... 
Some metrics could be found by monitoring the system globally, but 
metrics from the inside could drastically help to admin Galène.

(2) I do not see any privacy issue with those kind of very general 
statistics.

Otherwise, currently, those values are either retrieved theoretically 
from the code, or based on users experiences.

(3) Those computations may then be choosen by an admin : boolean to 
enable/disable it, interval between each computation (timer ? cron with 
another binary ?), metrics to monitor (config file or based on options). 
I think those computations are very basic but if you think it is really 
cumbersome for the code, just forget it.

Best regards,


>
> -- Juliusz

-- 
Dernat Rémy
Chef de projet SI, CNRS
Infrastructure des Systèmes d'Information ISI
ISEM Montpellier


[-- Attachment #1.2: Type: text/html, Size: 3534 bytes --]

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 3623 bytes --]

[Galene] Re: Statistics [was: Installation Script]

[Juliusz Chroboczek]

> (1) The way I see it, useful stats would be to add something that allow an
> admin to check the hardware/VM load compare to the number of rooms*users.

> eg: %%TIMESTAMP%% 4 rooms with a total of 130 users with a mean resolution ...
> and bandwidth ...

> A data with %users{Mic On/Webcam On} would also be useful, as well as % of drop
> packets.

You've got that already in /stats.  It would be quite easy to add
averages, although I'm not sure if the average is useful (the number of
outliers, as suggested by Michael, is perhaps more useful).

There should be no issue with periodically dumping them to a file (say,
every 30s or so).

-- Juliusz

[Galene] Re: Statistics

[Michael Ströder]

On 4/12/21 4:42 PM, Juliusz Chroboczek wrote:
> So if you have a clear idea of what statistics are (1) useful to the
> administrator, (2) don't impair user privacy, (3) are cheap to compute,
> please outline your design, and we'll think together about how to
> implement it without creating a point of contention.

Current logging and statistics is poor when trying to solve issues.

In my case I'd often like to see which user has e.g. a long downstream
RTT. Because I can then correlate that with user's feedback.

How about showing usernames in /stats instead of just the raw ID but use
the operator role to authorize who can see details of a certain group?
Then the group operator can only see the group users which he/she can
already see while being in the group.

Ciao, Michael.

[Galene] Re: Statistics

[Juliusz Chroboczek]

> How about showing usernames in /stats instead of just the raw ID

Using the id instead of the username is deliberate -- I want to be able to
check what's going on without knowing who is speaking with whom.  The latter,
I feel, would be a violation of privacy.

This is doubtless my personal feeling, so please feel free to add that
kind of monitoring to your copy of Galène.  For privacy reasons, however,
I won't be adding it to either the instances I deploy or to the master branch.

Sorry for that.

-- Juliusz

[Galene] Re: Statistics

[Michael Ströder]

On 4/12/21 6:10 PM, Juliusz Chroboczek wrote:
>> How about showing usernames in /stats instead of just the raw ID
> 
> Using the id instead of the username is deliberate -- I want to be able to
> check what's going on without knowing who is speaking with whom.  The latter,
> I feel, would be a violation of privacy.
> 
> This is doubtless my personal feeling, so please feel free to add that
> kind of monitoring to your copy of Galène.  For privacy reasons, however,
> I won't be adding it to either the instances I deploy or to the master branch.

I'd argue you get the best privacy by running your own Galène instance.

For this reason it should be more easy for an admin to sort out issues.
Otherwise people will switch over to whatever video-conferencing hosted
by a 3rd party.

You could make it configurable.

Ciao, Michael.

[Galene] Re: Statistics [was: Installation Script]

[Alexandre IOOSS]

On 4/12/21 4:42 PM, Juliusz Chroboczek wrote:
>> Would it be possible (if it is not already possible) to add these kind of
>> data/output in a log flat file ?
> 
> Similar things have been discussed before, see for example here:
> 
>    https://github.com/jech/galene/pull/23
> 
> The problem is that nobody seems to have a clear idea what statistics need
> to be provided, and how to provide statistics without sacrificing user
> privacy -- I administer three instances of Galène right now, and I don't
> want to know who is having a discussion with whom, it's none of my business.

What about showing on /stats:

  * Total number of users that joined a group since the instance started 
(incremental counter),
  * Total number of audio/video streams since the instance started,
  * Total connection to the TURN since the instance started,
  * Min/Max/Average RTT (this would be CPU heavy...?).

I believe we don't want to expose more as it would introduce privacy 
concerns.

> What is more, Galène is designed to scale well on multicore systems, and
> the patches to add statistics tend to introduce additional points of
> contention.  (Since Go doesn't provide either CPU-local or thread-local
> data, you'll need to implement sharding at the application level.)

`/stats` already exists. We could make it more easy to parse by a 
script. Do we need HTML on this page?

I am not really fan of having Galène pushing logs to a file, I would 
much rather keep the current "pull" system in which an admin has to go 
to `/stats`. It is then possible to fetch this page and save it to a log 
file (e.g. cron, systemd timer, monitoring system...).

Best,

-- 
Alexandre

[Galene] Re: Statistics [was: Installation Script]

[Dernat Rémy]

[-- Attachment #1.1: Type: text/plain, Size: 715 bytes --]

Hi,

Le 12/04/2021 à 18:44, Alexandre IOOSS a écrit :
>
> I am not really fan of having Galène pushing logs to a file, I would 
> much rather keep the current "pull" system in which an admin has to go 
> to `/stats`. It is then possible to fetch this page and save it to a 
> log file (e.g. cron, systemd timer, monitoring system...). 

Otherwise, just writing to stdout could also be useful (that is what is 
usually done for containers or systemd / journalctl).

BTW, I will double check it, but I may have an issue on my /stats page, 
as I do not see any metric.

Best regards,

-- 
Dernat Rémy
Chef de projet SI, CNRS
Infrastructure des Systèmes d'Information ISI
ISEM Montpellier


[-- Attachment #1.2: Type: text/html, Size: 1259 bytes --]

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 3623 bytes --]

[Galene] Re: Statistics [was: Installation Script]

[Juliusz Chroboczek]

> BTW, I will double check it, but I may have an issue on my /stats page, as I do
> not see any metric.

The stats page gives instantaneous information: you won't see anything if
there are no media flows flowing through Galène.  On a busy server, you'll
see a small number of statistics on every media flow (throughput, packet
loss, RTT, and jitter).

-- Juliusz

[Galene] Re: Statistics [was: Installation Script]

[Dernat Rémy]

[-- Attachment #1.1: Type: text/plain, Size: 777 bytes --]

Hi,

Ok, I understand the post-mortem issue; this is monitoring vs logging. 
Moreover the html page might be difficult to parse (need web scraping ?) 
to get those data for a longer period.

Best regards,

Rémy

Le 13/04/2021 à 12:20, Juliusz Chroboczek a écrit :
>> BTW, I will double check it, but I may have an issue on my /stats page, as I do
>> not see any metric.
> The stats page gives instantaneous information: you won't see anything if
> there are no media flows flowing through Galène.  On a busy server, you'll
> see a small number of statistics on every media flow (throughput, packet
> loss, RTT, and jitter).
>
> -- Juliusz

-- 
Dernat Rémy
Chef de projet SI, CNRS
Infrastructure des Systèmes d'Information ISI
ISEM Montpellier


[-- Attachment #1.2: Type: text/html, Size: 1391 bytes --]

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 3623 bytes --]

[Galene] Re: Statistics [was: Installation Script]

[Juliusz Chroboczek]

> the html page might be difficult to parse (need web scraping ?) to get
> those data for a longer period.

Solution 1: add a /stats.json in addition to /stats

Solution 2: create /stats.json, and replace /stats with a piece of
            JavaScript that fetches /stats.json and formats it.

Preferences?  Other ideas?

[Galene] Re: Statistics [was: Installation Script]

[Dernat Rémy]

[-- Attachment #1.1: Type: text/plain, Size: 586 bytes --]

Solution 2 for me :-)

+ writing to stdout ...?

Thank you so much !

Le 13/04/2021 à 14:11, Juliusz Chroboczek a écrit :
>> the html page might be difficult to parse (need web scraping ?) to get
>> those data for a longer period.
> Solution 1: add a /stats.json in addition to /stats
>
> Solution 2: create /stats.json, and replace /stats with a piece of
>              JavaScript that fetches /stats.json and formats it.
>
> Preferences?  Other ideas?

-- 
Dernat Rémy
Chef de projet SI, CNRS
Infrastructure des Systèmes d'Information ISI
ISEM Montpellier


[-- Attachment #1.2: Type: text/html, Size: 1191 bytes --]

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 3623 bytes --]

[Galene] Re: Statistics [was: Installation Script]

[Michael Ströder]

On 4/13/21 2:11 PM, Juliusz Chroboczek wrote:
>> the html page might be difficult to parse (need web scraping ?) to get
>> those data for a longer period.
> 
> Solution 1: add a /stats.json in addition to /stats
> 
> Solution 2: create /stats.json, and replace /stats with a piece of
>             JavaScript that fetches /stats.json and formats it.
> 
> Preferences?  Other ideas?

/stats?fmt=metrics

Prometheus/grafana combo is quite popular today.

https://prometheus.io/docs/guides/go-application/

https://github.com/prometheus/client_golang

Problem is to get the metrics labeling right because currently the
streams are grouped by random IDs in /stats. And many random labels
will result in separate many time-series...

Ciao, Michael.

[Galene] Re: Installation Script

[Fabrice Rouillier]

[-- Attachment #1: Type: text/plain, Size: 7402 bytes --]

Hi Remy,

Thanks

Note that you might directly contribute to the gitlab if you want.

For NGINX : great, this allow to type a simpler url without premising the 8443 port. I did try traffic which works also well as a reverse proxy, especially behind a domestic router with a unique IP ….

For certbot : yep this is also a good complement to have the automatic renewal.


Cheers,

Fabrice.
 
-------------------------
Fabrice Rouillier
fabrice@rouillier.fr

Bureau virtuel : https://www.rouillier <https://www.rouillier/>.fr/visio/fabrice





> Le 12 avr. 2021 à 09:56, Rémy Dernat <remy.dernat@umontpellier.fr> a écrit :
> 
> Hi Fabrice,
> 
> Thanks. That could be helpful.
> 
> Nevertheless, I have done a couple of things that could also be useful for your script. If you want, feel free to pick things/change your script accordingly.
> 
> First, there is a nginx configuration available from Yunohost package here : https://github.com/YunoHost-Apps/galene_ynh/blob/testing/conf/nginx.conf <https://github.com/YunoHost-Apps/galene_ynh/blob/testing/conf/nginx.conf>
> This configuration file was very useful to me. I am using it for proxying Galène on HTTPS. Here is a resulting configuration using it in/etc/nginx/sites-enabled/galene :
> 
> server {
>     listen 443 ssl default_server;
>     listen [::]:443 ssl default_server;
>     server_name galene.domain.tld visio.domain.tld;
>     ssl_certificate /etc/letsencrypt/live/galene.domain.tld/fullchain.pem;
>     ssl_certificate_key /etc/letsencrypt/live/galene.domain.tld/privkey.pem;
> 
>     location / {
>     
>       # Force usage of https
>       if ($scheme = http) {
>         rewrite ^ https://$server_name$request_uri <https://$server_name$request_uri/>? permanent;
>       }
>     
>       proxy_pass        https://127.0.0.1:8443 <https://127.0.0.1:8443/>;
>       proxy_redirect    off;
>       proxy_set_header  Host $host;
>       proxy_set_header  X-Real-IP $remote_addr;
>       proxy_set_header  X-Forwarded-Proto $scheme;
>       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
>       proxy_set_header  X-Forwarded-Host $server_name;
>       
>       # WebSocket support
>       proxy_http_version 1.1;
>       proxy_set_header Upgrade $http_upgrade;
>       proxy_set_header Connection "upgrade";
>     }
> }
> 
> To avoid any issue with let's encrypt renew crons and scripts (that would need another copy in ~galene/data), I added the acl package to allow galene user to read /etc/letsencrypt/ (instead of just copying the certs in /home/galene/data/:
> 
> apt-get install acl
> 
> setfacl -R -m u:galene:r /etc/letsencrypt
> 
> setfacl -m u:galene:x /etc/letsencrypt
> 
> setfacl -m u:galene:x /etc/letsencrypt/live
> 
> setfacl -m u:galene:x /etc/letsencrypt/live/galene.domain.tld
> 
> setfacl -m u:galene:x /etc/letsencrypt/archive/galene.domain.tld
> 
> setfacl -m u:galene:x /etc/letsencrypt/archive
> 
> ln -s /etc/letsencrypt/live/galene.domain.tld/chain.pem ~galene/data/chain.pem
> 
> ln -s /etc/letsencrypt/live/galene.domain.tld/privkey.pem ~galene/data/key.pem
> 
> ln -s /etc/letsencrypt/live/galene.domain.tld/privkey.pem ~galene/data/privkey.pem
> 
> ln -s /etc/letsencrypt/live/galene.domain.tld/cert.pem ~galene/data/cert.pem
> 
> ln -s /etc/letsencrypt/live/galene.domain.tld/fullchain.pem ~galene/data/fullchain.pem
> 
> 
> 
> Then, I also added a configuration file to source in /etc/default/galene, containing (after creating the /var/log directories) :
> 
> ARGS=" -memprofile /var/log/galene/mem.log  -cpuprofile /var/log/galene/cpu.log"
> 
> But you could add your turn configuration or whatever option...
> 
> Modification to the systemd script is very simple; just change:
> 
> ExecStart=/home/galene/galene
> 
> to:
> 
> EnvironmentFile=/etc/default/galene
> ExecStart=/home/galene/galene $ARGS
> 
> 
> 
> By the way, if anyone knows how to use these {cpu,mem} profile files ...?
> 
> Thanks,
> 
> Best regards,
> 
> Rémy
> 
> Le 12/04/2021 à 09:10, Fabrice Rouillier a écrit :
>> Hi,
>> 
>> I have implemented and tested a script to install Galene and a TURN that works behind a Router, even a domestic one.
>> 
>> You can doowload it here : https://gitlab.inria.fr/rouillie/visio/-/tree/master/galene <https://gitlab.inria.fr/rouillie/visio/-/tree/master/galene>
>> 
>> It works only for Ubuntu 20.04 or Debian buster for the moment, mainly because I am lazy :-) but it opens the door to an very easy deployment on a cloud or simply at home.
>> 
>> Cheers,
>> 
>> Fabrice
>> -------------------------
>> Fabrice Rouillier
>> fabrice@rouillier.fr <mailto:fabrice@rouillier.fr>
>> 
>> Bureau virtuel : https://www.rouillier <https://www.rouillier/>.fr/visio/fabrice
>> 
>> 
>> 
>> 
>> 
>>> Le 10 avr. 2021 à 16:41, Fabrice Rouillier <fabrice@rouillier.fr <mailto:fabrice@rouillier.fr>> a écrit :
>>> 
>>> Hi Juliusz,
>>> 
>>> Thanks for your help.
>>> See below for a simple setting that works with our external coturn.
>>> 
>>>> The problem is probably that your NAT doesn't implement hairpinning, so
>>>> Galène and the TURN server cannot communicate.
>>>>  Coturn is probably
>>>> connecting over IPv6, which the built-in server doesn't handle.
>>> 
>>> I have deactivated the IPV6 everywhere and double checked (inspecting about:websocket <about:websocket>) in Firefox that it does not use IPV6.
>>> 
>>>> Could you please check Galène's log for mentions of a relay test?  If the
>>>> server-side relay test (the one in Galène's log) fails, that's probably
>>>> indicative of a problem with hairpinning.
>>> 
>>> Server side : the relay-test do not fails.
>>> 
>>> Client side : 
>>>    - using the builtin turn, the relay test fails , 
>>>    - using coturn (on the same server) it works.
>>> 
>>>> Fabrice — if your instance of Galène is behind NAT, then I strongly
>>>> recommend running an instance of coturn on a host that is not behind NAT.
>>> 
>>> The following works : 
>>> 
>>> - port forwarding from my router (Freebox) to a Linux 20.04  VM (virtual box) : 
>>> 
>>>    49152-65535 (UDP/TCP)
>>>    3478 (UDP/TCP) 
>>>    8443 (UDP/TCP) 
>>> 
>>> - coturn on the VM with the following configuration
>>> 
>>>   listening-port=3478
>>>   fingerprint
>>>   lt-cred-mech
>>>   user=<TURN USER>:<TURN PASSWD>
>>>   server-name=<TURN SERVER NAME>
>>>   realm=<DOMAINE>
>>> 
>>> - galene on the VM launched with the option -turn auto 
>>>   and data/ice-servers.json with the following contents
>>> 
>>> [
>>>     {
>>>         "Urls": [
>>>             "turn:<TURN SERVER NAME>:3478",
>>>             « turn:<TURN SERVER NAME>:3478?transport=tcp"
>>>         ],
>>>         "username": "<TURN USER>",
>>>         "credential": "<TURN PASSWD>"
>>>     }
>>> ]
>>> 
>> 
>> 
>> 
>> _______________________________________________
>> Galene mailing list -- galene@lists.galene.org <mailto:galene@lists.galene.org>
>> To unsubscribe send an email to galene-leave@lists.galene.org <mailto:galene-leave@lists.galene.org>
> -- 
> Rémy Dernat
> Chef de projet SI
> IR CNRS - ISI / ISEM
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org


[-- Attachment #2: Type: text/html, Size: 25342 bytes --]

[Galene] Re: Installation Script

[Juliusz Chroboczek]

> https://gitlab.inria.fr/rouillie/visio/-/tree/master/galene

Thanks a lot, Fabrice.

1. I could easily add an option to Galène to restrict the range of
   ephemeral ports used — for example, you could set the range of ports to
   50000-60000, and only these ports would need to be forwarded.  Let me
   know if you think this is useful.

2. If you're using the built-in TURN server behind NAT (not recommended),
   you need to set up hairpinning in your NAT.  I don't see any good way
   around it.  Please mention it in the README.

Thanks again,

-- Juliusz

[Galene] Re: Installation Script

[Fabrice Rouillier]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Hi Juliusz

> 1. I could easily add an option to Galène to restrict the range of
>   ephemeral ports used — for example, you could set the range of ports to
>   50000-60000, and only these ports would need to be forwarded.  Let me
>   know if you think this is useful.
> 
It might be useful to make possible the use of 2 distincts coturn servers 
In my case I would like one for nextcloud and one for Galène.

Nextcloud uses the use-auth-secret 

Authentification and it turns out that I was unable to use it with Galene, event setting "credentialType": "hmac-sha1 » as described in your README file.

I can easily modify the script adding the two ports. 

> 2. If you're using the built-in TURN server behind NAT (not recommended),
>   you need to set up hairpinning in your NAT.  I don't see any good way
>   around it.  Please mention it in the README.
> 
The installation script  do not the use the built-in  turn but install and configure coturn. Galene is launched with the « -turn auto » option. 
Would you like me to add this information ?

> Thanks again,
> 
I am very pleased to contribute to the collective effort.


Fabrice.


[-- Attachment #2: Type: text/html, Size: 2410 bytes --]

[Galene] Re: Installation Script

[Juliusz Chroboczek]

> It might be useful to make possible the use of 2 distincts coturn servers 
> In my case I would like one for nextcloud and one for Galène.

> Nextcloud uses the use-auth-secret authentification and it turns out
> that I was unable to use it with Galene, event setting "credentialType":
> "hmac-sha1 » as described in your README file.

Strange.  Toke, can you confirm that it works for you?

> The installation script  do not the use the built-in  turn but install and
> configure coturn. Galene is launched with the « -turn auto » option.

Ah, I see.

> Would you like me to add this information ?

Please do.

-- Juliusz

[Galene] Re: Installation Script

[Toke Høiland-Jørgensen]

Juliusz Chroboczek <jch@irif.fr> writes:

>> It might be useful to make possible the use of 2 distincts coturn servers 
>> In my case I would like one for nextcloud and one for Galène.
>
>> Nextcloud uses the use-auth-secret authentification and it turns out
>> that I was unable to use it with Galene, event setting "credentialType":
>> "hmac-sha1 » as described in your README file.
>
> Strange.  Toke, can you confirm that it works for you?

Yup:
Apr 07 20:35:28 video galene[172961]: 2021/04/07 20:35:28 Relay test successful in 40.204714ms, RTT = 335.309<C2><B5>s

ice-servers.json contains:
[
    {
        "urls":["turn:my.turn.server:500",
		"turn:my.turn.server:443?transport=tcp"],
        "username":"galene",
        "credential":"censored",
        "credentialType":"hmac-sha1"
    }
]

and my.turn.server has this in turnserver.conf:

listening-port=500
tls-listening-port=443
use-auth-secret
static-auth-secret=censored

-Toke

[Galene] Re: Installation Script

[Michael Ströder]

On 4/12/21 4:40 PM, Fabrice Rouillier wrote:
> Nextcloud uses the use-auth-secret 
> 
> Authentification and it turns out that I was unable to use it with
> Galene, event setting "credentialType": "hmac-sha1 » as described in
> your README file.

It works for me every day.

Did you really comment out lt-cred-mech and add use-auth-secret in your
turnserver.conf? And did you correctly add the shared secret?

Ciao, Michael.

[Galene] Re: Installation Script

[Fabrice Rouillier]

> 
> It works for me every day.
> 
> Did you really comment out lt-cred-mech and add use-auth-secret in your
> turnserver.conf? And did you correctly add the shared secret?
> 

Made another try few minutes ago, it works. Sorry for the noise, I did certainly mix things between many combinations.


Fabrice


> Ciao, Michael.
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org

[Galene] ephemeral port range config

[Michael Ströder]

On 4/12/21 1:30 PM, Juliusz Chroboczek wrote:
> 1. I could easily add an option to Galène to restrict the range of
>    ephemeral ports used — for example, you could set the range of ports to
>    50000-60000, and only these ports would need to be forwarded.  Let me
>    know if you think this is useful.

Being a big fan of explicit configs I'd highly appreciate to have such a
config option.

Ciao, Michael.

[Galene] Re: Installation Script

[Fabrice Rouillier]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

Hi Juliusz

> 2. If you're using the built-in TURN server behind NAT (not recommended),
>   you need to set up hairpinning in your NAT.  I don't see any good way
>   around it.  Please mention it in the README.

I have added the following warning.

WARNING : as the default installation will install a turn server (coturn) on the same machine as galene, you might possibly face problems if hairpinning is not set in your NAT. Some options of coturn server are supposed to solve some of them but it is not clear that it will always work. Nevertheless, the script can easily be adapted in order to use an externat turn server. For the time being it has been successively tested on a public cloud (Scaleway) and behing a 10 years old freobox (v6).


All the best,

Fabrice.

> Thanks again,
> 
> -- Juliusz
> 


[-- Attachment #2: Type: text/html, Size: 2699 bytes --]