[Galene] Additional stats

[Galene] Additional stats

[Stefan Bethke]

Hi there,

long time lurker and occasional high-volume user (for chaos communication congress).

For this years' event, I would like to collect better stats about how many users are joining, and what their experience might be.

I see that there is /galene-api/v0/.stats, which provides headcount per room and general network stats. I have a small proxy that converts that to Prometheus format, so we can poll that and add it to the Prometheus/Grafana instance we're running for the event: https://github.com/stblassitude/galene-prom-exporter

I would be interested in making that endpoint an optional feature in Galéne, and I would additionally be interested in collecting ICE information.

Is there interest in such features, and would it be OK for me to use GenAI (Claude) for the implementation?


Regards,
stb

-- 
Stefan Bethke <stb@lassitu.de>   Fon +49 175 3288861

[Galene] Re: Additional stats

[Juliusz Chroboczek]

Hi Stefan,

> I would be interested in making that endpoint an optional feature in Galéne, and I would additionally be interested in collecting ICE information.
> 
> Is there interest in such features, and would it be OK for me to use GenAI (Claude) for the implementation?

Extra statistics would be useful.  However two caveats:

1. Respect user privacy.  For example, the current endpoint does not
   expose IP addresses or user names.  This is important: people are using
   galene.org for intimate couple chat (I know because the group names are
   exposed), and there is at least one priest using Galene for online
   confession (he's using his own instance, thankfully).

2. Don't harm performance, at least when the admin is not monitoring.
   Galene is very carefully constructed to scale across many cores.  For
   example, we avoid locking whenever possible, and we try to avoid atomic
   variables that are shared between more than a few threads.

   Note in particular that the Go Prometheus library uses global atomics,
   which is a bad idea, since it causes cacheline bouncing.  (The right
   solution would be to have per-thread or per-CPU counters, together with
   periodic reconciliation.)

I guess what I'm saying is that I'd love you to experiment, but please
don't be discouraged if I don't accept your first attempt.

-- Juliusz

[Galene] Re: Additional stats

[Sebastien Binet]

hi there,

On Tue Oct 20, 2026 at 16:15 CET, Juliusz Chroboczek wrote:
> Hi Stefan,
>
>> I would be interested in making that endpoint an optional feature in Galéne, and I would additionally be interested in collecting ICE information.
>> 
>> Is there interest in such features, and would it be OK for me to use GenAI (Claude) for the implementation?
>
> Extra statistics would be useful. However two caveats:
>
> 1. Respect user privacy. For example, the current endpoint does not
> expose IP addresses or user names. This is important: people are using
> galene.org for intimate couple chat (I know because the group names are
> exposed), and there is at least one priest using Galene for online
> confession (he's using his own instance, thankfully).
>
> 2. Don't harm performance, at least when the admin is not monitoring.
> Galene is very carefully constructed to scale across many cores. For
> example, we avoid locking whenever possible, and we try to avoid atomic
> variables that are shared between more than a few threads.
>
> Note in particular that the Go Prometheus library uses global atomics,
> which is a bad idea, since it causes cacheline bouncing. (The right
> solution would be to have per-thread or per-CPU counters, together with
> periodic reconciliation.)
>
> I guess what I'm saying is that I'd love you to experiment, but please
> don't be discouraged if I don't accept your first attempt.

if genAI assisted code were to be included into Galène, could you first tag the last release of Galène that happens to be untainted (and advertise it) ?

that would help people who do not want such things to taint their work/homestation.

thanks in advance.

cheers,
-s

[Galene] GenAI in Galene [was:Additional stats]

[Juliusz Chroboczek]

> if genAI assisted code were to be included into Galène, could you first
> tag the last release of Galène that happens to be untainted (and
> advertise it) ?

I do not intend to allow any GenAI code in the server.  This is not due to
any strong anti-AI stance, it's purely egoistical: I'm the person who will
end up having to maintain the slop.

(I'm a little less categorical about the UI parts, since I can understand
that people might not want to edit CSS by hand.  However, in my experience
GenAI-generated submissions tend to be overly verbose and redundant, and
the submitters disappear without trace when I ask them to fix the code.)

> that would help people who do not want such things to taint their
> work/homestation.

I'm probably somewhat less radical than you, but we're in general agreement.

-- Juliusz

[Galene] Re: Additional stats

[Stefan Bethke]

Hi there,

here is my proposal: https://github.com/jech/galene/pull/349

I will be running this code and testing it with Prometheus and Grafana for the next weeks, and eventually at our event on Dec 27th to 30th, so I will be able to report any practical findings from running it.

Let me know what you think.

Thanks,
Stefan

> On 20. Oct 2026, at 16:15, Juliusz Chroboczek <jch@irif.fr> wrote:
> 
> Hi Stefan,
> 
>> I would be interested in making that endpoint an optional feature in Galéne, and I would additionally be interested in collecting ICE information.
>> 
>> Is there interest in such features, and would it be OK for me to use GenAI (Claude) for the implementation?
> 
> Extra statistics would be useful.  However two caveats:
> 
> 1. Respect user privacy.  For example, the current endpoint does not
>   expose IP addresses or user names.  This is important: people are using
>   galene.org for intimate couple chat (I know because the group names are
>   exposed), and there is at least one priest using Galene for online
>   confession (he's using his own instance, thankfully).
> 
> 2. Don't harm performance, at least when the admin is not monitoring.
>   Galene is very carefully constructed to scale across many cores.  For
>   example, we avoid locking whenever possible, and we try to avoid atomic
>   variables that are shared between more than a few threads.
> 
>   Note in particular that the Go Prometheus library uses global atomics,
>   which is a bad idea, since it causes cacheline bouncing.  (The right
>   solution would be to have per-thread or per-CPU counters, together with
>   periodic reconciliation.)
> 
> I guess what I'm saying is that I'd love you to experiment, but please
> don't be discouraged if I don't accept your first attempt.
> 
> -- Juliusz

-- 
Stefan Bethke <stb@lassitu.de>   Fon +49 175 3288861

[Galene] Re: Additional stats

[Juliusz Chroboczek]

> here is my proposal: https://github.com/jech/galene/pull/349

Thanks, especially for mentioning explicitly that you used Claude to write
this code.

Unfortunately, your code has the typical flaws of LLM-generated code:

  - it's a single commit, with a verbose but useless commit message;

  - there are a number of changes that are unrelated to Prometheus, that
    are bundled into the same commit;

  - the comments are verbose and not useful;

  - the code is overly verbose, with no clear indication which features
    are actually useful;

  - doc strings and comments paraphrase the code rather than providing
    information that's useful to the reader.

For now, I suggest you extract the changes unrelated to Prometheus and
submit them in a separate PR.  Before we decide whether to work on
improving the code, I'd like to understand two things.

1. Do people actually feel that this kind of metrics are useful to include
   in Galene?  How do people use Prometheus usually, what insights does it
   give you, and are these insights useful?  Or do people just want
   Prometheus because it's what everyone does?

   (As far as I'm concerned, I optimise Galene by running it in production
   with profiling enabled, then spend some time poring over the profile.
   I find the output of a profiler much more useful, as it gives me
   a global view of the behaviour, not just the specific set of statistics
   that are exported by the monitoring interface.)

2. Are you actually using this code in production, and have you found the
   provided statistics useful for administering Galene?

Thanks again,

-- Juliusz

[Galene] Re: Additional stats

[Stefan Bethke]

Hi!

Let me the last point first:

> 1. Do people actually feel that this kind of metrics are useful to include
>   in Galene?  How do people use Prometheus usually, what insights does it
>   give you, and are these insights useful?  Or do people just want
>   Prometheus because it's what everyone does?
> 
>   (As far as I'm concerned, I optimise Galene by running it in production
>   with profiling enabled, then spend some time poring over the profile.
>   I find the output of a profiler much more useful, as it gives me
>   a global view of the behaviour, not just the specific set of statistics
>   that are exported by the monitoring interface.)
> 
> 2. Are you actually using this code in production, and have you found the
>   provided statistics useful for administering Galene?

I am running a test instance and are monitoring that through it. We will use this feature in the production setup December 27th to 30th at the 40th Chaos Communication Congress.

The reason I would like to have statistics, and a Prometheus metrics endpoint in particular:
* We want to understand if the server is running and serving audio as expected
* We want to understand if there are network issues degrading the user experience
* We would like some fancy graphs because they are cool

Information on the WebRTC/ICE performance in particular will be valuable to the NOC that runs the network in the lecure halls. The lecture halls range in size from about 4000 seats down to 500; having a significant amount of people getting on the Wifi and trying to listen to the audio streams requires correctly configured and placed access points. While the APs report on general quality aspects, having the user-visible application-level stats will show issues more immediately. For example, if a large download gets a large-ish number of retried packets, that only slows the download down a bit, but for an audio stream, that could make the difference between usable and unusable. At the past events, we did not have those metrics, so any reports on audio quality were anecdotal, and it was hard to correlate with any network-level metrics.

I do not run Galene outside of this type of event, so I cannot speak to other operators. From my experience running applications, having metrics available in a common format is appreciated quite a bit. But this really depends on the context: if you run a service on a small machine for a limited group of people, then this might not be useful to you at all, since running Prometheus and something like Grafana can take a lot of effort. If you're running it as a production service that many people rely on, and you have the resources and people to run an observability platform, (and you are running many services at the same time), then it can be very helpful.

I'm happy to let others say if they feel this would be helpful or not interesting at all.

> On 30. Oct 2026, at 12:15, Juliusz Chroboczek <jch@irif.fr> wrote:
> 
>> here is my proposal: https://github.com/jech/galene/pull/349
> 
> Thanks, especially for mentioning explicitly that you used Claude to write
> this code.
> 
> Unfortunately, your code has the typical flaws of LLM-generated code:
> 
>  - it's a single commit, with a verbose but useless commit message;
> 
>  - there are a number of changes that are unrelated to Prometheus, that
>    are bundled into the same commit;
> 
>  - the comments are verbose and not useful;
> 
>  - the code is overly verbose, with no clear indication which features
>    are actually useful;
> 
>  - doc strings and comments paraphrase the code rather than providing
>    information that's useful to the reader.
> 
> For now, I suggest you extract the changes unrelated to Prometheus and
> submit them in a separate PR.  Before we decide whether to work on
> improving the code, I'd like to understand two things.

Let me understand how you would like changes to be split up.

My design decision was to integrate the /metrics endpoint into the regular HTTP handler, and extend the authentication setup to control access to it. Other products have the /metrics endpoint on a separate handler (different port), and do not implement authentication, but rely on firewall rules to limit access to it. Running /metrics on its own handler and port would make the changes to the config and authentication redundant. If you would prefer to have it like that, I can certainly do that. Likewise, if the metrics endpoint is configured through command line options, the changes would not touch the config file.

The change in stats.go is to enable reuse of the code block from prometheus.go. If you'd rather have code duplication to keep the code completely separate, that is easy to change.

I'm happy to reword the pull request, commit messages, and coments, I'm happy to do that too. I honestly don't find the comments to be overly verbose?


Thanks,
Stefan

-- 
Stefan Bethke <stb@lassitu.de>   Fon +49 175 3288861

[Galene] Re: Additional stats

[Juliusz Chroboczek]

> I am running a test instance and are monitoring that through it. We will
> use this feature in the production setup December 27th to 30th at the
> 40th Chaos Communication Congress.

Wow!  That's excellent news.

> * We would like some fancy graphs because they are cool

That almost convinced me ;-)

Now, there are good reasons why I couldn't possibly apply your patch in
its current state.

  - It needs careful review in order to ensure that we don't expose private
    user data.  Respecting user privacy is an important feature of Galene,
    and one I won't compromise on.

  - There is no evidence that the provided metrics are the ones we want.
    Experience shows that once you include a feature, it's difficult to
    remove (XKCD 1172), so I'd like to make sure that we only include those
    metrics that we actually need.

  - It is verbose and redundant, as typical of Claude.  This makes it
    difficult to review, and will make it difficult to maintain in the
    long term.  This makes the first point difficult to satisfy.

I suggest the following:

1. Extract the non-Prometheus relasted changes, and submit them in
   a separate PR.  That will make your fork closer to upstream.

2. Run your code as a (friendly) fork until you get more experience.
   After running your code during CCC, you'll know which metrics are
   useful and which ones are missing.

3. After CCC, we can discuss whether we want to work on rewriting the code
   in a human-readable manner, or whether we want to give up.

-- Juliusz

[Galene] Re: Additional stats

[Stefan Bethke]

On 30. Oct 2026, at 13:46, Juliusz Chroboczek <jch@irif.fr> wrote:
> 
>> I am running a test instance and are monitoring that through it. We will
>> use this feature in the production setup December 27th to 30th at the
>> 40th Chaos Communication Congress.
> 
> Wow!  That's excellent news.
> 
>> * We would like some fancy graphs because they are cool
> 
> That almost convinced me ;-)
> 
> Now, there are good reasons why I couldn't possibly apply your patch in
> its current state.
> 
>  - It needs careful review in order to ensure that we don't expose private
>    user data.  Respecting user privacy is an important feature of Galene,
>    and one I won't compromise on.

I completely agree, and I wouldn't want to suggest code that would leak such info.

>  - There is no evidence that the provided metrics are the ones we want.
>    Experience shows that once you include a feature, it's difficult to
>    remove (XKCD 1172), so I'd like to make sure that we only include those
>    metrics that we actually need.

Good point. Happy to pick this up again after we've had practical experience in December.

>  - It is verbose and redundant, as typical of Claude.  This makes it
>    difficult to review, and will make it difficult to maintain in the
>    long term.  This makes the first point difficult to satisfy.

I am still not sure I follow. Can you point out an example?

> I suggest the following:
> 
> 1. Extract the non-Prometheus relasted changes, and submit them in
>   a separate PR.  That will make your fork closer to upstream.

The changes to the authentication code are necessary to implement access control for the metrics endpoint, but they do not provide any functionality by themselves. I'm happy to put them in a separate commit to make it clear that they are enabling changes, but I would suggest they should only go in together with the metrics endpoint.

> 2. Run your code as a (friendly) fork until you get more experience.
>   After running your code during CCC, you'll know which metrics are
>   useful and which ones are missing.
> 
> 3. After CCC, we can discuss whether we want to work on rewriting the code
>   in a human-readable manner, or whether we want to give up.

Sure thing.


Stefan

-- 
Stefan Bethke <stb@lassitu.de>   Fon +49 175 3288861

[Galene] Re: Additional stats

[Juliusz Chroboczek]

> The changes to the authentication code are necessary to implement access
> control for the metrics endpoint, but they do not provide any
> functionality by themselves. I'm happy to put them in a separate commit
> to make it clear that they are enabling changes, but I would suggest
> they should only go in together with the metrics endpoint.

Yes, that would be helpful.  Just make sure every commit compiles and
doesn't break any functionality.  I'll push them together, of course.

>>  - It is verbose and redundant, as typical of Claude.

> I am still not sure I follow. Can you point out an example?

After reading it more carefully, it's only the JSON handling at the
beginning that's clumsy, the rest is actually pretty good.  Sorry for
that.

-- Juliusz