From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) by mail.toke.dk (Postfix) with ESMTPS id 626167CA249 for ; Tue, 12 Jan 2021 20:06:02 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=E2Fdq+T6 Received: by mail-io1-xd34.google.com with SMTP id u26so6353086iof.3 for ; Tue, 12 Jan 2021 11:06:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=qyBO6jNVsdK/+I2QDsCTb2cMG8qYlcwZfQombqf+r20=; b=E2Fdq+T6j1PwDFk+DCWHND84n4boKOOR2c5Sr8w0+1BHPSJj6rLG5shQUgSgtFlf30 mnSwk1X1wCutBREDKo/BtIhttbTfpoCBDZXTxBnPds1CwyTDgUb4ahv2PzZo2JbwAqIo N70YgXfjYKllsqxhLHYq11cEhu1x67W6t+GDHCWhIhChlplUSJKTU+Q8Lz0PGnTP7Hji q6mipX9VvIIdh27EJnb/kTXENRtwiUnvUPftAIbXpL2/iHBASGH6euwgTMrRjgSqkZ4T 5kHEI+1jJI9B92YaaIvgnO2lDe0l8ohib0DtNnIsK9NgvslAXhf2rWivkyamdRxilAQ9 NGxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=qyBO6jNVsdK/+I2QDsCTb2cMG8qYlcwZfQombqf+r20=; b=bqqzyKA7ub204iQLDR+OLkK1Be298/r8COnuE7TtIc8VowgKWBkDMiIS1L2bhtmFU8 t9V7UwQNOS/9zEpkJXCLVvYfFTJjjbTNHUWGAsS4Srl9bimQyHm3lSNawQAuvr4y1yBX 4W5pibhDmlizfZEVbtamlHA2KMAwdvpJIEIb7C0t/8Nv6NCexZzyA66DLqNePxJZexBd ZkSUZDs9Ktzgb+Zbyk/XYapPr9JS5ZnaUT9c7nV60Kv2kBeRJp9TRm2RcCLgaEf4t+Us sX12V1GRjYTqqJap653061UXq8DfLiNX6HhkATTvD2hNoYh9rr+tXyrq+9uTM8XcitEo 4Qug== X-Gm-Message-State: AOAM530h9xRTmUPTpWL/S4Hl+V/u9a+nkMW9Z6EXKHNlUWLx6sny8zDT vlPtQLKQq86Li2Otfit6oyjHN1coUX11iXpuNWA= X-Google-Smtp-Source: ABdhPJwdtdLzAzn5SSpHF29lyznq2BncGUXnIMd2ehcwaaAIBhU1dUr7Zclv3hYkNc83phbRLDbThsDQXap/D/3YrKg= X-Received: by 2002:a6b:d010:: with SMTP id x16mr309747ioa.161.1610478359812; Tue, 12 Jan 2021 11:05:59 -0800 (PST) MIME-Version: 1.0 References: <87k0siqndj.wl-jch@irif.fr> <87pn2aup3o.fsf@toke.dk> <878s8yqcmh.wl-jch@irif.fr> <871reqqb52.wl-jch@irif.fr> In-Reply-To: <871reqqb52.wl-jch@irif.fr> From: Dave Taht Date: Tue, 12 Jan 2021 11:05:48 -0800 Message-ID: To: Juliusz Chroboczek Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: VDHUARCUMFJI5XEP4XSUTQJKODFJEADL X-Message-ID-Hash: VDHUARCUMFJI5XEP4XSUTQJKODFJEADL X-MailFrom: dave.taht@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: galene@lists.galene.org, Dave Taht X-Mailman-Version: 3.3.2 Precedence: list Subject: [Galene] Re: fq-codel trashing List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: On Tue, Jan 12, 2021 at 10:10 AM Juliusz Chroboczek wrote: > > > IF you have spare cpu, you can also run cake native no shaper and > > strip gro (maxpacket 64k indicates gro is on). > > GRO will only trigger for HTTP, WebSocket and TURN traffic, I believe; > most of Gal=C3=A8ne's traffic is RTP over UDP, with packets under 1200 by= tes. > > We just had a meeting with 70 people and at around 40 cameras switched on= , > Gal=C3=A8ne became unusable =E2=80=94 there were too many voice drops, wh= ich indicates > two issues: i have generally found vms perform badly for lots of small packets and r/t. please try cake. And collect a capture on the underlying hw if possible. > * I need to think of a better way of prioritising voice over video when > under load; I take it (I really didn't understand) that unbundling these two types is not currently feasible in the javascript or webbrowser. ?Still. I am pretty intersted in this layer of stuff but since adding crypto to rtp it's got really hard to look at it. > * there are fairness issues =E2=80=94 some clients were receiving okay-= ish > audio, others were not. collisions in fq_codel start to occur as the sqrt(1024) so at 70 users the odds are 2-3 of your sessions were colliding. Cake uses 8 way set associative. 0 collisions at this load. I imagine you were not tracking the actual backlog in fq_codel during this conference? (while :; do tc -s qdisc show dev bla >> whatever; done all that said, I tend to point fingers at loss at the virtio layer and underlying hw first just because I'm defensive about people blaming fq_codel for anything. :) I also hate vms for r/t traffic. you had a lot of GRO traffic it looked like. you can disable gso/gro via ethtool and stick with fq_codel or use cake with the > > Gal=C3=A8ne recovered after some people switched their cameras off, I did= n't > need to restart anything. At the highest point, Gal=C3=A8ne was at 270% = CPU, > and the TURN server was using another 50%. That's on a four-core VM. but it does sound like more virtual cores will help at this load? you in a position to profile? you are context switching like crazy most likely also. > > On the other hand, gro is saves on context switches in vms. > > $ PRODUCT INSTALL GALENE /CONFIGURATION=3DDEFAULT /LOG > > -- Juliusz --=20 "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729