[Galene] Re: fq-codel trashing

[Dave Taht <dave.taht@gmail.com>]

On Tue, Jan 12, 2021 at 7:55 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Juliusz Chroboczek <jch@irif.fr> writes:
>
> > Dave, Toke,
> >
> > We're having a large meeting on the lab's Galène server, and it looks like
> > fq-codel is trashing: the new_flow_count increases as fast as the packet
> > counter.  How do I fix that?

It is a pretty useless statistic. I never liked it. In this case it is
a positive sign you are
not filling the queue, not a negative one, as toke goes into more detail below.

If you are having fun with fq_codel, try cake instead. Lots of more stats,
some of which are way more relevant to your application.

tc qdisc add dev eth0 root cake bandwidth XMbit # besteffort if you
don't want to bother with diffserv

tc -s qdisc show


                   Bulk  Best Effort        Voice
  thresh      562496bit        9Mbit     2250Kbit
  target         32.3ms        5.0ms        8.1ms
  interval      127.3ms      100.0ms      103.1ms
  pk_delay        9.2ms        224us        494us
  av_delay        2.9ms         29us         35us
  sp_delay        153us          4us          3us
  backlog            0b           0b           0b
  pkts            74853     35044344       162590
  bytes        80542836   9388672264     27301751
  way_inds         2538      2129502        10508
  way_miss         2575       549864         7374
  way_cols            0            0            0
  drops              11         7987            0
  marks              22            1            0
  ack_drop            0      3616230            0
  sp_flows            1            1            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len         15605        25262         3169
  quantum           300          300          300





>
> That just sounds like wherever you're running FQ-CoDel is not actually
> the bottleneck? If there's no backpressure the queue will drain for each
> packet, so that by the time the next packet comes along it'll be
> considered "new" again.
>
> If you're running a few video flows on a gigabit link (and thus using a
> fraction of the bandwidth) that would be expected. But it only happens
> if you're also running a shaper (such as HTB or TBF) on the interface,
> since if FQ-CoDel is installed on a physical interface it will set the
> TCQ_F_CAN_BYPASS flag, which means that when the queue is completely
> empty it will be bypassed entirely...
>
> I just replicated the phenomenon like this:
>
> $ sudo tc qdisc replace dev testns root tbf rate 1gbit latency 10ms burst 1024
> $ sudo tc qdisc add dev testns parent 8002: fq_codel
> $ tc -s qdisc
> qdisc tbf 8002: dev testns root refcnt 2 rate 1Gbit burst 1000b lat 10ms
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc fq_codel 8003: dev testns parent 8002: limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
>
> $ ping -c 5 fc00:dead:cafe:1::2
> PING fc00:dead:cafe:1::2(fc00:dead:cafe:1::2) 56 data bytes
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=1 ttl=64 time=0.088 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=2 ttl=64 time=0.138 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=3 ttl=64 time=0.102 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=4 ttl=64 time=0.103 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=5 ttl=64 time=0.104 ms
>
> --- fc00:dead:cafe:1::2 ping statistics ---
> 5 packets transmitted, 5 received, 0% packet loss, time 4041ms
> rtt min/avg/max/mdev = 0.088/0.107/0.138/0.016 ms
>
> $ tc -s qdisc
> qdisc tbf 8002: dev testns root refcnt 2 rate 1Gbit burst 1000b lat 10ms
>  Sent 590 bytes 5 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc fq_codel 8003: dev testns parent 8002: limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
>  Sent 590 bytes 5 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 118 drop_overlimit 0 new_flow_count 5 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
>
>
>
> Whereas if fq_codel is the root qdisc:
>
> $ sudo tc qdisc replace dev testns root fq_codel
> $ tc -s qdisc
> qdisc fq_codel 8004: dev testns root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
>
> $ ping -c 5 fc00:dead:cafe:1::2
> PING fc00:dead:cafe:1::2(fc00:dead:cafe:1::2) 56 data bytes
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=1 ttl=64 time=0.085 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=2 ttl=64 time=0.098 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=3 ttl=64 time=0.098 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=4 ttl=64 time=0.097 ms
> 64 bytes from fc00:dead:cafe:1::2: icmp_seq=5 ttl=64 time=0.096 ms
>
> --- fc00:dead:cafe:1::2 ping statistics ---
> 5 packets transmitted, 5 received, 0% packet loss, time 4058ms
>
> $ tc -s qdisc
> qdisc fq_codel 8004: dev testns root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
>  Sent 590 bytes 5 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
>
>
> -Toke
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729