Re: [Iperf-users] Threading and CPU affinity in iperf2/iperf3

[Bob M. <bob...@br...>]

Hi Bruce,

We do -c localhost testing to help isolate performance issues too. I think
there are a few classes of bottlenecks that you and your team are aware of:

   - Core frequencies or cycle times, i.e. cpu limits
   - On chip accesses, including L1/L2 cache accesses
   - Pin i/o access, off chip but on board or CPU package, e.g. L3 cache
   and pcie bus memory
   - Network i/o accesses

It seems a bit of an art to find what is driving performance. I tried to
warn when network i/o is not driving a test's limits but not sure how good
my mechanism is so I don't really promote it much.

Bob

On Tue, Sep 26, 2023 at 10:25 AM Bruce A. Mah <bm...@es...> wrote:

> Hrm, no not doing zero-copy.
>
> We probably need to do some more testing to try to nail this stuff down a
> bit. The latest results we've been seeing (not by me personally) are less
> clear. I'll let the people who were actually doing this testing (or are
> closer to it) chime in if and when they feel like it.
>
> Anyways, I was mostly wondering when I started this thread if there's
> something obvious that we're missing. It doesn't sound like there is.
>
> I think running iperf2 or iperf3 from within numactl should be good for
> giving the needed CPU pinning functionality...I think anyways.
>
> Thanks!
>
> Bruce.
>
> If memory serves me right, Bob McMahon wrote:
>
> Iperf2 doesn't support zero copy. Are your tests using that? Did you try
> Iperf2 on your system? I can add an affinity option if you find it
> helpful.
>
> Bob
>
> On Mon, Sep 25, 2023, 7:14 AM Bruce A. Mah <bm...@es...> wrote:
>
>> Thanks Bob! Yes the effects of CPU affinity were new to us before testing
>> the multi-threading code. At least that's when I started noticing their
>> effects (or rather, a couple colleagues pointed this out to me). This
>> coincided with us getting results from some different test environments, so
>> it might be that we're just running in some new regime and thus seeing
>> effects that were hidden to us before.
>>
>> Bruce.
>>
>> If memory serves me right, Bob McMahon wrote:
>>
>> I should note that I mostly test on WiFi networks or 10G. There may be
>> improvements on 100G+ with CPU affinity and I just haven't tried it.
>>
>> Bob
>>
>> On Fri, Sep 22, 2023, 10:52 AM Bob McMahon <bob...@br...>
>> wrote:
>>
>>> Hi Bruce,
>>>
>>> Iperf 2 doesn't support CPU affinity. It lets the OS schedule threads on
>>> cores. I tried to use CPU affinity a few years ago and I didn't notice any
>>> performance impact. I was kinda of surprised by this. I have no idea why
>>> there is different behavior between 2 & 3 per this.
>>>
>>> Bob
>>>
>>> On Fri, Sep 22, 2023 at 10:29 AM Bruce A. Mah <bm...@es...> wrote:
>>>
>>>> Hey Bob--
>>>>
>>>> As we're doing some testing of the multi-threaded iperf3 in various
>>>> environments, we've observed (not surprisingly) that CPU pinning of threads
>>>> can have a significant impact on the throughput of tests. Generally we're
>>>> running on some recent Ubuntu distribution of Linux.
>>>>
>>>> I'm trying to understand a report that iperf2 seems to automatically Do
>>>> The Right Thing (tm) with respect to getting threads on the right CPU cores
>>>> (particularly on multiple CPU package systems). But by contrast, good
>>>> performance with iperf3 needs either the -A option for CPU affinity or
>>>> running iperf3 within an invocation of numactl. I haven't yet had the
>>>> opportunity to experiment with iperf2 much in this kind of environment.
>>>>
>>>> Is there some magic that iperf2 does to automatically figure out a good
>>>> placement for the threads it spawns? I've looked through the source code
>>>> (in particular compat/Thread.c) but I can't find anything applicable. I'd
>>>> love to do something similar, if indeed this exists, in iperf3.
>>>>
>>>> Thanks!
>>>>
>>>> Bruce.
>>>>
>>>
>> This electronic communication and the information and any files
>> transmitted with it, or attached to it, are confidential and are intended
>> solely for the use of the individual or entity to whom it is addressed and
>> may contain information that is confidential, legally privileged, protected
>> by privacy laws, or otherwise restricted from disclosure to anyone else. If
>> you are not the intended recipient or the person responsible for delivering
>> the e-mail to the intended recipient, you are hereby notified that any use,
>> copying, distributing, dissemination, forwarding, printing, or copying of
>> this e-mail is strictly prohibited. If you received this e-mail in error,
>> please return the e-mail to the sender, delete it from your computer, and
>> destroy any printed copy of it.
>>
>>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.