PC Engines APU NIC (RTL8111E) performance

[Momtchil Momtchev]

Hello,


Does anyone with a working knowledge of re(4) have any idea why the
PC Engines APU NICs perform so poorly in OpenBSD? Throughput is 300 to
320 MBit/s with about 30 pf rules, NAT and 10000 states when running
5.9. This is much less than an APU running Linux or FreeBSD or an
OpenBSD running on comparable CPU with another NIC. I wonder how do
other motherboards with that same NIC (it is very common on the lower
end) perform?

Here is my top output when running iperf routing through the APU :

CPU0 states: 1.0% user, 0.0% nice, 51.5% system, 26.9% interrupt,
20.6% idle
CPU1 states: 1.2% user, 0.0% nice, 24.2% system, 50.1% interrupt,
24.6% idle

[Darren Tucker]

Does anyone with a working knowledge of re(4) have any idea why the PC
Engines APU NICs perform so poorly in OpenBSD?

Most likely lack of hardware interrupt moderation in the driver.
There's code in re_setup_hw_im() that looks like might do something
plausible with the interrupt moderation register but AFAICT it'll
never be called because rl_imtype is always set to "RL_IMTYPE_SIM".

I tried to get hardware interrupt moderation working a while back but
it didn't seem to make a difference (which is probably an indication
that I did something wrong). I could dig up the patch if you'd like
to try it.

The other thing to be aware of is that if you're following current,
POOL_DEBUG is usually set in your config, which will be quite
expensive when pushing packets.

[Momtchil Momtchev]

Post by Darren Tucker

Does anyone with a working knowledge of re(4) have any idea why the PC
Engines APU NICs perform so poorly in OpenBSD?

Most likely lack of hardware interrupt moderation in the driver.
There's code in re_setup_hw_im() that looks like might do something
plausible with the interrupt moderation register but AFAICT it'll
never be called because rl_imtype is always set to "RL_IMTYPE_SIM".

What is the problem with software interrupt moderation? That it has
a fixed timer while the hardware one scales with the RX rate? This
shouldn't halve the performance? It should be more like 10% to 15% and
some latency benefit? I have also noticed that the TX rate is higher
than the RX rate (about 320 Mbit/s vs 260 Mbit/s). Could it be that the
FreeBSD driver uses MSI interrupts and the OpenBSD one does not?
PS. On the APU my interrupt rate is about 6000 IRQ/s when doing 320
MBit/s, this is one IRQ every 165us or one IRQ for about 3 or 4 packets.
I will make rl_sim_time tunable and I will test if it affects performance.

[Darren Tucker]

This post might be inappropriate. Click to display it.