Discussion:
PC Engines APU NIC (RTL8111E) performance
(too old to reply)
Momtchil Momtchev
2016-08-03 10:07:51 UTC
Permalink
Hello,


Does anyone with a working knowledge of re(4) have any idea why the
PC Engines APU NICs perform so poorly in OpenBSD? Throughput is 300 to
320 MBit/s with about 30 pf rules, NAT and 10000 states when running
5.9. This is much less than an APU running Linux or FreeBSD or an
OpenBSD running on comparable CPU with another NIC. I wonder how do
other motherboards with that same NIC (it is very common on the lower
end) perform?

Here is my top output when running iperf routing through the APU :

CPU0 states: 1.0% user, 0.0% nice, 51.5% system, 26.9% interrupt,
20.6% idle
CPU1 states: 1.2% user, 0.0% nice, 24.2% system, 50.1% interrupt,
24.6% idle
Darren Tucker
2016-08-04 07:13:54 UTC
Permalink
Does anyone with a working knowledge of re(4) have any idea why the PC
Engines APU NICs perform so poorly in OpenBSD?
Most likely lack of hardware interrupt moderation in the driver.
There's code in re_setup_hw_im() that looks like might do something
plausible with the interrupt moderation register but AFAICT it'll
never be called because rl_imtype is always set to "RL_IMTYPE_SIM".

I tried to get hardware interrupt moderation working a while back but
it didn't seem to make a difference (which is probably an indication
that I did something wrong). I could dig up the patch if you'd like
to try it.

The other thing to be aware of is that if you're following current,
POOL_DEBUG is usually set in your config, which will be quite
expensive when pushing packets.
--
Darren Tucker (dtucker at zip.com.au)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
Momtchil Momtchev
2016-08-04 12:46:44 UTC
Permalink
Post by Darren Tucker
Does anyone with a working knowledge of re(4) have any idea why the PC
Engines APU NICs perform so poorly in OpenBSD?
Most likely lack of hardware interrupt moderation in the driver.
There's code in re_setup_hw_im() that looks like might do something
plausible with the interrupt moderation register but AFAICT it'll
never be called because rl_imtype is always set to "RL_IMTYPE_SIM".
What is the problem with software interrupt moderation? That it has
a fixed timer while the hardware one scales with the RX rate? This
shouldn't halve the performance? It should be more like 10% to 15% and
some latency benefit? I have also noticed that the TX rate is higher
than the RX rate (about 320 Mbit/s vs 260 Mbit/s). Could it be that the
FreeBSD driver uses MSI interrupts and the OpenBSD one does not?
PS. On the APU my interrupt rate is about 6000 IRQ/s when doing 320
MBit/s, this is one IRQ every 165us or one IRQ for about 3 or 4 packets.
I will make rl_sim_time tunable and I will test if it affects performance.
Darren Tucker
2016-08-05 01:56:15 UTC
Permalink
On Thu, Aug 04, 2016 at 02:46:44PM +0200, Momtchil Momtchev wrote:
[...]
What is the problem with software interrupt moderation? That it has a
fixed timer while the hardware one scales with the RX rate?
The hardware moderation can do per-N-packets in addition to a timer.
This shouldn't
halve the performance? It should be more like 10% to 15% and some latency
benefit? I have also noticed that the TX rate is higher than the RX rate
(about 320 Mbit/s vs 260 Mbit/s). Could it be that the FreeBSD driver uses
MSI interrupts and the OpenBSD one does not?
Dunno. If I knew what the cause was I'd have fixed it :-(
PS. On the APU my interrupt rate is about 6000 IRQ/s when doing 320
MBit/s, this is one IRQ every 165us or one IRQ for about 3 or 4 packets. I
will make rl_sim_time tunable and I will test if it affects performance.
I dug up my patch. If you're experimenting, making the value used to
set the RL_IM register tunable then seeing what impact various values
have on throughput would be interesting.

Index: dev/ic/re.c
===================================================================
RCS file: /cvs/src/sys/dev/ic/re.c,v
retrieving revision 1.192
diff -u -p -r1.192 re.c
--- dev/ic/re.c 20 Apr 2016 12:15:24 -0000 1.192
+++ dev/ic/re.c 5 Aug 2016 00:31:04 -0000
@@ -747,7 +747,7 @@ re_attach(struct rl_softc *sc, const cha
sc->rl_flags |= RL_FLAG_PHYWAKE | RL_FLAG_PHYWAKE_PM |
RL_FLAG_PAR | RL_FLAG_DESCV2 | RL_FLAG_MACSTAT |
RL_FLAG_CMDSTOP | RL_FLAG_AUTOPAD | RL_FLAG_JUMBOV2 |
- RL_FLAG_WOL_MANLINK;
+ RL_FLAG_WOL_MANLINK | RL_FLAG_HWIM;
sc->rl_max_mtu = RL_JUMBO_MTU_9K;
break;
case RL_HWREV_8168E_VL:
@@ -821,13 +821,19 @@ re_attach(struct rl_softc *sc, const cha
/* Reset the adapter. */
re_reset(sc);

- sc->rl_tx_time = 5; /* 125us */
- sc->rl_rx_time = 2; /* 50us */
- if (sc->rl_flags & RL_FLAG_PCIE)
- sc->rl_sim_time = 75; /* 75us */
- else
- sc->rl_sim_time = 125; /* 125us */
- sc->rl_imtype = RL_IMTYPE_SIM; /* simulated interrupt moderation */
+ if (sc->rl_flags & RL_FLAG_HWIM) {
+ /* hardware interrupt moderation */
+ sc->rl_imtype = RL_IMTYPE_HW;
+ sc->rl_tx_time = 5; /* 125us */
+ sc->rl_rx_time = 2; /* 50us */
+ } else {
+ /* simulated interrupt moderation */
+ sc->rl_imtype = RL_IMTYPE_SIM;
+ if (sc->rl_flags & RL_FLAG_PCIE)
+ sc->rl_sim_time = 75; /* 75us */
+ else
+ sc->rl_sim_time = 125; /* 125us */
+ }

if (sc->sc_hwrev == RL_HWREV_8139CPLUS)
sc->rl_bus_speed = 33; /* XXX */
@@ -2233,6 +2239,8 @@ re_stop(struct ifnet *ifp)
void
re_setup_hw_im(struct rl_softc *sc)
{
+ u_int16_t im;
+
KASSERT(sc->rl_flags & RL_FLAG_HWIM);

/*
@@ -2258,11 +2266,15 @@ re_setup_hw_im(struct rl_softc *sc)
* Currently we only know how to set 'timer', but not
* 'number of packets', which should be ~30, as far as I
* tested (sink ~900Kpps, interrupt rate is 30KHz)
- */
- CSR_WRITE_2(sc, RL_IM,
- RL_IM_RXTIME(sc->rl_rx_time) |
- RL_IM_TXTIME(sc->rl_tx_time) |
- RL_IM_MAGIC);
+ *
+ * According to the Linux driver, supposedly:
+ * (TxTimer << 12) | (TxPackets << 8) | (RxTimer << 4) | RxPackets
+ * Linux uses hard coded 0x5151.
+ */
+ im = RL_IM_TXTIME(sc->rl_tx_time) | RL_IM_TXPKTS(4) |
+ RL_IM_RXTIME(sc->rl_rx_time) | RL_IM_RXPKTS(4);
+ printf("setting interrupt moderation %hx\n", im); /* XXX */
+ CSR_WRITE_2(sc, RL_IM, im);
}

void
Index: dev/ic/rtl81x9reg.h
===================================================================
RCS file: /cvs/src/sys/dev/ic/rtl81x9reg.h,v
retrieving revision 1.98
diff -u -p -r1.98 rtl81x9reg.h
--- dev/ic/rtl81x9reg.h 20 Apr 2016 12:15:24 -0000 1.98
+++ dev/ic/rtl81x9reg.h 5 Aug 2016 00:31:04 -0000
@@ -570,7 +570,9 @@

#define RL_IM_MAGIC 0x5050
#define RL_IM_RXTIME(t) ((t) & 0xf)
+#define RL_IM_RXPKTS(t) (((t) & 0xf) << 4)
#define RL_IM_TXTIME(t) (((t) & 0xf) << 8)
+#define RL_IM_TXPKTS(t) (((t) & 0xf) << 12)

struct rl_chain_data {
u_int16_t cur_rx;
--
Darren Tucker (dtucker at zip.com.au)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
Loading...