Freezing VMs on Bytemark Hosting

[Edd Barrett]

Hi,

This is very much off-topic, and a long shot.

I have a VM hosted at Bytemark, which seems to have started freezing
about once a week. It stops responding to the network, and if I bring up
the console, I see the login prompt with a flashing cursor, but it is
not responsive to key-presses.

I have a support ticket open, but we are not sure if it's an OpenBSD
problem, or something on their end. The VM is running 5.9-stable with
all patches applied. FWIW, Bytemark uses KVM + Qemu, so this question may
extend to ARP networks VMs too(?).

Wondering if anyone else here is hosting on Bytemark (or ARP) and had a
similar issue, or even a workaround.

Like I said, long shot.

Cheers

[Mark Carroll]

Post by Edd Barrett
I have a support ticket open, but we are not sure if it's an OpenBSD
problem, or something on their end. The VM is running 5.9-stable with
all patches applied. FWIW, Bytemark uses KVM + Qemu, so this question may
extend to ARP networks VMs too(?).
Wondering if anyone else here is hosting on Bytemark (or ARP) and had a
similar issue, or even a workaround.

I am running vanilla 5.9-stable on Bytemark's BigV, pretty up to date
with patches, and haven't seen any such problems. Feel free to e-mail
privately if you want to probe any particular aspects of it.

-- Mark

[Mark Lee Smith]

I have a host running OpenBSD 5.9 -stable on Vultr which has been doing the
same thing; freezing once a week. Is there any information that I could
provide if/when it happens again?

All the best,

Mark

Post by Mark Carroll

Post by Edd Barrett
I have a support ticket open, but we are not sure if it's an OpenBSD
problem, or something on their end. The VM is running 5.9-stable with
all patches applied. FWIW, Bytemark uses KVM + Qemu, so this question may
extend to ARP networks VMs too(?).
Wondering if anyone else here is hosting on Bytemark (or ARP) and had a
similar issue, or even a workaround.

I am running vanilla 5.9-stable on Bytemark's BigV, pretty up to date
with patches, and haven't seen any such problems. Feel free to e-mail
privately if you want to probe any particular aspects of it.
-- Mark

[Edgar Pettijohn]

Sent from my iPhone

Post by Mark Lee Smith
I have a host running OpenBSD 5.9 -stable on Vultr which has been doing the
same thing; freezing once a week. Is there any information that I could
provide if/when it happens again?
All the best,
Mark

I do as well, but haven't experienced this.

Edgar

Post by Mark Lee Smith

Post by Mark Carroll

Post by Edd Barrett
I have a support ticket open, but we are not sure if it's an OpenBSD
problem, or something on their end. The VM is running 5.9-stable with
all patches applied. FWIW, Bytemark uses KVM + Qemu, so this question may
extend to ARP networks VMs too(?).
Wondering if anyone else here is hosting on Bytemark (or ARP) and had a
similar issue, or even a workaround.

I am running vanilla 5.9-stable on Bytemark's BigV, pretty up to date
with patches, and haven't seen any such problems. Feel free to e-mail
privately if you want to probe any particular aspects of it.
-- Mark

[Edd Barrett]

Post by Mark Lee Smith
I have a host running OpenBSD 5.9 -stable on Vultr which has been doing the
same thing; freezing once a week. Is there any information that I could
provide if/when it happens again?

Interesting...

Does Vultr use kvm+qemu?

At the moment I am trying to narrow down what the freeze could possibly
be related to. I really have no idea.

I think I am going to throw up another VM and deploy a DEBUG kernel on
it, to see if it gives any clues.

What have you tried so far btw?

[Abel Abraham Camarillo Ojeda]

Post by Edd Barrett

Post by Mark Lee Smith
I have a host running OpenBSD 5.9 -stable on Vultr which has been doing the
same thing; freezing once a week. Is there any information that I could
provide if/when it happens again?

Interesting...
Does Vultr use kvm+qemu?

dmesg from vultr attached.

I'm not having problems using it.

Post by Edd Barrett
At the moment I am trying to narrow down what the freeze could possibly
be related to. I really have no idea.
I think I am going to throw up another VM and deploy a DEBUG kernel on
it, to see if it gives any clues.
What have you tried so far btw?
--
Best Regards
Edd Barrett
http://www.theunixzoo.co.uk

[demime 1.01d removed an attachment of type application/octet-stream which had a name of dmesg]

[andrew fabbro]

You're running on KVM, which probably means you're using virtio. Have you
set the 0x2 flag on the vio driver?

I experienced hangs on my KVM-hosted OpenBSD VMs until I read the vio(4)
man page:

http://man.openbsd.org/vio.4

"The *vio* driver provides support for the virtio(4)
<http://man.openbsd.org/virtio.4> network interface provided by bhyve, KVM,
QEMU, and VirtualBox.
Setting the bit 0x2 in the flags disables the RingEventIndex feature. This
can be tried as a workaround for possible bugs in host implementations of
*vio* at the cost of slightly reduced performance."

An example of how to do this:

http://blather.michaelwlucas.com/archives/2083

Post by Edd Barrett
Hi,
This is very much off-topic, and a long shot.
I have a VM hosted at Bytemark, which seems to have started freezing
about once a week. It stops responding to the network, and if I bring up
the console, I see the login prompt with a flashing cursor, but it is
not responsive to key-presses.
I have a support ticket open, but we are not sure if it's an OpenBSD
problem, or something on their end. The VM is running 5.9-stable with
all patches applied. FWIW, Bytemark uses KVM + Qemu, so this question may
extend to ARP networks VMs too(?).
Wondering if anyone else here is hosting on Bytemark (or ARP) and had a
similar issue, or even a workaround.
Like I said, long shot.
Cheers
--
Best Regards
Edd Barrett
http://www.theunixzoo.co.uk

[Edd Barrett]

Hi,

Post by andrew fabbro
You're running on KVM, which probably means you're using virtio. Have you
set the 0x2 flag on the vio driver?
I experienced hangs on my KVM-hosted OpenBSD VMs until I read the vio(4)
http://man.openbsd.org/vio.4
"The *vio* driver provides support for the virtio(4)
<http://man.openbsd.org/virtio.4> network interface provided by bhyve, KVM,
QEMU, and VirtualBox.
Setting the bit 0x2 in the flags disables the RingEventIndex feature. This
can be tried as a workaround for possible bugs in host implementations of
*vio* at the cost of slightly reduced performance."
http://blather.michaelwlucas.com/archives/2083

This is very good to know.

However, I don't think this is what I am experiencing. When I started
having problems I switched the VM into "legacy mode". Once I did this,
vio0 became em0, and the problem persisted.

I don't think the issue it network related. The VGA console is not
responsive to key-presses at the login prompt.

Cheers

[Mark Lee Smith]

Sorry if this message comes as a duplicate, I notice that I didn't CC misc.

Thanks for your response Andrew and Abel.

@Andrew what do you mean by "hangs"? Did the whole machine lock up or was
it just the network that was down?

@Abel Your dmesg is from OpenBSD 5.7?

Here's the dmesg from OpenBSD 5.9 -stable running on one of the two hosts I
have on vultr that have been freezing roughly once per week.

This happened again this morning.

Does anyone have any advice on how to proceed and or collect any relevant
information from the host? I'm available to help as much as possible but I
don't know where to start.

@Edd It's probably too soon to tell but have you had any luck so far?

OpenBSD 5.9 (GENERIC) #8: Thu Jul 14 20:12:37 CEST 2016
***@stable-59-amd64.mtier.org:
/binpatchng/work-binpatch59-amd64/src/sys/arch/amd64/compile/GENERIC
real mem = 788398080 (751MB)
avail mem = 760397824 (725MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf6300 (9 entries)
bios0: vendor SeaBIOS version "rel-1.8.2-0-g33fbe13 by qemu-project.org"
date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Virtual CPU 714389bda930, 2400.39 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOV
BE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ABM,FSGSBASE,BM
I1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0

On Fri, 29 Jul 2016 at 15:07 Abel Abraham Camarillo Ojeda <

Post by Abel Abraham Camarillo Ojeda

Post by Edd Barrett

Post by Mark Lee Smith
I have a host running OpenBSD 5.9 -stable on Vultr which has been doing

the

Post by Edd Barrett

Post by Mark Lee Smith
same thing; freezing once a week. Is there any information that I could
provide if/when it happens again?

Interesting...
Does Vultr use kvm+qemu?

dmesg from vultr attached.
I'm not having problems using it.

Post by Edd Barrett
At the moment I am trying to narrow down what the freeze could possibly
be related to. I really have no idea.
I think I am going to throw up another VM and deploy a DEBUG kernel on
it, to see if it gives any clues.
What have you tried so far btw?
--
Best Regards
Edd Barrett
http://www.theunixzoo.co.uk

[Mark Lee Smith]

Note to self: I shouldn't attempt to reply to emails when I'm tired in the
future. Here is the full dmesg for my host OpenBSD 5.9 -stable running on
vultr, in case it didn't come through in my previous email.

OpenBSD 5.9 (GENERIC) #8: Thu Jul 14 20:12:37 CEST 2016
***@stable-59-amd64.mtier.org:
/binpatchng/work-binpatch59-amd64/src/sys/arch/amd64/compile/GENERIC
real mem = 788398080 (751MB)
avail mem = 760397824 (725MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf6300 (9 entries)
bios0: vendor SeaBIOS version "rel-1.8.2-0-g33fbe13 by qemu-project.org"
date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Virtual CPU 714389bda930, 2400.39 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOV
BE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ABM,FSGSBASE,BM
I1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom
removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int
9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 56:00:00:2d:1c:60
virtio0: apic 0 int 11
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 15360MB, 512 bytes/sector, 31457280 sectors
virtio1: apic 0 int 11
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 10
virtio3 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio3
virtio3: apic 0 int 10
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 1: density unknown
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet"
rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (6d14ab39e519f0a2.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
clock: unknown CMOS layout

Post by Mark Lee Smith
Sorry if this message comes as a duplicate, I notice that I didn't CC misc.
Thanks for your response Andrew and Abel.
@Andrew what do you mean by "hangs"? Did the whole machine lock up or was
it just the network that was down?
@Abel Your dmesg is from OpenBSD 5.7?
Here's the dmesg from OpenBSD 5.9 -stable running on one of the two hosts
I have on vultr that have been freezing roughly once per week.
This happened again this morning.
Does anyone have any advice on how to proceed and or collect any relevant
information from the host? I'm available to help as much as possible but I
don't know where to start.
@Edd It's probably too soon to tell but have you had any luck so far?
OpenBSD 5.9 (GENERIC) #8: Thu Jul 14 20:12:37 CEST 2016
/binpatchng/work-binpatch59-amd64/src/sys/arch/amd64/compile/GENERIC
real mem = 788398080 (751MB)
avail mem = 760397824 (725MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0: vendor SeaBIOS version "rel-1.8.2-0-g33fbe13 by qemu-project.org"
date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Virtual CPU 714389bda930, 2400.39 MHz

FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOV
BE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ABM,FSGSBASE,BM
I1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT

Post by Mark Lee Smith
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0
On Fri, 29 Jul 2016 at 15:07 Abel Abraham Camarillo Ojeda <

Post by Mark Lee Smith

Post by Edd Barrett

Post by Mark Lee Smith
I have a host running OpenBSD 5.9 -stable on Vultr which has been

doing the

Post by Edd Barrett

Post by Mark Lee Smith
same thing; freezing once a week. Is there any information that I could
provide if/when it happens again?

Interesting...
Does Vultr use kvm+qemu?

dmesg from vultr attached.
I'm not having problems using it.

Post by Edd Barrett
At the moment I am trying to narrow down what the freeze could possibly
be related to. I really have no idea.
I think I am going to throw up another VM and deploy a DEBUG kernel on
it, to see if it gives any clues.
What have you tried so far btw?
--
Best Regards
Edd Barrett
http://www.theunixzoo.co.uk