Discussion:
sshd 7.8p1 close connection from VMware Fusion NAT Port Forwarding
Zach Cheung
2018-08-27 06:15:02 UTC
Permalink
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but via
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.

Any idea about this bug?

VMware Fusion NAT Port Forwarding setup:
VMware Fusion – NAT Port Forwarding 101 | www.weatherhead.net
http://www.weatherhead.net/2015/06/21/vmware-fusion-nat-port-forwarding-101/

Arch openssh PKGBUILD:
PKGBUILD\trunk - svntogit/packages.git - Git clone of the 'packages'
repository
https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/openssh

Here are ssh and sshd logs:
OpenSSH 7.8p1 close connection from VMware Fusion NAT Port Forwarding
https://gist.github.com/ZachCheung/9e48769067ac6681c419fb46f480fb90
Stuart Henderson
2018-08-27 09:20:54 UTC
Permalink
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but via
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Stuart Henderson
2018-08-27 09:29:02 UTC
Permalink
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but via
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Sorry, should be separated by whitespace: "IPQoS lowdelay throughput".
(This restores the pre-7.8 setting).
Zach Cheung
2018-08-27 10:13:45 UTC
Permalink
Sorry, add -o 'IPQoS=lowdelay throughput' to ssh?
if so, still not working, here are log:
https://gist.github.com/ZachCheung/9e48769067ac6681c419fb46f480fb90#file-ssh-7-8p1_with_ipqos-log
Post by Stuart Henderson
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but
via
Post by Stuart Henderson
Post by Zach Cheung
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Sorry, should be separated by whitespace: "IPQoS lowdelay throughput".
(This restores the pre-7.8 setting).
_______________________________________________
openssh-unix-dev mailing list
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Stuart Henderson
2018-08-27 10:53:13 UTC
Permalink
Post by Zach Cheung
Sorry, add -o 'IPQoS=lowdelay throughput' to ssh?
Try in sshd_config on the server.
Post by Zach Cheung
https://gist.github.com/ZachCheung/9e48769067ac6681c419fb46f480fb90#file-ssh-7-8p1_with_ipqos-log
Post by Stuart Henderson
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but
via
Post by Stuart Henderson
Post by Zach Cheung
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Sorry, should be separated by whitespace: "IPQoS lowdelay throughput".
(This restores the pre-7.8 setting).
_______________________________________________
openssh-unix-dev mailing list
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Zach Cheung
2018-08-28 02:01:51 UTC
Permalink
Hi Stuart,

It works, thanks a lot.
Post by Stuart Henderson
Post by Zach Cheung
Sorry, add -o 'IPQoS=lowdelay throughput' to ssh?
Try in sshd_config on the server.
https://gist.github.com/ZachCheung/9e48769067ac6681c419fb46f480fb90#file-ssh-7-8p1_with_ipqos-log
Post by Zach Cheung
Post by Stuart Henderson
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest
with
Post by Zach Cheung
Post by Stuart Henderson
Post by Stuart Henderson
Post by Zach Cheung
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS
Sierra
Post by Zach Cheung
Post by Stuart Henderson
Post by Stuart Henderson
Post by Zach Cheung
(10.12.6) host to Arch guest via local NAT port forwarding failed,
but
Post by Zach Cheung
Post by Stuart Henderson
via
Post by Stuart Henderson
Post by Zach Cheung
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Sorry, should be separated by whitespace: "IPQoS lowdelay throughput".
(This restores the pre-7.8 setting).
_______________________________________________
openssh-unix-dev mailing list
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Damien Miller
2018-08-28 04:17:55 UTC
Permalink
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but via
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Do you have any insight into what is breaking here? I don't believe
changing the default DSCP values should break connections...

-d
Job Snijders
2018-08-28 09:51:29 UTC
Permalink
Post by Damien Miller
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but
via
Post by Stuart Henderson
Post by Zach Cheung
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Do you have any insight into what is breaking here? I don't believe
changing the default DSCP values should break connections...
I suspect VMWare Fusion has a very broken NAT implementation, where they
seem to hash packets to identify flows on (part of) the DSCP field.

Kind regards,

Job
Stuart Henderson
2018-08-28 10:15:38 UTC
Permalink
Post by Damien Miller
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest with
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
(10.12.6) host to Arch guest via local NAT port forwarding failed, but via
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Do you have any insight into what is breaking here? I don't believe
changing the default DSCP values should break connections...
I think it's probably a NAT bug in VMware Fusion. tcpdump might
give more clues as to how it's broken (maybe it's mangling packets,
maybe it's just rejecting them) but actually fixing it would need
VMware's involvement.

Short description: OpenSSH 7.8 started marking packets with DSCP
(af21 for interactive, cs1 for bulk) instead of IP TOS ("lowdelay"
for interactive, "throughput" for bulk). VMware Fusion with NAT
port-forwarding to sshd in the guest fails with OpenSSH 7.8.
It should be possible to replicate this failure with older OpenSSH
(6.0 or newer) by using "IPQoS af21 cs1" in sshd_config in the guest.

Unless any VMware people are reading this, it's probably best if one
of their customers reports it as a bug, I can't imagine it would be
that complicated to fix, the problem will be getting the report past
front-line support and on to the right person.
Zach Cheung
2018-08-28 14:24:04 UTC
Permalink
confirmed that sshd 7.7p1 with "IPQoS af21 cs1" also closed connection.
Post by Zach Cheung
Post by Damien Miller
Post by Stuart Henderson
Post by Zach Cheung
After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest
with
Post by Damien Miller
Post by Stuart Henderson
Post by Zach Cheung
OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS
Sierra
Post by Damien Miller
Post by Stuart Henderson
Post by Zach Cheung
(10.12.6) host to Arch guest via local NAT port forwarding failed,
but via
Post by Damien Miller
Post by Stuart Henderson
Post by Zach Cheung
Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
problem.
Any idea about this bug?
I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Do you have any insight into what is breaking here? I don't believe
changing the default DSCP values should break connections...
I think it's probably a NAT bug in VMware Fusion. tcpdump might
give more clues as to how it's broken (maybe it's mangling packets,
maybe it's just rejecting them) but actually fixing it would need
VMware's involvement.
Short description: OpenSSH 7.8 started marking packets with DSCP
(af21 for interactive, cs1 for bulk) instead of IP TOS ("lowdelay"
for interactive, "throughput" for bulk). VMware Fusion with NAT
port-forwarding to sshd in the guest fails with OpenSSH 7.8.
It should be possible to replicate this failure with older OpenSSH
(6.0 or newer) by using "IPQoS af21 cs1" in sshd_config in the guest.
Unless any VMware people are reading this, it's probably best if one
of their customers reports it as a bug, I can't imagine it would be
that complicated to fix, the problem will be getting the report past
front-line support and on to the right person.
_______________________________________________
openssh-unix-dev mailing list
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Loading...