openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6

[m***@rocketmail.com]

openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)



Hi all,


first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)

As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
this is a specific openssh issue or something else?


Thanks in advance,
Michael

[Vincenzo Romano]

Post by m***@rocketmail.com
openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
Hi all,
first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
this is a specific openssh issue or something else?
Thanks in advance,
Michael

Can you try the same transfer with a (totally) different protocol,
like FTP, NFS, CIFS?
This is to exclude any network-related slowdown.

[m***@rocketmail.com]

Vincenzo,


thanks for answering !!!

As I found out before that this slow down only happens for encrypted transmissions, I've followed your suggestion and
tested with a https-based download, from my Nextcloud instance on my VPS.

Same slow speed when connecting via IPv6, and as fast as expected when connecting via IPv4.

Just to confirm: There's NO dependency/relation between openssh and https, on Linux, correct? (Web Server is Apache)
If NO, sorry for wasting your time, and again THANK YOU for leading me into another direction!


Regards,
Michael

Post by Vincenzo Romano

Post by m***@rocketmail.com
openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
Hi all,
first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
this is a specific openssh issue or something else?
Thanks in advance,
Michael

Can you try the same transfer with a (totally) different protocol,
like FTP, NFS, CIFS?
This is to exclude any network-related slowdown.

[Vincenzo Romano]

Post by m***@rocketmail.com
Vincenzo,
thanks for answering !!!
As I found out before that this slow down only happens for encrypted transmissions, I've followed your suggestion and
tested with a https-based download, from my Nextcloud instance on my VPS.
Same slow speed when connecting via IPv6, and as fast as expected when connecting via IPv4.
Just to confirm: There's NO dependency/relation between openssh and https, on Linux, correct? (Web Server is Apache)
If NO, sorry for wasting your time, and again THANK YOU for leading me into another direction!
Regards,
Michael

libssl could be a common factor.
Try FTP, please.

Post by m***@rocketmail.com

Post by Vincenzo Romano

Post by m***@rocketmail.com
openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
Hi all,
first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
this is a specific openssh issue or something else?
Thanks in advance,
Michael

Can you try the same transfer with a (totally) different protocol,
like FTP, NFS, CIFS?
This is to exclude any network-related slowdown.

[Vincenzo Romano]

Il giorno mer 7 nov 2018 alle ore 19:38 Vincenzo Romano

Post by Vincenzo Romano

Post by m***@rocketmail.com
Vincenzo,
thanks for answering !!!
As I found out before that this slow down only happens for encrypted transmissions, I've followed your suggestion and
tested with a https-based download, from my Nextcloud instance on my VPS.
Same slow speed when connecting via IPv6, and as fast as expected when connecting via IPv4.
Just to confirm: There's NO dependency/relation between openssh and https, on Linux, correct? (Web Server is Apache)
If NO, sorry for wasting your time, and again THANK YOU for leading me into another direction!
Regards,
Michael

libssl could be a common factor.
Try FTP, please.

Nope, libssl isn't a common library.
Again , try FTP.

Post by Vincenzo Romano

Post by m***@rocketmail.com

Post by Vincenzo Romano

Post by m***@rocketmail.com
openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
Hi all,
first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
this is a specific openssh issue or something else?
Thanks in advance,
Michael

Can you try the same transfer with a (totally) different protocol,
like FTP, NFS, CIFS?
This is to exclude any network-related slowdown.

--
Vincenzo Romano - NotOrAnd.IT
Information Technologies
--
NON QVIETIS MARIBVS NAVTA PERITVS

[Philipp Marek]

Post by m***@rocketmail.com
first post to list, hopefully on-topic. Haven't found anything on the
net, tried to ask at first in OpenSuse forums a
while ago
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
and today
opened a bug in OpenSuse's Bugzilla
(https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
As written in bug report, as general network issues could be excluded,
any hints where to start, how to narrow down if
this is a specific openssh issue or something else?

Your IPv6 _route_ has 3 times the latency:

$ time /usr/sbin/traceroute vserver.domain.tld
...
7 domain.tld (IPv4) 26.002 ms !X 24.859 ms !X 25.230 ms !X

$ time /usr/sbin/traceroute6 vserver.domain.tld
...
8 domain.tld (IPv6) 80.430 ms !X 79.301 ms !X 80.444 ms !X

The big time waster seems to be this link:
6 nug-d-i40-v6.telia.net (2001:2000:3018:8d::1) 32.046 ms 23.850
ms 24.097 ms
7 contabo-ic-305268-ffm-b11.c.telia.net (2001:2000:3080:953::2)
74.829 ms 77.704 ms 78.066 ms

I'll now speculate that you might as well have some MTU mismatches,
meaning that packages have to get split up along the way, further
slowing down your connection.

Do you have native IPv6 all the way, or is there an 6-in-4 tunnel
inbetween (from the fritzbox on)?

Please try
# ping -M do -s <size> <ip>
for sizes between 1300 and 1500 and find the largest (even) packet size
that still works, and then do the same with ping6 as well.

Also note whether some packetsize works, but just larger ones vanish
along the way.

[m***@rocketmail.com]

Servus Philipp,


Unfortunately the traceroute(6) results are both more or less random. Sometimes traceroute "hangs" a while, wherever,
sometimes traceroute6. Sometimes traceroute is faster, sometimes traceroute6. Not reliable.

Your MTU question, tried as adviced:

Maximum size for IPv4 is 1466, and max size for IPv6 is 1444. Exceeding these values leads to a "ping: local error:
Message too long, mtu=1492" in both cases.

My Fritzbox says "Native IPv6", no tunnel, no underlying "IPv4 via DS-Lite". These are the default settings for ISP
"Deutsche Telekom", VDSL-50.


VINCENZO: Tried as adviced to test with FTP, installed pure-ftpd and opened VPS's ftp port in firewall. Maximum IPv4
speed was much slower (1.5-2 MB/sec) than other tests, maybe temporary issue or virtualization. But again: IPv6 was MUCH
slower than IPv4.

Post by m***@rocketmail.com
first post to list, hopefully on-topic. Haven't found anything on the
net, tried to ask at first in OpenSuse forums a
while ago
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
and today
opened a bug in OpenSuse's Bugzilla
(https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
As written in bug report, as general network issues could be excluded,
any hints where to start, how to narrow down if
this is a specific openssh issue or something else?

  $ time /usr/sbin/traceroute vserver.domain.tld
  ...
   7  domain.tld (IPv4)  26.002 ms !X  24.859 ms !X  25.230 ms !X
  $ time /usr/sbin/traceroute6 vserver.domain.tld
  ...
   8  domain.tld (IPv6)  80.430 ms !X  79.301 ms !X  80.444 ms !X
   6  nug-d-i40-v6.telia.net (2001:2000:3018:8d::1)  32.046 ms  23.850 ms  24.097 ms
   7  contabo-ic-305268-ffm-b11.c.telia.net (2001:2000:3080:953::2)  74.829 ms  77.704 ms  78.066 ms
I'll now speculate that you might as well have some MTU mismatches,
meaning that packages have to get split up along the way, further
slowing down your connection.
Do you have native IPv6 all the way, or is there an 6-in-4 tunnel
inbetween (from the fritzbox on)?
Please try
   # ping -M do -s <size> <ip>
for sizes between 1300 and 1500 and find the largest (even) packet size
that still works, and then do the same with ping6 as well.
Also note whether some packetsize works, but just larger ones vanish
along the way.

[Philipp Marek]

Post by m***@rocketmail.com
Unfortunately the traceroute(6) results are both more or less random.
Sometimes traceroute "hangs" a while, wherever,
sometimes traceroute6. Sometimes traceroute is faster, sometimes traceroute6. Not reliable.

That might just as well be DNS issue; the reported latency
values are the interesting data.

Post by m***@rocketmail.com
Maximum size for IPv4 is 1466, and max size for IPv6 is 1444.
Message too long, mtu=1492" in both cases.

Hmmm, the IPv6 header is 20 bytes longer than IPv4 -- so there
are still 2 bytes missing.

Please try to set the MTU manually smaller, eg. to 1400,
and test once more.

[m***@rocketmail.com]

Philipp,


thank you so much for your valuable hints!!

BUT as this list's topic is OpenSSH, and as I now know that my issue has NOTHING to do with OpenSSH, I would like to
stop here, not wasting anyone's time.

Thanks to Vincenco's FTP hint I've now tested with a Debian based "rescue live system" instead of my OpenSuse based VPS,
on server side, offered by my VPS's hosting company. And on the client side with my Android smartphone and the (FOSS)
"Ghost Commander" instead of my OpenSuse ***@home. To be sure that my issue has also nothing to do with OpenSuse, as my
preferred Linux flavor.

And in fact it has nothing to do with either OpenSSH or OpenSuse....


Thanks again,
Michael

Post by Philipp Marek

Post by m***@rocketmail.com
Unfortunately the traceroute(6) results are both more or less random.
Sometimes traceroute "hangs" a while, wherever,
sometimes traceroute6. Sometimes traceroute is faster, sometimes
traceroute6. Not reliable.

That might just as well be DNS issue; the reported latency
values are the interesting data.

Post by m***@rocketmail.com
Maximum size for IPv4 is 1466, and max size for IPv6 is 1444.
Message too long, mtu=1492" in both cases.

Hmmm, the IPv6 header is 20 bytes longer than IPv4 -- so there
are still 2 bytes missing.
Please try to set the MTU manually smaller, eg. to 1400,
and test once more.