File downloads are truncated on Three Broadband

[electricworry]

I'm looking for some verification about whether the issue I have is isolated to me, my area, or if it's a general Three-wide problem as I think it is.

I use Three 5G broadband and I'm about 50 metres away from the gNodeB so I've got excellent uninterrupted signal. It's not a Layer 1 problem I'm facing. The problem I have is that TCP connections are terminated prematurely (i.e. a RST packet is sent) before all data is received. Here's a simple test to verify if you have the problem or not.
The following command will attempt to download an 8MiB file (all NULs) from a website in AWS. It should work the same on Linux, MacOS, and modern Windows computers just the same. For me, I get the error "curl: (18) transfer closed with XXXXXX bytes remaining to read", which is the problem.

curl -H "Connection: close" https://electricworry.net/test-8 -o test-curl

If you're not comfortable connecting to my server, the following third-party download test should produce the same result (it does for me!):

curl -H "Connection: close" https://files.testfile.org/ZIPC/15MB-Corrupt-Testfile.Org.zip -o test-curl

When I tested, I collected a packet capture at both sides and I can see that my server sends the whole 8MiB file in the TLS session and then terminates the connection with a RST packet at the end (which it does because we sent a "Connection: close" header). However on my client side, only half of the file comes through before the session is impolitely terminated.

Would people on Three 5G broadband mind testing please to help confirm/deny whether this is a general problem or an individual one?

I've done a lot of testing over the past month and I've got a hypothesis.

• Comparing the server and client packet captures, the packets do not match up; the sequence and ack numbers - though they start the same - end up being very different. It appears that something in the middle is buffering the stream and ACKing the packets on my behalf.
• The problem only happens when I'm on my Three 5G Broadband service. If I take my laptop into work, the problem is gone. The problem doesn't occur when I use my Giffgaff mobile as a hotspot either.
• The problem exists on all websites (I suffer *a lot* from Ubuntu APT packages being half-downloaded and rejected on my workstation).
• Since the times on my server and client are synchronised as good as possible with NTP I can compare progress of the stream at both sides. When my server has finished transmitting (and received the final ACK) it correctly sends a RST packet according to the standard. However, at that same time on the client all of the stream has not been received (we're about half-way) and I certainly haven't sent an ACK for it. then a RST comes in tearing down the session before it's finished and truncating the download.
• The problem only happens if "Connection: close" header is used. If "Connection: keep-alive" is used, then it's the responsibility of the client to terminate the connection once it's done. In this case, no problem! However, a lot of things don't use that. A web browser generally uses keep-alive for efficiency - hence 99% of users won't encounter or know about the problem - but a lot of systems (e.g. APT, Ansible) will use "close", which is why it's such a problem for me in my work.
• Changing APN and PDP type in the router has zero impact; it doesn't matter whether I'm using IPv4, IPv6, IPv4v6, APN "3internet", "3secure", or "three.co.uk". The problem for me is general.

Ultimately, my hypothesis is that Three have some sort of connection buffering to optimise the user experience or maybe to prevent wasted re-transmissions, but there's a glaring bug in it whereby it resets the connection and discards the buffer it holds for the session once the server has closed the connection. This would make sense for an ISP based solely on a Radio Area Network because if clients exist in grey spots where the connection can go down momentarily much of the time it is helpful to buffer the lost packets for the clients rather than have the server spamming their link with retries of the unACKd packets (and further polluting the radio waves). So I think Three ACKing the packets on my behalf is by design. Only the implementation is bad and it mistakenly assumes it can throw away the buffer when the server terminates the connection.

Any help/testing/solidarity would be much appreciated because Three technical support have been zero help since I raised it with them over a month ago. I sent over detailed evidence, but all they can muster is a call occasionally to incorrectly restate the problem and ask if I'm still having it. Really awful experience; I've never seen a team so completely unable to escalate to responsible people who might actually be able to help eventually.

[TheEnglishman]

@electricworry 

I’ve also been through support and while they were great, they aren’t network engineers and aren’t testing over a 5G router.  I similarly raised a complaint but the response was the same as yours - put up with it or leave.  I’m using a ZTE MC888A and it has the latest firmware from Three.  And yes, I’ve rebooted it!  🙂

Currently,

Wordpress won’t download, as per example

Brizy won’t download from the ‘Plugins’ section of Wordpress or via the website download link

Elementor won’t download from the ‘Plugins’ section of Wordpress (I didn’t try directly from the website)

Scrypted Core plugin won’t download (I’m guessing from GitHub)

I’ve also got issues updating Pi.Hole

If I switch to Virgin Broadband everything works.

I’d do a load of wireshark captures and comparisons etc but I’d only replicate what you found and waste a day doing it.  If Three ask for such info I’ll do it but otherwise there’s no point.

It’s really sad but as you say, this isn’t going to be an issue for 99% of customers so it’s unlikely it’ll get fixed, even in the medium term so I’m going to have to end the trial at the end of the month and go elsewhere for internet.

Take care, thanks for putting in the work and have a great summer, now the sun’s shining!

[electricworry]

I called up support again to see where the support case is, but it seems that the previous case has been dropped.

I raised a complaint stating that support has been pretty much non-existent. I set my expectations thus: I need to know whether Three are willing to escalate the problem to properly understand it and decide whether they are (a) not interested in fixing the problem because it won't affect 99% of users or (b) are interest in fixing the problem and want to work to its resolution. It was agreed in scenario a, I will be released from my contract, which is fine; in the economics of running a company deciding not to fix an issue that most people won't notice because they're mostly on YouTube might be good business sense. In scenario b, they deserve some lenience to work at it as a lower priority issue.

Meanwhile I was put through to technical support to go through the process again, and my call has ostensibly been escalated again for a callback. It remains to be seen whether it gets in front of the right person or not, but I'll give it one more shot. Hopefully it will reach a team that deals with tricky issues that require some understanding of network protocols and not just someone that will tell me there are no signal problems in my area.

@PeteG, you said that you'd speak to another team internally. If that's the case, I would be happy to relay the information again so they can decide whether investigating it is of interest. Obviously I'm not looking for an urgent fix. This isn't like a car that doesn't work at all and needs urgent attention, more like a car that doesn't drive on B roads so needs resolved eventually.

If there's no more movement on it I'll most likely be ditching the contract and going back to FTTC because despite being slower, it didn't break Internet protocols that have been around for the lifetime of the Internet.

[PeteG]

Hello. 

Sorry for the lack of updates. I haven't heard anything back yet, but the issue has been highlighted, and this thread and your comments have been used to detail the issue as well.

It's worth noting that the team you speak to on the phone about your account won't be able to see what's going on, and they won't be able to provide any updates on it. There's a lot of cogs in the machine, I don't know how long it's going to take to get an update back on it, sorry. 

Pete. 

[electricworry]

Thanks Pete. That's the most promising news yet. It's not a showstopper, so this is the sort of response I was hoping for. Presumably you'll get updates? Please do keep me informed as you hear more.

[Gardinerr]

I can replicate the issue with a NR5103e router running dual-stack IPV4/IPV6 by executing the same curl commands (curl version 8.7.1 on macOS Monterey 12.7.6). So it is likely not a router issue. However, if I use the same curl commands to download files hosted on an Apache server (I used the Apache Server Project as an example), then these download correctly i.e. no truncation. I am wondering if the problem might lie with the web server end, perhaps files hosted on nginx and the way buffering is set-up?

[jr0]

good point, and that made me have a peek at nginx latest release notes.
Which URL did you use to test?

but I guess electricworry made up his mind

[electricworry]

It is not what @Gardinerr suggests. To prove it's not some Apache vs Nginx difference, I went ahead and switched my webserver (which is only up for testing anyway) from Nginx to Apache.

If you visit https://electricworry.net/ you'll clearly see it's an Apache installation. The only thing I've done to it is add an SSL certificate so that it's representative of the problem.

$ curl -H "Connection: keep-alive" https://electricworry.net/test-8 -o test-curl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8192k 100 8192k 0 0 2705k 0 0:00:03 0:00:03 --:--:-- 2706k

$ curl -H "Connection: close" https://electricworry.net/test-8 -o test-curl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
51 8192k 51 4195k 0 0 2205k 0 0:00:03 0:00:01 0:00:02 2204k
curl: (18) transfer closed with 4092646 bytes remaining to read

@Gardinerr, want to try it too?

[Gardinerr]

I tried both commands but they failed (after about a minute) with a SSL connect error:

LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to electricworry.net:443 

 I thought this might be something to do with the way the Mac resolves hostnames under IPV6. Consequently, I switched to an IPV4-only connection but got the same error.

[electricworry]

Sorry, definitely not a problem at your end. The server fell over (which is what always happens when I run Apache on low RAM server, due to massive probing from botnets).

It's running again. You've got from now until next time it falls over (could be hours or days); after that I'm switching back to Nginx.