File downloads are truncated on Three Broadband

[electricworry]

I'm looking for some verification about whether the issue I have is isolated to me, my area, or if it's a general Three-wide problem as I think it is.

I use Three 5G broadband and I'm about 50 metres away from the gNodeB so I've got excellent uninterrupted signal. It's not a Layer 1 problem I'm facing. The problem I have is that TCP connections are terminated prematurely (i.e. a RST packet is sent) before all data is received. Here's a simple test to verify if you have the problem or not.
The following command will attempt to download an 8MiB file (all NULs) from a website in AWS. It should work the same on Linux, MacOS, and modern Windows computers just the same. For me, I get the error "curl: (18) transfer closed with XXXXXX bytes remaining to read", which is the problem.

curl -H "Connection: close" https://electricworry.net/test-8 -o test-curl

If you're not comfortable connecting to my server, the following third-party download test should produce the same result (it does for me!):

curl -H "Connection: close" https://files.testfile.org/ZIPC/15MB-Corrupt-Testfile.Org.zip -o test-curl

When I tested, I collected a packet capture at both sides and I can see that my server sends the whole 8MiB file in the TLS session and then terminates the connection with a RST packet at the end (which it does because we sent a "Connection: close" header). However on my client side, only half of the file comes through before the session is impolitely terminated.

Would people on Three 5G broadband mind testing please to help confirm/deny whether this is a general problem or an individual one?

I've done a lot of testing over the past month and I've got a hypothesis.

• Comparing the server and client packet captures, the packets do not match up; the sequence and ack numbers - though they start the same - end up being very different. It appears that something in the middle is buffering the stream and ACKing the packets on my behalf.
• The problem only happens when I'm on my Three 5G Broadband service. If I take my laptop into work, the problem is gone. The problem doesn't occur when I use my Giffgaff mobile as a hotspot either.
• The problem exists on all websites (I suffer *a lot* from Ubuntu APT packages being half-downloaded and rejected on my workstation).
• Since the times on my server and client are synchronised as good as possible with NTP I can compare progress of the stream at both sides. When my server has finished transmitting (and received the final ACK) it correctly sends a RST packet according to the standard. However, at that same time on the client all of the stream has not been received (we're about half-way) and I certainly haven't sent an ACK for it. then a RST comes in tearing down the session before it's finished and truncating the download.
• The problem only happens if "Connection: close" header is used. If "Connection: keep-alive" is used, then it's the responsibility of the client to terminate the connection once it's done. In this case, no problem! However, a lot of things don't use that. A web browser generally uses keep-alive for efficiency - hence 99% of users won't encounter or know about the problem - but a lot of systems (e.g. APT, Ansible) will use "close", which is why it's such a problem for me in my work.
• Changing APN and PDP type in the router has zero impact; it doesn't matter whether I'm using IPv4, IPv6, IPv4v6, APN "3internet", "3secure", or "three.co.uk". The problem for me is general.

Ultimately, my hypothesis is that Three have some sort of connection buffering to optimise the user experience or maybe to prevent wasted re-transmissions, but there's a glaring bug in it whereby it resets the connection and discards the buffer it holds for the session once the server has closed the connection. This would make sense for an ISP based solely on a Radio Area Network because if clients exist in grey spots where the connection can go down momentarily much of the time it is helpful to buffer the lost packets for the clients rather than have the server spamming their link with retries of the unACKd packets (and further polluting the radio waves). So I think Three ACKing the packets on my behalf is by design. Only the implementation is bad and it mistakenly assumes it can throw away the buffer when the server terminates the connection.

Any help/testing/solidarity would be much appreciated because Three technical support have been zero help since I raised it with them over a month ago. I sent over detailed evidence, but all they can muster is a call occasionally to incorrectly restate the problem and ask if I'm still having it. Really awful experience; I've never seen a team so completely unable to escalate to responsible people who might actually be able to help eventually.

[penguin30]

Just want to add my voice here that I also have this problem. I had it with the ZTE MC801A and I now have the same issue with the Greenpacket outdoor hub. It affects me on both my Linux computers and on Android (it affects some of the appstore/package managers I use - it takes many attempts per package to update).

This problem deserves more attention than Three are giving it - the "connection: close" header is set by urllib in Python, which underpins a huge amount of software. There are various mentions across the internet of this problem, and quite often it turns out that the user is on a 5G connection. I'd put money in them being Three customers.

I'm tech savvy - after finding my own way to the same conclusion as @electricworry I edited my local Python installation to remove that header, but that isn't a reliable solution for me, and not everyone is able to do it. I'm mostly happy with my Three broadband, but I can't recommend it to anyone else while this is an issue.

[electricworry]

Welcome. It's nice to hear from others that are coming to the same conclusion.

I think you're right that there will be a lot more people than realised suffering from this problem but unable to get any progress because of a lack of technical knowledge to investigate. A lot of people will hit the problem, seek help on the forum of whatever product they're trying to use, and eventually they will realise the only variable different about their setup compared to everyone without the issue is 5G: to which they may assume there's something magically unreliable about 5G as opposed to Three having a specific issue on their service.

It would be good if you encounter more people in that boat to send them here. I don't really spend any time on forums and just on the web to (try and) get work done.

[Justmakeitworkk]

I'm still experiencing this issue with Unity despite the numerous testing provided by OP. What more should one need to start fixing the issue? It's been more than 3 months already.

[electricworry]

I think @RRayner is correct. Waiting on a fix from Three is futile at this point and you'd be best either using a VPN (yuck) or changing ISP. I had thought that the thing stopping a fix was no-one had yet presented a good batch of evidence about the problem to Three. However, I did that months ago and despite an interest from @PeteG who tried to escalate it (but can say nothing about said escalation) nothing has come of it. No requests for further tests, no acceptance and timeline to fix.

I think that with the Three/Vodafone merger ongoing this may never get resolved. If the 5GC/EPC cores of the two networks are getting merged in the future (I'm guessing that this issue is a core problem) then why bother, right? If Three users are eventually going to be moved over to the Vodafone core and that doesn't have the problem, why fix the Three one?

Anyway that's just conjecture. I've grown to accept that the Three service I subscribe to is just broken and I workaround it as best I can. I hope that FTTP is available where I live in the next couple of years and I can move on.

[RRayner]

It's a pain for sure, the only work-around I've found is using a vpn to do unity installs. It seems to bypass the issue somehow, so its definitely on Three's end.

[RRayner]

Brill, we've just joined Three with 5G broadband and wfh. It's our only internet option, literally. And this issue is causing the Unity hub to get stuck at 99%. Absolutely infuriating. I can manually download, but it's annoying as hell installing all the packages individually. The fact this has been a known issue since 2020 is appalling, sort it out Three.

[electricworry]

At least you're part of a growing number of people in this thread talking about it. I've provided lots of technical evidence about the problem but so far I've not had any confirmation that the team has seen/reproduced this issue which is pretty rotten.

Out of interest, where did you find out that it's a known issue since 2020?

[RRayner]

For some reason I swear I read a post on the Unity forum about it, but must've misread the year! This thread discussing it is from Apr 2023. So the issue has been around for a least 2 years.

https://discussions.unity.com/t/unity-editor-download-stuck-at-99/914753

[electricworry]

Ah, thanks. That's interesting. I agree, pretty bad.