File downloads are truncated on Three Broadband

[electricworry]

I'm looking for some verification about whether the issue I have is isolated to me, my area, or if it's a general Three-wide problem as I think it is.

I use Three 5G broadband and I'm about 50 metres away from the gNodeB so I've got excellent uninterrupted signal. It's not a Layer 1 problem I'm facing. The problem I have is that TCP connections are terminated prematurely (i.e. a RST packet is sent) before all data is received. Here's a simple test to verify if you have the problem or not.
The following command will attempt to download an 8MiB file (all NULs) from a website in AWS. It should work the same on Linux, MacOS, and modern Windows computers just the same. For me, I get the error "curl: (18) transfer closed with XXXXXX bytes remaining to read", which is the problem.

curl -H "Connection: close" https://electricworry.net/test-8 -o test-curl

If you're not comfortable connecting to my server, the following third-party download test should produce the same result (it does for me!):

curl -H "Connection: close" https://files.testfile.org/ZIPC/15MB-Corrupt-Testfile.Org.zip -o test-curl

When I tested, I collected a packet capture at both sides and I can see that my server sends the whole 8MiB file in the TLS session and then terminates the connection with a RST packet at the end (which it does because we sent a "Connection: close" header). However on my client side, only half of the file comes through before the session is impolitely terminated.

Would people on Three 5G broadband mind testing please to help confirm/deny whether this is a general problem or an individual one?

I've done a lot of testing over the past month and I've got a hypothesis.

• Comparing the server and client packet captures, the packets do not match up; the sequence and ack numbers - though they start the same - end up being very different. It appears that something in the middle is buffering the stream and ACKing the packets on my behalf.
• The problem only happens when I'm on my Three 5G Broadband service. If I take my laptop into work, the problem is gone. The problem doesn't occur when I use my Giffgaff mobile as a hotspot either.
• The problem exists on all websites (I suffer *a lot* from Ubuntu APT packages being half-downloaded and rejected on my workstation).
• Since the times on my server and client are synchronised as good as possible with NTP I can compare progress of the stream at both sides. When my server has finished transmitting (and received the final ACK) it correctly sends a RST packet according to the standard. However, at that same time on the client all of the stream has not been received (we're about half-way) and I certainly haven't sent an ACK for it. then a RST comes in tearing down the session before it's finished and truncating the download.
• The problem only happens if "Connection: close" header is used. If "Connection: keep-alive" is used, then it's the responsibility of the client to terminate the connection once it's done. In this case, no problem! However, a lot of things don't use that. A web browser generally uses keep-alive for efficiency - hence 99% of users won't encounter or know about the problem - but a lot of systems (e.g. APT, Ansible) will use "close", which is why it's such a problem for me in my work.
• Changing APN and PDP type in the router has zero impact; it doesn't matter whether I'm using IPv4, IPv6, IPv4v6, APN "3internet", "3secure", or "three.co.uk". The problem for me is general.

Ultimately, my hypothesis is that Three have some sort of connection buffering to optimise the user experience or maybe to prevent wasted re-transmissions, but there's a glaring bug in it whereby it resets the connection and discards the buffer it holds for the session once the server has closed the connection. This would make sense for an ISP based solely on a Radio Area Network because if clients exist in grey spots where the connection can go down momentarily much of the time it is helpful to buffer the lost packets for the clients rather than have the server spamming their link with retries of the unACKd packets (and further polluting the radio waves). So I think Three ACKing the packets on my behalf is by design. Only the implementation is bad and it mistakenly assumes it can throw away the buffer when the server terminates the connection.

Any help/testing/solidarity would be much appreciated because Three technical support have been zero help since I raised it with them over a month ago. I sent over detailed evidence, but all they can muster is a call occasionally to incorrectly restate the problem and ask if I'm still having it. Really awful experience; I've never seen a team so completely unable to escalate to responsible people who might actually be able to help eventually.

[jr0]

@TheEnglishman 
FYI I tried myself. I'm using a hotspot from my phone - currently 4G - and not even once it failed.

What setup do you have?

[TheEnglishman]

It only materialises through the router - if I connect to a hotspot on my phone (same 3 tower)  or a Virgin Router everything works. 

[electricworry]

Yep agreed. jr0 is trying to be helpful and probably has a hard time believing that we're not doing something dumb, but this is definitely 100% a problem; I've been doing network engineering for too long to make a dumb mistake on this sort of problem (I hope! 😀).

I don't have a Three mobile phone to test on, all I know is that with the mobile broadband service served over a ZTE router this issue is 100% repeatable, and now we've got two people reporting the same issue.

[jr0]

I guess you narrowed it down to the device/router 

What router brand and model? Was it supplied by Three 

[electricworry]

Hiya. Yes it was supplied by Three. It's a ZTE Three 5G Hub MC801A, specifically MC801AHW-1.0.0 on software version BD_UKH3GMC801AV1.0.0B15.

[TheEnglishman]

Yes!

I’m having a nightmare trying to get Wordpress and various plugins installed.  Sometime (rarely) it works then most of the time it doesnt

curl -o wp.zip https://downloads.wordpress.org/release/en_GB/wordpress-6.7.2.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
93 33.7M 93 31.7M 0 0 8121k 0 0:00:04 0:00:04 --:--:-- 8122k
curl: (18) transfer closed with 2129919 bytes remaining to read

Infuriating and I’ve no idea where to go with this

[electricworry]

I responded to this earlier but my post seems to have disappeared. In summary what I said was...

Thanks for testing. You've got the same issue as me. The test case you supplied is interesting in that it doesn't matter what request header you send, the server chooses to use "Connection: close" as the mode and so it happens (close to) 100% for that URL. If we look at a packet capture (where we know the TLS keys) here's what happens (see bold text):

GET /release/en_GB/wordpress-6.7.2.zip HTTP/1.1
Host: downloads.wordpress.org
User-Agent: curl/8.5.0
Accept: */*
Connection: keep-alive

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 11 Mar 2025 17:44:25 GMT
Content-Type: application/zip
Content-Length: 35434101
Connection: close
Cache-control: private
Content-Disposition: attachment; filename=wordpress-6.7.2-en_GB.zip
Last-Modified: Tue, 11 Feb 2025 17:06:24 GMT
Accept-Ranges: bytes
Content-MD5: 09dce658065deafab53de05acc1f0fd8
Alt-Svc: h3=":443"; ma=86400
X-nc: MISS ord 4

So that's a great test case as there's no workaround!

[electricworry]

Sorry, I was away for a long weekend.

Yes, you've got the same problem as me. In the case of your download, the server you're communicating with doesn't care about the "Connection" header. I've tested both ways; the response always contains "Connection: close", so you're always going to have the problem regardless of what headers you put in the request. I've tested and here's the request/response (see the bits I've made bold):

GET /release/en_GB/wordpress-6.7.2.zip HTTP/1.1
Host: downloads.wordpress.org
User-Agent: curl/8.5.0
Accept: */*
Connection: keep-alive

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 11 Mar 2025 17:44:25 GMT
Content-Type: application/zip
Content-Length: 35434101
Connection: close
Cache-control: private
Content-Disposition: attachment; filename=wordpress-6.7.2-en_GB.zip
Last-Modified: Tue, 11 Feb 2025 17:06:24 GMT
Accept-Ranges: bytes
Content-MD5: 09dce658065deafab53de05acc1f0fd8
Alt-Svc: h3=":443"; ma=86400
X-nc: MISS ord 4

So that's actually a great test case, thank you, as it happens all the time and there's no workaround. (i.e. you can't have someone saying "well don't use 'Connection: close' then!"

Thank you for getting in touch and confirming that it's not just me that has the issue!

What make of router were you supplied by Three? Is it the ZTE one? Because it's always possible that it's that that's throwing away the connection once the remote side sends the RST packet. It's a firewall so the idea that it's doing buffering of the connection and has a crap implementation is completely feasible. It's ZTE equipment after all.

[jr0]

what is the result when you don't send that header? like
curl -O https://www.abc.com/abc 
or
curl -o myfile https://www.abc.com/abc

[electricworry]

Sorry, I've been away for a long weekend.

In the case you propose, if the header "Connection: close" isn't sent then it's up to the server whether to use "Connection: close" or "Connection: keep-alive". In the case of the server I'm testing with, the absence of that request header defaults to "Connection: keep-alive", in which case it works.

I touched upon that in my original post:

---

The problem only happens if "Connection: close" header is used. If "Connection: keep-alive" is used, then it's the responsibility of the client to terminate the connection once it's done. In this case, no problem! However, a lot of things don't use that. A web browser generally uses keep-alive for efficiency - hence 99% of users won't encounter or know about the problem - but a lot of systems (e.g. APT, Ansible) will use "close", which is why it's such a problem for me in my work.

---

So, just to be extremely accurate about the problem:

# The following request doesn't work
curl -H "Connection: close" https://electricworry.net/test-8 -o test-curl
# The following request does work.
curl https://electricworry.net/test-8 -o test-curl

I've 100% confirmed this. I've got a packet capture running at both sides, and I can read the contents of the TLS connection (by using libcurl's SSLKEYLOGFILE variable to get the keys used). I think that's pretty compelling evidence that rules out all sorts of things.

• Three and all routers in-between the client and the server have no knowledge about the contents of the stream. That includes my 5G ZTE router; it can't see inside the TLS connection.
• When "Connection: close" is used, the only change to the contents of the TLS stream is what comes back from the server in the response headers. The file requested is sent completely by the server in both cases.
• BUT when "Connection: close" is used, the server automatically closes the TCP connection at the end once it's received an ACK (from the mystery box in the middle) and my connection is rudely torn down early. On the other hand, when that header is not used, and my server defaults to using "Connection: keep-alive", the server politely waits for the client to close the connection (which only happens when the full file has been received.