This is my first real post using WordPress vs. my legacy blog. We’ll see how it turns out.
When I moved to Seattle in early 2014 I realized I had two non-wireless Internet options at my townhome in the High Point neighborhood of West Seattle: CenturyLink DSL or Comcast DOCSIS. I chose a medium-tier plan with Comcast Xfinity that included TV (my wife would ultimately end up paying for this because I don’t watch broadcast TV) and did the self install. I remember checking CenturyLink’s options but I recall the max plan being something on the order of 5 Mbps downstream and a measly upstream so I didn’t really give them much consideration. Plus, I liked the < 10 ms RTT to the CMTS that DOCSIS typically provides vs. 20-25 ms RTT that I’ve seen on most DSL connections. The Xfinity connection was solid and I had no complaints. However, I wanted to get public IPs for at least two devices (CPEs) and this required a business class connection through Comcast Business. I added the 50/10 service from Comcast Business in April of 2014 and they assigned me a static IPv4 /29 for use. I kept the Xfinity service and terminated it in a Linux container. I had to lease a modem because I think it ran some dynamic routing (RIP?) with the CMTS. The modem I got was the NETGEAR CG3000DCR, which had some software issues for awhile but those got sorted within a few months.
The Comcast Business service worked great. Latency and variability (for the most part) was low and I had ~10 ms reachability to things in Seattle (work VPN, Vultr VPS hosting, etc.). My SmokePing graphs don’t go back too far but it was something like this the whole time:
BTW, the SmokePing graphs I have here are either to the first hop routers or to a nearby anycasted IP like Google’s 8.8.8.8 or CloudFlare’s 1.1.1.1. I don’t only ping routers because I am aware of CoPPs and how they may affect ICMP responses but if the latency to the first hop router vs. a nearby anycasted address is about the same, I choose the first hop router.
It was pretty expensive, though. I paid $150/mo for it but the technical support (I only needed it once to get them to add some DNS PTRs, which I had to read over the phone, ugh) was worlds better than Xfinity. I ended up switching to a subsidized plan through my employer for $120/mo and upgraded to 75/15 in late 2018.
So, yeah, I was a DOCSIS fan. I didn’t care that the media was shared and that my neighbors and I fought over the same the RF spectrum. It worked great. I mostly (there were a few months when they had some hub problems) had a good experience with DOCSIS provided by Time Warner Cable in Charlotte, NC for 9 years, as well.
The COVID-19 work from home guidance from my employer was announced on March 4, 2020. I actually worked from home that day for other reasons and took PTO for the rest of the week (it my wife and my 5 year wedding anniversary, which, after some canceled plans.. spent at home). My first day of mandated WFH was March 9, 2020. The Comcast Business connection worked great and my daily video conferences didn’t hiccup at all. However, at noon on March 20, 2020 I noticed a huge increase of latency and upstream (only!) packet loss:
What the heck? I figured it was a temporary blip and a node or amp had failed so I let it go 24 hours. However, it persisted. I saw it on my Xfinity connection as well and was able to (with a client-side VPN) reproduce it from my /neighbors/ connection using xfinitywifi. So, it wasn’t just my unit. TV was still fine and I attributed that to the fact that the loss and latency was only in the upstream direction. This, of course, was maddening because on video conferences I could see and hear everyone but they couldn’t see or hear me clearly! The latency and loss subsided overnight and was only apparent between around 0900 and 2330 PDT. It felt like either something had gone bad and was falling over due to load. Here’s a shot showing an example 0000 to 2359:
So, I called Comcast Business on Saturday evening and the CSR who I talked to verified my modem levels were fine but there was definitely packet loss that he could reproduce. He had me reboot my modem but that didn’t help (I knew it wouldn’t). He couldn’t do anything more for me so he scheduled a tech. to come out to my home on that coming Wednesday (March 24, 2020). I had him schedule it a few days out just in case if it cleared up I could cancel the appointment.
It didn’t clear up. The tech. came out and due to COVID-19 he couldn’t enter my home. However, he said he checked my modem levels and they were OK and he also replaced an old splitter on the side of my townhome just to make sure I was getting the best levels possible. He couldn’t do much else for me but did say that a ticket had been filed the prior day (Tuesday, the 23rd) for packet loss and latency and it was assigned to maintenance so a line tech. would be working on it. I asked him if he knew if this was definitely due to some bad hardware or if this was indeed due to network congestion. He said he couldn’t answer directly because they were being told to not use certain terms but he strongly led me to believe it was something wrong that needed to be fixed. I was reassured by this because the hardest part was over – convincing Comcast that there’s a problem and it’s not due to something silly in your house or your “unsupported operating system” (ugh, I’ve been through that garbage). Comcast acknowledged there was a problem and they were going to fix it! Woo!
Well, I’ll cut to the chase. It never got fixed.
I called in half a dozen times over the next two and a half weeks to get a status on the maintenance ticket, which each CSR said usually is resolved within 72 hours. Each time when I gave the maintenance ticket number to the CSR they had zero visibility into it and said there were no notes in there.
Even though every interaction with our neighborhood’s NextDoor site has been wholly negative, I decided to crowdsource some information and posted some details about my Comcast issues. Over the next day or two I got some replies from folks at the far end of High Point indicating they had no issue but I did get a few replies from folks closer to my block indicating they had the same issues and those issues lined up with the timeline I listed. Well, at least I had more evidence it was not just me or my block. I also fired off a tweet on March 22nd and kept it updated.
Governor Jay Inslee’s official stay at home order went into effect on March 26, 2020. The packet loss didn’t get any worse, surprisingly.
When the issues first started I checked CenturyLink’s site and found that I could now get 140/20 Mbps VDSL2 service to my unit. The block next to mine got built up with Polygon NW townhomes since 2014 and I believe a new DSLAM was installed on the property. I kept that information in my back pocket because I was still scared of a 20 ms latency hit (or more!?) to the DSLAM. I have a clear view of downtown Seattle from my balcony and didn’t feel it was right to pay a > 20 ms RTT just to SSH to a box in the Westin Building! Anyway, I put that aside and ordered the 140/20 Mbps CenturyLink DSL service on March 27th. The installation was scheduled for April 7th. At this point I mentally decided to nuke Comcast Business regardless if they resolved the issue or not. I didn’t need 3x Internet connections.
Anyway, I was going to still see if I could get the Comcast issue sorted because I intended to keep my Xfinity service as a backup (or promote it back to primary if the DSL service sucked). So, I kept calling Comcast. The CSR on March 30th gave me the option to escalate so she submitted an escalation request and said a “supervisor” would call me back that day or the next. To cut to the end for a second – I never got a call back regarding it.
When I called in yet again on April 6, 2020 before speaking to a CSR I got a notification that there was going to be maintenance on April 7 between 0000 and 0600 local time and it would interrupt my Internet connectivity. Woo! They were going to fix my problems! I asked the CSR if this was related the maintenance ticket but the CSR didn’t know and said they thought it might be. It was becoming clear that either Comcast couldn’t give me any information about the problem and even could not tell me if the problem was a legitimate hardware failure or related to congestion, so I just waited for the next morning.
When I checked on things in the morning the loss and latency still were present and I never saw a drop in Internet service overnight. Either nothing had been done or the maintenance was completely unrelated to my problems.
On April 7, 2020 I got the CenturyLink DSL installed. It works great. I get around 6.6ms RTT to the CenturyLink BRAS in Tukwila (even through their crappy NAT box!). I switched IPv4 Internet over to it but we’ll talk more about that in a little bit.
So, I called Comcast Business again on the 8th. I provided the maintenance ticket again for the $LOSTCOUNT time and the CSR indicated that the maintenance ticket was closed on April 1st at 0407. What the heck? The CSRs just strung me along for the ride for these weeks because apparently the maintenance ticket that was opened was not related to the work that was done on April 7th and had nothing to do with my problems. I explained my situation and asked to cancel the service. I was given a national account e-mail address to use to request service termination because it was a subsidized account. I sent the e-mail that day. On April 10th my account was terminated (abruptly, I might add) and I returned the modem that same day. One good thing here is that I explained the situation in the cancellation e-mail and I was told that my account would be credited back to March 20th. This is a good gesture. That being said, I haven’t received the last invoice for the account at the time of writing.
I restructured part of my network and put the DSL connection into a Linux container (LXC) as well, just like the Xfinity connection. This gave me more flexibility to do fancy routing things. I advertised a default route into OSPF from it for IPv4 traffic (double-NAT) and for IPv6 traffic used a combination of BGP LOCAL_PREF and AS_PATH prepending (read more about my network here – IPv6 is all tunneled because I have an ARIN PI block and my own ASN) to make outbound IPv6 traffic go through the DSL connection and inbound still take the Xfinity connection. I did this because the Comcast loss was only in the upstream direction and my Xfinity plan provided 300 Mbps downstream. So, for IPv4 I had 140/20 Mbps but for IPv6 I essentially had 300/20 Mbps. Due to the tunnel overhead and differing latency TCP has to deal with realistically this gets me around 175-190 Mbps downstream.
Anyway, the ~6.6 ms best case RTT to the BRAS (I assume that is the first L3 hop and it’s located south of me in Tukwila, WA) is really good for DSL. The VDSL2 configuration on the ZyXel C3000Z modem indicated the loop length was between 1110 and 1120 ft. Here’s an MTR to Google:
Here’s a SmokePing graph to the BRAS:
There’s a little spike in the evenings but it’s short-lived so I’m not too worried.
I put the Linux container’s IP in the “DMZ” on the ZyXel modem. I’m not sure if it’s still doing port translation or only if needed but I don’t really care. I’d like to terminate the PPPoE connection in the Linux container and turn on transparent mode but I haven’t been able to get the PPPoE password yet and the prior instructions I found on a web search no longer work on the C3000Z because the “sh” command in the Telnet/SSH remote management now requires a password. Supposedly I can just chat with a CenturyLink representative and they’ll give it to me. It’s on my list of things to do. Right now, I make sure my tunnels are set to an effective MTU of 1492 to account for the PPPoE overhead. The C3000Z has some TCP MSS clamping to 1452 I believe.
I’d still love to get the Comcast issues sorted. The NextDoor thread progressed to a point where some neighbors called Xfinity and got a service credit for their trouble because they were told that they don’t have personnel to fix the problem. I don’t know if this was just a generic “go away” response or if this is the case or if they just don’t know the cause of the problems. I still doubt it’s congestion from the stay-at-home traffic because it started so abruptly at noon on the 20th. It should have gotten worse day by day. One neighbor ended up getting a CenturyLink installation appointment scheduled and presumably will be switching away from Xfinity. Others are apparently too far away from the DSLAM and have been quoted 1.5 Mbps (yes, 1 point 5 megabits per second) service so that’s a non-starter.
There’s CenturyLink GPON / FTTH in other parts of West Seattle but not in High Point, where all utilities are subterranian and I doubt they’ll want to (or even be given the OK from the various owners of the shared property in the neighborhood) dig up conduit to install fiber. Maybe if one day the NIMBYs in Seattle will allow for it we’ll get some 5G mmWave deployments and some real competition for high-speed residential Internet access.