After spending 7 years, 4 months, and 1 day in the Emerald City, my wife and I moved back to the east coast to a neighborhood (actually, a census designated place or CDP) named Brambleton in the Commonwealth of Virginia, located within the area that’s considered Northern Virginia (or NoVA) during the summer of 2021. Seattle had its ups and downs for us and I will sincerely miss the natural beautify of the Pacific Northwest, the WSDOT ferry rides to Bainbridge Island, and the quirky little eateries (and distilleries!) in the area. I won’t miss the broken politics, misguided legislation, and broken bridges. We purchased a new construction townhouse within walking distance to the Brambleton Town Center and sold our home in the High Point neighborhood of West Seattle (ironically, we used the same great realtor to buy it in 2014 and sell it this year).
We just moved in!
Our new home has Verizon FiOS (now stylized as Fios, for some reason) and I opted to upgrade to the highest speed plan, which, according to Verizon, is 940 Mbps downstream and 880 Mbps upstream. This is a welcome change from the troubles I had with Internet connectivity while in Seattle. The only downside is that there’s no IPv6 and there’s also no alternative due to a multi-year HoA agreement with Verizon. My cellular backup is still through Verizon Wireless, so I probably need to change that one of these days. Starlink is always an option, I suppose, but at $100/mo it’s not really a cost-effective backup. Latency on the FiOS connection is fairly low but the GPON network provides more jitter than VDSL2 or even DOCSIS, which is surprising:
SmokePing Plot to 8.8.8.8
It would be nice if Verizon would finally fix their equipment so it doesn’t keep breaking ICMP traceroutes, though.
tl;dr Yes, SIP INVITEs are blocked via what seems to be a packet size filter but some things are inconsistent.
Back in early 2020 I switched from Comcast Business to CenturyLink DSL for my main Internet connection. I still have Xfinity since my wife uses it for TV and her package comes with a 300 Mbps plan (that use for VPNs and some fun downstream-only load-balancing.. but that’s another topic). Anyway, when I flipped Internet IPv4 access over to the DSL connection I noticed that my outbound SIP connections to Vitelity (via my Asterisk server) were failing and timing out. SIP REGISTERs were fine so I could still receive incoming calls. I did a 30 second debug with tcpdump, saw packets going out but nothing coming back in. PPPoE adds an 8 byte overhead and the packets going out were 1492 bytes so I figured they would fit and that CenturyLink was just blocking SIP because they are “The Phone Company” and probably wanted me to buy a land line or something. I internally advertised Vitelity’s block from the LXC instance attached to my Xfinity connection and called it a day.
I was chatting with folks at work this past week regarding this and it prompted me to dig a little deeper. What I observed in early 2020 didn’t paint the whole picture. The SIP INVITEs are being sent to outbound.vitelity.net, which has a few A RRs all out of the 64.2.142.0/24. The SIP packets themselves are huge, and even fragment on my local network since the total length of the first packet (including IP headers) is 1634 bytes. So, this means the SIP packets are being fragmented going out my Xfinity connection and still working fine:
(remus:19:27:PST)% sudo tcpdump -i enp3s0f1 -qvn net 64.2.142.0/24 tcpdump: listening on enp3s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes 19:27:21.045354 IP (tos 0x0, ttl 62, id 34953, offset 0, flags [+], proto UDP (17), length 1500) 98.247.161.61.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1472 19:27:21.045360 IP (tos 0x0, ttl 62, id 34953, offset 1480, flags [none], proto UDP (17), length 180) 98.247.161.61 > 64.2.142.188: ip-proto-17 19:27:21.228064 IP (tos 0x20, ttl 48, id 51882, offset 0, flags [none], proto UDP (17), length 567) 64.2.142.188.5060 > 98.247.161.61.5060: UDP, length 539 19:27:21.228368 IP (tos 0x0, ttl 62, id 34986, offset 0, flags [none], proto UDP (17), length 452) 98.247.161.61.5060 > 64.2.142.188.5060: UDP, length 424 19:27:21.228593 IP (tos 0x0, ttl 62, id 34987, offset 0, flags [+], proto UDP (17), length 1500) 98.247.161.61.5060 > 64.2.142.188.5060: UDP, bad length 1824 > 1472 19:27:21.228598 IP (tos 0x0, ttl 62, id 34987, offset 1480, flags [none], proto UDP (17), length 372) 98.247.161.61 > 64.2.142.188: ip-proto-17 19:27:21.311301 IP (tos 0x20, ttl 48, id 51883, offset 0, flags [none], proto UDP (17), length 492) 64.2.142.188.5060 > 98.247.161.61.5060: UDP, length 464 19:27:23.510342 IP (tos 0x20, ttl 48, id 51885, offset 0, flags [none], proto UDP (17), length 882) 64.2.142.188.5060 > 98.247.161.61.5060: UDP, length 854
So, fragmentation and packet size may not be the problem. SIP packets halfway through the outgoing call setup are even larger, past 1800 bytes! Well, fragmentation usually stinks but it seems to work fine here.
First, why are these SIP packets so large? It looks like it’s because of the list of advertised available codecs. Here’s a full decode of the first message (my authentication data is temporarily scrambled, FWIW):
I could probably dust off my Asterisk configuration and figure out how to reduce the number of useless codecs that are advertised but since this works via my Xfinity connection I’m not going to focus on that.
Here’s what it looks like out my DSL connection:
(discovery:19:42:PST)% sudo tcpdump -i ppp0 -qvn net 64.2.142.0/24 tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 19:43:13.716582 IP (tos 0x0, ttl 61, id 27388, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1464 19:43:13.716624 IP (tos 0x0, ttl 61, id 27388, offset 1472, flags [none], proto UDP (17), length 188) 97.113.163.82 > 64.2.142.188: ip-proto-17 19:43:14.216162 IP (tos 0x0, ttl 61, id 27435, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1464 19:43:14.216200 IP (tos 0x0, ttl 61, id 27435, offset 1472, flags [none], proto UDP (17), length 188) 97.113.163.82 > 64.2.142.188: ip-proto-17 19:43:15.217157 IP (tos 0x0, ttl 61, id 27666, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1464 19:43:15.217197 IP (tos 0x0, ttl 61, id 27666, offset 1472, flags [none], proto UDP (17), length 188) 97.113.163.82 > 64.2.142.188: ip-proto-17
And, just as a sanity check, I see it out the physical Ethernet interface that the PPP daemon is using, too:
(discovery:19:43:PST)% sudo tcpdump -i eth1 -qvn tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 19:43:13.716605 PPPoE [ses 0x9e8a] IP (tos 0x0, ttl 61, id 27388, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1464 19:43:13.716661 PPPoE [ses 0x9e8a] IP (tos 0x0, ttl 61, id 27388, offset 1472, flags [none], proto UDP (17), length 188) 97.113.163.82 > 64.2.142.188: ip-proto-17 19:43:14.216182 PPPoE [ses 0x9e8a] IP (tos 0x0, ttl 61, id 27435, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1464 19:43:14.216208 PPPoE [ses 0x9e8a] IP (tos 0x0, ttl 61, id 27435, offset 1472, flags [none], proto UDP (17), length 188) 97.113.163.82 > 64.2.142.188: ip-proto-17 19:43:15.217178 PPPoE [ses 0x9e8a] IP (tos 0x0, ttl 61, id 27666, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 64.2.142.188.5060: UDP, bad length 1632 > 1464 19:43:15.217204 PPPoE [ses 0x9e8a] IP (tos 0x0, ttl 61, id 27666, offset 1472, flags [none], proto UDP (17), length 188) 97.113.163.82 > 64.2.142.188: ip-proto-17
If you look at the packet lengths, the Linux LXC instance attached to my DSL line is re-fragmenting the packet that’s originally fragmented from my Asterisk server to something that will fit into 1492 MTU. It’s 1492+188 instead of 1500+180, now. Yes, fragmentation is working as expected even though it’s kinda horrid!
So, I was wondering how CenturyLink might be blocking the SIP traffic? Just blocking UDP/5060 traffic might be an easy choice, so I tested this using NetCat to a VM of mine, remembering to set both the source and destination ports to 5060 (discovery is the DSL LXC instance and dax is the remote VM):
(dax:22:51:EST)% sudo tcpdump -qvi vtnet0 -n port 5060 tcpdump: listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:51:26.612983 IP (tos 0x0, ttl 57, id 30455, offset 0, flags [DF], proto UDP (17), length 32) 97.113.163.82.5060 > 45.32.4.66.5060: UDP, length 4
Okie doke. This is not the way they’re blocking it. How about large UDP packet sizes like the kind we saw with SIP and some fragmentation? For this we’ll use hping3:
--- 45.32.4.66 hping statistic --- 2 packets transmitted, 0 packets received, 100% packet loss round-trip min/avg/max = 0.0/0.0/0.0 ms
hping3 will automatically fragment if you specify a data size that results in the generated packet exceeding the interface MTU. 1,600 bytes does this for us.
But, I’m only seeing the fragments on my VM:
(dax:23:01:EST)% sudo tcpdump -qvi vtnet0 -n host 97.113.163.82 tcpdump: listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes 23:01:39.219056 IP (tos 0x0, ttl 57, id 25, offset 1472, flags [none], proto UDP (17), length 156) 97.113.163.82 > 45.32.4.66: ip-proto-17 23:01:40.219297 IP (tos 0x0, ttl 57, id 25, offset 1472, flags [none], proto UDP (17), length 156) 97.113.163.82 > 45.32.4.66: ip-proto-17
Verifying that my LXC instance is actually sending the packets correctly, I tcpdump ppp0:
(discovery:20:01:PST)% sudo tcpdump -qv -i ppp0 -n host 45.32.4.66 tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 20:01:39.051291 IP (tos 0x0, ttl 64, id 25, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5060 > 45.32.4.66.5060: UDP, bad length 1600 > 1464 20:01:39.051362 IP (tos 0x0, ttl 64, id 25, offset 1472, flags [none], proto UDP (17), length 156) 97.113.163.82 > 45.32.4.66: ip-proto-17 20:01:40.051589 IP (tos 0x0, ttl 64, id 25, offset 0, flags [+], proto UDP (17), length 1492) 97.113.163.82.5061 > 45.32.4.66.5060: UDP, bad length 1600 > 1464 20:01:40.051620 IP (tos 0x0, ttl 64, id 25, offset 1472, flags [none], proto UDP (17), length 156) 97.113.163.82 > 45.32.4.66: ip-proto-17
Yep, this is fine. So, looks like CenturyLink is blocking huge packets. Just to round this out, I try without a fragmented packet:
Well, of course, I need to know, where are they blocking these packets and how large of a packet can get through? I started with one byte less than the full payload:
That’s the packet. So, now that that’s answered, where are they blocking it? We can use MTR for this. Here’s the same full UDP packet with both source and destination port of 5060:
My VM filters UDP/5060 but we do see it in tcpdump and the traceroute does get through CenturyLink’s network. Looks like it’s the BRAS itself or something between that and me. But wait, as I was messing around I up-arrowed and I’m seeing 1492 being passed now too:
I went through all the other tests again and made sure they were consistent and they seemed to be. However, the traceroute probes seemed to introduce some weird behavior. The only thing that’s different between successive runs of MTR is the IP identification (ID) fields. Linux uses a common counter for UDP so if MTR is the only application using UDP the IDs will increment sequentially. Any other UDP application running in parallel will cause these IDs to start skipping numbers. hping3 allows the use of a random IP ID per packet, so CenturyLink maybe allowing special IP IDs seems to be a dead end since the random IP ID is the default for the tests I ran above.
So, I’m pretty confused. The only thing that results in inconsistent behavior is traceroute, which uses a TTL of 1 and then incrementing the TTLs until it reaches the destination. Maybe whatever filter CenturyLink is using does something special with low TTLs? I don’t know.
Regardless, even though we see inconsistent behavior it’s pretty clear that my CenturyLink DSL connection is filtering SIP INVITEs and that’s not good.
I am indeed using a CenturyLink-branded Zyxel C3000Z modem in bridge mode so it’s possible there’s some packet filter embedded into the firmware even though the PPP connection is terminated on my own equipment. Or, it could be filtered at the BRAS or DSLAM.
I haven’t really seen much on the Internet to confirm my findings (maybe nobody cares about SIP on a DSL connection?) and I don’t know if this also happens on CenturyLink’s FTTH/GPON offerings. The same DSL modem I am using is also used for the GPON service as well and also requires PPPoE, so maybe it’ll act the same.
Do any readers have the same problem? Or, different symptoms?
I decided to try Signal today. The security and privacy features are probably its best selling point as well as it being FOSS. However, it’s got some rough edges that you should probably be aware of before deleting all your other messaging accounts.
First, here’s a little background. I’m kinda old so I’ve used a fair amount of computer mediated communication systems over the years. I started with instant messaging on AOL in the early 1990s and continued using the standalone service (AIM) for awhile as well. Toward the late 1990s I had fun with ICQ. In the 2000s I picked up IRC and Lily, which is a chat service created by some RPI developers. I use IRC and Lily primarily in a terminal, which fits my usage patterns for other things. I ran my own XMPP (jabberd) server for awhile and was able to talk to folks who used Google Talk until Google decided to kill their XMPP S2S interface and re-invent the product as Google Hangouts. I also picked up Facebook Messenger as well because a few of my friends use it. I tried out Telegram in early 2020 but didn’t end up using it for anything. SMS is also there for correspondence with one or two folks as well as all those insecure 2FA things and a whole lotta spam.
For work, I’ve used Microsoft OCS/Lync at two jobs as well as Amazon Chime and Slack.
I’ve never used WhatsApp or SnapChat.
Here’s my current state of my non-work messaging:
IRC – Mostly idling and periodic chats
Lily – Mostly idling and periodic chats
Facebook Messenger – Daily chats
Google Hangouts – Daily chats
SMS – Daily spam, some chat, and horrible 2FA
Google has made it clear that they are going to kill Hangouts or at least change it into something I won’t like, so I’ve decided I will use Hangouts until it no longer works. I won’t be using any future chat products by Google.
With the pending (although it now seems delayed?) WhatsApp ToS change and the increasing popularity of Signal, I figured I’d give it a go.
Account creation was easy but used my phone number as my identity. This rubbed me the wrong way because I would like to think phone numbers are ephemeral. It’s not supported to change phone numbers, currently. I started with the Android app, created my PIN (4 digits or optional alphanumeric), and was off to the races.
The app found about a dozen or two contacts from my address book that are already using signal, including some good friends and family members. This was encouraging.
I was away from my Linux workstation and using my MacBook so I installed the macOS app and activated it using a QR code read by my phone. Pretty easy so far. I then later went to install the Linux app on Debian, and ran into some issues.
While Signal’s site indicates that it provides binaries for 64-bit Debian-based distributions I had to add an APT source that referenced the xenial distribution, indicating it was Ubuntu-centric. Broadcasting support for Debian-bases distributions and then being centered around Ubuntu is a pet peeve of mine. Anyway, I use Debian testing, which is a rolling snapshot of the next stable distribution (bullseye, at the moment) and I ran into dependency problems when trying to install signal-desktop:
The following packages have unmet dependencies: signal-desktop : Depends: libappindicator1 but it is not installable
Well, that’s nice. According to a bug report this is due to libappindicator1 being deprecated. The workaround was to change sources to sid and try the install again, which worked. I, of course, changed my sources back to testing. I’m sure things will break again in the future.
The Signal app for Linux looks like it’s nothing more than an Electron-style wrapper that uses Chrome or Chromium under the hood:
Meh, I don’t really care but I’m a little bit disappointed, especially because there is no actual web interface offered. It only takes up 332 MiB RSS, which is nice. It could be worse!
There’s a Pidgin plugin that I need to try. It looks like it has a hard dependency on signald to do anything, which I’m not familiar with at all. More things to do later, I suppose.
So, everything was mostly peachy, right?
It was until I decided to fire up signal on my 2nd phone, an iPhone Xs. Yes, I carry two phones because iOS offers some things that Android doesn’t and vice versa. I can’t decide which platform is best for me so I have selected both.
My iPhone has a different phone number, of course, so I plugged in my original phone number when starting up the iOS Signal app. Everything seemed to work fine until I realized that the Signal app on my other devices and Android phone started kicking out API errors. After trying to figure what was going on I found the page that indicates that more than one phone is not supported.
Huhwha? While this is not a deal-breaker I also realized that Android tablets are not supported either. I don’t get it, why can’t Signal on my iPhone be activated the same way the macOS and Linux clients were activated? I have a feeling the answer is security-related, but I can’t actually figure it out.
Lastly, to finish this up, I was curious who hosted Signal. I tcpdump’ed some DNS requests from a fresh Wi-Fi connection from my Android phone to see the following:
16:09:00.217072 IP 10.3.6.114.48167 > 10.3.5.1.53: 7640+ AAAA? textsecure-service.whispersystems.org. (55) 16:09:00.260375 IP 10.3.5.1.53 > 10.3.6.114.48167: 7640 0/1/0 (140) 16:09:00.261707 IP 10.3.6.114.25681 > 10.3.5.1.53: 18696+ A? textsecure-service.whispersystems.org. (55) 16:09:00.389951 IP 10.3.5.1.53 > 10.3.6.114.25681: 18696 2/0/0 A 76.223.92.165, A 13.248.212.111 (87) 16:09:00.697416 IP 10.3.6.114.39256 > 10.3.5.1.53: 21129+ AAAA? storage.signal.org. (36) 16:09:00.843944 IP 10.3.5.1.53 > 10.3.6.114.39256: 21129 4/0/0 AAAA 2001:4860:4802:34::15, AAAA 2001:4860:4802:38::15, AAAA 2001:4860:4802:32::15, AAAA 2001:4860:4802:36::15 (148) 16:09:00.845337 IP 10.3.6.114.16859 > 10.3.5.1.53: 41372+ A? storage.signal.org. (36) 16:09:00.882592 IP 10.3.5.1.53 > 10.3.6.114.16859: 41372 4/0/0 A 216.239.34.21, A 216.239.36.21, A 216.239.38.21, A 216.239.32.21 (100)
This mostly matches up with what’s detailed here regarding firewall settings for Signal. It looks like *.signal.org and *.whispersystems.org are the main domain names. Right now, textsecure-service.whispersystems.org. is not dual-stacked, which means it won’t work in an IPv6-only environment. Also, looking up the addresses that the two names resolve to gives me:
That’s some nice big tech right there! Ah well, at least Signal is end-to-end encrypted so I don’t have to care who or what is in the middle.
I’ll keep an eye out for multi-phone support as well as IPv6 server support and the ability to change phone numbers in the future. For now, Signal seems like a clear privacy-centric alternative to other things like WhatsApp, FB Messenger, and Google Hangouts (and whatever will replace it).
I mentioned that I did the Big Sur upgrade in a previous blog entry and that it was working out alright. However, I’m going to have to change my mind on this one. I don’t think macOS is an operating system that meets my needs anymore. The amount of hackery needed to get certain components of the OS to work the way I desire is adding up and probably compromising the security of the OS in the process, not to mention just annoying the heck out of me.
Sleep Annoyances
On my 2017 MacBook there is no option to prevent the computer from sleeping while on battery power.
I don’t see any “computer sleep” slider here, do you?
This results in active SSH connections dying at some time after the display shuts off. When I say active I mean there is activity on the terminal, for example htop running. The odd thing is that it’s not a complete sleep, SmokePing indicates that my machine will still ping over Wi-Fi (albeit with heightened latency due to Wi-Fi power saving that cannot be disabled, yep, another thing!) but TCP communications are cut off. I cannot SSH into the machine, either. I have to use sudo pmset sleep 0 on every reboot to get around this stupidity and ultimately this needs to go into some sort of login script since cron is gone.
DNS Issues and Launch Daemons
I have to hack up the mDNSResponder (DNS resolver) property list file to make macOS resolve short names with a dot in them. I expect a DNS query for foo.vpn will result in my DNS search prefix appended but it doesn’t do that by default (mostly because it’s not appropriate for the general public, but it’s important for me and my particular network setup). Again, there’s no GUI option for this so in the past I’ve modified the com.apple.mDNSResponder.plist in /System/Library/LaunchDaemons, added -AlwaysAppendSearchDomains to the ProgramArguments key and re-enabled it through launchctl. Now, though, starting in Catalina /System is non-writable without some major hacking of disabling SIP and SSV, the latter which will break your OS and cause boot loops if you do it wrong. The only middle ground here is to disable SIP and then unload and load a copy of the property list file out of /Library/LaunchDaemons. Unfortunately, none of these launchd changes persist between reboots (I don’t know why, probably some additional security layer that’s working against me) so I have to do this again at every boot or put it in some sort of login script.
IPv6 Issues
I also want to disable RFC 4941 (IPv6 privacy extensions) because I like the original EUI-64 behavior and I think that privacy extensions are privacy theater. Naturally, macOS doesn’t let me change this behavior (same with iOS and iPadOS) so I have to create my own /etc/sysctl.conf and add stuff to it:
Just as with the above, on any incremental upgrade these changes will need to be reapplied but at least macOS will throw the file on your desktop after that upgrade indicating it doesn’t like you modifying things.
Notarization
Nah, not gonna talk about this. Search the web for both good information and misinformation on this one and make up your own mind on it.
In general, macOS is just doing the opposite of what I want and it’s getting worse with every release. Maybe the problem here is that I want macOS to be Linux or even something resembling a Unix-based operating system and it’s just not anymore. It’s being bogged down by heavy-handed mitigations to protect the user from their own naivete, which has an effect of restricting power users from doing what they want. Maybe this is due to the departure of Jordan Hubbard or just an effect of macOS completing the move over the last two decades from a niche operating system used by schools and graphic designers to an operating system for the masses.
I haven’t written a blog entry since April of 2020. It’s not that I haven’t had much to say about all the horrible things 2020 has brought but I just haven’t thought the Internet needed more commentary about things that have been commented on to death. I’m still alive and really don’t have anything interesting to report but this blog needed activity, so here’s some randomness.
First, because it’s a substring of the title of this blog entry, Still Alive is an epic and moving piece of EDM by Ashley Wallbridge & Evan Henzi. He wrote it after narrowly winning a battle with meningitis. It, along with You’ll Be OK and Elise by Gareth Emery, got quite a bit of runtime this year on pretty much all systems I own that are capable of playing digital audio.
Since some of y’all know I love vintage Unix workstations and servers, I picked up an HP C8000 on eBay halfway through the summer. It was new in box and shipped from Germany. Due to problems with TnT (Track & Trace, owned by FedEx) it took a couple of weeks to arrive. The C8000 is a PA-RISC system and a model of the HP 9000 line of servers and workstations that were scrubbed in 2008. The machine has 2x PA-8900 CPUs @ 1 GHz, 8 GiB of RAM, and 2x Ultra320 SCSI drives. It was a top-of-the-line system for the early 2000s and still feels pretty quick & responsive (at least on the CLI and in CDE).
I spent about two weeks getting HP-UX 11.11 (unfortunately, it’s the last version with support for the C8000) installed and configured to my liking. The machine initially did not boot and after messing with it a bit I just decided to do a fresh install of HP-UX. I have a feeling there was something in the BCH that was wiped out when the CMOS battery died a few years ago. Anyway, after the installation I used LVM to split up /var, /home, /usr, etc. across the two SCSI drives because I didn’t care about redundancy. The HP-UX Porting and Archive Center provided me enough familiar FOSS packages to install so I could feel at home. The only thing I wasn’t able to get working was IPv6 (HP-UX_11i_v1_IPv6NCF11i_B.11.11.0705_HP-UX_B.11.11_32+64.depot) because I couldn’t get the patch installed due to dependency hell. I learned quite a bit about how swinstall worked and a little bit about the hardware during the process. Unfortunately, the C8000 pumped out much too much heat for my home office so during a warm week without A/C in September I decided to shut it down. I’ll mess with it again once I move some things around in the house.
I am NOT caught up with Star Trek Discovery. I lost interest toward the end of season 2 but, as with many other TV series, I figured I needed to probably watch it twice to appreciate it. So, I vowed that before season 3 came out I would re-watch seasons 1 and 2. Well, that didn’t happen. I’m on episode 6 of season 2 right now so I’ve got a few episodes to go before I can start season 3. That being said, I think I need to tell Google News and other sources that I’m temporarily not interested in anything Star Trek because one of these days I’m going to accidentally read a real spoiler.
I have been watching Lower Decks. It’s entertaining, so far. I think I’m a couple episodes behind on that, too.
My wife got me The Bartesian for my birthday this past August! The machine itself isn’t too fancy and does what it’s supposed to do—make mixed drinks. Similar to a Keurig, it can be a little bit messy. That might only be a problem for me, due to OCD. Depending on use, it’s a wallet suck through both the spirits required as well as the capsules. I stocked up, though:
Speaking for a second more about R3COH, my wife (wait, there’s a theme here?) got me a Christmas present last year in the form of a variety of mini-bottles leading up to Christmas Day. Most of the spirits I’ve encountered before except for a few, which included Green House Gin. Western Washington has more than a few distilleries that produce gin and I’ve tried many of them (Big Gin is usually my go-to). However, Green House is probably the best gin I’ve had—it’s got a ton of botanicals and the juniper flavor is just the right strength. I highly recommend trying it out if you’re into such things.
In other news, I upgraded my 2017 Retina MacBook from Mojave to Big Sur (11.0.2) a few days after it came out. I usually wait until the dust settles but I felt like going for it early this year. I always do a wipe and fresh install because I don’t want to deal with any leftover “garbage” or badly-migrated settings from the previous release. Anyway, most things work and I don’t miss any 32-bit applications. I think the only real applications I run on macOS nowadays are:
Microsoft Office 2019
VLC
Chromium
Kindle
Intel Power Gadget
Xcode (to support MacPorts builds only)
MacPorts
TunnelBlick
I usually make some minor hacks to macOS to get things to work the way I like, which include things like turning off some animations and making the DNS resolver always append the search domains:
Most of these hacks still work in Big Sur, which is nice. That being said, net-snmp does not compile correctly due to Xcode adhering to C99 standards so I submitted a bug report to MacPorts. I use net-snmp as an include to my mrtg-rmt script, which allows me to report some system health information. At least iStats still mostly works:
% istats
--- CPU Stats ---
CPU temp: 25.63°C ▁▂▃▅▆▇
--- Fan Stats ---
Total fans in system: 0
--- Battery Stats ---
Battery health: unknown
Cycle count: 107 ▁▂▃▅▆▇ 10.7%
Max cycles: 1000
Current charge: 4851 mAh ▁▂▃▅▆▇ 100%
Maximum charge: 4959 mAh ▁▂▃▅▆▇ 89.4%
Design capacity: 5550 mAh
Battery temp: 23.8°C
For more stats run `istats extra` and follow the instructions.
Also, I didn’t see Blink Lite in the App Store anymore, which was my SIP client on macOS. That’s not much of a loss since I probably only used it once or twice a year, max.
Local stuff. Let’s see, the West Seattle Bridge has been closed since March and only this month did Mayor Durkin decide to pursue the repair option. The expectation is that traffic will return to the bridge in early to mid 2022. Also, I bought westseattlebridge.com for no reason. No, I didn’t bother with an SSL certificate because I’m not really using the site for anything, at the moment. Well, that’s not true, it’s a bookmark for the official SDOT page, which has a much longer URL.
No blog entry in 2020 would be complete without some mention of COVID-19! My wife, a few co-workers, and I all believe we got a bout of COVID-19 in January. I have asthma and had what might be called a miniature to medium attack, which was very unusual. I, along with the others, got over it just fine, though. My wife works at Harborview Medical Center and goes into work every day but I’ve been working from home since late March with an expected return sometime during the summer of 2021. For many years I was diametrically opposed to WFH and still believe the lack of casual human interaction takes a toll on productivity and innovation but this year has taught me that it’s not as bad as I thought. I, along with the team I’m on, are still wildly productive. We have a virtual lunch once a week where we catch up on things that might happen in a hallway conversation. It feels forced but it works out alright. I still visit the office once or twice a month.
Speaking of, since I’m home a lot more lately I’ve taken up walking all over West Seattle. I usually just walk High Point, Gatewood, and Fairmount Park but once a week I’ll make it down to Lowman Beach and periodically will walk Highland Park. The beach is nice:
We put up the Christmas tree a few days after Thanksgiving, this year. I didn’t setup the train this year due to apathy:
I also didn’t do a time lapse as I’ve done in years past.
That’s it for now. Actually, a number of items above are now out-of-date because I left this blog entry in draft mode, untouched, for about two weeks. Sorry about that!
This is my first real post using WordPress vs. my legacy blog. We’ll see how it turns out.
When I moved to Seattle in early 2014 I realized I had two non-wireless Internet options at my townhome in the High Point neighborhood of West Seattle: CenturyLink DSL or Comcast DOCSIS. I chose a medium-tier plan with Comcast Xfinity that included TV (my wife would ultimately end up paying for this because I don’t watch broadcast TV) and did the self install. I remember checking CenturyLink’s options but I recall the max plan being something on the order of 5 Mbps downstream and a measly upstream so I didn’t really give them much consideration. Plus, I liked the < 10 ms RTT to the CMTS that DOCSIS typically provides vs. 20-25 ms RTT that I’ve seen on most DSL connections. The Xfinity connection was solid and I had no complaints. However, I wanted to get public IPs for at least two devices (CPEs) and this required a business class connection through Comcast Business. I added the 50/10 service from Comcast Business in April of 2014 and they assigned me a static IPv4 /29 for use. I kept the Xfinity service and terminated it in a Linux container. I had to lease a modem because I think it ran some dynamic routing (RIP?) with the CMTS. The modem I got was the NETGEAR CG3000DCR, which had some software issues for awhile but those got sorted within a few months.
The Comcast Business service worked great. Latency and variability (for the most part) was low and I had ~10 ms reachability to things in Seattle (work VPN, Vultr VPS hosting, etc.). My SmokePing graphs don’t go back too far but it was something like this the whole time:
BTW, the SmokePing graphs I have here are either to the first hop routers or to a nearby anycasted IP like Google’s 8.8.8.8 or CloudFlare’s 1.1.1.1. I don’t only ping routers because I am aware of CoPPs and how they may affect ICMP responses but if the latency to the first hop router vs. a nearby anycasted address is about the same, I choose the first hop router.
It was pretty expensive, though. I paid $150/mo for it but the technical support (I only needed it once to get them to add some DNS PTRs, which I had to read over the phone, ugh) was worlds better than Xfinity. I ended up switching to a subsidized plan through my employer for $120/mo and upgraded to 75/15 in late 2018.
So, yeah, I was a DOCSIS fan. I didn’t care that the media was shared and that my neighbors and I fought over the same the RF spectrum. It worked great. I mostly (there were a few months when they had some hub problems) had a good experience with DOCSIS provided by Time Warner Cable in Charlotte, NC for 9 years, as well.
The COVID-19 work from home guidance from my employer was announced on March 4, 2020. I actually worked from home that day for other reasons and took PTO for the rest of the week (it my wife and my 5 year wedding anniversary, which, after some canceled plans.. spent at home). My first day of mandated WFH was March 9, 2020. The Comcast Business connection worked great and my daily video conferences didn’t hiccup at all. However, at noon on March 20, 2020 I noticed a huge increase of latency and upstream (only!) packet loss:
What the heck? I figured it was a temporary blip and a node or amp had failed so I let it go 24 hours. However, it persisted. I saw it on my Xfinity connection as well and was able to (with a client-side VPN) reproduce it from my /neighbors/ connection using xfinitywifi. So, it wasn’t just my unit. TV was still fine and I attributed that to the fact that the loss and latency was only in the upstream direction. This, of course, was maddening because on video conferences I could see and hear everyone but they couldn’t see or hear me clearly! The latency and loss subsided overnight and was only apparent between around 0900 and 2330 PDT. It felt like either something had gone bad and was falling over due to load. Here’s a shot showing an example 0000 to 2359:
So, I called Comcast Business on Saturday evening and the CSR who I talked to verified my modem levels were fine but there was definitely packet loss that he could reproduce. He had me reboot my modem but that didn’t help (I knew it wouldn’t). He couldn’t do anything more for me so he scheduled a tech. to come out to my home on that coming Wednesday (March 24, 2020). I had him schedule it a few days out just in case if it cleared up I could cancel the appointment.
It didn’t clear up. The tech. came out and due to COVID-19 he couldn’t enter my home. However, he said he checked my modem levels and they were OK and he also replaced an old splitter on the side of my townhome just to make sure I was getting the best levels possible. He couldn’t do much else for me but did say that a ticket had been filed the prior day (Tuesday, the 23rd) for packet loss and latency and it was assigned to maintenance so a line tech. would be working on it. I asked him if he knew if this was definitely due to some bad hardware or if this was indeed due to network congestion. He said he couldn’t answer directly because they were being told to not use certain terms but he strongly led me to believe it was something wrong that needed to be fixed. I was reassured by this because the hardest part was over – convincing Comcast that there’s a problem and it’s not due to something silly in your house or your “unsupported operating system” (ugh, I’ve been through that garbage). Comcast acknowledged there was a problem and they were going to fix it! Woo!
Well, I’ll cut to the chase. It never got fixed.
I called in half a dozen times over the next two and a half weeks to get a status on the maintenance ticket, which each CSR said usually is resolved within 72 hours. Each time when I gave the maintenance ticket number to the CSR they had zero visibility into it and said there were no notes in there.
Even though every interaction with our neighborhood’s NextDoor site has been wholly negative, I decided to crowdsource some information and posted some details about my Comcast issues. Over the next day or two I got some replies from folks at the far end of High Point indicating they had no issue but I did get a few replies from folks closer to my block indicating they had the same issues and those issues lined up with the timeline I listed. Well, at least I had more evidence it was not just me or my block. I also fired off a tweet on March 22nd and kept it updated.
Governor Jay Inslee’s official stay at home order went into effect on March 26, 2020. The packet loss didn’t get any worse, surprisingly.
When the issues first started I checked CenturyLink’s site and found that I could now get 140/20 Mbps VDSL2 service to my unit. The block next to mine got built up with Polygon NW townhomes since 2014 and I believe a new DSLAM was installed on the property. I kept that information in my back pocket because I was still scared of a 20 ms latency hit (or more!?) to the DSLAM. I have a clear view of downtown Seattle from my balcony and didn’t feel it was right to pay a > 20 ms RTT just to SSH to a box in the Westin Building! Anyway, I put that aside and ordered the 140/20 Mbps CenturyLink DSL service on March 27th. The installation was scheduled for April 7th. At this point I mentally decided to nuke Comcast Business regardless if they resolved the issue or not. I didn’t need 3x Internet connections.
Anyway, I was going to still see if I could get the Comcast issue sorted because I intended to keep my Xfinity service as a backup (or promote it back to primary if the DSL service sucked). So, I kept calling Comcast. The CSR on March 30th gave me the option to escalate so she submitted an escalation request and said a “supervisor” would call me back that day or the next. To cut to the end for a second – I never got a call back regarding it.
When I called in yet again on April 6, 2020 before speaking to a CSR I got a notification that there was going to be maintenance on April 7 between 0000 and 0600 local time and it would interrupt my Internet connectivity. Woo! They were going to fix my problems! I asked the CSR if this was related the maintenance ticket but the CSR didn’t know and said they thought it might be. It was becoming clear that either Comcast couldn’t give me any information about the problem and even could not tell me if the problem was a legitimate hardware failure or related to congestion, so I just waited for the next morning.
When I checked on things in the morning the loss and latency still were present and I never saw a drop in Internet service overnight. Either nothing had been done or the maintenance was completely unrelated to my problems.
On April 7, 2020 I got the CenturyLink DSL installed. It works great. I get around 6.6ms RTT to the CenturyLink BRAS in Tukwila (even through their crappy NAT box!). I switched IPv4 Internet over to it but we’ll talk more about that in a little bit.
So, I called Comcast Business again on the 8th. I provided the maintenance ticket again for the $LOSTCOUNT time and the CSR indicated that the maintenance ticket was closed on April 1st at 0407. What the heck? The CSRs just strung me along for the ride for these weeks because apparently the maintenance ticket that was opened was not related to the work that was done on April 7th and had nothing to do with my problems. I explained my situation and asked to cancel the service. I was given a national account e-mail address to use to request service termination because it was a subsidized account. I sent the e-mail that day. On April 10th my account was terminated (abruptly, I might add) and I returned the modem that same day. One good thing here is that I explained the situation in the cancellation e-mail and I was told that my account would be credited back to March 20th. This is a good gesture. That being said, I haven’t received the last invoice for the account at the time of writing.
I restructured part of my network and put the DSL connection into a Linux container (LXC) as well, just like the Xfinity connection. This gave me more flexibility to do fancy routing things. I advertised a default route into OSPF from it for IPv4 traffic (double-NAT) and for IPv6 traffic used a combination of BGP LOCAL_PREF and AS_PATH prepending (read more about my network here – IPv6 is all tunneled because I have an ARIN PI block and my own ASN) to make outbound IPv6 traffic go through the DSL connection and inbound still take the Xfinity connection. I did this because the Comcast loss was only in the upstream direction and my Xfinity plan provided 300 Mbps downstream. So, for IPv4 I had 140/20 Mbps but for IPv6 I essentially had 300/20 Mbps. Due to the tunnel overhead and differing latency TCP has to deal with realistically this gets me around 175-190 Mbps downstream.
Anyway, the ~6.6 ms best case RTT to the BRAS (I assume that is the first L3 hop and it’s located south of me in Tukwila, WA) is really good for DSL. The VDSL2 configuration on the ZyXel C3000Z modem indicated the loop length was between 1110 and 1120 ft. Here’s an MTR to Google:
Here’s a SmokePing graph to the BRAS:
There’s a little spike in the evenings but it’s short-lived so I’m not too worried.
I put the Linux container’s IP in the “DMZ” on the ZyXel modem. I’m not sure if it’s still doing port translation or only if needed but I don’t really care. I’d like to terminate the PPPoE connection in the Linux container and turn on transparent mode but I haven’t been able to get the PPPoE password yet and the prior instructions I found on a web search no longer work on the C3000Z because the “sh” command in the Telnet/SSH remote management now requires a password. Supposedly I can just chat with a CenturyLink representative and they’ll give it to me. It’s on my list of things to do. Right now, I make sure my tunnels are set to an effective MTU of 1492 to account for the PPPoE overhead. The C3000Z has some TCP MSS clamping to 1452 I believe.
I’d still love to get the Comcast issues sorted. The NextDoor thread progressed to a point where some neighbors called Xfinity and got a service credit for their trouble because they were told that they don’t have personnel to fix the problem. I don’t know if this was just a generic “go away” response or if this is the case or if they just don’t know the cause of the problems. I still doubt it’s congestion from the stay-at-home traffic because it started so abruptly at noon on the 20th. It should have gotten worse day by day. One neighbor ended up getting a CenturyLink installation appointment scheduled and presumably will be switching away from Xfinity. Others are apparently too far away from the DSLAM and have been quoted 1.5 Mbps (yes, 1 point 5 megabits per second) service so that’s a non-starter.
There’s CenturyLink GPON / FTTH in other parts of West Seattle but not in High Point, where all utilities are subterranian and I doubt they’ll want to (or even be given the OK from the various owners of the shared property in the neighborhood) dig up conduit to install fiber. Maybe if one day the NIMBYs in Seattle will allow for it we’ll get some 5G mmWave deployments and some real competition for high-speed residential Internet access.
