I mentioned that I did the Big Sur upgrade in a previous blog entry and that it was working out alright. However, I’m going to have to change my mind on this one. I don’t think macOS is an operating system that meets my needs anymore. The amount of hackery needed to get certain components of the OS to work the way I desire is adding up and probably compromising the security of the OS in the process, not to mention just annoying the heck out of me.
Sleep Annoyances
On my 2017 MacBook there is no option to prevent the computer from sleeping while on battery power.
I don’t see any “computer sleep” slider here, do you?
This results in active SSH connections dying at some time after the display shuts off. When I say active I mean there is activity on the terminal, for example htop running. The odd thing is that it’s not a complete sleep, SmokePing indicates that my machine will still ping over Wi-Fi (albeit with heightened latency due to Wi-Fi power saving that cannot be disabled, yep, another thing!) but TCP communications are cut off. I cannot SSH into the machine, either. I have to use sudo pmset sleep 0 on every reboot to get around this stupidity and ultimately this needs to go into some sort of login script since cron is gone.
DNS Issues and Launch Daemons
I have to hack up the mDNSResponder (DNS resolver) property list file to make macOS resolve short names with a dot in them. I expect a DNS query for foo.vpn will result in my DNS search prefix appended but it doesn’t do that by default (mostly because it’s not appropriate for the general public, but it’s important for me and my particular network setup). Again, there’s no GUI option for this so in the past I’ve modified the com.apple.mDNSResponder.plist in /System/Library/LaunchDaemons, added -AlwaysAppendSearchDomains to the ProgramArguments key and re-enabled it through launchctl. Now, though, starting in Catalina /System is non-writable without some major hacking of disabling SIP and SSV, the latter which will break your OS and cause boot loops if you do it wrong. The only middle ground here is to disable SIP and then unload and load a copy of the property list file out of /Library/LaunchDaemons. Unfortunately, none of these launchd changes persist between reboots (I don’t know why, probably some additional security layer that’s working against me) so I have to do this again at every boot or put it in some sort of login script.
IPv6 Issues
I also want to disable RFC 4941 (IPv6 privacy extensions) because I like the original EUI-64 behavior and I think that privacy extensions are privacy theater. Naturally, macOS doesn’t let me change this behavior (same with iOS and iPadOS) so I have to create my own /etc/sysctl.conf and add stuff to it:
Just as with the above, on any incremental upgrade these changes will need to be reapplied but at least macOS will throw the file on your desktop after that upgrade indicating it doesn’t like you modifying things.
Notarization
Nah, not gonna talk about this. Search the web for both good information and misinformation on this one and make up your own mind on it.
In general, macOS is just doing the opposite of what I want and it’s getting worse with every release. Maybe the problem here is that I want macOS to be Linux or even something resembling a Unix-based operating system and it’s just not anymore. It’s being bogged down by heavy-handed mitigations to protect the user from their own naivete, which has an effect of restricting power users from doing what they want. Maybe this is due to the departure of Jordan Hubbard or just an effect of macOS completing the move over the last two decades from a niche operating system used by schools and graphic designers to an operating system for the masses.
I haven’t written a blog entry since April of 2020. It’s not that I haven’t had much to say about all the horrible things 2020 has brought but I just haven’t thought the Internet needed more commentary about things that have been commented on to death. I’m still alive and really don’t have anything interesting to report but this blog needed activity, so here’s some randomness.
First, because it’s a substring of the title of this blog entry, Still Alive is an epic and moving piece of EDM by Ashley Wallbridge & Evan Henzi. He wrote it after narrowly winning a battle with meningitis. It, along with You’ll Be OK and Elise by Gareth Emery, got quite a bit of runtime this year on pretty much all systems I own that are capable of playing digital audio.
Since some of y’all know I love vintage Unix workstations and servers, I picked up an HP C8000 on eBay halfway through the summer. It was new in box and shipped from Germany. Due to problems with TnT (Track & Trace, owned by FedEx) it took a couple of weeks to arrive. The C8000 is a PA-RISC system and a model of the HP 9000 line of servers and workstations that were scrubbed in 2008. The machine has 2x PA-8900 CPUs @ 1 GHz, 8 GiB of RAM, and 2x Ultra320 SCSI drives. It was a top-of-the-line system for the early 2000s and still feels pretty quick & responsive (at least on the CLI and in CDE).
I spent about two weeks getting HP-UX 11.11 (unfortunately, it’s the last version with support for the C8000) installed and configured to my liking. The machine initially did not boot and after messing with it a bit I just decided to do a fresh install of HP-UX. I have a feeling there was something in the BCH that was wiped out when the CMOS battery died a few years ago. Anyway, after the installation I used LVM to split up /var, /home, /usr, etc. across the two SCSI drives because I didn’t care about redundancy. The HP-UX Porting and Archive Center provided me enough familiar FOSS packages to install so I could feel at home. The only thing I wasn’t able to get working was IPv6 (HP-UX_11i_v1_IPv6NCF11i_B.11.11.0705_HP-UX_B.11.11_32+64.depot) because I couldn’t get the patch installed due to dependency hell. I learned quite a bit about how swinstall worked and a little bit about the hardware during the process. Unfortunately, the C8000 pumped out much too much heat for my home office so during a warm week without A/C in September I decided to shut it down. I’ll mess with it again once I move some things around in the house.
I am NOT caught up with Star Trek Discovery. I lost interest toward the end of season 2 but, as with many other TV series, I figured I needed to probably watch it twice to appreciate it. So, I vowed that before season 3 came out I would re-watch seasons 1 and 2. Well, that didn’t happen. I’m on episode 6 of season 2 right now so I’ve got a few episodes to go before I can start season 3. That being said, I think I need to tell Google News and other sources that I’m temporarily not interested in anything Star Trek because one of these days I’m going to accidentally read a real spoiler.
I have been watching Lower Decks. It’s entertaining, so far. I think I’m a couple episodes behind on that, too.
My wife got me The Bartesian for my birthday this past August! The machine itself isn’t too fancy and does what it’s supposed to do—make mixed drinks. Similar to a Keurig, it can be a little bit messy. That might only be a problem for me, due to OCD. Depending on use, it’s a wallet suck through both the spirits required as well as the capsules. I stocked up, though:
Speaking for a second more about R3COH, my wife (wait, there’s a theme here?) got me a Christmas present last year in the form of a variety of mini-bottles leading up to Christmas Day. Most of the spirits I’ve encountered before except for a few, which included Green House Gin. Western Washington has more than a few distilleries that produce gin and I’ve tried many of them (Big Gin is usually my go-to). However, Green House is probably the best gin I’ve had—it’s got a ton of botanicals and the juniper flavor is just the right strength. I highly recommend trying it out if you’re into such things.
In other news, I upgraded my 2017 Retina MacBook from Mojave to Big Sur (11.0.2) a few days after it came out. I usually wait until the dust settles but I felt like going for it early this year. I always do a wipe and fresh install because I don’t want to deal with any leftover “garbage” or badly-migrated settings from the previous release. Anyway, most things work and I don’t miss any 32-bit applications. I think the only real applications I run on macOS nowadays are:
Microsoft Office 2019
VLC
Chromium
Kindle
Intel Power Gadget
Xcode (to support MacPorts builds only)
MacPorts
TunnelBlick
I usually make some minor hacks to macOS to get things to work the way I like, which include things like turning off some animations and making the DNS resolver always append the search domains:
Most of these hacks still work in Big Sur, which is nice. That being said, net-snmp does not compile correctly due to Xcode adhering to C99 standards so I submitted a bug report to MacPorts. I use net-snmp as an include to my mrtg-rmt script, which allows me to report some system health information. At least iStats still mostly works:
% istats
--- CPU Stats ---
CPU temp: 25.63°C ▁▂▃▅▆▇
--- Fan Stats ---
Total fans in system: 0
--- Battery Stats ---
Battery health: unknown
Cycle count: 107 ▁▂▃▅▆▇ 10.7%
Max cycles: 1000
Current charge: 4851 mAh ▁▂▃▅▆▇ 100%
Maximum charge: 4959 mAh ▁▂▃▅▆▇ 89.4%
Design capacity: 5550 mAh
Battery temp: 23.8°C
For more stats run `istats extra` and follow the instructions.
Also, I didn’t see Blink Lite in the App Store anymore, which was my SIP client on macOS. That’s not much of a loss since I probably only used it once or twice a year, max.
Local stuff. Let’s see, the West Seattle Bridge has been closed since March and only this month did Mayor Durkin decide to pursue the repair option. The expectation is that traffic will return to the bridge in early to mid 2022. Also, I bought westseattlebridge.com for no reason. No, I didn’t bother with an SSL certificate because I’m not really using the site for anything, at the moment. Well, that’s not true, it’s a bookmark for the official SDOT page, which has a much longer URL.
No blog entry in 2020 would be complete without some mention of COVID-19! My wife, a few co-workers, and I all believe we got a bout of COVID-19 in January. I have asthma and had what might be called a miniature to medium attack, which was very unusual. I, along with the others, got over it just fine, though. My wife works at Harborview Medical Center and goes into work every day but I’ve been working from home since late March with an expected return sometime during the summer of 2021. For many years I was diametrically opposed to WFH and still believe the lack of casual human interaction takes a toll on productivity and innovation but this year has taught me that it’s not as bad as I thought. I, along with the team I’m on, are still wildly productive. We have a virtual lunch once a week where we catch up on things that might happen in a hallway conversation. It feels forced but it works out alright. I still visit the office once or twice a month.
Speaking of, since I’m home a lot more lately I’ve taken up walking all over West Seattle. I usually just walk High Point, Gatewood, and Fairmount Park but once a week I’ll make it down to Lowman Beach and periodically will walk Highland Park. The beach is nice:
We put up the Christmas tree a few days after Thanksgiving, this year. I didn’t setup the train this year due to apathy:
I also didn’t do a time lapse as I’ve done in years past.
That’s it for now. Actually, a number of items above are now out-of-date because I left this blog entry in draft mode, untouched, for about two weeks. Sorry about that!
This is my first real post using WordPress vs. my legacy blog. We’ll see how it turns out.
When I moved to Seattle in early 2014 I realized I had two non-wireless Internet options at my townhome in the High Point neighborhood of West Seattle: CenturyLink DSL or Comcast DOCSIS. I chose a medium-tier plan with Comcast Xfinity that included TV (my wife would ultimately end up paying for this because I don’t watch broadcast TV) and did the self install. I remember checking CenturyLink’s options but I recall the max plan being something on the order of 5 Mbps downstream and a measly upstream so I didn’t really give them much consideration. Plus, I liked the < 10 ms RTT to the CMTS that DOCSIS typically provides vs. 20-25 ms RTT that I’ve seen on most DSL connections. The Xfinity connection was solid and I had no complaints. However, I wanted to get public IPs for at least two devices (CPEs) and this required a business class connection through Comcast Business. I added the 50/10 service from Comcast Business in April of 2014 and they assigned me a static IPv4 /29 for use. I kept the Xfinity service and terminated it in a Linux container. I had to lease a modem because I think it ran some dynamic routing (RIP?) with the CMTS. The modem I got was the NETGEAR CG3000DCR, which had some software issues for awhile but those got sorted within a few months.
The Comcast Business service worked great. Latency and variability (for the most part) was low and I had ~10 ms reachability to things in Seattle (work VPN, Vultr VPS hosting, etc.). My SmokePing graphs don’t go back too far but it was something like this the whole time:
BTW, the SmokePing graphs I have here are either to the first hop routers or to a nearby anycasted IP like Google’s 8.8.8.8 or CloudFlare’s 1.1.1.1. I don’t only ping routers because I am aware of CoPPs and how they may affect ICMP responses but if the latency to the first hop router vs. a nearby anycasted address is about the same, I choose the first hop router.
It was pretty expensive, though. I paid $150/mo for it but the technical support (I only needed it once to get them to add some DNS PTRs, which I had to read over the phone, ugh) was worlds better than Xfinity. I ended up switching to a subsidized plan through my employer for $120/mo and upgraded to 75/15 in late 2018.
So, yeah, I was a DOCSIS fan. I didn’t care that the media was shared and that my neighbors and I fought over the same the RF spectrum. It worked great. I mostly (there were a few months when they had some hub problems) had a good experience with DOCSIS provided by Time Warner Cable in Charlotte, NC for 9 years, as well.
The COVID-19 work from home guidance from my employer was announced on March 4, 2020. I actually worked from home that day for other reasons and took PTO for the rest of the week (it my wife and my 5 year wedding anniversary, which, after some canceled plans.. spent at home). My first day of mandated WFH was March 9, 2020. The Comcast Business connection worked great and my daily video conferences didn’t hiccup at all. However, at noon on March 20, 2020 I noticed a huge increase of latency and upstream (only!) packet loss:
What the heck? I figured it was a temporary blip and a node or amp had failed so I let it go 24 hours. However, it persisted. I saw it on my Xfinity connection as well and was able to (with a client-side VPN) reproduce it from my /neighbors/ connection using xfinitywifi. So, it wasn’t just my unit. TV was still fine and I attributed that to the fact that the loss and latency was only in the upstream direction. This, of course, was maddening because on video conferences I could see and hear everyone but they couldn’t see or hear me clearly! The latency and loss subsided overnight and was only apparent between around 0900 and 2330 PDT. It felt like either something had gone bad and was falling over due to load. Here’s a shot showing an example 0000 to 2359:
So, I called Comcast Business on Saturday evening and the CSR who I talked to verified my modem levels were fine but there was definitely packet loss that he could reproduce. He had me reboot my modem but that didn’t help (I knew it wouldn’t). He couldn’t do anything more for me so he scheduled a tech. to come out to my home on that coming Wednesday (March 24, 2020). I had him schedule it a few days out just in case if it cleared up I could cancel the appointment.
It didn’t clear up. The tech. came out and due to COVID-19 he couldn’t enter my home. However, he said he checked my modem levels and they were OK and he also replaced an old splitter on the side of my townhome just to make sure I was getting the best levels possible. He couldn’t do much else for me but did say that a ticket had been filed the prior day (Tuesday, the 23rd) for packet loss and latency and it was assigned to maintenance so a line tech. would be working on it. I asked him if he knew if this was definitely due to some bad hardware or if this was indeed due to network congestion. He said he couldn’t answer directly because they were being told to not use certain terms but he strongly led me to believe it was something wrong that needed to be fixed. I was reassured by this because the hardest part was over – convincing Comcast that there’s a problem and it’s not due to something silly in your house or your “unsupported operating system” (ugh, I’ve been through that garbage). Comcast acknowledged there was a problem and they were going to fix it! Woo!
Well, I’ll cut to the chase. It never got fixed.
I called in half a dozen times over the next two and a half weeks to get a status on the maintenance ticket, which each CSR said usually is resolved within 72 hours. Each time when I gave the maintenance ticket number to the CSR they had zero visibility into it and said there were no notes in there.
Even though every interaction with our neighborhood’s NextDoor site has been wholly negative, I decided to crowdsource some information and posted some details about my Comcast issues. Over the next day or two I got some replies from folks at the far end of High Point indicating they had no issue but I did get a few replies from folks closer to my block indicating they had the same issues and those issues lined up with the timeline I listed. Well, at least I had more evidence it was not just me or my block. I also fired off a tweet on March 22nd and kept it updated.
Governor Jay Inslee’s official stay at home order went into effect on March 26, 2020. The packet loss didn’t get any worse, surprisingly.
When the issues first started I checked CenturyLink’s site and found that I could now get 140/20 Mbps VDSL2 service to my unit. The block next to mine got built up with Polygon NW townhomes since 2014 and I believe a new DSLAM was installed on the property. I kept that information in my back pocket because I was still scared of a 20 ms latency hit (or more!?) to the DSLAM. I have a clear view of downtown Seattle from my balcony and didn’t feel it was right to pay a > 20 ms RTT just to SSH to a box in the Westin Building! Anyway, I put that aside and ordered the 140/20 Mbps CenturyLink DSL service on March 27th. The installation was scheduled for April 7th. At this point I mentally decided to nuke Comcast Business regardless if they resolved the issue or not. I didn’t need 3x Internet connections.
Anyway, I was going to still see if I could get the Comcast issue sorted because I intended to keep my Xfinity service as a backup (or promote it back to primary if the DSL service sucked). So, I kept calling Comcast. The CSR on March 30th gave me the option to escalate so she submitted an escalation request and said a “supervisor” would call me back that day or the next. To cut to the end for a second – I never got a call back regarding it.
When I called in yet again on April 6, 2020 before speaking to a CSR I got a notification that there was going to be maintenance on April 7 between 0000 and 0600 local time and it would interrupt my Internet connectivity. Woo! They were going to fix my problems! I asked the CSR if this was related the maintenance ticket but the CSR didn’t know and said they thought it might be. It was becoming clear that either Comcast couldn’t give me any information about the problem and even could not tell me if the problem was a legitimate hardware failure or related to congestion, so I just waited for the next morning.
When I checked on things in the morning the loss and latency still were present and I never saw a drop in Internet service overnight. Either nothing had been done or the maintenance was completely unrelated to my problems.
On April 7, 2020 I got the CenturyLink DSL installed. It works great. I get around 6.6ms RTT to the CenturyLink BRAS in Tukwila (even through their crappy NAT box!). I switched IPv4 Internet over to it but we’ll talk more about that in a little bit.
So, I called Comcast Business again on the 8th. I provided the maintenance ticket again for the $LOSTCOUNT time and the CSR indicated that the maintenance ticket was closed on April 1st at 0407. What the heck? The CSRs just strung me along for the ride for these weeks because apparently the maintenance ticket that was opened was not related to the work that was done on April 7th and had nothing to do with my problems. I explained my situation and asked to cancel the service. I was given a national account e-mail address to use to request service termination because it was a subsidized account. I sent the e-mail that day. On April 10th my account was terminated (abruptly, I might add) and I returned the modem that same day. One good thing here is that I explained the situation in the cancellation e-mail and I was told that my account would be credited back to March 20th. This is a good gesture. That being said, I haven’t received the last invoice for the account at the time of writing.
I restructured part of my network and put the DSL connection into a Linux container (LXC) as well, just like the Xfinity connection. This gave me more flexibility to do fancy routing things. I advertised a default route into OSPF from it for IPv4 traffic (double-NAT) and for IPv6 traffic used a combination of BGP LOCAL_PREF and AS_PATH prepending (read more about my network here – IPv6 is all tunneled because I have an ARIN PI block and my own ASN) to make outbound IPv6 traffic go through the DSL connection and inbound still take the Xfinity connection. I did this because the Comcast loss was only in the upstream direction and my Xfinity plan provided 300 Mbps downstream. So, for IPv4 I had 140/20 Mbps but for IPv6 I essentially had 300/20 Mbps. Due to the tunnel overhead and differing latency TCP has to deal with realistically this gets me around 175-190 Mbps downstream.
Anyway, the ~6.6 ms best case RTT to the BRAS (I assume that is the first L3 hop and it’s located south of me in Tukwila, WA) is really good for DSL. The VDSL2 configuration on the ZyXel C3000Z modem indicated the loop length was between 1110 and 1120 ft. Here’s an MTR to Google:
Here’s a SmokePing graph to the BRAS:
There’s a little spike in the evenings but it’s short-lived so I’m not too worried.
I put the Linux container’s IP in the “DMZ” on the ZyXel modem. I’m not sure if it’s still doing port translation or only if needed but I don’t really care. I’d like to terminate the PPPoE connection in the Linux container and turn on transparent mode but I haven’t been able to get the PPPoE password yet and the prior instructions I found on a web search no longer work on the C3000Z because the “sh” command in the Telnet/SSH remote management now requires a password. Supposedly I can just chat with a CenturyLink representative and they’ll give it to me. It’s on my list of things to do. Right now, I make sure my tunnels are set to an effective MTU of 1492 to account for the PPPoE overhead. The C3000Z has some TCP MSS clamping to 1452 I believe.
I’d still love to get the Comcast issues sorted. The NextDoor thread progressed to a point where some neighbors called Xfinity and got a service credit for their trouble because they were told that they don’t have personnel to fix the problem. I don’t know if this was just a generic “go away” response or if this is the case or if they just don’t know the cause of the problems. I still doubt it’s congestion from the stay-at-home traffic because it started so abruptly at noon on the 20th. It should have gotten worse day by day. One neighbor ended up getting a CenturyLink installation appointment scheduled and presumably will be switching away from Xfinity. Others are apparently too far away from the DSLAM and have been quoted 1.5 Mbps (yes, 1 point 5 megabits per second) service so that’s a non-starter.
There’s CenturyLink GPON / FTTH in other parts of West Seattle but not in High Point, where all utilities are subterranian and I doubt they’ll want to (or even be given the OK from the various owners of the shared property in the neighborhood) dig up conduit to install fiber. Maybe if one day the NIMBYs in Seattle will allow for it we’ll get some 5G mmWave deployments and some real competition for high-speed residential Internet access.
