Creative Articulation

Late Summer 2025 GenAI Tooling Review

2025-08-12T00:00:00+00:00

It’s been just eight months since my 2024 year-end GenAI tool review</a>, and the landscape already feels unrecognizable. I am posting this update to record how much has changed, mainly so that I can amuse myself by revisiting this in the future and marveling at how primitive things were back in the antediluvian GenAI epoch of 2025.

Tools I’ve Added</h2>
Claude Code</h3>
Around May of 2025 I made mention of using Claude Code in my With, By, or For</a> post; I had already by that point found myself preferring Claude Code for much of my LLM-augmented programming work, but I still had a few Cursor windows open on projects that I just started with Cursor and kept in Cursor due to inertia. But now I’m using Claude Code exclusively for all of my LLM programming assistant and vibe-coding needs.
Anecdotally, I’m not alone. By now Claude Code has exploded in popularity, with many users who would actually prefer a VS Code fork like Cursor putting up with the terminal aesthetic simply because Claude Code is a better tool (having said that I think there are now VS Code extensions that embed the Claude Code engine so you don’t have to touch that icky terminal if you don’t want to). It’s even penetrated the lazy vibe-coder zeitgeist, such that I’ve noticed the developers in my company who used to make slop PRs with Cursor and Copilot now use Claude Code instead.
Anthropic didn’t even invent terminal-based LLM coding assistants. I recall playing with the open-source aider</a> shortly after OpenAI released the GPT-3.5 API. I found it clunky and limiting and not smart enough to do anything meaningful, and went back to what was then the state of the art, copy-pasting text between a terminal window and the ChatGPT web interface. There were several terminal-based LLM clients of various kinds, most of which I never tried since Aider was such a bust. When I learned about Claude Code on Hacker News, I fully expected it to suck. Then I was blown away by its capabilities almost immediately.
I can’t say exactly how Anthropic was able to make such a great coding assistant that runs in the terminal. I have to assume it was built by a few independent developers without the supervision of a PM, as it’s hard to imagine a Product organization ever allowing a team to build a TUI as their first foray into LLM coding assistants. But kudos to those developers, they really nailed the text-mode UX, and it keeps getting subtly better over time without making jarring changes that completely break existing workflows and muscle memory (Cursor, I’m glaring at you!). Most importantly, though, somehow the agentic plumbing and specialized tools and prompts in Claude Code really make the Anthropic models sing. It’s still an overconfident and eager junior developer who will happily lie to try to please you, but it’s as if this junior developer has better instincts and is generally smarter than the ones that inhabit the other LLM coding assistant tools that I’ve used. That’s particularly wild because in fact it’s not true: Cursor uses the same Anthropic models as Claude Code, but sucks so much more!
I don’t have rigorous evals for performance on my coding tasks; it’s purely subjective vibe checks. If I were to invent a metric, it would be to analyze the chat transcripts with Cursor vs Claude Code and count how often I use expletives. I haven’t done that analysis but I would be shocked if the ratio isn’t at least 5:1 in favor of Claude Code. It’s not that the underlying model doesn’t make stupid mistakes all the time; it absolutely does. It fails to apply changes, fucks up tool calls, and is constantly forgetting what the current directory is. But if you let it run with auto-accept enabled (or for those of us who like to live `--dangerously-skip-permissions</code>, with training wheels completely removed (IYKYK)), it for the most part figures it out.`
`A month or two ago, I switched from metered use of Claude Code to their $200/mo Max subscription. The month before I switched, I spent almost $500 in Anthropic API usage, entirely due to Claude Code. The Max subscription is a great deal right now, to the point that I’m afraid Anthropic might come to their senses and jack up the price.`
`I may write a separate post to capture my current techniques for getting the most out of Claude Code, but suffice it to say that if you have not tried this yet, you need to stop reading this article and set up Claude Code. In the meantime, Anthropic has published some great practical guidance on how to get the most out of Claude Code based on their own internal dogfooding:`
Claude Code: Best practices for agentic coding</a></li> How Anthropic teams use Claude Code</a></li> Common workflows</a></li> </ul> Tools I’ve Discarded</h2> Cursor</h3> I was a pretty enthusiastic early adopter of Cursor last year, and encouraged several of my team members in Elastio to try it. I got a lot of utility out of it, which I recounted at length in my previous tool review post. But it has long-since been discarded. I already explained what motivated me to move to Claude Code in the Claude Code section above</a>; refer back to that section for those details. Part of the problem with Cursor is that, once Anthropic showed me what it could be like to access SOTA LLM programming assistants inside a terminal integrated with the rest of my terminal tools like tmux</code>, nvim</code>, and zsh</code>, having to touch the mouse and click around in Cursor felt like puttering around on a little scooter after a brisk ride on a Ducati. I didn’t know how much I wanted a terminal-based LLM assistant until I used one, and then I couldn’t go back. But the bigger driver of the change from Cursor to Claude Code wasn’t the improved ergonomics of the terminal, it was the fact that Claude Code just seemed to perform so much better. This was kind of surprising, since when I used Cursor it was almost always with Anthropic models, the same ones that power Claude Code. Why does Cursor suck and Claude Code is so good? I think the issue is the economics of the two products. When Claude Code first came out, you gave it an Anthropic API key and paid for every token it consumed. That got expensive very quickly, but it also meant that there were no tricks being played on Anthropic’s to economize on tokens to save money; quite the contrary the more tokens the tool uses the more they earn. Whereas Cursor got a flat $20/mo from me, but had to pay Anthropic for every token. They did have some rate limiting but clearly it wasn’t enough, and they had to do what they could to minimize the amount of context sent to Anthropic, resulting in much shittier performance in the tool. I believe they now have a different pricing model whereby you pay for usage, and for all I know maybe that has improved the quality of the LLM’s assistance, but I don’t care. Claude Code just feels more solid, and like it was built by people who used it for serious software engineering, while Cursor felt to me like the kind of tool you use to vibe-code some slop so you can performatively ship fast and call yourself “cracked” on social media while script kiddies download the API tokens your slop leaked all over the Internet. Tools That I Tried and Hated</h2> Gemini CLI</h3> Once Anthropic’s Claude Code had shown Product teams that there was an enthusiastic market for terminal-based coding assistants, the other big AI players rushed to ship their own me-too offerings. The one I most eagerly anticipated was Google’s CLI, which they ended up calling “Gemini CLI</a>”, after their series of frontier models of the same name. The Gemini Pro models, most recently 2.5 Pro, are widely hyped on Twitter and YouTube. They have something like a 1M token context window, and have some impressive multimodal capabilities. LLM grifters</del> influencers are constantly gushing about the galaxy-brain capabilities of this model. It’s hard sometimes to sift through the slop and figure out who is actually smart enough that a model being smarter than they are on some task that they gave it is actually a compelling endorsement. But the Gemini models are not a scam; they really do seem to perform well on programming-related tasks, at least in my limited testing. Back when I was still using Cursor, sometimes if it shat the bed running Claude Sonnet I’d switch over to Gemini Pro 2.5 and get a much smarter result, so I took it for granted that the Gemini CLI tool was going to be similarly competitive. As an added bonus, my company has some generous GCP credits, so Gemini CLI is effectively free for me, unlike Claude Code which I have to pay for. So all Gemini CLI needed to do was be about as good as Claude Code, and I’d switch to it. My God in heaven, it was not even close. Gemini CLI sucks. The UX sucks; it lacks the elegant but text-native polish of Claude Code. But worse than that, the performance on agentic coding tasks is…I think in this case it’s appropriate to use the word “retarded”. Just like I am amazed that Claude Code gets better performance from the same models that Cursor is using under the covers, I’m astonished that Gemini CLI is able to make Gemini 2.5 Pro suck as much as it does. I immediately configured it to use gemini-2.5-pro</code>, and gave it some simple Python and Rust tasks. Not even ones that Claude Code struggled with, just whatever I was working on at the time. It made stupid decisions, failed to make use of available information, simply forgot or ignored guidance in the prompt, and went in circles. It was utterly useless. Sometimes if the other models I have access to get stuck on something, I’ll try giving it to Gemini 2.5 Pro, but using the Vertex AI web interface. It typically gives me some useful output, even if it’s not able to solve the problem itself. So something about the way the CLI wrapper prompts or invokes Gemini seems to lobotomize it. Just an epic fail on Google’s part. My friends who are also eager early adopters of LLM assistants, some professional SWEs and some closer to a PM level of sophistication, have all reported the same results. It’s actually impressive how a seemingly-capable model can be made to be so stupid given the right scaffolding. Jules</h3> Jules</a> is another Google AI project, this one is an AI coding agent that runs on its own given a GitHub repo and a task description. Behind the scenes Google spins up a VM, checks out the repo, and sets loose an LLM agent in that sandbox environment where it can work as long as it needs to in order to accomplish the task. This is in the same product category as OpenAI Codex (no, not that Codex</a>), the widely hyped Devon, one of the dozens of different products sold under “Copilot”, and approximately millions of other frantic me-too AI startup cash grabs. It’s clear to me that, as a category, this is going to be a big part of the future of LLM coding assistants. There are a lot of advantages to this approach. If the tooling is good enough, agents could take a crack at dozens or hundreds of issues from the backlog, particularly chores like updating dependencies and making minor text changes, then produce PRs for human engineers to review. I’ve heard of teams inside OpenAI (meaning, teams with effectively infinite budget for AI spend) spawning 10 or more instances of these agents for a single task, then reviewing all of the solutions and picking the best one. Even if a problem is too hard for an agent to one-shot, you could leave PR comments just like you would with a human colleague, and maybe it’ll get it right after a few tries. That’s how I envision these tools working someday. Perhaps there are tools that work like that today. However Jules is not such a tool. Jules feels to me as if it were built by people who had no prior experience building software for a living. Or at least, no experience with the typical software lifecycles around backlog items, branches, pull requests, and eventual merges. If I’m being more generous, it feels like it was built by people whose bosses were anxious about getting their promotions in an era at Google where “AI” is the answer and no one gives a fuck what the question was, and who noticed that no exec had yet laid claim to building a background autonomous LLM programming agent, so they decided that this crude demo whipped up over a weekend to show the basic idea should be shipped to production without any regard to whether or not it fucking worked. It is an embarrassment, one which makes me feel actual pity for whoever works in the utterly dysfunctional org that allowed this to ship. I gave it very simple tasks, like to update a Node.js dependency and use a new field added in the updated version of the dependency. It made the code change easily enough, so I assume it’s using Gemini 2.5 Pro without whatever makes Gemini CLI suck so much. But everything about the interaction sucked: It was a pain in the ass to configure Jules to build my project; it wasn’t smart enough to use GitHub Actions, I had to use a very rough UI to give it build commands to run inside of its VM to set up the environment</li> The process of testing those build commands to see if they worked was clumsy and confusing, and they go in a tiny text area with minimal context clues helping you figure out what to type there.</li> When it thinks it has finished the task, you have the option to make a PR, view the branch, and that’s it. Did it completely fuck this up and you want to give it more guidance? Too bad. Do you want to make a PR, but then your GHA CI checks are all red and you want Jules to look at those failures and fix them? Too bad.</li> From time to time my auth token would expire and I would be unable to access Jules because I’m in Europe now and it’s US-only, even though my company is American and once I am authenticated it will work, I’d still have to use a VPN to get past the stupid geo-blocking to complete the auth challenge. Google of course has the technical ability to be smarter about this, but evidently no one cares enough to bother.</li> </ul> The “agent runs a branch in a sandbox and ships a PR” pattern is the future. Jules is not. Even if they somehow fix the leadership failure that led to this shipping, knowing Google, by the time it starts to get good they’ll kill it off. Tools I’m Still Using Daily</h2> These were all in the year-end tool review, although my use of them has evolved somewhat since then, they’re still daily drivers. Claude Desktop (Max plan)</h3> If I need an LLM to do something for me other than writing or debugging code, I first reach for Claude Sonnet 4 or Opus 4.1 in Claude Desktop. Here is a selection of prompts from my actual Claude Desktop history: Which breaker on this Hungarian panel is the stove? </li> Read the read me at https://github.com/etw11/DunedinPACNI and summarize </li> Give me a command to grow / to use the extra space on this device: (followed by lsblk</code> output from the system) </li> Who is playing at the Danube arena in Budapest tonight? </li> In my vmagent log output I see a lot of this [a bunch of log output snipped] What’s the problem? Suggest fixes </li> Help me troubleshoot an Ethernet issue. I have cat5e installed in my apartment by the builders. In each room are two rj45 jacks. All lines terminate in a wiring closet. I’ve crimped rj45 connectors onto the lines. Line labeled 3 goes to 3 jack, and 4 goes to the 4 jack. 3 and 4 are connected to an Eero pro 6e to its 2.5 gbit and 1 gbit ports, respectively. 3 (on the 2.5gbit port) is the wan and connects to the internet provider. 4 on the 1gbit port connects to a gige switch in the wiring closet to provide wired internet to some devices. The wan port works and has a 2.5gbit speed. The lan port detects no connection at all. I used a cable tester from the 4 cable in the wiring closet to the 4 jack, and to the other end of the patch cable connecting the 4 jack to the eero. All eight lines are good. However whether I connect to the 1gbit port on the eero or a desktop gigabit switch for testing purposes, I can’t get any link light. Not even a degraded 10 or 100mbit link. I’ve tested each cable separately with a cable tester. They all pass. I know the eero 1gbit port works as it was working in my old apartment two days ago. Help me theorize as to the problem and diagnostic next steps </li> Consider this tree: [output of tree</code> command in a directory with nested directories and files that have _</code> appended to their extensions] What’s a shell one-liner to rename each file in place, to strip the _ off of the end of the filename </li> </ul> My preference for the Anthropic models for these use cases stems from the fact that I use them for my LLM coding needs as part of Claude Code, and I have a Max subscription so I also have very generous limits in the Desktop app. I’ve never hit a usage limit, however I do hit outages much more often than I’d like. ChatGPT Desktop (Plus plan)</h3> Though my default go-to models when I want an LLM to do something for me or answer a question for me are still the Anthropic models, I keep a paid version of ChatGPT around for a few reasons: Anthropic is not very reliable; I often get network or server errors trying to submit prompts using Claude Desktop, and I’m not going to just wait around until Anthropic gets their shit together, so I fall back to an OpenAI model</li> My wife has been habituated to use ChatGPT so if I turn this off I’ll have to set her up with her own account or let her use my Claude account. Keeping this is the path of least resistance.</li> Sometimes the Claude models fail to perform well on a task and I’ll give the task to OpenAI. I especially liked the o3</code> model for advanced reasoning; it was very slow but seemed to be more resourceful and had more powerful reasoning capabilities. In the last week or so OpenAI removed all of those and now I only have GPT 5, which is making me strongly consider cancelling the subscription and giving Gemini a try.</li> If I have some random idea or complex question about how things work, I like to feed it to o3</code> and sometimes have a back-and-forth session as I explore it. In most cases I’m sure Claude Opus could perform well in this role, but I just fell into the habit of using o3</code> as the slow but wise sage that I can consult when I want to deepen my understanding of the world. With the removal of o3</code> and the forced migration to GPT 5, I may take that as an occasion to switch this use case to Anthropic.</li> It sometimes happens that Claude Desktop is busy on a research task and I have another thing I want to research at the same time. It’s easier to just Command-Tab over to ChatGPT and initiate the research there.</li> </ul> Here are some examples of prompts I’ve used with the various OpenAI models recently: On on the latest macOs. my macbook pro is configured to connect to my home wifi. when I’m docked to my thunderbolt dock, I also have a gigabit ethernet connection. How do I make sure that the ethernet connection is used when available, and wifi used otherwise? </li> [With some photo attachments] The first photo is a weird plastic thing that installs in the corner of the Schrack distribution panel cover in the second photo. When the circular component is turned it extends a rod vertically which operates as a hinge. This one has been broken off. Find me a replacement online from some place that ships to Budapest. Note that it failed this task; it claimed to find a parts kit but the kit did not contain the hinge that I needed.</li> </ul> </li> Consider the latest USB PD standard, which IIRC can deliver up to 240W of power. Would it be physically possible to make a mock cordless tool battery for major 18v and 12v platforms like Makita and DeWalt, where in place of an actual chemical battery is a USB-C trigger board connected to a USB PD power supply. Is there enough current and voltage capacity in the latest spec to provide the same amount of energy as the chemical batteries? </li> I have a DD1802H smart home RF remote control that is programmed correctly to control five motorized shutters in my apartment. It works well. I also have a DD1805H RF remote that I want to set up to exactly clone the DD1802H. That is, I don’t want to have to pair the 1805H with each shutter’s controller again; I want to make the DD1805H use the exact same signals that the DD1802H already sends, so that it will also work with each of the five motorized shutters. Figure out how I do that. </li> Consider this idea I’m writing up: [multiple paragraphs of text from a journal entry in which I describe an idea I had] I don’t want to build that if it already exists. Find what open source tools are available that do some or all of this, and review their features, strengths, and weaknesses. </li> </ul> Perplexity (Pro plan)</h3> I would estimate that about 80% of the searches that I would once have done with Kagi (for which I also maintain a paid subscription), I now do with Perplexity, either Pro or Research. As the Internet descends further and further into the abyss of AI slop and weaponized SEO hacks, a tool that sorts through search results and gets to the information I actually want is well worth the $20/mo. Unfortunately, all of the LLM caveats apply. Not only does it outright hallucinate from time to time, but it’s not the best judge of character, and will regurgitate claims from pages matching search terms uncritically. It’s not unusual to have to fall back to Kagi for searches where Perplexity gets…well, perplexed. The Research feature is all of that, but more so. It’s incredibly valuable for finding sources of information that I wouldn’t have found myself without exhaustive searching, but it’s not at all good at separating out the bullshit from what’s real, and it often gets confused by competing claims in search results. Often, the only valuable output from a Research run is just the list of links it’s drawing on, which I can then read myself to get the information I needed. I would share a few examples from recent Perplexity activity, but the Perplexity Electron app is user-hostile and prevents me from selecting and copying text from a Perplexity session. All I can do is use the Share feature to generate a link to it. The cynic in me suspects that some sociopath in Product had some incentive to drive engagement on shared Perplexity sessions and realized making those links the only way to share information would tweak the stats. Damn you! As much value as I get from Perplexity the service, I would welcome a complete rewrite of the Perplexity Mac app. I hate it. The inability to copy-paste text is unforgivable, but it seems that whenever I Command-Tab over to it after it’s been out of focus for a few hours, I get the spinning volleyball of death and force quit rather than wait for it to right itself. This is on a 2024 M4 MBP with 48GB of RAM where absolutely nothing else, including DaVinci Resolve Studio, lags at all, so it takes a special kind of idiot to suck wind on this system. That said, if there is a credible competitor to Perplexity (other than the web search and research features in the other LLM apps that I use), I’m not aware of it. I hope to see more competition in this space, as I don’t think Perplexity is particularly great at what it does, the mere fact that it works at all has value but I’m sure it can be done better. Predictions for H2 2025</h2> Background coding agents will hit mainstream workflows. One vendor will ship a credible “branch-per-issue” loop that knows about existing CI/CD infrastructure; the rest will copy it. Jules will continue to suck until Google kills it.</li> The Claude Max plan will be enshittified somehow. Anthropic already announced more restrictive usage limits for Pro, adding not only limits within five-hour windows but also weekly usage limits. They claim that this won’t impact most users, and for now this is specific to Pro and not Max, but given the economics of hosted LLMs I think further arbitrary usage restrictions are inevitable.</li> Vibe-coded slop will get worse. The economic and psychological incentives for lazily turning off your brain and vibe-coding slop that you then ship are simply too strong, and the consequences of doing so still too diffuse and remote. Even if there were zero hype around LLMs and no frenzy of AI FOMO intoxicating every exec and investor on the planet, I think you’d still see widespread vibe-coding simply because it’s easier than having to think for yourself, and you can potentially work multiple jobs at once shipping slop at each one. Add in the AI mandates from execs and investors and I see no hope that software doesn’t get much worse very quickly. I don’t think we will ever recover, but I do hope at some point in the coming years the damage will be sufficiently undeniable and MBA programs will teach enough cautionary case studies that most execs and engineering managers will have learned not to tolerate vibe-coding slop-merchants (or, at the very least, the vibe-coding slop-merchants will be forced to make some minimal effort to hide the telltale signs of slop which might incidentally make them less sloppy).</li> LLMs will continue to not be capable of replacing software engineers, and will continue to get more useful in the hands of competent professionals. More leverage for pros; more mess from everyone else. This is a double-edged sword. I’m glad that there will still be professional opportunities for people like me in the future, but I also am pretty sure I will not enjoy those opportunities due to all of the aforementioned slop.</li> LLM performance will gradually increase, which will continue to give me the ability to knock out quick tasks and explore random ideas that I would never have been able to justify spending my own time on before LLM coding agents were developed. This is the positive side to the negative take in the previous bullet. I’m very excited about what I can do when augmented by a SOTA LLM coding agent. My lament is entirely concerned with what others will do with the same tools.</li> Valuations on AI plays will continue to be insane, and I will continue to seethe with jealousy and resentment as I compare my equity stake and comp at Elastio with packages at even the dumbest me-too LLM wrapper startups.</li> </ul>

With, By, or For - Three ways I use LLMs as a software engineer 2025-05-18T00:00:00+00:00 Like most everyone in my company (and, if you’re reading this, probably yours as well), my colleagues and I have been enthusiastically adopting AI tech, particularly LLMs, since ChatGPT 3.5 first showed us what was possible with next-token prediction and insane amounts of compute and training. In the 2+ years since then, I’ve used quite a few tools and numerous models, and tried to find how to get the most productive value out of LLMs as a software engineer and engineering leader. In that time I’ve learned quite a bit about how to get value out of LLMs without succumbing to either of the most common failure modes: outright dismissal of modern LLMs as “stochastic parrots”, and breathless idiotic “AI cloned AirBnb in one shot devs are NGMI!!!” clickbaiting. This article is a distillation of my thinking on using LLMs as a software engineer, as of May 2025. Disclaimer: AI technology is changing very rapidly. I expect I’ll read this article five years from now and scoff at how primitive our tools were and how clueless I was. Consider this article a snapshot of the state of my use of LLMs as of this moment in time. I’m neither an AI doomer nor an AI hypebeast, I’m a working SWE and engineering leader who has to deal with these technologies whether I like it or not and is trying to find ways to get the most value from LLMs and avoid drowning in AI slop. tl;dr - Three ways I use LLMs as an SWE</h2> I break down the ways I use LLMs as an SWE into three categories, in the order in which I discovered them: Code is written with LLMs - By this I mean I’m doing the thinking and writing the code, but consulting with an LLM on specific problems or to explore an idea that I’m not yet ready to commit to coding. What ten years ago would have been done with a mix of Google and Stack Overflow, I now do with LLMs. I use Claude Desktop, ChatGPT Desktop, and Perplexity in this role.</li> Code is written by LLMs - This refers to modern agentic coding tools, including Cursor’s Agent mode, Claude Code, and the like. In this use case, the LLM is writing the code, I am driving it with prompts. I may jump in and modify code myself here and there, but the majority of the code is authored by the LLM.</li> Code is written for LLMs - This refers to choices of tooling, language, infrastructure, and documentation with the express purpose of improving the productivity and accuracy of agentic coding tools. The agentic tools don’t know enough to do this themselves, so you as the developer need to go to the effort to give the agents what they need but don’t yet know they require in order to be productive.</li> </ul> None of these is better or worse than the other; you can do stupid things and hurt yourself with all three of these techniques, and you can get value out of LLMs without using any of them. This is the combination that works for me in May 2025. The rest of this article is an elaboration on the three techniques. With</h2> This was the original use case for LLMs in the ChatGPT 3.5 era, when all we had was the chat interface. You’d prompt it to do something, it would spit out some code or a command, which you’d copy-paste or modify or just run, if you got an error you’d copy-paste that back into the session, ad infinitum. Or you’d use it as a stochastic Google, asking it questions that it hopefully will produce the right answers to. As far as I can tell, for the vast majority of people who use AI, this continues to be the primary workflow. This can go badly wrong when you don’t understand the limitations of the LLM, but it can also be immensely helpful and save a ton of time if you know what you’re doing. For my day-to-day coding with LLMs, I use Claude Desktop and the latest thinking model, which as of now is Sonnet 3.7 (update: now Sonnet 4 and Opus 4 are out). If I don’t like what I’m getting, or if Claude is down or I’m throttled, I will use the ChatGPT Desktop or Mobile app. Both of these are $20/mo and are well worth the price for the value I get. If the task involves up-to-date information or searching the Internet is for some reason key to success, I also have a Perplexity Pro subscription which I use constantly. Here are a few real-world examples pulled from my recent history in the aforementioned applications: (Claude) - Getting recommendations on how to set up a new Typescript project in 2025 using the latest tooling and tsconfig.json</code> values</li> (Claude) - Feed in an OpenAPI specification for an API that is very poorly designed, pointing out some of the things I don’t like about it, and getting a more comprehensive list of problems in Markdown format with specific endpoints called out</li> (Claude) - What are the options for automatically generating OpenAPI specifications for REST APIs built on Node with Fastify (as part of my effort to unfuck the aforementioned poorly designed OpenAPI spec)</li> (Claude) - Explain how FF3-1 format-preserving encryption and Feistel networks work, after I ran across the concept somewhere and wanted to understand how it actually worked. This ended up being a long conversation with many follow-up questions after which I felt I understood the concept well enough to evaluate its use for my particular case</li> (Claude) - How to make Jest tests fail with a meaningful error message in a particular failure case that I wasn’t sure how best to represent (I have almost no experience with Typescript or Node tooling so this was a noob question)</li> (Claude) - Write a shell script to list processes by the tmux session and window that they are running in. It took a lot of back-and-forth to get something I could live with, but I’m reasonably happy with the result</a>. I don’t often need this but when I do it’s great having it available. Sadly this needs to work on Mac as well so it’s limited to some pretty old Bash syntax and thus performs very poorly.</li> (OpenAI) - Refresh my memory on random forest networks, and how they are implemented using modern Python ML frameworks</li> (OpenAI) - Paste in an error from the Rust compiler and help me understand what the problem actually is (interestingly OpenAI o4-mini-high</code> utterly failed to analyze this correctly and came up with a completely ridiculous but plausible-sounding explanation for the cause; thankfully I’m a very experienced Rust programmer so I was able to recognize this and went about solving the problem myself)</li> (Perplexity) - Search for help solving a problem with Claude Code caused by a new release that didn’t work with Google Vertex AI yet</li> (Perplexity) - Figure out how I can install a Typescript “binary” program into the PATH on Windows using pnpm</code></li> (Perplexity) - Find sources for a large Ukrainian language corpus as part of a research project related to detecting tampering with data</li> </ul> By</h2> Starting in Q4 last year, “agentic AI” became all the rage, and agentic features started to appear in Cursor and similar tools. AI influencers fell over each other to be the first to state the obvious, that 2025 was to be the “year of agentic AI”. In February 2025, Andrej Karpathy christened the name vibe coding</a> to refer to the low-effort generation of AI slop code that was already by then rampant. AI grifters on YouTube and X performatively gaped at the ease with which primitive agentic coding tools turned a screenshot of some web app into a React application, heralding the end of software engineering and the urgency with which one must join their Patreon or perish. If you’ve experimented with these tools, you’ve likely noticed how quickly and confidently they produce garbage code that you wouldn’t accept from the greenest junior developer. You can be forgiven for dismissing agentic coding tools as gimmicks hyped by charlatans who need to somehow justify their absurd VC investments, for indeed they are in many cases exactly that. However, I have been able to get some valuable output from them, and whether you like it or not your laziest and least-capable colleagues are churning out code written by AI anyway so you may as well come to terms with it now. As of right now my go-to agentic coding tool is Claude Code, although I still pay $20/mo for Cursor and occassionally use it still. Claude Code uses the Anthropic APIs, which are billed per token, so comparing it to the $20/mo Cursor subscription isn’t really fair, but I’m doing it anyway. My company has some generous Google GCP credits, and Claude Sonnet 3.7 is available via Vertex AI, so for us in particular Claude Code is “free”, in the sense that it doesn’t use up any of our runway. Paying Cursor per token plus their 10% markup would happen with actual dollars, and Claude Code works very well for me so I haven’t put any effort into exploring other options yet.</del> UPDATE: It turns out that GCP credits do not apply to third-party models hosted in Vertex AI. Google’s documentation on this is weaselly hence the confusion. I racked up a $500 Anthropic model usage bill on GCP in June, which our credits did not cover. My advice as of August 2025 is to just pay for the Claude Max plan if you want to use Claude Code in anger. Either way it’s more expensive than $20/mo for Cursor, but it’s my blog and I can make unfair comparisons if I want to. Cursor’s agent mode has come a long way since I first mentioned it in my year-end GenAI tooling review</a>. There’s even (finally) a background mode so you can have multiple agents churning on tasks. However, I have grown very tired of the throttling on Cursor when I use up all of my “fast” credits, which happened usually within a few days of the start of the billing cycle. I also get the sense that Cursor is motivated to minimize the amount of tokens that they pay for on the $20/mo plans which may explain the poor performance I experienced. But Cursor is a VS Code fork, and sometimes I prefer those ergonomics to that of the terminal, which is when I still find myself reaching for Cursor. Also, since I do most of my day-to-day work with Claude Code, Cursor throttling is less of a problem for me. As for how to get decent code written by LLMs, the best practices in the Anthropic Claude Code best practices</a> doc are what helped me finally get some decent results. That doc is very specific to Claude Code, but the section “3. Try common workflows” seems like broadly applicable guidance that will improve results with other agentic coding tools that work similarly to Claude Code. Thanks to that Anthropic guidance, I’ve been able to get several useful results out of LLMs that were done faster than I could have done on my own, even taking into account rework and time spent reviewing and correcting the code. Here are a few examples: Write some complex Python scripts implementing an Apache Beam job on Google Dataflow. I didn’t know anything about Beam or Dataflow before I started, therefore the bar for “what I could have done on my own” was so low that even with several stumbles the LLM was able to get me something good enough much faster than I could myself. Also these are internal tools for testing models, they’re not customer facing nor are they mission-critical, so I felt more at ease mostly vibe-coding the scripts.</li> Read a complex and very poorly constructed Typescript codebase to figure out how some APIs work. I am only vaguely familiar with Typescript and my web development days are 15 years behind me already, so having the agent explore the code, summarize what it found, and point me to actual code lines where I could see things for myself was a huge time savings.</li> Plan, execute, and test a substantial refactoring of aforementioned Typescript codebase to correct a pretty serious structural deficiency in how it was constructed. I can’t emphasize enough the importance of “plan” here. In fact this was done over multiple agent sessions; the first one was entirely dedicated to authoring and refining a Markdown plan document. By the time the document was done, the context window was already full. Each subsequent step was implemented with another session and a fresh context window, but that was fine because I just instructed the agent to read the plan doc, and pick up where we left off before. Here again, if I were a master Typescript REST API developer, I’m not entirely sure that the agent would have been faster than me doing it myself. But as it was, the agent gave me the ability to quickly and thoroughly refactor a foreign codebase in a language that I don’t know well, in just a day.</li> Write a Rust program to do a complex analysis on a memory dump. I am a very strong Rust programmer, I could easily do this one myself, but in this case it was a weekend, my mental energy level was low, and I wanted to try an experiment using an agent on a language and stack that I knew very well. If I had been fresh and highly motivated, I definitely could have finished this task faster than the agent did, but under the circumstances having the agent was the difference between doing it and not getting around to it. It didn’t do a very good job initially, but thankfully the Rust compiler is so fastidious that I only rarely had to intervene to nudge it in the right direction. This program was also a research project and doesn’t operate on customer data, so I had no reservations at all about vibe-coding it.</li> </ul> If you do nothing more than follow the Anthropic best practices with Claude Code and make a good-faith effort learn the nuances of how the various coding models work, I think you’ll get good results at least some of the time. This is especially true if you use agents to do tasks that otherwise would not be done at all for reasons of mental energy or familiarity with a codebase or tech stack. However, I would also urge you to augment the Anthropic best practices by investing heavily in the technique described in the next section. Your codebase needs to be written for AI, since it’s probably inevitable at this point that at least parts of it are going to be written by AI. For</h2> Already in last December’s year-end GenAI tooling review</a>, the kernels of what became “coding for AI” were present: The corollary of the previous bullet is that generating docs optimized for LLM consumption will be much more important, particularly for new tools and languages. I think it’s inevitable that software development agents will need to get much better at looking up documentation, and when they do the extent to which that documentation is easily consumed by whatever mechanism they use will be important. Right now it seems like dumping all documentation content into a big Markdown file is a pretty good approach, but I bet this will be refined over time. This applies not just to developer docs but also end-user docs as well. On the plus side, perhaps this will finally be the death of product docs locked away behind a login?</li> </ul> </blockquote> In the intervening 5 months working with agentic systems, it’s become abundantly clear that that prediction was right but also insufficient: in fact, not just docs help LLMs, but anything that can be invoked as a tool that will provide actionable feedback on their output. In fact, it turns out that the things I’ve been doing throughout my career to harden a code base against the predations of eager juniors (and incompetent offshore “experts” brought in for the latest “this-time-it’s-different” management cost-cutting scheme, but I digress), also go a long way to making agents more useful. Every programmer I know who was done anything with LLMs in the last two years has inevitably characterized the experience as that of working with an eager junior, tireless and overconfident and all too willing to lie to you if it thinks that will make you happy. If you, like me, enjoy the experience of mentoring a promising and eager junior as he or she grows into a more capable programmer, then you will probably protest that a stochastic parrot wrapped in an agentic framework is something entirely different and qualitatively inferior. I won’t argue that point, but just like an eager junior (or the latest outsourcing scammer), LLMs have no actual understanding of anything they write, and they lack any judgement by which to evaluate what they have built. If you force them to get their code to pass a type check or compilation step, a linter, a beautifier, unit tests, integration tests, maybe some dynamic analysis, you automate much of the tediuous and error-prone verification work, so that by the time it gets to you for review you at least know you won’t see any mistakes any previous steps can catch. In the case of coding agents, this isn’t just a way to spare yourself the brunt of their vibe-coded stupidity. In many cases it seems that this feedback cycle somehow guides the agent along a random walk to more likely arrive at an acceptable answer. I suppose if you think of the underlying LLM as a stochastic parrot, then it makes sense that the more guardrails you put in place the more values the stochastic parrot will sample from the parts of the desired region of the solution space, thus increasing the odds that it eventually produces something that’s at least acceptable. Here are some of the actual things I’ve put in place in codebases where I want to enable (or in some cases, lack the power to prohibit) productive use of coding agents: Maintain a CLAUDE.md (for Claude Code) or Cursor Rules (for Cursor) or whatever your agent uses. You can make agents read a README or some other document, but they will always look at the agent-specific documentation and load some or all of it into the context window. This is where you should put exactly the kinds of instructions that you would write for a new junior. Explain what the project is, what it does, how to compile it, what the coding standards are, how to run the tests, etc. As you see the agent doing something stupid, update the rules doc accordingly. For example, I had a Python project where Cursor kept trying to add dependencies by just doing a pip install</code>, but this was a uv</code> project with a pyproject.toml</code>. I wrote a Cursor rule that explicitly forbade ever running pip install</code> or creating a requirements.txt</code> and stating that all dependencies must go into pyproject.toml</code>. On a Rust project we used workspace dependencies but agents always try to add dependencies to the Cargo.toml</code> for the crate that will use the dependency, so I wrote a script that worked like cargo add</code> but worked how I wanted, and put some text in the rules file forbidding direct edits to Cargo.toml</code> when to add dependencies and requiring the use of script instead. As with bargain-basement offshore multi-tasking engineers, you will likely never stop discovering new agent misbehavior that needs to be explicitly prohibited. But if you keep up this discipline you will be rewarded with better agent performance. And unlike those bargain-basement offshore multi-tasking engineers, I find that coding agents actually try to follow the rules at least 75% of the time. Plus, these days it’s a virtual certainty that your gullible management hiring the cheapest offshore “help” is going to end up unknowingly hiring human wrappers around a coding LLM anyway, so you’re helping their performance as well. </li> If something can be done deterministically, do that - By this I meant that it’s better to have an entry in the agent’s rule file that says “always run frobnulator.sh</code> to check your work for errors before considering a task complete” where frobnulator.sh</code> contains all of the commands the agent should run, than to write “always run foo</code> and also bar</code> and then baz</code> to check your work for errors before considering a task complete”. It takes fewer tokens, and gives the agent less opportunity to fuck up. Maybe it ran foo</code>, that passed, then bar</code> failed, then it makes a fix for that failure which happens to break foo</code> but it doesn’t re-run foo</code> and declares victory when bar</code> passes. As it is, it will not 100% of the time run frobnulator.sh</code>, but the odds are better when it’s just one simple command to run. In a Rust codebase recently I used just</code> to make a command vibecheck</code> that ran a multitude of checks, cargo check --tests</code> and clippy</code> and a few quick unit tests and some more expensive unit tests and some integration tests. In the agent’s rules file I just had “Run just vibecheck</code> to compile and run all tests after every task”. Using just</code> for this is especially great because just</code> is smart enough to walk up the directory tree until it finds a Justfile</code>, so no matter how confused the LLM is about the current directory as long as it’s somewhere in the workspace this will work. This is also helpful in cases where there are multiple ways to run things, like in a Python project where you want to use uv run</code>. Agents can try to get clever and run a tool the wrong way or with the wrong version of Python, which will cause them to get wrapped around that axle, so it’s better to constrain them as much as you can. </li> Do as much static type checking as possible - Let the compiler or type checker do as much of the verification work as possible to catch LLM fuckups. If it’s a Python project, use mypy</code> and require types everywhere. If it’s Typescript, configure the compiler as strictly as possible and prohibit any</code>. If you have the pleasure of working on a Rust codebase, make sure you run not only the compiler but also clippy</code> with at least the default lints, and consider enabling some additional ones on a case-by-case basis. In all cases, make sure whatever compiler or linter you use is configured to fail the build on any warnings as well as all errors, and obviously make sure that running this is a stated requirement in the agent’s rules file. </li> Automate all the tests - You really should be doing this anyway, but coding agents make it even more valuable to have reliable, quick, and automated tests. If you have flaky tests, they will absolutely drive a coding agent mad. If you don’t have any automated tests, then the coding agent will very happily “fix” things by disabling authentication or turning off a feature or silently eating an exception. The good news is that one of the most compelling use cases for coding agents is the generation of unit tests. The Anthropic best practices guide that I linked above goes into their recommendations on this in more detail, but suffice it to say that in 2025 there is no excuse for a project not having at least some basic unit tests covering the majority of the application’s functionality. If engineering leadership or the Product org push back and insist that your JIRA tickets are higher priority, show them this post and explain how they are failing to capitalize on AI-powered efficiency gains by not having reliable tests, and ask them how they will explain to the CEO why they are not utilizing GenAI to drive engineering efficiencies. If that doesn’t result in an immediate test automation mandate then you and I clearly have very different professional milieus. The more bugs you can protect against with automated tests, the better. I think of it in adversarial terms: there’s an AI run amok intent on ruining your project with subtly-wrong vibecode, and your best defense is your own army of (deterministic) machines to catch stochastic fuckups before they can land in master</code>. As an added bonus, these tests help you and any other humans on your team as well, none of whom are infallible. </li> Don’t assume the agent has run any of your checks - Most of the time the agents will follow instructions about mandatory checks, but in my experience even with Claude Code it’s not 100% of the time. So you still need to have CI and that CI still needs to run at least all of the same checks that the agent is required to run, and if those checks fail it still needs to block the pull request. </li> CI should be able to deploy the whole stack and test it - Arguably this is just part of the previous point, but I want to call it out specifically. In many projects, for various reasons, it’s not practical for developers to run the entire product or solution on their local systems. If they can, that’s ideal, because then you can do that as part of the “automate all the tests” point and you’re done. But even if not, there’s still immense value in having a CI step that deploys the whole solution and runs it, as realistically as possible, with some automated tests that can verify the actual behavior of the whole system. I know, if you don’t have this yet it’s quite a bit of effort to set up. But coding agents now are starting to move out of the developer environment and onto separate dedicated compute, where they grind asynchronously on a Github issue and prepare a pull request for you to review at your leisure. That means that if you have CI checks that can catch problems, even if they take 30 minutes to run and aren’t practical to make the agent do itself, as long as they block the PR from landing and provide some meaningful failure message when they break, this can still save some of your time by catching stupid things that the other layers of testing somehow missed. And here again, this is a good thing to have even if GenAI turns out to have been an opioid fever dream and you go back to writing your own code. </li> Accept that vibe-coded garbage is going to get into the codebase and know what to do when that happens - If you are one of those teams that is always shipping, and do your debugging in prod, you hopefully have already ensured that you can quickly recover from a bad push, whether it’s a 100% organic human-originated fuckup or the cheap AI-powered kind. What changes when using coding agents is the volume of (often dubious) code that can be produced. This phenomenon is addressed in a great article titled The Coming Knowledge-Work Supply-Chain Crisis</a> which I urge you to read. In the past, without any GenAI tools, teams could do everything right, and shit still broke in production due to human error. Coding agents, if they live up to even a fraction of their creators’ hype, will dramatically increase the amount of code being produced, and I see no reason to believe that code will be any less fallible than the code humans write; I’m confident that it will in fact be worse. So you need to be able to recover from bad commits quickly. In my case with Elastio most of our product is installed in customer accounts, so we don’t have the luxury of pushing that code to prod dozens of times a day. That fact, combined with the nature of Elastio and its role in our customers’ security postures, means there is a high bar for quality in much of the code base. For this kind of product, you should consider the provenance of all code very carefully and make it clear which named developer is personally responsible for which pull request, regardless of where it came from. I would suggest that low-effort vibe-coded PRs that waste reviewers’ time and risk the integrity of the product should be prohibited, and perpetrators punished when caught, although I’m also aware in most orgs that’s just a fantasy. </li> Use a beautifier but only at the end of tasks - LLMs produce some really ugly code, particularly if you are sensitive to things like trailing whitespace. In the LLMs’ defense, humans are also pretty terrible at this. Fortunately we have beautifiers now. Definitely use one to check at the CI stage that new code has been formatted according to the standards. But there’s a nuance that I ran into when using this with Claude Code specifically, that probably applies to other agents as well. When the agent is working on a file, it loads part of the file into its context window, which lets it rewrite that part of the file with some other content relatively efficiently. There’s a safety feature in the tool that Claude Code uses to write to a file, that detects when what the LLM thinks is currently there doesn’t match. There are very good reasons for that check, which I’m sure is there to avoid an otherwise common LLM failure mode. However if the beautifier has run since the LLM wrote the code, this means it will fail making subsequent writes and thus has to load the file again. That not only slows down the agent but also uses up more context window tokens. So resist the urge to make the beautifier part of the standard tools that you make the agent run as part of its work. They don’t provide any feedback that would help the agent anyway, so it’s better to do that in a commit hook, or as some explicit final step separate from the code checks. </li> </ul> I’m sure over time we’ll discover more techniques for building guardrails to keep the agents on the road. I personally would love to find a way to detect the zero-value comments that LLMs are so fond of injecting into the code, as well as the deletion of valuable comments that for whatever reason they seem to deem unnecessary. Conclusion</h2> The tone of this text may suggest that I’m an AI skeptic and possibly even a curmudgeonly gray beard gatekeeper nostalgic for the days of punch cards and walking to school uphill both ways in the snow. That is very much not the case. I am bullish on AI tools in general, I already get a lot of value in the tools as they exist today, and it seems certain that they will continue to increase in capability. But I am also a jaded and cynical SWE who can clearly see the lazy and careless use of AI by people whose unaugmented abilities are low enough that they are not capable of evaluating the slop that their GenAI tooling is producing in their name. This is already wasting my time by making me read LLM-generated slop docs and looking at vibe-coded PRs from idiots who don’t know or possibly don’t care how obvious it is that their incompetence in their actual job is matched only by their inability to prompt LLMs to do their job for them. I’ve written this article for others like me who feel the same way. You cannot stop this AI revolution, you cannot hide from slop, but I urge you to keep an open mind and try to regard this new technology with the wonderment that I remember in my youth as I first discovered programming and then the Internet. There is real value to be had here, and it’s worth your time to figure out how to take advantage of it. Statement of Provenance</h2> I wrote the text of this article entirely myself, by thinking thoughts and translating those thoughts into words which I typed with my hands on a keyboard. Any emdashes, proper grammar and spelling, or use of the words “underscore” and “delve” is entirely coincidental. OpenAI’s GPT 4.5 model was used to copyedit a draft of this text for typos and sloppy or lazy writing. Its feedback was read by me with my eyeballs, the proposed changes were considered by me with my own brain, and changes that I agreed with were again made with my own hands. All thoughts and clever turns of phrase are my own, and do not necessarily reflect the opinions of Elastio, nor did they emerge from a high-dimensional latent space on NVIDIA silicon. My take on the American foreign policy shift on Ukraine 2025-03-04T00:00:00+00:00 The American political Left and Right are full of takes on the recent public takedown of Zelenskyy by Trump and Vance in the White House. I’m a natural born American citizen and am politically receptive to America First governance in principle, and also a legal permanent resident of Ukraine where I have made a life for myself since 2018 and where most of the engineers I employ in my startup still reside. I should also note that my wife is Ukrainian. Therefore I have an unusual point of view in this debate, and I can’t restrain myself from weighing in. A review of my posts and social media activity will reveal a spoiler of sorts: I fully and enthusiastically support Ukraine for both selfish and patriotic reasons, and if I were president I would provide such lavish military aid to Ukraine that she either wins or Putin makes his final mistake on this earth and throws a hot rock at the problem. I’m not writing this post to reiterate that position, which I think I’ve made clear already. I feel compelled to write this post because the discourse from the Left and the Right is so histrionic and misinformed that I have no choice but to intervene. </a> Over the last 8 or so years, it’s become a cliche to claim that anyone who is against a policy position held by the American left is a Russian bot. It’s about as worn out an insult now as “fascist”. This is unfortunate, because there actually are fascists in the world, and there actually are Russian bots operated by Russian intelligence and influence operations to sow discord and spread Russian propaganda to serve their own interests. Diluting the impact of these terms reduces our ability to recognize and call out legitimate threats. I see that happening now as it relates to the invasion of Ukraine. In my own feed on X, anecdotally I have recently observed a lot more anti-Ukraine, anti-Zelenskyy commentary that is repeating Kremlin talking points. Some of this is surely actual Americans sharing their actual thoughts (perhaps influenced by contact with prior propaganda campaigns, perhaps not), and a lot of it is actual Russian bots. There’s no point in arguing with bot accounts; they are tireless and they outnumber humans. But it was unwise and ultimately destructive to the cause to just write off any skepticism towards Ukraine, her cause, the correct level of economic and military aid, or the winnability of the war as the mindless ramblings of Russian bot accounts and Putin’s fellow travelers. Now we get to hear from the people who came to power on the shoulders of a base that is at least skeptical of our role in the war in Ukraine, and it sucks. Unfortunately I think Zelenskyy or at least those advising him made a grievous error in their approach to the new American administration. It’s so obvious to Ukrainians that their cause is just, the cause of freedom and democracy in the face of authoritarian aggression, that to question it is to reveal oneself as an agent of Putin. I fully understand that position, and were it not for my exposure to patriotic Americans who are skeptical of foreign adventures, I’d probably hold the same position. Unfortunately, Ukraine needs foreign support, and the European nations with their hollowed out military industries appear to be unable to shoulder that burden on their own. Zelenskyy dismissed any skepticism about Ukraine aid or talk of peace when he should have engaged with it behind the scenes and before the White House event that felt more like a public execution at the end. To my many friends in Ukraine asking me to explain Trump’s policy position: I can’t; not in logical terms anyway. You shouldn’t have to justify why American and European interests are served by a Ukrainian victory and threatened by a Ukrainian defeat. But you should be used to it by now. The US and the European nations have been slow-walking aid since the early days of the invasion. Remember 700 Javelins? Remember how HIMARS, ATACMS, Abrams, and F-16s were out of the question? And that was with an administration whose party was enthusiastically pro-Ukraine and anti-Russia, but afraid to “escalate” and end up in a nuclear conflict. You should also know that American intervention in World War I and even World War II was controversial at the time. If Japan had not bombed Pearl Harbor, we might not have entered WWII at all. Was that because Americans were pro-Nazi or pro-Hitler? No, clearly not. But Americans are safe behind two oceans on a vast and rich continent, and for many it’s hard to understand why they should pay for, and in some cases send their children to die for, foreign causes. It isn’t fair that you should have to make the case for your own continued existence, but geopolitics are not fair. Your continued existence now depends on Ukraine’s ability to make this case repeatedly and convincingly at every turn. If there’s one thing that I wish my Ukrainian friends and the Ukrainian leadership would take away from this, it’s that Americans skeptical of our aid to Ukraine (yes, in many cases influenced by Russian propaganda) are not automatically KGB agents or lovers of Putin or victims of Kremlin blackmail. You can act like there’s no possible way that a good person could make up their mind that America shouldn’t support Ukraine in her struggle, but if you do that you are not helping your cause, and you play into Russia’s divide-and-conquer strategy. For you it’s a question of your own survival, but for most Americans with no direct ties to Ukraine it’s a question of what to spend our tax dollars on, at a time when American finances are unsustainably red. And for those on the American right who have either recently adopted Trump’s anti-Ukraine position, or have always been skeptical of Ukraine aid, I try (often in vain) to explain the following: Ukraine didn’t start this war, and neither did US policies. I’ve seen claims made that this war was an inevitable consequence of the encroachment of NATO on areas of Russian influence, that the US promised not to grow NATO beyond its 1994 borders, and that the CIA orchestrated the overthrow of Yanukovich which precipitated the invasion of Crimea and Donbas in 2014 and eventually the full-scale invasion in 2022. I’ve recently read propaganda claims to the effect that Putin offered Zelenskyy just terms for peace in 2022 (“just” here in the Russian sense meaning “fuck you later”) and he didn’t take them therefore Ukraine is responsible for this war now. I think it’s useful to try to understand the Russian motivations for the actions that they take, just like it was useful post-9/11 for Americans otherwise unfamiliar with Islamic terrorism to ask why Al Qaeda hated America so much. But when the result of that analysis is “Putin did X because he felt Y when the US did Z”, it does not follow that “X is America’s fault”. NATO didn’t conquer and occupy Estonia or Poland or Finland or any of the new members; they all petitioned to join NATO because of their long and painful experience of Russian aggression and dominion, in the hope that NATO Article 5 and the nuclear security umbrella would protect them from any future aggression. NATO countries do not preemptively invade and conquer their peaceful neighbors, and if one were to try to do so there would certainly not be any Article 5 security assistance from the alliance. Russia’s talking points to the contrary are their own internal propaganda dating back to the Soviet era, and do not justify Russian aggression in Ukraine. Russia started this war. Russia could make this war end tomorrow by withdrawing to Ukraine’s 2022 borders. That they do not do so is Putin’s decision, and is not the fault of either the US or Ukraine. </li> Zelenskyy and Ukrainians in general really do want peace. Ukraine has already lost a devastating number of her best people, both at the front and to Russian attacks on civilians. Ukrainian men of fighting age not already in military service live in fear of conscription and a violent death or dismemberment at the front. No one has greater desire for peace than the Ukrainians, the victims of Putin’s war of aggression. What Zelenskyy opposes, and what the Ukrainian people overwhelmingly oppose, is the typical Western idea of “peace”, which means the bad guy signs a document and makes promises to not be bad anymore, democracy is saved, business as usual can resume. They oppose it because they understand perfectly well that Russian leadership generally and Putin specifically will sign or promise or swear whenever it’s in their interest to do so, and then renege on their obligations as soon as it suits them. So a peace in which Russia and Ukraine sign documents saying they won’t fight and Russia won’t invade any further and Ukraine gives up the territory she lost is tantamount to surrender, because Russia will get what she wanted in this war, re-build her military capacity, and come back at her leisure to finish the job. Hand-wavy “security guarantees” won’t cut it, which is why so many Ukrainians long for NATO membership. It’s obvious to me that Ukraine will not be admitted to a NATO in which America is a member, but it’s understandable why that’s their desire. Short of Article 5 treaty obligations, there is little historical reason to believe the American or European forces would die on the battlefield to save Ukraine in some future Russian revanche. </li> Ukraine is not a dictatorship and Zelenskyy is not a dictator. It’s true that Zelenskyy declared martial law when the war started, and has renewed that declaration whenever it has expired. Martial law is the legal mechanism in the Ukrainian constitutional system for placing the country on a war footing. It would be hard to argue that Ukraine is not at war; this is clearly not a blatant power grab or a coup. The Ukrainian courts have affirmed the plain language of the Ukrainian constitution which does not permit elections to be held under martial law for what should be obvious logistical reasons. If 20% of US territory was under occupation and we held an election in which the occupied territories could not participate, would that be a legitimate election? Ukraine’s courts and constitution take the position that it is not, and whether you agree or disagree I don’t think it’s an unreasonable or despotic position to take, particularly since this law was ratified by democratically elected representatives. If the suspension of elections and the censorship of opposition parties funded by and allied with the aggressor nation cause you to think that Ukraine must not be a democracy and has fallen to authoritarian dictatorship, then I urge you to make inquiries as to the British political situation during WWII, and even the years after the war ended. Perhaps your position is that the US was wrong to take Britain’s side in that war? </li> Ukraine did ban political parties and media outlets that were directly tied to Russian interests, and she did expel the Russian Orthodox Church (technically it’s called the “Ukrainian Orthodox Church” which is affiliated with Moscow, while the actual Orthodox faith in Ukraine is represented by the Orthodox Church of Ukraine, a different religious organization entirely and not subject to any legal sanction) and establish a legal framework for the expropriation the Russian church’s property. To American sensibilities, this is prima facie evidence of a totalitarian dictatorship and not something democracies would do. But these facts have been misrepresented in Western media as if Zelenskyy banned Christianity and purged the government of all opposition. That is not at all the case. Religious liberty in Ukraine is alive and well. While the Orthodox faith is the majority religion in Ukraine, many Ukrainians practice Roman Catholocism, various Protestant faiths, of course there is a large Jewish population and a Jewish holy site in Uman (Zelenskyy himself is Jewish), there are a lot of Muslims particularly in the Crimea, and no small number of atheists. The state doesn’t interfere with the practice of religion, whatever it is. Ukraine also has diverse political parties, including parties of candidates who opposed him in the last election and in many cases still do. Look up Petro Poroschenko’s and Yulia Tymoshenko’s recent political activity if you don’t believe me. Certainly, any banning of a religion or a political party on the basis of affiliation with an enemy would run afoul of the First Amendment in America and should be fiercly resisted. But that’s my American opinion. The reality is that Americans are unusually, uniquely free in this regard. State crackdowns on certain political ideas and imagery are more common in Europe. If you doubt that, try to start a Nazi party in Germany or France and see how far you get. The fact is that Ukraine had some political parties aligned with Russia and in some cases operating as clandestine foreign influence operations for Russia, as was the Russian Orthodox Church in Ukraine, giving material aid and comfort to the enemy. These crackdowns were popular among the Ukrainian population, so while they are technically authoritarian they were also democratic. To be clear, I dislike these laws and think that they were a strategic error in pursuit of a feel-good populist win domestically; there seems to be a lot of that going around these days though. If you find that to be beyond the pale, let me assure you that Russia is far worse in this regard, and some Western European nations are not much better. </li> Ukraine hasn’t stolen half of the military aid. While Ukraine has a problem with corruption, and her officer ranks definitely include many traitors who steal whatever they can even when it means starving the war effort, the extent of this problem has been deliberately misrepresented by Russian propaganda. Zelenskyy famously said that half of the aid was missing, by which he meant that half of what had been promised to Ukraine had not been delivered. Much of the US aid is actual weapons and material purchased by the US government in the US; there probably is some corruption involved there but it’s the corruption of US defense contractors and not at all limited to the Ukraine war. </li> Corruption in Ukraine is still widespread, but unlike Russia they have made substantial progress in rooting it out even in the years that I’ve lived there. If the concern is that corrupt officials are stealing some fraction of the aid we’re sending (see bullet point above) then by all means let’s stand up an org in-country like SIGIR in Iraq and put a boot on the neck of corruptioners [“corruptioners” is a loan word from Ukrainian; you know a country has a corruption problem when they have a specific word for “person who knowingly profits from corruption” - ed]. If you want the names of some bloodthirsty Ukrainians with an accounting background and a burning desire to root out corruption in the military reach out to me privately I’ll put you in touch with a few good people. </li> Ukrainians in Donbas and Crimea did not vote to join Russia. Before 2014, it was true that the political sentiment and voting patterns in both Donbas and Crimea were largely pro-Russian. I believe Yanukovich won those oblasts handily in the election that brought him to power. But all of that changed, starting in 2014 and definitely in 2022. The “elections” that Russia held in the occupied territories to determine if they should join the Russian federation were as fair and trustworthy as any Russian election result in recent history, which is to say not at all fair or trustworthy. There was extensive evidence of election tampering and intimidation, including videos of Russian soldiers bringing ballots to individual apartments to be filled out in their presence, quite literally at gunpoint. There is no democratic justification for anything Russia has done in Ukraine, least of all the annexation of the Donbas and Crimea. </li> Ukrainians are not ready to give up the occupied territories, even though it’s obvious to everyone that there is no military path to their liberation (short of my hypothetical administration’s “damn the hot rocks and full speed ahead” policy articulated in the intro). With my Ukrainian friends who know me well enough to know that I’m not a Kremlin agent, I even admit as much. But can you blame them for holding onto this position? How willing would we be to give up Alaska in some alternate reality where Russia invaded and occupied it? We all naively thought that the bad old days when the violent invasion and occupation of parts of sovereign nations was a legitimate and effective growth strategy were behind us. I don’t have a brilliant political proposal for how to solve this problem, but if the result is that Russia gets to keep the territories that she has violently annexed and occupied, I can guarantee that her appetite will not stop there. If you don’t believe me, look up “Neville Chamberlain” and “peace in our time” for an instructive historical example. </li> </ul> So, as I scream into the void, unheard and yearning for a catharsis that pointless acts seldom provide, here are my requests for both the Ukrainians waging the information sphere campaign for Ukraine’s survival, and for the politicians on the skeptical American right. First, for the Ukrainians: Enough with the nonsense Trump/Vance/Musk-as-Russian-agents narrative. Believe that privately if you want, but the only people this reaches are people who already hated them and already supported you, and it alienates a lot of potential allies.</li> In fact, completely stop the name-calling when politicians or other public figures question their country’s support for Ukraine. Yes, some of them are literally paid Kremlin assets, some of the rest are idiots feeding from the Russian propaganda trough, but I promise you many of them are neither. Engage them on the facts if you have the patience, or ignore them if you don’t. Your histrionics are understandable given the horrors of the war three years in, but they are not bringing you any closer to victory and actually play into Russia’s hand. Don’t fall for it.</li> Make a heroic effort to patiently explain why it’s in America’s and others’ interest to support Ukraine, and why a ceasefire or a “peace” deal is not only not in Ukraine’s interest but also not in Europe’s or America’s. I know it’s maddening to have to argue for your continued existence, and it’s not fair that the world works this way, but it’s also not fair that Putin invaded and so many Ukrainians have been killed so by now you understand that the world isn’t fair. Do you want to be righteous and indignant or do you want to actually win over potential allies? If you can’t bear to do this in a way that will convince a skeptical interlocutor, it’s better to not engage at all.</li> Zelenskyy needs to wear the suit and tie and come crawling back to the Trump White House. It grinds my gears too, and you can burn Trump in effigy on the streets of Kyiv after the war is over, but at this point your choice is Trump or Putin. They will both put you on your knees. One of them won’t let you get back up again.</li> Introduce the death penalty for corruption in time of war. My guess is that informally this is already practiced in some cases in the military, but it really needs to be official. I know the European sensibilities recoil at capital punishment, but corruption in the military is still ongoing and it’s a huge propaganda boost for Russia, therefore it’s tantamount to treason. Yes, the executions will also feed Russian propaganda campaigns, but the corruption is worse, and the US also has the death penalty for treason so your biggest key ally is in no position to object. I suggest a format like the Nuremberg trials, complete with international judges, to make sure that the trials are fair and biased towards acquittal without being influenced by oligarchs’ money.</li> </ul> For the American skeptics on Ukraine: Visit Ukraine, at least Kyiv and if you have the nerve then go to Kharkiv or Kherson closer to the front. By all means, while there seek out Ukrainians who will confirm your priors or express any affinity for Russia or an unconditional surrender to Russia, if you can find them. Ukraine isn’t North Korea; you can walk around on your own and talk to people without SBU handlers giving them the stink eye. If you don’t believe me about that then send out some aides or fixers to do it for you and report back. Make sure you’re there during a Russian barrage and spend some time in an air raid shelter. Feel free to put on a ballistic helmet and body armor and take the photo op; just crop out the Ukrainian grannies in the background pointing and laughing. Try to explain your position to the Ukrainians you meet there, tell them the things you believe about Ukraine and Zelenskyy and Russia and Putin, and listen to their response. Then come back home and see if your position has changed.</li> Reflect on your objections to support for Ukraine. Are they based on Russian propaganda talking points that I’ve addressed above? If not, try a thought experiment: apply your objections to American foreign policy in the summer of 1941 vis a vis Britain. Are you then opposed to American involvement in WWII? If so are you comfortable with this position?</li> If you are still against American aid for Ukraine, please articulate why the fall of Ukraine to Russian aggression is not bad for American interests at home and abroad. Or perhaps alternatively, why you think a Russian victory in Ukraine serves our interests.</li> Setting aside NATO treaty obligations, are there any European countries whose defense against a Russian invasion you would be willing to support at least to the extent that the US has supported Ukraine thus far? How close of an ally and how old of a democracy must a country be before it’s worthy of American military support against unprovoked aggression? What quality is Ukraine missing that makes it unworthy of American support?</li> If you support continued US aid to Israel against her enemies, consider how Ukraine’s situation is any different. When Biden was threatening to suspend arms shipments to Israel and demanding an immediate ceasefire with Hamas, did you support that position? Now imagine that Hamas outnumbers Israel by population 4 to 1 and occupies 20% of her territory, as Russia does Ukraine. Do you make any distinction between Ukraine and Israel?</li> If you insist on peace for Ukraine on your own terms, ask yourself why. What is different about this peace deal, compared to the countless times Russia has agreed to ceasefires or DMZs only to keep fighting? Is your position based on trust that Russia honors her agreements? Is it based on some vague hand-wavy idea, not ratified by the Senate as a treaty obligation, that we would intervene if Russia attacks again? If so, then what prevents Russia from using the peace to rearm and correct the mistakes she made in 2022? If you don’t have solid answers not based on wishful thinking or historical illiteracy, then your position is basically that of the America First movement in 1941 before Pearl Harbor: </li> </ul> As for me, I don’t know what more I can do but argue on the Internet, seeth with impotent rage, and send money to guys I know who are fighting the Russians. If you want to turn some dollars into Russian casualties, there are many units that you can support directly. I can personally vouch for and unreservedly recommend supporting the 92nd Separate Mechanized Brigade “Achilles”</a>, or if you prefer a more quid pro quo arrangement, buy some sweet merch in the 92 OMB store</a>. Make sure you click “Одяг” (“clothes”) and pick up a “RSN PZD” t-shirt (if you know, you know). UPDATE: The always-provocative Handwaving Freakoutery posted a politically neutral mathematical analysis of the relative and absolute contributions of various NATO powers to Ukraine</a>. If you care as I do about the survival and flourishing of an independent Ukrainian state, it makes for grim reading. If you enjoy owning the libs on Ukraine, you should enjoy this. It turns out that Trump’s claims about the US’s outsized contribution to Ukraine aid specifically and NATO securiy in general are not some figment of a deranged mind, but supported by NATO data. In particular, this graph ruined my day: </picture> </a> </figure> This confirms my position that I’ve articulated above, that Zelenskyy needs to put on the suit and tie and crawl back to the White House on hands and knees in the hopes of turning US aid to Ukraine back on. The European nations at this moment (with notable and heroic exceptions of the Poles and Estonians, with whom I would advise potential invaders not to fuck) do not support Ukraine with money the way they claim to support Ukraine with words. I see no reason to believe that Trump being a dick will suddenly shock them into cutting back on their welfare states and whatever else they spend their money on and fully funding Ukraine’s war effort. The one and only chance for Ukraine’s survival as a free and independent state is for someone to convince Trump that Ukraine is worth the cost to America to save. I and many Americans already believe that wholeheartedly, but it doesn’t matter what we believe right now. The only thing that matters is convincing Trump to change his mind. Perhaps as a gesture of goodwill, Ukraine could rename the Batkivshina Mat statue in Kyiv to Dyadya Trump and give it a Trumpian haircut. Then rename John McCain Street in central Kyiv to Donald Trump Boulevard. I’m joking to make the point, which is to damn your national pride for the moment and swallow whatever bitter pills are needed to save the country. If I were president of Ukraine, that’s what I’d do, even though it meant living out the remainder of my days as a pariah and laughtingstock in my own country. What’s the alternative? Heroically falling to Russian dominion for another 100+ years? 2024 Year-end GenAI Tooling Review 2024-12-31T00:00:00+00:00 I don’t normally spend any time on navel-gazing posts about tools that I use, to say nothing of year-end predictions. However I want to record the GenAI tooling I’m using at the end of 2024 for my day to day software engineering work, mainly because it’s such a fast-moving field that I suspect it will be amusing to revisit this in a year or five and marvel at how primitive our lives were. At the same time, I’m also curious to see how some of my predictions about the future of the field hold up over time. With that preamble out of the way, on to the listicle: GenAI Tooling I’m Using Daily</h2> Cursor</h3> I’m not exactly on the cutting edge with this one. I don’t recall when I discovered Cursor; sometime earlier in 2024 on a Hackernews thread, most likely. I did an eval of an early version and found it to be an incremental improvement on the prevailing GenAI dev workflow at the time, which was copy-pasting code from Claude and then copy-pasting compiler errors and program output back into Claude when something went wrong. However, what has changed recently, and what leads me to gladly pay the $20/mo for Cursor, is the new “agent” feature in Composer. “Agentic” is the hotness right now, to the point that if your AI grift</del> startup doesn’t have an agentic story then you’re not going to be taken seriously. In Cursor’s case the agentic features are pretty simple, but they also unlock a more productive way of working with LLMs. Cursor can now not only make changes to code files as part of conversations in Composer, but it can run commands in the shell (subject to user approval) and see the results automatically. It sounds like a small thing, something you can already do by copy-pasting between Claude or ChatGPT and existing tools, but for me it’s removed annoying friction and lets me use existing tooling to help the LLM correct its inevitable hallucinations and screw-ups. Now that this feature is available, it’s more important than ever to have tooling in place like static analysis tools and automated test suites. I can tell Cursor to write some bit of code, and then tell it to run the tests or run a linter, knowing that this will surface a lot of the typical LLM fuckups that I’ve come to expect when programming with GenAI. It’s not perfect of course, but when it works I can almost mindlessly Command-Enter repeatedly to approve the model’s various flights of fancy and let it figure out the details as it runs afoul of clippy or a test or the compiler itself. It’s become a cliche at this point to characterize current SOTA LLMs as eager, tireless, but often incompetent junior developers. That goes double now, when using the agent feature. Just like you would harden a repo against the well-intended predations of junior devs with branch protection and a bunch of checks in the CI system, that same effort pays dividends when using LLMs. Today, the agent workflow in Cursor is interactive, but companies are already starting to sell junior dev AI services that operate entirely in the background on a Github issue, potentially coming back hours later with a PR. The more you can automate checking the work of the LLM with existing tooling, the more likely these tools (well, future iterations of them anyway; right now they’re still pretty raw) will be able to provide some value. Speaking of cliches, another one is that the use of Cursor to help take on unfamiliar tasks, languages or frameworks empowers one to take on work that otherwise would be too time-consuming or intimidating. I can confirm this as well. My preferred language today is Rust, which I know very well (and which LLMs generally don’t know very well, presumably due to lack of training data). However, sometimes as punishment for sins I must have committed in a past life, I need to work with languages that aren’t Rust. Most recently Python. Being able to have the LLM guide me through the subtleties of the language and which packages are available for what tasks is a huge unlock. The resulting code is not great in many cases, and I’m sure professional Python devs would cringe at my output, but this isn’t a shipping product it’s internal tooling and glue and such where getting something done matters a lot more than stylistic purity, and the LLM makes the difference in many cases between a quick-and-dirty script existing and helping get things done, and not having anything at all. Claude</h3> I have access to the Claude Sonnet 3.5 model as part of my Cursor subscription, but I also pay Anthropic $20/mo for access to the Claude model on my own. I need this for a few reasons: First, I use LLMs for non-programming tasks as well. To give but one example, I currently live in Budapest but speak no Hungarian. I can take a photo of some document or sign posted by the entrance of my building, and Claude will not only translate it but also explain it and answer questions. My wife uses my account as well, interacting in Russian and Ukrainian, to similar good effect (don’t tell Anthropic please!). Second, in Cursor’s presentation of Claude one doesn’t get the raw model; Cursor has extensive prompting in place to guide the model to the task at hand. I find that this often works great, but sometimes confuses the model and results in it doing dumb things that obviously won’t work. When I see this happening, I’ll sometimes pop over to Claude directly and tell it what I’m trying to do and have it generate a prompt and code example for use with an LLM, then paste that back into the Composer conversation and get things back on track. Finally, Anthropic frequently release cool new functionality that is only in the app, or at least starts there, and I like to be able to pick up and play with new stuff as it lands. ChatGPT</h3> I recently re-activated my $20/mo ChatGPT subscription, which I had canceled once Claude 3.5 Sonnet took over as the SOTA model for dev tasks. Part of this was to play with the stuff they announced rapid-fire at the end of the year, and part of it was to be able to play with o1 (having done so I don’t think it’s superior to Sonnet for my work). I probably ought to cancel this again, but I’m keeping it around for the rare cases where I want to play Sonnet against another model to sanity-check its analyses. Perplexity</h3> I have found myself using Perplexity search by default now, unless I’m doing a very mechanical lookup that I know Kagi will find quicker (I also pay for Kagi). The Pro search is clearly better, and does a pretty good job of sifting through SEO crap and blogspam to get to meaningful content. I find this is particularly true if I want to shop for something and I want to find the best option. Long gone are the days where one could search “best gonkolator” and actually get actionable and unbiased results back that will help you figure out who sells the best gonkolator. Instead one must take an adversarial approach, interrogating each result on the assumption that it’s a bad actor trying to trick you into clicking an affiliate link. For the most part, Perplexity Pro does the first pass on its own, making it much easier to sift through what’s left. GenAI Predictions for 2025</h2> In no particular order: 2025 is the year of agentic systems. This isn’t so much a prediction as a parroting of the Zeitgeist on X at the moment. However one defines “agentic”, Cursor’s primitive agent feature has shown that the way to at least ameliorate the limitations of current SOTA LLMs is to plug them into deterministic systems that can automatically call out their bullshit hallucinations and keep them on a somewhat straight and narrow path. To the point that I doubt this distinction will exist for much longer; systems that use GenAI and don’t suck completely will just obviously be built as agents incorporating tool use.</li> AI junior devs will be widely adopted. Not because it’s a good idea, or because it will be a net positive for engineering productivity, but because the pressure on management to replace expensive and annoying developers with cheap and compliant AI will be too powerful to resist. To be clear, I’m bullish on GenAI tooling for devs in general, and use multiple tools every day to increase my productivity. But nothing that I have experienced in my long career gives me any reason to believe that adoption of these new technologies will be principled, measured, and rational. I fully expect that part of the job of senior engineers will be wrangling the army of AI agents submitting PRs based on issues and requirements documents that themselves were generated by AI tools, and I fully expect that to suck. It won’t be politically acceptable to turn this off until enough negative experience permeates the MBA Zeitgeist and managers can be confident that they’re not missing out on a hot new trend. That is multiple years away unfortunately.</li> Software will get a lot worse. Since I use LLMs to help me write code every day, in multiple languages, I have a pretty good sense of what they’re capable of. Not a day goes by that I don’t get code from a SOTA model that is very obviously wrong but compiles. In all cases “obviously wrong” includes stylistic, structural, and nominative defects, but in many cases also subtle functional bugs as well. I’m quite certain that many developers will just Command-Enter until the code seems to work, and then ship it. At scale this will show up as flaky, confusing, slow, and in some cases utterly broken software. Some of that software might even be my own, if I am not constantly vigilant and never complacent. It’s tempting to rail against the injustice but I don’t think that changes the outcome.</li> New tooling and languages will be much harder to get adopted. Already I think a compelling steelman argument against adopting Rust in projects that otherwise are a good fit for it is that LLMs don’t do a very good job of writing Rust since there’s comparatively less Rust code in public for LLMs to train on. As more and more of the code that gets shipped is built with LLMs by developers who don’t understand the code that’s being written, the ability to outsource thinking to the GenAI tools will be an important consideration for teams that don’t have the luxury of employing motivated humans to write and review every line of code for correctness. This will impact languages but also tools. In the old days, part of the calculus when choosing a tool or a language was the ability to hire developers with those skills; now it will be the extent to which the foundational models that power modern AI agents will be able to effectively analyze and generate code with those tools or languages. I suppose if you hate having to learn a new Javascript bundler every year, this is good news, but I think it’s a net negative.</li> The corollary of the previous bullet is that generating docs optimized for LLM consumption will be much more important, particularly for new tools and languages. I think it’s inevitable that software development agents will need to get much better at looking up documentation, and when they do the extent to which that documentation is easily consumed by whatever mechanism they use will be important. Right now it seems like dumping all documentation content into a big Markdown file is a pretty good approach, but I bet this will be refined over time. This applies not just to developer docs but also end-user docs as well. On the plus side, perhaps this will finally be the death of product docs locked away behind a login?</li> It will turn out that letting LLMs build entire codebases with little human understanding of what is being built will result in a mess that the LLM itself can’t work with. This year there was a lot of breathless “holy shit!” posting on X in which an AI tool took a spec like “here’s a screenshot of Facebook now write the code for it” and produced the right HTML, CSS, and React. I’ve also seen is claimed that no one will pay for SaaS products anymore, since they can just tell ChatGPT to write whatever they need. I am quite confident that this is nonsense, and that the people making these claims are either grifters, ignorant of how complex software is built and operated, or in many cases likely both. However those of us naysayers will be ignored in favor of the sugar rush of exciting AI future vibes, and funded companies will make a big deal out of using AI to build the whole product. At some point, I think likely later in 2025, the failure of this approach will become impossible to deny, but in the meantime expect a lot of breathless “the end is nigh for software engineers” takes.</li> Developers without AI augmentation will be at a huge disadvantage. There is still a large population of developers posting on HN and elsewhere who insist that they see no value in GenAI tooling for the software engineering profession, and are much more productive without it. Without passing judgement on the merits of those statements one way or the other, I predict that this will be a position one will need to keep to oneself when looking for software dev jobs in 2025. As for myself, I can honestly say that I get a lot of value out of Cursor and SOTA models and I look forward to synergistically working tirelessly to increase shareholder value by leveraging disruptive AI technologies to push the boundary of what is possible whilst constantly shipping.</li> </ul> Generating OpenGraph image cards for my Zola static blog with Cloudflare Functions and Rust 2024-11-16T00:00:00+00:00 Not long ago I resumed posting stuff on 𝕏. I noticed that when I posted links to my blog posts, they rendered in a really ugly way that looked sloppy to my eyes. Comparing this to how links to, for example, Github repos were rendered, it was a night and day difference. One lazy weekend I decided that my blog, a very lonely corner of the Internet with basically no traffic, must have a fancy OG card for every post! Not for any practical reason, just that it’s mine and I like nice things and there was no one around to stop me. The same “because I can” reasoning further moved me to do this in Rust and WASM, although there would have probably been easier ways to go about this. Before this project, this is what the preview looked like when posting a URL to one of my blog posts in Slack: </picture> </a> </figure> The Slack preview actually doesn’t look that bad, but on 𝕏 it’s just dreadful. Here is the same URL, but in the 𝕏 post composer: </picture> </a> </figure> Compare this with what it looks like when posting a Github repo URL on 𝕏: </picture> </a> </figure> You can imagine the jealousy I felt, seeing how cool Github was able to make their repo URLs render in preview, and how plain and ugly my own posts looked by comparison. Or, maybe you’re a normal person and can’t imagine caring about such a small detail, in which case just believe me when I say that this really bugged me and I had no choice but to do something about it. I quickly learned that Slack, 𝕏, and pretty much anything else that generates preview thumbnails of URLs use the OpenGraph</a> metadata property og:image</code> in a given HTML page, which if present is assumed to contain the URL to the image that should be used as the preview for that page. 𝕏 also has its own meta property, twitter:card</code> and twitter:image</code>, but the idea is the same, and 𝕏 falls back to the OG properties if none of its own are present. So in the head</code> HTML element, one needs only place a few tags like this:  <meta property="og:image" content="https://..." /> <meta property="og:image:width" content="1200" /> <meta property="og:image:height" content="600" />  <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:image" content="https://..." /> </code></pre> and whatever is at the URL specified in content</code> will be rendered as the preview. Sources seem to vary on what supported image formats are allowed, but PNG definitely works everywhere. My blog uses the Zola</a> static site generator to generate HTML from my Markdown blog posts, and Cloudflare</a> Pages to host the resulting static assets. One very easy thing I could do would be to create an image that looks nice enough to represent all posts on my site, and modify the Zola HTML template for my blog to point og:image</code> and twitter:image</code> metadata properties to that image. The problem with that is that each post would always have the same image, so the image could not have any post-specific details. That’s lame. GitHub didn’t do that; they automatically generated a preview image for each repo based on some information about that repo. I wanted the same! Because my blog is statically generated by Zola, and Zola doesn’t have any built-in support for generating post-specific preview images, I decided I’d try using Cloudflare Functions</a>, a serverless function product from Cloudflare that is part of their Cloudflare Pages product that I was already using. Unfortunately, Cloudflare Functions uses Javascript, so I started looking into ways I could generate preview images with Javascript, much as it pained me to do so. It turned out that many people are as fastidious as I am and wanted their own cool custom OG preview images, so there is a proliferation of Javascript-based solutions. To pick but one, Vercel published a OG Image Generation</a> function that does this. Their docs described a particular feature that caught my eye: @vercel/og uses Satori and Resvg to convert HTML and CSS into PNG</li> </ul> </blockquote> I’m not much of a designer, particularly not of the pixel-pushing variety, so the idea of making a template in HTML describing how my preview image should look and getting an image with a raster rendering of that HTML seemed like the right approach. But I didn’t want to use Javascript, I wanted to use Rust! So, rather than solving this problem easily but unsatisfactorily in a few minutes with off-the-shelf Javascript, I started to explore how I might achieve this using Rust, with the caveat that whatever I came up with had to be callable from Cloudflare Functions. Those of you who stubbornly want to use Rust to do everything probably already know about WASM. As it turns out, last year Cloudflare posted a blog about using Cloudflare Pages Functions with Webassembly</a>. It turned out that Cloudflare Pages Functions are just a special-case of the more general Cloudflare Workers, the latter also supporting WASM. This post showed how to write a Rust function, compile it to WASM, and call it from a Cloudflare Pages Function written in Typescript. Yes! I set about writing some Rust code that would render HTML to an image. I very quickly ran into a roadblock: there are no such libraries in Rust, as rendering HTML turns out to be hard. I quickly relaxed my requirements and decided that SVG is also fine, which led me to resvg</a>. Recall that Resvg was mentioned in the Vercel docs as well. The difference is that in Javascript-land they make use of Satori</a> which renders HTML/CSS to SVG, and then Resvg renders that SVG to a PNG. I’d already accepted using SVG to describe my OG preview image, so I was fine with just using Resvg directly. It’s a great project, with strong SVG support and is pure Rust which means compiling it to WASM isn’t a challenge. Once I was sure SVG rendering would work, I used Claude Sonnet 3.5 to generate some candidate SVG layouts. Making the most beautiful OG image wasn’t the goal here; the goal was to have something unique to me that could be used as a template for any post on my site. In the end I came up with this: </picture> </a> </figure> Making this into a template was a bit more of a challenge. I used the Askama</a> crate to compile the SVG template into the Rust binary and present a typed Rust struct as the interface to generating the output SVG: </iframe> The SVG itself is modified to use the Jinja2 template syntax which Askama also supports, which is validated at compile time but evaluated at runtime to produce dynamic SVG based on the inputs specified in the struct above. A few lines from the SVG template illustrate this concept: </iframe> You might notice that the inputs are post_title_lines</code> and post_description_lines</code>, instead of just a post title and description. Why? Because Resvg (and maybe SVG in general?) doesn’t implement advanced text rendering, including wrapping text in a bounding box. We take this for granted in HTML but it’s actually rather complex, particularly with arbitrary fonts and glyphs. I wasn’t about to implement a typesetting mechanism in SVG just for this vanity project, so I came up with an ugly hack that mostly works: through trial and error, I figured out how many A</code> characters fit on one line in my particular template before they go too far to the right and overflow the image boundary. I then made a const</code> for that number, and used the textwrap</a> crate to perform the much simpler task of breaking up text at word boundaries such that no line has more than the max number of characters: </iframe> This is, I want to emphasize, fucking hideous! The algorithm assumes all letters have the same width, but I am using a variable-width font (about which more later), so it will wrap on some boundaries too early and make for oddly short lines. It’s not the algorithm that your browser or your word processor uses when typesetting text. But it’s also just a few lines of Rust, and unless you’re looking closely at the generated OG image cards, you probably won’t notice that the word wrapping is a bit funky. All of this worked great when compiled for the Apple Mac Rust target and run in a test locally. Then I tried to run it under WASM, and immediately stumbled: You see, I haven’t shared the full SVG template. It refers to two external resources, one directly and one somewhat indirectly: </iframe> Can you spot them both? The obvious one is the reference to devil.svg</code>, an SVG version of the iOS smiling devil emoji that has been my logo since before emojis existed (search “Simon Jester” if you’re curious). The indirect one is the font reference, Noto Sans</code>. On my Mac the devil.svg</code> file is in the Git repo alongside the template, and I had installed the free Noto Sans font long ago. But when code runs in WASM, it runs in a sandbox isolated from both the filesystem and the system fonts; in fact when Resvg is compiled for WASM it doesn’t even have a system font API to link to. I started to despair, and wondered if maybe I should just generate a static OG image for the entire site and be done with it. No one will ever notice. But, no! I will notice. I resolved to dig into the Resvg code and find a solution. And find I did! First, I modified the Rust code to compile into the Rust binary both the devil.svg</code> file and the Noto Sans</code> font file: </iframe> Then I took advantage of the very well designed Options API in the usvg</a> crate, which is the component that Resvg uses to parse SVG and prepare it for rendering. This API design appeared to anticipate my needs exactly, and provided a way to intercept and handle what would otherwise be outgoing network requests to retrieve external resources (devil.svg</code> in this case), and another function to populate the fonts DB with custom font data in a Vec<u8></code>: </iframe> With this, I had WASM code running that could generate an SVG from my template for a given site title, post title, and post description, apply word-wrapping, and render that SVG to a PNG, with the text rendered in a custom embedded font! But immediately I saw something I disliked: the resulting PNG was 600KiB! For a 1200x600 OG title card that was mostly just text and some simple vector graphics! I started to dig into this, and learned that users of PNGs who really care about file size optimize the PNGs to make them a bit smaller, at the expense of some up-front compute cost to do the optimization. As it happened, one of the most popular tools was called oxipng</code>, and if you thought maybe the “oxi” prefix indicates a Rust tool, then you thought right! The oxipng</a> crate is also shipped at a CLI binary, but in my case I wanted to compile the functionality directly into the WASM binary. Again fortune smiled, as oxipng is pure Rust and also easily compiles for WASM. The code to get oxipng in the loop to post-process the PNG that Resvg generated from the SVG was a bit complex, owing to the way Resvg works. Once you have your SVG parsed in memory in a usvg::Tree</code> struct, you allocate a “pixmap” (basically a buffer meant to hold pixels) using the tiny-skia</a> crate, and you call resvg::render</code> passing in the SVG tree struct and the pixmap that you want Resvg to render into: </iframe> The result isn’t a PNG file; it’s a pixmap containing the raw pixel values for the rasterized SVG. tiny-skia</code> has an optional feature that will produce a PNG directly from that pixmap, which is what I used originally that produced a very unoptimized PNG. I wanted oxipng to do the PNG creation so that it could optimize the output, so I disabled the feature in tiny-skia</code> that implements direct PNG support, and wrote a Rust function pixmap_to_optimized_png</code> that takes a tiny-skia</code> pixmap and creates an optimized PNG. pixmap_to_optimized_png</code> is a bit long, I don’t want to embed the whole thing here. You can go to the repo and see the whole thing here</a>. There is some trickery required to convert the pixmap into just raw pixel values in a Vec<u8></code>, after which it’s pretty straightforward: </iframe> This worked, although not as well as I hoped. PNGs were now around 400KiB, and the generation step took almost 2 seconds when run on Cloudflare! The free tier on Cloudflare allows a miserly 10ms of CPU usage per function call, so I had to switch to the $5/mo “paid” plan to up that to 15 minutes. Given the disappointing performance, it was clear that caching would be required. Cloudflare Pages has a serverless product called “KV” which, as the name implies, is a key-value store that is well-suited for use as a cache. On the $5/mo paid plan that I had to upgrade to anyway, KV monthly limits are crazy: 10M reads, 1M writes, 1GB of data. So I can use KV “for free” for caching. However there was no chance that I could integrate with KV directly from Rust, as the WASM code doesn’t have any network access. So, reluctantly, I started to adapt the Typescript wrapper around my Rust WASM masterpiece to implement caching. You can read the entire Typescript file here</a> if you’re interested. I certainly am not. But I do want to point out a couple of interesting bits: The way I wanted this to work is by referring to the og-image</code> Cloudflare function in the meta</code> tags that point to the OG image card in the header. For example something like this: <meta property="og:image" content="https://127.io/og-image?p=/2024/11/01/whats-in-my-air-raid-bug-out-bag/" /> <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:image" content="https://127.io/og-image?p=/2024/11/01/whats-in-my-air-raid-bug-out-bag/" /> </code></pre> Note that the only information passed on the query string to og-image</code> is just the path to the post. But recall that the template takes parameters site_name</code>, post_title_lines</code>, and post_description_lines</code>. Where do those come from? Well, a naive way would have been to pass them as a query string parameter directly, like: <meta property="og:image" content="https://127.io/og-image?site_name=127.io&post_title=foo&post_description=bar" /> </code></pre> But the problem with that is that I have asshole friends who would abuse this to generate OG title cards with my blog’s branding for posts I never wrote, like “10 ways Golang is better than Rust (#7 will shock you!)” or “Star Wars Episode 1 is the best Star Wars! Fight me!”. Or they might learn that it takes 2 seconds of Cloudflare Pages compute time to generate an image, and spam my site with requests for unique title/description combinations to drive up my Cloudflare bill. Or some unscrupulous copy-cat might decide that my blog’s livery is simply dazzling and use the og-image</code> generator to generate OG title cards for their own site without my permission! Now admittedly, this last one opens the unscrupulous copy-cat up to some very amusing trolling opportunities if I were to discover what they had done, but I don’t have time for any of that nonsense. So I want to be able to limit og-image</code> to generating only title cards for posts that actually exist on my site. So, I decided to compile in to the Cloudflare Functions bundle the list of all posts, their titles and descriptions. I made a custom Zola template called og-metadata.html</code>, which does that. The Zola template is in my blog’s content repo which is private, but here’s a snippet for those who are interested: { "default": { "title": {{ config.title | json_encode | safe }}, "description": {{ config.description | json_encode | safe }} }, {%- for page in pages %} {%- if page.path %} {{ page.path | json_encode | safe }}: { "title": {{ page.title | default(value="") | json_encode | safe }}, "description": {% if page.description %} {{ page.description | json_encode | safe }} {% elif page.summary %} {{ page.summary | striptags | trim | truncate(length = 1000) | json_encode | safe }} {% else %} {{ page.content | striptags | trim | truncate(length = 1000) | json_encode | safe }} {% endif %} } {%- if not loop.last %},{%- endif %} {%- endif %} {%- endfor %} } </code></pre> It produces JSON output that looks something like this: { "default": { "title": "Creative Articulation", "description": "Musings and misadventures of an expat enterpreneur" }, "/2024/11/01/whats-in-my-air-raid-bug-out-bag/": { "title": "What's in my air raid bug-out bag?", "description": "I went overboard prepping for a very specific threat scenario, specifically: large-scale Russian drone or missile attacks on Kyiv. The result is this bug-out bag.\n" }, // ... } </code></pre> Why is it called og-metadata.html</code> if it’s a JSON output? Well, that brings us to the first hideous workaround to barely-documented Cloudflare limitations, to wit: </iframe> The comment explains the weird name. This embeds the contents of the generated public/og-metadata.html</code> file which Zola creates at build time by applying the og-metadata.html</code> template, and makes it available in the function Typescript as a string variable. I made a Typescript type to represent the shape of this JSON: </iframe> and at function load time I parse the JSON into this struct: </iframe> There are not even 3 dozen posts as of this writing, so the overhead of this is minimal. I then use the path passed in on the query string as p</code> to look up the metadata in this JSON: </iframe> A couple things to note: If the p</code> variable contains a path that doesn’t correspond to a known post, the code silently falls back to default</code> which is just the OG card for the main 127.io site. Asshole friends trying to mess w/ this will not get far, as that default</code> will be served from the cache very quickly and will not contain anything amusing or incriminating.</li> I use a cheap hashing function fnv1a</code> (I just noticed the comment says “SHA-256” but it’s not) to hash the post title and metadata and include this in the cache key. This ensures that if I change existing post metadata for some reason, the old cached OG card will not be served any more.</li> </ul> There’s code that queries the KV namespace for the cache key, generates the image if not found, etc but that’s all very straightforward. Look in the repo if you want to read the whole thing. The one final thing I want to point out, relating to using Rust code in Cloudflare functions via WASM: I linked earlier to a Cloudflare blog post about using Rust via WASM in Cloudflare Functions</a>. In that post, this is the code proffered to load a WASM module for use in Typescript: // functions/api/distance-between.js import wasmModule from "../../pkg/distance.wasm"; export async function onRequest({ request }) { const moduleInstance = await WebAssembly.instantiate(wasmModule); const distance = await moduleInstance.exports.distance_between(); return new Response(distance); } </code></pre> This code wasted hours of my time, because it works in this particular blog post example but is wrong! That distance_between</code> function is written in Rust and compiled to WASM. Here’s the Rust declaration: fn distance_between(from_latitude_degrees: f64, from_longitude_degrees: f64, to_latitude_degrees: f64, to_longitude_degrees: f64) -> f64 { //... } </code></pre> Note that all arguments and the return value are scalars, f64</code> in this case. Because of this, calling the WASM export distance_between</code> directly works fine. But here is my generate_og_image</code> function as declared in Rust: </iframe> Notice JsString</code> and Vec<u8></code>. These are not scalar types. They cannot be used directly from Javascript! That’s why when you run wasm-pack</code> as part of the build process to compile your Rust code and generate a WASM module, it doesn’t just generate the WASM file:  ll pkg --no-icons .rw-r--r--@ 427 rupert 16 Nov 18:23 og_generator.d.ts .rw-r--r--@ 162 rupert 16 Nov 18:23 og_generator.js .rw-r--r--@ 5.4k rupert 16 Nov 18:23 og_generator_bg.js .rw-r--r--@ 4.2M rupert 16 Nov 18:23 og_generator_bg.wasm .rw-r--r--@ 468 rupert 16 Nov 18:23 og_generator_bg.wasm.d.ts .rw-r--r--@ 397 rupert 16 Nov 18:23 package.json .rw-r--r--@ 758 rupert 16 Nov 18:17 README.md </code></pre> See those .js</code> and .d.ts</code> files? Those are important! They wrap the WASM in Typescript/Javascript wrappers, that transform input arguments from Javascript into the WASM types that Rust expects, and output values from Rust’s WASM representation to Javascript. Very important! That is why in my Typescript function you see the WASM being imported differently: </iframe> Maybe in a follow-up post I’ll go into more detail on what’s going on here. For now let’s just say that this is the way wasm-pack</code> intends for you to consume the generated WASM module in this case. But wait! The above still won’t work right. Because the generated wrappers have to be initialized explicitly, due to some pecularity of how Cloudflare Workers is unlike the other Javascript environments that wasm-pack</code> is generating for: </iframe> Now, at last, this will work! You can use wrangler</code> to deploy this to Cloudflare. After all of this, I can finally hold my head high when posting links to my blog posts on social media. Feast your eyes on the fully operational 127.io OG images: In Slack: </picture> </a> </figure> On 𝕏: </picture> </a> </figure> Dazzling, I’m sure you’ll agree! As mentioned, it takes ~2 seconds to generate a single OG image. 𝕏 seems to timeout very aggressively when requesting OG card images, and even if it didn’t 2 seconds is a long time to wait for visual feedback when pasting a URL in somewhere that uses these images as a URL preview. So at the end of this project I put a step in the Github Actions workflow that runs when I publish new blog content, to use that same JSON file og-metadata.html</code> to get a list of post URLs and use curl</code> to request the OG card for each one, which pre-heats the cache. It is not lost on me that I could just as easily have implemented static generation of the OG image assets in a simple Rust program running under GHA, solving this problem in a much more straightforward and performant way, without having to pay an extra $5/mo to Cloudflare or jump through numerous hoops with underdocumented Cloudflare features and the challenges of running Rust under WASM. But I’ve learned a lot about WASM, SVG, PNG, Cloudflare Functions, and the risks of indulging one’s obsessive tendencies. Maybe, in the end, the real OG image card is the friends we made along the way! I'm leaving Ukraine (for the winter at least) 2024-11-10T00:00:00+00:00 </picture> </a> Prince Volodymyr's statue has watched Ukraine's enemies attack Kyiv for well over 150 years. He was there in 1918 for the German occupation, and clashes between the Ukrainian Nationalist Army, the Red Army, and the imperial remnant White Army. He was there in 1941 when the Nazis took Kyiv, and he was still there in 1943 when the Soviets took back the parts of the city that they had not destroyed outright. He was there in 2014 for the Maidan protests, counter-protests, and violence. He was there in 2022 when Russian forces attempted to invade Kyiv only to be fought back by fierce Ukrainian resistance. He's still there now, and I'm sad to say that he hasn't seen the last of it yet.</figcaption> </figure> It’s a sad day today, as I crossed the border from Ukraine into Hungary, having packed up most of my belongings in Kyiv and either placed them in storage there or had them sent to Hungary. I finally decided that it was time to leave as the tempo of attacks has increased, and I increasingly suffer the effects of sleep disturbance caused by the sound of explosions and air defenses, and the stress of not knowing what the future will hold but being pretty sure it’s going to be worse. I have posted previously about what daily life is like</a> here, and specifically about how loss of power isn’t really a problem since the blackouts so far are rolling blackouts and my Ecoflow system is enough to power essential devices for lights, computers, and Internet. But winter is another story as there’s heat to think about, plus winter increases load on the already-struggling grid, PLUS Russia seems intent on stepping up the infrastructure attacks this winter. It’s all become too much. I first moved to Kyiv in 2018, it was here that I started Elastio</a> and hired the initial engineering team, it was here that I bought my apartment, it was here that I lived through the COVID-19 pandemic, it was here that I lived until early February 2022 when I left ahead of the Russian invasion, and it was to here that I returned in August of last year. Many of my friends and colleagues remain here, unable or unwilling to leave. As I cross the border into a country where I know for sure that any loud noise I hear is definitely not enemy fire, I feel guilty and ashamed. Guilty that I will now live in relative comfort in Budapest with reliable electricity and heat, where it’s been over 30 years since the last civilian was killed by the Russian military. Ashamed that I wasn’t willing to stick it out, especially considering that I lived in the center of Kyiv under a robust air defense umbrella while many Kyivans have it much worse and are not fleeing. I intend to live here in Budapest at least until spring, then will re-assess based on the situation in Ukraine and also my professional situation. I have nothing against the Hungarian people, but Hungary is definitely not Ukraine, and I don’t have any desire to live here long-term. Soon I will need to decide whether to return to Ukraine, or go back to the US after having lived abroad for 6+ years. UPDATE 2024-12-01: Unfortunately it looks like this decision was prescient. Since I left, Ukraine’s civilian infrastructure has been under considerably heavier bombardment than most of the rest of 2024. Rolling blackouts are back, and the invincibility points are once again opened around the country. I still have my apartment in Kyiv for now, where my Ecoflow backup infrastructure is located, so I can see when it switches to battery power, which is quite often in the last few days. Seeing that my concerns were well-founded doesn’t make me feel any less guilty or any less ashamed, but it is some consolation at least that I didn’t bail over imaginary fears. What's in my air raid bug-out bag? 2024-11-01T00:00:00+00:00 My earlier post about my air raid response protocols</a> prompted some interest in what specifically is in my “alarm suitcase” (тривожний чемодан in Ukrainian), or what English-speaking preppers would typically call a “bug-out bag” or “BoB”. This post is a quick bag dump for those of you who are interested in such things. Let me first make clear that I do not know any Kyivans, either Ukrainian or foreign, who have put as much thought and effort into a bug-oug bag as I have. I do not hold up my kit as typical of what civilians in Kyiv will have by the door, and honestly if you’re a foreigner in Kyiv in the center of the city surrounded by air defenses, you probably should not follow my example. I have a pretty strong prepper streak by nature, and I like to be prepared for various eventualities however unlikely they may be. I’ve always maintained a bug-out bag when living in peaceful American cities, so why would I do any less when living in an actual war-torn country where actual hostile forces fire actual ordnance at civilian targets every day? Next let me be explicit about the threat model here. This bug-out bag is not in preparation for the typical bug-out scenarios in the US, like heavy weather, riots, civilizational collapse, zombie apocalypse, or boisterous American football hooligans running amok. It’s not intended to sustain me on foot across long marches through the wilderness or an urban landscape, nor is there any offensive component at all. The scenario here is that Russian bombardment necessitates taking shelter in an underground bomb shelter (in my case that is a Kyiv Metro station), possibly for a day or more, and with a non-zero chance that my apartment could be destroyed and I would have only those possessions that I placed in this bag. There is also a vanishingly small but very much non-zero possibility of a nuclear weapon being used against targets far enough away that I’m not immediately vaporized, but close enough that I have to worry about the fallout. The threat model presumes that civilization remains intact and that the Ukrainian government and economy are still functioning enough to provide food and shelter in the aftermath of any attack, albeit possibly delayed by several hours up to a couple of days. If you live in a stable Western country, particularly in the US, it’s very unlikely that most of the considerations that went into this particular bugout kit will apply to you. Finally, for some of these items I will link to the item on Amazon or other sites. I don’t belong to Amazon’s affiliate program, so I don’t get anything if you buy via those links. They’re just for convenience. With that preamble out of the way, here is my air raid bug-out bag, fully packed in the state in which I keep it by the door at all times: </picture> </a> Specialized air raid bug-out bag, packed</figcaption> </figure> The backpack is a 5.11 Tactical RUSH 24</a> 37 liter pack. I’ve owned it for many years, it’s well made and holds up to abuse. Before the war I used this as a sort of range bag, which is why it has a name tape in Ukrainian on it. I haven’t removed that because it’s not an opsec issue in my case and I think it looks cool. I haven’t weighed the pack fully-loaded, but I would guess it’s between 30 and 40 pounds. I’ve worn this pack while walking on the treadmill at max incline for 45 minutes, which was challenging but didn’t kill me. I can climb the 12 flights of stairs from the street to my apartment while wearing this pack and not die, although it’s not a joyful experience. I would not want to walk all day in this pack, and I couldn’t run very far at all with it on, but as I’ve noted above this pack’s use case doesn’t involve either of those things. I have a few pieces of gear affixed to the outside of the pack: Lashed to the left and right sides of the backpack via MOLLE, I have bottled water stored in Vanguest Hydra</a> collapsible bottle holders. These are great; they hold the bottles securely (I cinch the shock cord around the neck of the bottle, so they are fully retained even when running or if the backpack is upside-down), and when empty they velcro flat against the side of the bag.</li> Affixed to the MOLLE field at the lower back of the pack is a Dark Angel Medical EDC Trauma Kit</a>. This has a CAT Gen 7 tourniquet, trauma shears, and inside it contains nitrile gloves, a compression bandage, a pair of Hyfin chest seals, and hemostatic gauze. Given that the threat scenario here is large-scale bombardment of civilian population centers, it’s not impossible that someone (possibly me) would sustain injuries before making it to shelter. This minimal kit is at least enough to stop serious bleeds.</li> The longer nylon straps below the trauma kit are add-ons from 5.11 called the RUSH Tier System Kangaroo Kit</a>. These are intended to be used to add another “kangaroo” pouch to the pack, but I use them to lash the Helinox chair (below) to the pack securely with quick-disconnect buckles for convenience.</li> The black bag labeled “Helinox” contains a Helinox Sunset Chair</a>. Laugh all you want you armchair commandos, but know this: once you’re in the shelter, you do a lot of sitting around waiting. The Metro stations don’t have any seating, so you’re either sitting on the stairs or the floor. This gets very old very quickly, at which point you really wish you had a chair to sit in. This Helinox is very lightweight (remember I’m not marching with this bag, I’m taking it to a shelter a few hundred meters away) and easy to deploy. When I am sitting in this chair in the shelter chilling comfortably, the Ukrainians who wisely also brought camp chairs and I look at each other knowingly, while everyone else shifts uncomfortably on the cold hard ground wondering where their lives went wrong.</li> Not visible in the “packed” photo but located behind the Helinox bag, I have lashed a Kifaru Woobie</a> in its compression sack to the loops at the bottom of the RUSH pack which are there for this specific purpose. This is lashed with GEAR AID Fast Wrap velcro straps</a> which also are fed through the straps sewn into the Kifaru compression sack so it’s very securely affixed. This is useful as a pillow, and for long stays in the Metro, or air raids that happen at ungodly hours like 4AM one sometimes wants to just lie down and go back to sleep. In winter having a blanket makes that much more comfortable. The Kifaru woobie is quite possibly the epitome of blanket technology; impossibly light and yet warm and very durable. It makes a Rumpl look like a scratchy wool blanket from the Civil War.</li> </ul> That’s it for the outside of the pack; let’s dump it and go over the contents: </picture> </a> Everything you need to live in comfort and style for 24 hours in the Kyiv metro</figcaption> </figure> Going left-to-right and top-to-bottom: Ziploc bag with fresh underwear, socks, and a t-shirt.</li> Helinox chair described above</li> Kifaru woobie likewise</li> Gear Doctors self-inflatting sleeping pad</a> - This is, admittedly, a luxury item. But when you take this guy out of the bag, toss it on the floor, open the valve and watch him self-inflate, it’s worth it. As I alluded to above, air raids often happen at ungodly hours, so you’re tired and cranky and want to go back to sleep. This mattress is easy to deploy, and gives you a place to lie down in relative comfort. Combine that with the Kifaru woobie and it’s possible to get a bit of fitfull sleep.</li> USGI MRE (menu 12; elbow macaroni in tomato sauce) and First Strike Ration - I’ve never had to stay in the shelter long enough that food became a serious concern, but it has happened earlier in the war and could possibly happen again. These two items are shelf-stable for years, and I know that I don’t mind eating them. Combined this is well north of 5K calories, with plenty of small items to share with fellow vault-dwellers.</li> Illumination - I admittedly have gone overboard on lighting, but I have a phobia about being trapped underground in the dark. Metro stations get priority for electricity, but Russians specifically target power plans and other electrical infrastructure and it’s not impossible for a metro station to lose power for a time. So I have taken the three-is-one-and-two-is-none approach to lighting: NiteCore LR60 USB-C rechargeable LED lantern</a> - This is a very versatile lantern, which has a long life at the lowest brightness and can also be used to illuminate a large area. If the lights go out it’ll be a freakout, so being able to cast a lot of light around might provide some bit of comfort to my neighbors as well.</li> COAST HL8R LED headlamp</a> - This is possibly among the highest quality head lamps on the market. It’s definitely an extravagance, but I had it in my hurricane preps from long ago and brought it with me to Ukraine. I shouldn’t have to explain the utility of a headlamp in this situation.</li> Fenix LD15R right-angle flashlight</a> - this tiny flashlight has a convenient clip that you can clip to the backpack shoulder strap. Because it’s a right-angle light, the beam will shine ahead of you. It’s cheap enough that I could loan it or give it out to a needy neighbor if necessary, while my remaining lighting options will provide me with the illumination I need.</li> </ul> </li> Two bottles of water (stored in the Vanquest Hydra pouched described above)</li> No-name black sleeping mask with ear plugs - The Metro stations are usually rather brightly lit in most places, and very noisy. If one is groggy and annoyed and needs to sleep, even with luxuries like a sleeping mattress and woobie, it will be hard without a sleeping mask and ear plugs.</li> Toilet paper - the metro stations don’t have toilet facilities normally accessible to passengers, but during air raids I think they can make them available on request. Do you really want to find out if there’s any toilet paper left?</li> N99 respirators - Metro stations have ventilation, but it’s possible there will be smoke and dust to get to the metro station, or afterwards when the air raid alarm is canceled. These obviously aren’t enough to breath in heavy smoke in a confined space, but they will be enough to protect my lungs if the air is breathable but full of nasty particulates.</li> GQ GMC-500Plus Geiger Counter</a> - Laugh all you want, but Russia threatens nukes pretty regularly. One of these days they might just be dumb enough to throw a hot rock my way. I have read the Good News About Nuclear Destruction</a>, so I know that a nuclear strike isn’t automatically instant death for the entire planet, or even for the entire city that was attacked. In the first few days after an attack, avoiding fallout is critical. I don’t want to have to guess whether I’m doing a good job or not; this Geiger counter will confirm my level of exposure. Also in the nylon case is a reference card with various levels of radiation exposure and their lethality, to help make an informed decision about what to do in a given situation.</li> IOSAT Potassium Iodide pills</a> - These are potentially life-saving in the event of a nuclear blast, and they are cheap and lightweight so anyone who carries a freaking Geiger counter should definitely also carry these!</li> Wise Owl Outfitters Camping Towel</a> - Bring a towel</li> Various small sustainment items Pepcid AC antacid tablets</li> Wet Ones antibacterial wipes</a> - my hands quickly get dirty down in the metro and there is not usually a convenient place to wash them.</li> FOMIN paper soap sheets</a> - these are handy little “paper” slips of soap. If you have water but don’t have soap, you get your hands wet and this paper slip disolves into antibacterial soap. A single little box has 100 sheets. As long as you can keep the sheets absolutely dry until you’re ready to use, it comes in handy in places with running water but no soap.</li> Lip balm</li> Medicines Advil liquigels</li> My prescription meds, enough doses for a few days</li> </ul> </li> PPEasy disposable urine bags (I bought these in Ukraine; on Amazon a similar product is Disposable Urine Bags</a>) - Hopefully I don’t need to explain what these are for or why one might want them in a crowded air raid shelter with limited toilet facilities…</li> </ul> </li> </ul> Not visible for privacy reasons: Cash in USD and EUR</li> Must-have documents like Ukrainian residency documents, marriage certificate, apartment deed, passport, driver’s license, etc</li> </ul> That’s everything. I hope you never find yourself in a situation where you can say that you found this post helpful. Quick and dirty hack to automate tedious expense report task 2024-10-18T00:00:00+00:00 I really dislike business travel, but I really REALLY dislike expense reports. They are always so tedious, made worse by user-hostile war crimes like Concur (cursed be its name!). My most recent trip from Kyiv to Boston and back, however, presented a particularly unappealing task: nearly a dozen receipts spread across four currencies and two languages. Thankfully Elastio is not yet a big enough company to waste money on Concur licenses, and we still have an informal process whereby we email receipts to our accountant. However, for her sanity and mine, I needed to make a spreadsheet organizing each expense, including its description, amount, date, and some reference back to the PDF containing the receipt (recall that these receipts are in multiple languages). The prospect of manually entering all this data into a spreadsheet was, to put it mildly, unappealing. It would take me at least 10 minutes of tedious labor. So I decided to spend 15 minutes fucking about with Anthropic Claude Sonnet 3.5 to make it do it for me! The Problem</h2> Here’s what I was dealing with: 11 individual receipts</li> 4 different currencies (USD, EUR, HUF, UAH)</li> Documents in both English and Ukrainian</li> A mix of formats: hotel bookings, flight receipts, train tickets, and taxi fares</li> </ul> I can read the languages involved, but our accountant doesn’t read Ukrainian, and I’m not sure what our rules are for currency conversion. I decided to leverage the power of LLMs, specifically in this case Claude 3.5 Sonnet. The obvious solution that doesn’t work.</h2> At first I thought this would be extremely trivial. I just dragged all of the PDF receipts over into Claude’s web UI and… But I have more than 5 PDFs! FML! The Quick and dirty solution that does work</h2> When it comes to being lazy, I can be very persistent. I invoked GPT-4o from my terminal (this would have also worked with Claude) to find out what the terminal one-liner to concat a bunch of PDFs would be. It came up with: pdfunite *.pdf output.pdf </code></pre> But I don’t seem to have pdfunite</code> in my path. A follow-up question revealed that it’s part of the poppler</code> Brew package on macOS: brew install poppler </code></pre> Having run pdfunite</code> successfully, I had concatenated a bunch of small PDFs into one big 8MB PDF with all of my receipts. I wasn’t sure if Claude would be smart enough to understand that this was in fact multiple receipts, especially since some of them were multi-page receipts (Bolt, why!??). To ensure it had plenty of context, I used eza --oneline --no-icons *.pdf</code> to make a list of all of the PDF file names that went in to the monster PDF, and hoped for the best: From the attached PDF containing all receipts from my recent business trip, generate a CSV with the following: Date of expense</li> Description</li> Amount</li> Currency</li> </ul> I had to concatenate all receipts into one PDF file due to limitations on attachments. This PDF contains the following individual PDFs: [List of PDF filenames produced with `eza --oneline --no-icons *.pdf`] </code></pre> </blockquote> After getting the initial results, I was pleased to discover that my hack actually worked! But the description didn’t quite match the file names, and yet the description was more informative than the file names that I used, so I decided the Excel sheet should contain both the PDF file name of the receipt, and a description of the expense. That was easy: Add another column, after “Date”, called “Filename” with the name of the PDF containing that particular expense, from the list I provided earlier. </blockquote> The Results</h2> The LLM produced a neatly formatted CSV that looked like this (most rows redacted): Date,Filename,Description,Amount,Currency 2024-09-26,2024-09-24 Train from Kyiv to Western Ukraine.pdf,Train from Kyiv to Kolomyia,1689.65,UAH 2024-09-12,2024-09-12 BUD to BOS Flight Receipt.pdf,Air France flight from Budapest to Boston,996.10,USD 2024-10-12,2024-10-13 Taxi from BUD airport.pdf,Taxi from Budapest Airport,12630.00,HUF 2024-10-19,2024-10-19 Train from Western Ukraine to Kyiv - Part 1.pdf,Train from Uzhhorod to Kyiv,858.98,UAH </code></pre> To my amazement, Claude correctly figured out which original PDF filename went with which expense in the concatenated PDF, even though there was nothing in the concatenated PDF to link any particular page to its original file name! This quick hack saved me at least 10 minutes of mind-numbing tedium, and took no more than 15 minutes including fucking around with the PDF concatenation. If this were a thing I had to do regularly, I could get Claude to package that up into a shell script and it would take a minute at most to prepare the spreadsheet. In fact, I’ve spent much more time writing this up than I did actually doing it. I just wanted to capture the satisfaction of turning a tedious task into the much more interesting task of automating the tedium. I also think it will be fun to read this in a year or two, when presumably this kind of thing is built into Office and kids point and laugh at the old people who gasped in wonder at the capabilities of primitive LLMs. Even in that bright future, though, Concur will be a piece of shit! Fuck you Concur!! My Air Raid Response Protocols 2024-08-31T00:00:00+00:00 In a previous post</a> I made mention of the air raid alarms that we have all become accustomed to hearing in Ukrainian cities almost daily. This post expands on that topic, describing how I gauge the risk of and air raid and how I respond based on that risk. This is just my own opinion, trying to make do in a difficult situation as best that I can. My colleagues and friends in Ukraine all have different approaches to this threat. Immediate Response to Air Raid Alarm</h2> I find out about an air raid in one of two ways (sometimes both simultaneously): an alert on my phone from the Digital Kyiv mobile app, or the sound of the air raid sirens on the street outside. At this point in the war, in central Kyiv at least, the air raid alarm can be caused by a wide range of signals, which vary in specificity and risk. So, my first step is to try to understand why this signal has been triggered. To my knowledge, there aren’t real-time official government sources for this information (if you’re aware of something, get in touch!). There are, however, a number of Telegram channels that I look to for information. A few if you’re curious are: Радник (“Radnik”)</li> @war_monitor</li> @insiderUKR</li> </ul> By the time the air raid alarm sirens are audible, there’s usually something in one of these channels as to why. If not, I can usually find something within a minute or two of the alert. The key is to determine what is causing the alert, because that determines my reaction to the threat. I evaluate this based on two factors: What kind of threat is it?</li> How specific is it?</li> </ul> I’ll share some examples below. Russian Military Aviation Taking Off</h3> </picture> </a> Alert about a MiG-31 taking off</figcaption> </figure> Often when Ukrainian air defense radar detects Russian combat aircraft taking off from Russian air bases, if those aircraft are capable of carrying air-launched missiles that can threaten Ukraine, an air raid alert will be triggered for a large part of the country, just in case. Ukrainian defenders can’t know immediately if the aircraft is on a combat mission, or if so what the target is, so if there’s a potential threat they trigger the alarm. If I see this is the reason for the alert, I take no action. I consider the risk to be minimal, and not worth the disruption. Ballistic Missile Launch (Generic)</h3> </picture> </a> Alert about a ballistic missile threat from the north</figcaption> </figure> This indicates a ballistic missile was detected (in this case from the North but they can come from the East or from Crimea in the South). This is usually a very broad alert as well, because ballistic missiles move very fast there’s not much warning, so the alert can apply to the whole country or several oblasts. This gets my attention, and makes me more wary. If possible I’ll try not to stand next to an exterior window, and if I’m out walking I may switch to the underground passageways if I’m in an area that has them. But I don’t consider it a big risk. Ballistic Missile Launch (Specific)</h3> </picture> </a> Alert about a ballistic missile threat in Kyiv specifically, advising to take shelter.</figcaption> </figure> This gets more attention. The threat is specifically to the Kyiv region. My response is the same as a generic ballistic missile alert, but I am more nervous. Multiple Shahed Drones in Kyiv</h3> </picture> </a> Alert about multiple Shahed kamikaze drones in Kyiv</figcaption> </figure> This I consider to be a more dangerous situation. Particularly if it’s a large number of kamikaze drones. The AFU air defenses are particularly good at shooting these down, but even when shot down they still have to land somewhere, and sometimes their warhead hasn’t detonated before they hit the ground. The risk of falling debris from a successful interception by air defense exists for the ballistic missile scenarios as well, but given the larger number of drones the odds that some debris will hit me go up. This is just the math of the situation. Shaheds are also much cheaper than ballistic or cruise missiles, so Russia fields more of them. In this case I will make sure I’m not close to any windows, I’ll close any windows that are cracked (of course explosives can blow out the window when it’s closed, but it will take more force to blow out a closed window than one that’s cracked), and may move to a position in the apartment where there are thick brick walls between me and the outside. A direct hit from above is probably game-over no matter what, but lateral impacts could be survivable if one is shielded from the blast and not sheltering directly at the point of impact. On a somewhat humorous note, the Iranian bastards who sell these drones to Putin call them “Shahed”, which I think is the term in Islam for a holy warrior who martyrs himself. However, when writing messages about drone threats, Radnik calls them “шлюхеді” (pronouned “shlyuhedy”), which is a portmanteau of an offensive Ukrainian word for a promiscuous woman, and “Shahedi” (plural form of “Shahed”). It makes me smile every time I read those alerts; the juvenile vulgarity is funny when juxtaposed with the seriousness of the topic. Mass Attack</h3> </picture> </a> Screenshot of the Radnik telegram channel in the morning of Monday 26 August 2024, as a massive Russian attack on Ukraine was underway. 1) Cruise missiles vectoring towards Kyiv from the Chernihiv region 2) 6 Tu-22M3 bombers reached missile launch range in Kursk, Russia 3) 4x Tu-95MC bombers taking off from Engels air base, Russia</figcaption> </figure> I can’t tell you what I would do if I saw this, because I found out about this mass attack when I woke up to the sounds of explosions, either from AFU air defense batteries or from Russian cruise and ballistic missile impacts. I like to think I’d go to the shelter proactively. Mitigating Factors</h2> One of my rules is that if I hear explosions, either air defense batteries firing or ordnance landing, I go to the shelter. My thinking here is that when something has already kicked off close enough that I can hear it, the odds of additional impacts dramatically increase, as does the risk. I’ll talk more about the protocol for bugging out to the shelter a bit later. A mitigating factor in the other direction is sleep. While it’s possible to configure the Digital Kyiv app to play a loud alarm sound when there’s an air raid alarm, regardless of the time of day or night, I turned that off after a couple of days here. It’s simply unbearable. It’s not possible to get a normal night’s sleep, and in most cases the cause of the alarms are such that my standard protocol is to take no immediate action anyway. It happens that most Russian attacks are in the early morning, between 4AM and maybe 7AM, so it happens often that I ignore air alarms by virtue of being asleep, and discover the threat when I awake to the sound of an explosion. Bugging Out</h2> If I hear an explosion, it’s time to go to the shelter. It’s a simple rule, that can be evaluated even when one is groggy from just waking up. When this happens my brain tries to negotiate it’s way out of this. It thinks “maybe that was just a car back-firing. it wasn’t very loud. it sounded far away.” If I’m very tired I sometimes let it get away with this, until I hear another explosion that is undeniably an explosion. But in any case, at some point, booms == bug out. I have a pre-arranged plan for how this works. My wife’s and my bug-out bags (remember from a prior post, Ukrainians call these “alarm suitcases”) are next to the door, as are a pair of 5.11 pants, socks, and a shirt. I jump out of bed, grab my phone and laptop, stow my laptop in a dedicated pocket in my bug-out back, put on my 5.11 pants, shirt, and socks, then boots and an ankle-mounted tourniquet. We leave the apartment, lock the door, and make haste down the stairs (I live on the 6th floor of an apartment building with no elevator), and across the street to the nearest Metro station. During this process I always feel silly, because inevitably there will be people out on the street not reacting at all. I feel like I’m the one chicken-shit who can’t handle a little bit of exploding Russian ordnance, and feel ashamed. But then I get into the metro and it’s packed w/ people taking shelter, and I don’t feel so stupid after all. It’s a bizarre psychological quirk. It happens every time. Sheltering</h2> Once I arrive in the shelter, I feel a sense of relief. This metro station is deep underground, dug specifically to provide shelter against NATO nukes in the event that WW3 kicked off in the Ukrainian SSR. I was not in Kyiv in the early days of the war, when people were living in metro stations for days on end under relentless Russian barrage. I presume it was terrifying then. But now, the mood in the metro during an air raid is probably not what you’d expect. It’s quite casual. If you didn’t know the context, you’d think it’s just a bunch of Ukrainians waiting their turn at the DMV to renew their driver’s licenses, only for some reason doing it in a metro station. Having done this a few times now, I carry some preps to make it more comfortable. I may do a separate post about the actual contents of my bug-out bag, but for now I’ll say that I have a packable camp chair that I deploy and can relax in comfort as I wait for the threat to pass. </picture> </a> POV: Monday morning, 26 August 2024, a massive Russian attack on Ukraine is underway and you're chilling in the Metro station in your camp chair.</figcaption> </figure> As you can see in the photo, many Kyivans have the same idea. There are also some very compact folding camp stools that were donated to Ukraine from somewhere, that they pass out during air raids for people who don’t have their own chairs. I sat on one of those during my first air raid, and ordered my wife and I camp chairs while we were still underground. All-Clear</h2> When I see the all-clear signal for an air raid that I didn’t do anything to respond to, it’s a kind of smug relief. Something like “ha, I rolled the dice and won! it wasn’t anything to worry about!”. When I get that message from inside the shelter, it’s a different kind of relief, like the recess bell ringing at school. Typically I’ll be in the shelter for 30 minutes to a few hours, and it gets very dull. Especially since these air raids almost always happen when I’m asleep, so I’m tired and groggy and not in any mood to doom-scroll or get some work done. Everyone generally gets it all at once, and there’s a mass exodus out of the shelter. It takes me a couple of minutes to pack up my fancy camp chair, so I’m usually one of the last ones out. I have never emerged from the shelter nervous about what I might find, worried that maybe my apartment building was hit and all of my stuff is scattered across the courtyard in a smouldering mess. Mainly because I think I would notice it even from underground. In my bed, I can feel the rumbling in the ground when a train goes through the metro 100s of feet below me; I’m fairly sure a Russian warhead exploding where my bed is would be noticeable from inside the metro station as well. In fact, I always emerge strangely carefree, as if the threat is definitely passed, there’s nothing to worry about, we survived, now let’s get on with our day. Often the wife and I will get breakfast afterward, if restaurants are open yet. Aftermath</h2> You might imagine that this whole life-and-death experience is psychologically draining, and maybe leaves some emotional scars. But it’s not like that. It’s…mundane. It’s an annoyance, like having to wait a long time for a bus. The worst part is the sleep disruption, since as I’ve noted these attacks usually happen in the early hours of the morning. I know I’ll be mostly useless the whole day due to the sleep deprivation. I can’t complain about this to my Ukrainian colleagues since they obviously live through it (although most of them just stay in bed and try to ignore the explosions), and I can’t complain about it to my American colleagues because they wouldn’t understand and I don’t want to alarm people unnecessarily. I almost never find out details about what happened in the air raid. Were those explosions Ukrainian air defense batteries, or Russian ordnance blown up mid-air, or a successful hit on a Russian objective? Where were the explosions relative to me? What was the damage? This is intentional, and in fact it’s illegal to post information on social media that would help the Russian forces perform a post-attack BDA (Battle Damage Assessment). However, sometimes you know. In the case of the Monday attack, it was widely publicized. </picture> </a> Trajectories of Russian ordnance in mass attack on Monday 26 August 2024</figcaption> </figure> </picture> </a> Infographic showing the largest rocket attacks on Ukraine in the war. The attack on Monday 26 August 2024 was the largest of the whole war, with 127 rockets launched at Ukraine (102 of which were intercepted), as well as 109 Shahed drones (of which 99 were intercepted).</figcaption> </figure> Strangely, after surviving one of these attacks I feel some sense of accomplishment. I didn’t do anything, I didn’t help shoot down any Shaheds or rescue any elderly from a bombed-out building or even find someone’s lost cat in the shelter. But nonetheless there is a feeling that my mettle was tested and I passed somehow. War does strange things to the mind. What daily life is like for an expat in Kyiv in August 2024 2024-08-24T00:00:00+00:00 Today, 24 August, is the day when Ukraine celebrates her independence. Happy Independence Day, Ukraine! I haven’t posted anything about daily life in Kyiv, either before the war started or after. Walking around the city center today I thought it might be interesting to describe what it’s like here, from an expat’s perspective. Daily life is mostly normal</h2> Even though a brutal large-scale war rages on a few hundred miles away, life goes on. People still go to work, they still want to relax on the weekends, they still go to restaurants and bars and cafes. My favorite places to go from before the war started are still there and are still my favorite places to go. </picture> </a> Kyivans out enjoying the nice weather on Independence Day, on Independence Square. "Майдан Незолежності" in Ukrainian; this was the location of the revolution in 2014, which is why that revolution is sometimes called the "Euromaidan" for short ("maidan" just means "square"; "Euro" because the proximate cause of the revolution was a desire to integrate further with the EU). Officially Ukrainians call it the "Революція Гідності" or "Revolution of Dignity".</figcaption> </figure> Those of you fortunate enough to have never been in a conflict zone might find this hard to believe. I bet, however, that Israelis understand what I’m talking about, as will anyone who’s been in the rear echelons of a war zone. I had intended to include some more photos of normal life, but they were just so normal and boring that I changed my mind. If you’ve lived in any European capital you would immediately recognize life here. It’s, for the most part, normal. But not entirely, which brings us to this post… Air Raid Alarms</h2> Though Kyiv is far from the front, it’s well within range of a variety of Russian weapons, including Iskander ballistic missiles, various cruise missile varieties, the “Kindzhal” hypersonic ballistic missile, and Shahed drones from Iran (which Russia pretends are their own weapons system called “Геран-2”, literally “Geranium 2” LOL). Recently, Putin made some deal with Kim Jong Un for the use of North Korean ballistic missiles. As a resident of a city that these missiles are launched at, let me first say “fuck you KJU!”, but also, thanks for being such an incompetent shit-tier despot that your incompetent weapons industry can field such unreliable ballistic missiles. It’s likely that some of these NoKo missiles has hit its target (see above “fuck you KJU!”), but the Ukrainian government isn’t shy about disclosing the times when they fail or just blow up mid flight. Institutional incompetence FTW for once! Anyway, the reality of life here is that there are air raid alarms at all hours of the day or night, sometimes multiple per day. </picture> </a> Five air raid alarms in one day on the "Киів Цифровий" ("Digital Kyiv") mobile app.</figcaption> </figure> There are mobile apps that alert you when there’s an air raid alarm in the area, although unless you’re inside somewhere the sirens are very audible. They sound just like the air raid sirens in every nuclear apocalypse movie you’ve ever seen. They only sound for a few minutes though, presumably to preserve the sanity of the survivors. Most Kyivans by now just ignore these alarms. That might be hard for you to imagine, but IMHO this is actually a healthy adaptation. The alarms come at all hours, but especially late and night and early in the morning. If every time there was an air raid alarm you woke up, grabbed your essentials (you do have an air raid bag packed in advance, right?), and ran to the shelter (which is in most cases either a musty basement or a metro station; in my case the latter), and sat on the floor in either hot stuffy or cold temperatures for an hour or two or three, you would have gone mad by now! If you still find that hard to imagine, you should realize that “air raid alarm” doesn’t mean “a missile is inbound directly to my location”. It could mean that a jet capable of launching a Kindzhal took off from an airbase in Crimea, so the whole country is under an alert. Or it could mean that Shaheds were spotted outside the city. One time I heard an air raid alarm go off, and less than 30 seconds later one of the fancy new Zircon hypersonic ballistic missiles destroyed a building a few miles away, with an explosion so loud that I would have sworn it was next door to me. According to the Digital Kyiv app, there have been 1222 air raid alarms in Kyiv during last 2+ years of war. Out of those 1222, very few impacted the capital; even fewer in the center where I live and where there is stronger air defense than probably anywhere else in Europe. And as I mentioned, life goes on; people need to work, their kids go to school, etc. You can’t blame the Ukrainians for ignoring these most of the time. However, those “very few” impacts do weigh on my mind… The psychology of being under an air raid alert is very odd for me as an expat foreigner. I have not been able to fully adopt the “IDGAF” nonchalance of my Ukrainian colleagues. Every time I hear an air raid alarm, I immediately look at the Telegram channels where various people post about why an alarm has sounded. I try to gauge the level of risk, whether I should at least go sit in the more secure part of my apartment away from windows (I almost never do that since it’s so inconvenient to pick up and move my workspace), or just close the windows and remember that none of us get out of this life alive. My personal rule is that if there are audible explosions, it’s time to make haste to the shelter. This is hardly a 100% risk-free plan (recall that Zircon I mentioned that hit with < 30 seconds notice, and no sound before it impacted its target), but it’s pragmatic. If I required a situation in which there is 0% chance of being killed by Russian ordnance, I would not be here. In cases of a massive drone or missile strike, the first explosions are likely the sounds of air defense batteries opening up, and the assault could last for multiple hours; in that case I’ll be safe underground. I’ve also developed a sensitivity to sound as a result of being here for a while. If I hear a car backfire or a delivery truck drop a load, or the trash truck dropping our trash bins late at night, I immediately alert and think it was an explosion. Similarly, any sound that sounds like an air raid alarm gets my attention. Sometimes as I’m laying in bed trying to go to sleep, I’ll swear I hear an alarm, and have to check the Kyiv Digital app and satisfy myself that there is not any threat and it’s all in my head. Part of my coping strategy for this stress is preparing for possible adverse outcomes, which gives me a feeling of having some control over the situation. I, like most Kyivans, have a bag packed in case I need to run to the shelter. I would call this a “bug-out bag”; they call it “alarm suitcase” (тривожний чемодан). Maybe I’ll do a post some day about what’s in my bug-out bag. For now, suffice it to say it only bears a passing resemblance to what an American prepper would pack, because it’s for a very specialized bug-out scenario. </picture> </a> My and my wife's bug-out bags, in front of additional preps in case of loss of water and shortage of food.</figcaption> </figure>Ukrainian Losses</h2> Of course, the human cost of the war is incalculable. </picture> </a> This ground on Independence Square is densely packed with small Ukrainian flags memorializing the war dead, as well as large flags of units, countries whose citizens have died in the fight, and other symbols.</figcaption> </figure> I don’t know anyone who hasn’t lost someone in the war. One of my acquaintances has been killed already, and several more are serving in the AFU now. This level of loss leaves a mark on the people who suffer it, which will be felt for generations. As the war rages on, the government deliberately avoids talk of losses, and crows about victories while usually staying mum on what they cost. That is probably the right strategy, but still there must be some immediate acknowledgement of the suffering and loss. Once such acknowledgement is on Independence Square. I don’t think there is literally one small flag for every Ukrainian KIA, but even so it’s a striking visualization of the scale of the loss. Russian losses are said to be from 3x to an order of magnitude higher. I have no sympathy for Russian casualties, however. I doubt many Ukrainians do either. The Nazi invasion of the Soviet Union was over 80 years ago, and you’d still be hard-pressed to find anyone in the former Soviet republics to voice any kind of sympathy for the German soldiers who perished on the Eastern front. I would be surprised if Ukrainian hearts soften any sooner for the 21st century’s facist stormtrooper, the Russian invader. Speaking of Ukrainian attitudes towards the Nazis vis a vis Russians, there’s an effort underway to purge Soviet and Russian culture from Ukrainian civic life, about which more later. Visible Reminders of War</h2> Even in the center of the capital, which Russia never occupied, there are many reminders of the destruction they have wrought, and the armed struggle of the Ukrainians to remain free. </picture> </a> The Samsung building, near the central train station in Kyiv. It was damaged in a Russian missile strike in 2022. I don't know what the weapon was, or its intended target, but this building is across the street from a power plant (a smoke stack of which is visible to the right of the frame) so maybe it was collateral damage. Photo taken July 2024.</figcaption> </figure> Though the Ukrainians try to rebuild destroyed buildings, that doesn’t always happen, nor is it a quick process. The Samsung building is boarded up and closed. I’m not sure what the plan is for it. Perhaps it doesn’t make sense to repair it when the risk of another strike remains. For now it’s a striking reminder of the war. Equally striking is what you can’t see: Ukrainian monuments are packed in sandbags and barriers to protect them from shrapnel in the event of an attack. </picture> </a> Statue of Taras Shevchenko, hardened against Russian ordnance.</figcaption> </figure> You’ll have to trust me that this is actually a larger-than-life statue of Taras Shevchenko, a famous Ukrainian poet, in the park named after him in the center of the capital. Pretty much any statue of any cultural significance is wrapped up this way. To my knowledge, Kyiv hasn’t lost any statues to Russian attacks as yet, and they are clearly intent on keeping it that way. Throughout the city one frequently sees fortifications, usually unmanned and dismantled but sometimes very much manned. </picture> </a> Collection of concrete barricades and tank traps, rusting in a parking lot in central Kyiv, hopefully never to be needed again.</figcaption> </figure> At the start of the war, there were checkpoints all over (called “blockposts” in Ukrainian for some reason), and key roads were blocked by tank traps and barriers, like I remember from my days in Baghdad. I have only gone through one checkpoint in my time here, in the Pechersk district very close to the Ukrainian parliament building. They are however still in operation throughout Ukraine, enforcing Ukraine’s mandatory registration for military service for Ukrainian men as much as guarding against Russian inflitrators. De-russification</h2> The historical reality is that modern-day Ukraine and modern-day Russia were, up until very recently, very close in terms of politics, culture, and geopolitical outlook. While the Ukrainians have a distinct Ukrainian language (to any Russian bot who claims it’s just a dialect of Russian, it sure doesn’t feel like I’m learning a dialect when I struggle to get out a few words of Ukrainian), Russian is widely spoken and a lot of Russian cultural touchstones are instantly recognizable to Ukrainians. Arguably, Russians were invented here in Ukraine in the Kievan Rus a millenia ago. Ukraine was, it must also be said, the enemy of NATO and especially the United States while it was a part of the Soviet Union. An American executive with whom I was slightly acquainted years ago served in the US Air Force as a bomber pilot during the Cold War. When it came up that I was in Ukraine, he joked that he knew Kyiv very well–from the air. He was familiar with the strategic targets that he would be bombing if WW3 kicked off. I mention this because my early years visiting Kyiv (I first came here in 2009) came at a time when Ukraine was ruled by Russian-backed politicians, where Russian language media was widespread, the Russian language was almost always spoken in the capital, and travel for work and pleasure between Ukraine and Russia barely registered as an international trip. Americans were regarded with the same skepticism that my parents would have regarded Russians in the 80s. Since 2014 but especially 2022 that’s all changed, utterly. De-russification is in full force. If you were very familiar with the old Kyiv of brotherhood with Russia, it is a striking change (but, it must be said, completely understandable and richly deserved). Here’s but one example: At the Motherland Memorial (Батківшина-Мати), in 2010 I took this picture of two Russian tanks: </picture> </a> Soviet T-80 (I think?) (left) and T-62 (right) tanks on display at the Motherland Ukraine monument in Kyiv, September 2010.</figcaption> </figure> At the time I was struck by how they have painted the tanks in whimsical colors, as if to make fun of the horrors that were visited upon the Soviet Union in WW2. I even commented on this at the time to my Ukrainian colleague. There was a feeling that this was all very firmly in the past, not to be forgotten sure, but also not important to think about often. Contrast that with this same location, a month ago: </picture> </a> Soviet tanks augmented by a trophied Russian tank taken from a battlefield in Ukraine, on display at the Motherland Ukraine monument in Kyiv, July 2024.</figcaption> </figure> A very different vibe, if you ask me. This is no longer a distant cultural memory of a threat long past and barely even conceivable to the modern mind. This is the re-awakening of the old enemy, a fight that was thought won which must now be fought once again. An unforgivable sin that has been committed, and that will be avenged. That’s not all that’s changed at the Motherland monument. It used to list the names (written in Russian) of the major cities of the Soviet Union where WW2 was fought. Kyiv was on the list, as was Moscow, Minsk, and several others. They’re still there, but the text is obscured by camouflage netting: </picture> </a> Russian names of Soviet cities involved in WW2, covered as part of de-russification, at the Motherland Ukraine monument in Kyiv, July 2024.</figcaption> </figure> One more major change. See if you can spot the difference between the photo above and this one: </picture> </a> Motherland statue, Kyiv, September 2010. Until 2023, the Motherland statue in Kyiv had the Soviet coat of arms on her shield. It was modified in 2023, replacing the Soviet symbol with the Ukrainian trident, as part of de-russification and de-communization of Ukrainian civic life.</figcaption> </figure> The Ukrainian government finally replaced that very Soviet symbol on the shield w/ the symbol of Ukrainian statehood, the trident. This pissed off a lot of Russians, who without any sense of irony thought it was outrageous to vandalize a cherished monument to the Great Patriotic War. AFU Recruitment</h2> When I first came to Ukraine in 2009, there was no emphasis whatsoever on military service. They had inherited the Soviet system of conscription, but I saw no signs of any effort on the part of AFU to recruit soldiers to the ranks. Serving in the Army was not a particularly prestigious profession, and those that could sought to avoid it. Contrast that to now. Due to the general mobilization, technically all fighting age men 25 and up can be called up, with the exception of those who are medically unfit, the father of 3 or more children, and a few other edge cases. It’s now possible, with some complications AFAIK, to volunteer to join not just the Army, but a specific unit and a specific job. Some men calculate that it’s better to choose your unit and MOS than to get drafted and end up in some frontline assault group with all of the risks that entails. A consequence of this policy change is that individual units can recruit specifically for themselves. They advertise all over. </picture> </a> Recruiting poster for the 3rd separate assault brigade (III окурема штурмова бригада), which reads "fly into the third assault [brigade]" ("залітай у третю штурмову").</figcaption> </figure> This one is from a very well-known brigade, the third separate assault brigade (3ОШБ), which has fought in some of the hottest spots of the war. As the name implies, they are a trigger-pulling front-line unit. Like any armed formation in this conflict, they are on the cutting edge of drone warfare, by necessity, and they need drone operators. There’s also a darker side to recruitment, for which I don’t have any photos: there are units of the military whose job it is to draft men into the army, and sometimes those men don’t want to go. This leads to various potential confrontations, and there are many stories of corruption and abuse by these units. Not having anything to fear from these units is part of the way the expat experience differs from that of the typical Ukrainian man. That, and the fact that I can come and go across the border as I please. Critical Infrastructure</h2> Part of Russia’s strategy has been and continues to be the destruction of Ukrainian civil infrastructure, with a particular emphasis on electrical infrastructure. I read a report in the Kyiv Post</a> which claimed that Ukraine may have lost 80% of its power generation capacity due to a combination of Russian attacks and Russian occupation. Whatever the number, the Ukrainian electrical grid is under significant strain. As a result, rolling blackouts are common. In my specific location in the center of the city, power is pretty reliable. The worst I’ve experienced since returning was July 2024, during a heat wave, when I had 2 hours of electricity in every 8. In that case, the Digital Kyiv app shows the schedule for my address for the day, and it’s usually pretty accurate. Being able to know when the outages will happen, and having them spread out with brief periods of restored power in between, goes a long way to making them bearable. </picture> </a> My backup power system, an Ecoflow Delta Max portable power station, and 2 additional batteries. This gives me 6KWh of total backup power.</figcaption> </figure> What also makes them bearable is that it doesn’t actually impact my day that much. I have an Ecoflow Delta Max power station, and I recently obtained two additional external batteries for it. With this system, my Internet connection, a Synology server, and my refrigerator are all powered off of the Ecoflow. During the heat wave, I also ran a mini split A/C unit in my room for brief periods of time to cool off when the heat became uncomfortable. The Ecoflow and batteries can recharge within a two hour window. With the rolling blackouts we had, I needed backup power for about 6 hours, followed by two hours of power which recharged the Ecoflow batteries. If I had a 24 hour outage, I’d be in trouble. But the regular, scheduled, rolling blackouts we’ve had so far aren’t a problem; I quickly got used to it and they cause me little more than a slight annoyance. Asute readers might ask “but what about your ISP”? ISPs have adapted to the conditions, and many of them offer fiber installations with extensive backup battery power at each building. If you are fortunate enough to have such a provider (as far as I know the waiting list now is months long), you will still have Internet access as long as you can power your own router/wifi access point. I and several of my colleagues work uninterrupted through blackouts, which is the only way I’m able to continue to live here. Running generators in an apartment is not legal, nor is it safe. I’m sure it happens, particularly outside the center, but I’ve never felt the desire to do so. For businesses, especially retail businesses, it’s another story. Here in the center, most of them have generators by now, which are placed on the street with a cable run inside. Some are the little gas generators that you can buy at Home Depot in the US, but more and more I see proper industrial generators with longer duty cycles and greater reliability, and a wiring job that might have some chance of passing a US code inspection. You can tell when the power is out in an area by the sound; if there’s a deafening roar of generators, you can be sure that the power is out. This does mean that walking around the city when there’s no power isn’t very pleasant, due to the noise and also the air pollution. I don’t like it either, but the Ukrainians are just adapting to the hardships imposed upon them by Russia. Curfew</h2> In most of Ukraine there is a curfew every night, from 12AM to 5AM. Only authorized vehicles and personnel are allowed outside during curfew. Anyone caught out without authorization is subject to detention. This has a massive impact on the logistics of urban life. Bars and clubs get started much earlier, because around 11PM they have to close. Restaurants also can’t be open very late, and if you are hungry for delivery you’d better get your order in by 9PM or so or you might not get it at all. When visiting a friend’s place, as the evening gets late you have to either cut things short and go home, or you’re going to be spending the night. And there’s a mad rush for taxis starting around 11PM, so if you lost track of time and it’s 11:30 or 11:40 you’ll be lucky to get a taxi at all and it’ll be priced accordingly. I don’t understand the military purpose of the curfew at this stage in the war. During the Battle of Kyiv in the early days it made sense, as enemy agents and saboteurs were actively operating in the city. Does it really meaningfully improve the security of the population now? I’m not sure. Conclusion</h2> This post ended up a lot longer than I intended. Maybe I’ll do a follow-up on air raid preps and procedures, I didn’t cover that as much as I wanted to. I moved to Kyiv 6 years ago, to work on a new idea I had for a startup. I greatly enjoyed my time in Kyiv, right up until the full-scale invasion started in February 2022 and ruined it all. I returned to Kyiv in September 2023. I don’t know for how much longer I will remain here. But I will always have a fondness in my heart for the city of Kyiv and the Ukrainian people, and I hope someday soon this war will all be just a bad memory that old people go on and on about but the young find boring and irrelevant. Calling SOAP APIs from Rust 2024-08-10T00:00:00+00:00 Approximately 15 years ago, I was running the engineering team at AppAssure Software, working on what was at the time a cutting-edge new feature: the ability to restore our backups of your physical servers, to a VMWare virtual machine! This required solving many challenging problems, not least being figuring out the ways in which your Windows install had to be manipulated to ensure it would properly boot into a guest VM when it had spent its life up to that point inhabiting a physical Dell or HP or Supermicro server. However, such low-level technical challenges paled in comparison to the horror of integrating with the VMWare vSphere SOAP API! I can’t remember every hard thing I did over my long career, but that VMWare API integration stands out for how much of a slog it was. So you can imagine my surprise when I found myself, in 2024, working on a cutting-edge new feature for my current company (Elastio Software</a>), and I once again needed to programmatically control a VMWare ESXi hypervisor, this time from Rust. I figured “obviously 15 years have passed, it’ll be a nice REST API now, probably with OpenAPI definitions, this should take a day or two to get working”. As I started to dig into the VMWare documentation, I was shocked to find that the diabolical SOAP API from all those years ago is still the primary way to interact with VMWare vSphere! VMWare have since published Python and Go SDKs to wrap the worst of its many excesses, and the docs are quite a bit better now, but it’s still a SOAP API, with a WSDL definition, and all the attendant complexity that implies! In my travels through the VMWare docs (made harder by the recent Broadcom acquisition which summarily broke all vmware.com URLs) I noticed that there is the beginnings of a REST/JSON based API, but it’s new in ESXi 8, and what I’m doing needs to work with 7.x as well, so I’m stuck with SOAP. This led me to look around for SOAP implementations in Rust. Not surprisingly, it’s slim pickings. If you’re even a little bit younger than me you probably don’t even know what SOAP is, but even if you do, you probably never once thought to yourself “you know what, it would be fun to write the first and only SOAP client implementation in Rust!” I certainly didn’t. Why spend energy and time on something that is so obviously a dead end, something that manifests the worst of early 2000’s architecture astronaughtics and object-oriented excess? I’m pleased to report that someone at least took a crack at it, albeit with a number of limitations: Github user mibes</a> wrote zeep</a>, which conceptually is a bit like PROST, Rust’s protobuf library. You feed Prost a .proto</code> file describing a Protobuf schema, and it generates some Rust files with structs that correspond to the PB messages, and can serialize/deserialize to/from Protobuf. Zeep does the same, except it’s input is a WSDL or XSD schema file, and it generates Rust code that can serialize/deserialize to/from the XML types that are defined therein. Unlike Prost, Zeep does attempt to generate SOAP client code. I was able to invoke Zeep from build.rs</code> pretty easily, but when I fed it the insanely huge VMWare vSphere WSDL, it failed in a few unfortunate ways: It could not traverse multiple levels of <include></code> elements, which the VMWare WSDL has in abdunance</li> It didn’t property keep track of the type names and aliases of the complexType</code> types the WSDL and XSD files defined, resulting in a lot of code that would not compile</li> It doesn’t have a good way to implement XSD’s very 90s Java EE style of inheritance in Rust terms, so several key polymorphic types that the vSphere API uses extensively were not propertly expressed in Rust.</li> </ul> I tried to work around some of these by writing some Rust code using the syn</code> crate to parse the generated Rust and then modify it to hack around some of the flaws in the generated code, but with each horrific and depraved act I hated myself more, and only uncovered yet more flaws in the generated code. I then thought I’d take a different approach, and use the xml-schema</a> crate from media-io</a> to generate just the Rust structs that correspond to the XML types in the XSD, and then write the SOAP client code myself. This, sadly, was also a failed experiment. First I manually extracted the XSD type definitions from the WSDL XML, and combined it with type definitions in the other XSD files that the WSDL file includes, into one big hairy ball of XSD that I could later on sell to a hostile regime as a bioweapon. I then tried to get xml-schema</code> to generate Rust structs from this XSD, but ultimately it failed in similar ways. On the plus side, as a result of my experiments with Zeep and xml-schema, I had a bunch of Rust structs that sort-of resembled the XSD types. I say “a bunch”; I am not exaggerating. The WSDL and XSD files that define the vSphere SOAP API are almost 6MB of XML. The generated Rust structs were in a single .rs</code> file that was 9MB large!!! Neither rust-analyzer</code> nor rustc</code> were amused in the slightest by this abomination, and I wouldn’t say it was my proudest moment either. Compile times were miserable, my Neovim editing experience was almost as bad as mid-2000s Eclipse, I still didn’t have types I could actually use with the real SOAP API due to the aforementioned limitations. The one advantage I had this time compared to the integration I build 15 years ago was that this time I only needed to use a tiny fraction of the API. Enough that I could probably hand-roll the structures I needed, using the ones generated by zeep</code> as the starting point. It would be a bit tedious, but compared to forking Zeep and fixing the issues there it would be much faster (and, yes, more selfish). Over a couple of late nights, wove an intricate tapestry of profanity as inscrutable and arcane as SOAP itself, and I also got a working SOAP client implementation going. Maybe someday we’ll open-source the code, but then someone might see it and judge me for the horrible things I had to do, but you weren’t there and you don’t know what it was like so don’t judge me! If you are reading this because you are faced with a similar situation, I have the following advice: Don’t. Surely there’s a newer REST API, or a different product that has a better API, or a different profession you could take up that doesn’t involve SOAP APIs.</li> Failing that, try to use zeep</code> to generate SOAP client code and structs for your API. My sense is that Zeep only struggled in my case because of the insane surface area and complexity of the vSphere API. If zeep</code> doesn’t panic, generates code that compiles, and you can make it work with your API, then you have dodged a bullet and should go star the zeep</code> project on Github to show the author your gratitude for the horrors that they have saved you from.</li> If zeep</code> doesn’t work for you, try hacking around your source WSDL/XSD, commenting-out problematic structures, until it can generate something that compiles, and then manually modify the generated code to try to get the specific operations that matter to you to work right. Beware in particular XSD inheritance where a xsd:complexType</code> extends</code> some other type. If this inheritance is used in a polymorphic way where an operation takes the base type but does different things depending on which subtype you pass in (or returns a subtype in a similar way), you will need to roll your own YaSerialize</code> and YaDeserialize</code> implementation that looks at the xsi:type</code> attribute to determine which concrete type to use.</li> </ul> To the authors of the zeep</code> and xml-schema</code> projects, I salute you for your efforts, and I hope that you are masochistic enough to continue to maintain and improve these projects, so that others may not have to suffer as I have. I am a permanent resident of Ukraine, finally! 2023-09-30T00:00:00+00:00 I’m shocked and horrified to realize that it was almost FIVE long years ago that I celebrated the receipt of my Ukraine temporary residence permit</a>! When I think back to that time, I can scarcely recognize the life I lived back then. I haven’t blogged much about the intervening five years, but a quick summary of events includes: Bought an apartment in Kyiv</li> Lived through the pandemic and lockdowns (thankfully the ungovernable Ukrainian people largely ignored the strict lockdown rules so this was not nearly as bad for me as it was for many in the world)</li> Survived a health crisis (and no, it was not due to aforementioned flouting of lockdown rules!)</li> Co-founded Elastio</a>, and subsequently raised seed and Series A funding rounds</li> Got married</li> 1.5 years (and counting) of full-scale war with Russia (most of which I spent outside of Ukraine, although I write these words from Kyiv)</li> </ul> During almost all of this, my application for permanent residency on the basis of investment in Ukraine (to wit: aforementioned apartment purchase) has been in some way or another in progress. If I had competent legal representation from the start, and there had not been a global pandemic, and a hoard of fascist Orcs had not invaded Ukraine, perhaps this would not have taken so long. Nonetheless, the wait is finally over: I am officially a permanent resident of Ukraine: Official Ukraine Permanent Residency Permit This permit is good for ten years, and my legal residency is, well, permanent. This doesn't change much for me day to day, but it means I don't have to go through the residency renewal process every three years, and it means I can open a Monobank account, and I think it means I'm legally able to purchase firearms in Ukraine. </figcaption> </figure> My public comments on the Rust Foundation Proposed Trademark Policy 2023-04-14T00:00:00+00:00 The Rust Foundation ruffled a lot of feathers when they posted a draft Trademark Policy document that is…well..let’s say it’s heavy-handed to an extent that seems far out of proportion to the necessary protection of a legal trademark. A kerfuffle has ensued, and the Foundation wrote a blog post</a> kind-of explaining their position, but without actually acknowledging any of the concerns about the policy. I was so upset by the imperious tone and jack-booted authoritarian particulars of this policy that I actually wrote up a formal comment which I submitted to the foundation via the feedback form linked in the blog post. I encourage anyone else who was as outraged as I was by this proposed policy to comment as well. The full text of my comments as submitted are reproduced below: I read the blog post at https://blog.rust-lang.org/inside-rust/2023/04/12/trademark-policy-draft-feedback.html, which explains that this policy derives from the starting principle that Rust should be a trademark owned by the Foundation, and under US (and maybe international) trademark law that obligates the Foundation to take some minimum steps to defend that trademark lest trademark protection be forfeited. I don’t doubt that this statement is true, and I don’t have any objection in principle to protecting the trademark. However I think it’s important to remember that the raison d’etre of the Foundation is not the defense of the Rust trademark, therefore the goal of maintaining the trademark must be balanced against other more important considerations. I’m not a Foundation member, just a passionate user of Rust for many years, so I must humbly submit that one such more important consideration must be preserving the freedom of the Rust community to write, share, talk about, debate, and proselytize the Rust language and ecosystem to a diverse audience the world over. I hope that’s not a controversial position to take. Evaluating the proposed policy through that lens, I find much there that I disagree with. I love using Rust, have built a startup around Rust, pay dozens of programmers to write production Rust code full time, and want very much to continue to do that indefinitely. My initial reaction to the trademark policy draft was very…emotional…and even now having had some time to think about it, you’ll no doubt hear the emotion in my words below. I’m operating from the assumption that the parties involved are acting in good faith, and are genuinely receptive to thoughtful commentary, and I hope you will take my (sometimes emotional) comments in the spirit of cooperation in which they are intended. My goal here in taking the time to write this up is to help prevent the Foundation from going in a direction that I think will be detrimental to the community, and will definitely be detrimental to my continued investment in the Rust ecosystem. Let’s first start with the requirement that blog posts must explicitly not be endorsed by the Rust Foundation. This imperious demand triggers a strong emotional reaction for me, which I’ll do my best to control as I register my objections. The required disclosure language is provided in the policy along with the conditions under which it must be displayed. So if I’m writing a “Why I Love Rust” or “Rust Sucks” or “Rust is the savior of the downtrodden Go developers toiling endlessly under the lash of Rob Pike’s hubris”, I must first recite the Foundation-mandated incantation to make it clear to my (apparently presumed to be very dim-witted) audience that the screed that follows does not bear the imprimatur of the Rust Foundation. That is (I’m desperately trying to remain civil here)…suboptimal. Not even the demon spawn Oracle lawyers had the audacity to make that demand of the Java community, and yet the Rust Foundation’s lawyers apparently take the position that failure to adopt this policy will surrender the trademark. If that’s true (and I’m quite sure it’s not), then surrender trademark protection. It’s not worth it. Next we’ll address the guidance that the word “Rust” should not appear in crate names, supposedly because this creates the impression that such crates are official or blessed by the Foundation in some way. This is, again….suboptimal. The lawyers are effectively saying that Rust developers who want to share the work they’ve done, freely, by publishing their crates to crates.io, must not use the word “Rust” in the names of their crates. The Foundation graciously allows us the use of “RS” instead. That’s so absurd that when I first heard the claim that this was the policy, I assumed it was clickbait and overstated the actual language. But having read it, I see that this is actually a policy that someone thought was reasonable and consistent with the minimum trademark protection necessary to retain the right to the mark. I will say again here, that I don’t believe your lawyers if they’re claiming that you must do this or you will surrender the trademark, but if that is in fact the case, surrender the trademark. This kind of “rules-are-rules” blind policy-making is what I expect from a Fortune 500 HR department, not the people who are supposed to be the stewards of the most loved programming language. If you’re really worried about distinguishing between official Rust crates and third party crates, then finally get around to having namespaces on crates.io, just like Github and Docker and countless other services do. Then you can say that the rust/</code> prefix means Rust Foundation official crates. I’m then free to make anelson/rust-sandbox</code> without getting a C&D from your very clearly humorless attorneys. After having taken a few moments to breathe deeply and get my blood pressure back to safe levels, let’s proceed. The next outrageous diktat I’d like to address is related to domain names. This doesn’t impact me personally, I don’t use any domain names with “rust” in them, but the absurdity of the claim cries out for comment. The idea that the mere the act of registering and serving content from a domain name containing the string “rust” is an infringement of the mark is so outrageous that I hope it was simply not intended that way and will be corrected forthwith. The alternative, that the Foundation and its attorneys think their stewardship of the Rust community includes prohibiting the creation of Rust websites that have “rust” in the domain name, is too depressing to contemplate. If there are any other widely used open source projects whose leadership places their boot on the collective necks of their community in this regard, I am not aware of it (and were I made aware of it, I would certainly avoid it). Up to this point, I’ve been pointing out policy elements that could perhaps be represented as part of the minimal defense of the trademark necessary to retain ownership of the mark. I’ve made it clear my skepticism of this representation, but my objections boiled down to a disagreement as to what constitutes a minimal defense. However I’m sad to say the policy goes well beyond that good-faith-but-debatable position, into territory which bears an uncomfortable resemblance to activism. To wit: Section 5.2.1 regarding User Groups, graciously permits Rust user groups to call themselves Rust user groups, but only if they formally adopt and enforce a “robust Code of Conduct”. I did not go to law school, and to the extent that I practice law on Internet forum posts it’s without a license, however I am quite sure that trademark law does not consider the application of a Code of Conduct when evaluating trademark claims. Setting aside for the moment that the policy does not define “robust Code of Conduct”, I find the prescription that only user groups who adopt such a code may be permitted to call themselves Rust user groups to be an absurd overreach on the part of the Foundation. What possible purpose could this diktat serve? Perhaps the Foundation members like robust codes of conduct and think they are great and that all spheres of human social activity should be governed by them. Maybe that’s even a good idea. But what is that opinion doing in a trademark policy?? It is completely inappropriate, and in my mind belies an attempt to use the cudgel of trademark law to impose the political preferences of Foundation members on the community, whether that community likes it or not. Suffice it to say that this community member, at least, does not like it. Were I younger and more impulsive this alone would motivate me to start the Outlaw Rust User Group, an illegal gang taking pride in turning its back on the strictures of moral society, setting fire to their Codes of Conduct, and rolling dirty on the open road, always one step ahead of the their nemesis the Rust Foundation corporate counsel and their evil sidekicks the Tone Police. As it is I’ll have to settle for shit-posting on the Internet, but I’m dismayed it’s even come to that. I’ve saved the bulk of my ire, and what remains of my vascular health, for Section 5.3.1, regarding events and conferences. Here the trademark enforcement mask drops, revealing naked political activism beneath. The ridiculous prohibition on the use of the word “Rust” in a Rust event pops up here again, but I’ve registered my displeasure at that elsewhere in these remarks. I want to draw attention to the list of minimum requirements that the Foundation will impose when deciding whether a particular event will be blessed or not, in particular: prohibit the carrying of firearms, comply with local health regulations, and have a robust Code of Conduct </blockquote> I’ve already taken issue with the “robust CoC” bit earlier, so let’s focus on the other two, regarding the carrying of firearms and compliance with local health regulations. It’s clear that “compliance with local health regulations” is a reference to the pandemic-era mask and vaccine passport requirements. This is an absurd addition to a trademark policy. If there is a local health regulation that has the force of law, then it seems pointless to add a requirement to “follow this particular category of local law” to the list. We can assume that obeying the law is required. If the regulations don’t have the force of law, but are recommendations, then who are the Rust Foundation to presume to dictate which local recommendations event organizers must comply with? Why is this even in here? Is it meant to be a stick in the eye to all of those renegade contrarian Rustaceans who defiantly met to talk about Rust without masks or vaccine passports in 2021? If so my outrage stems in part from the fact that I was not invited. However upsetting the local health regulations is, the prohibition on the carrying of firearms is an order of magnitude moreso. I assume this language came from an American, as I doubt anyone from another country would even think this is the kind of thing that must be said. This presumed American presumably doesn’t like firearms and probably would prefer to ban them entirely, but failing that they must at the very least be prohibited in any gatherings bearing the imprimatur of the Rust trademark. I must ask: why?? If a particular event organizer wants to ban the carrying of firearms, there is a legally enforceable mechanism to do that in every US state. I’m not aware of an epidemic of gun battles at Rust conferences breaking out over heated debates about the right way to implement HKTs or whether the current async syntax was a mistake, so clearly this not a reaction to a real problem but rather is a projection of some authors’ political preferences. That any member of the Foundation would consider it appropriate to insert their own political biases into a trademark policy document calls into question their suitability for the roles they presently occupy. Maybe firearms have no place in Rust conferences. I think that’s a very broad statement that’s hard to defend at the margins, but it’s a political position that I’m sure many Rustaceans share. Perhaps alcohol doesn’t belong in Rust conferences; after all it kills a lot of people every year and seems to have little or no health benefits. Serving meat is perhaps not appropriate at Rust conferences, since it’s offensive to vegans and anyway consumes a lot more resources than plant matter. Sugary drinks might ought to have no place at Rust conferences, since they spike insulin levels, and are very bad for one’s dental health. Smoking within 100 feet of a Rust conference is definitely out of the question for what are hopefully obvious reasons. Invoking the Judeo-Christian God in a Rust conference would definitely not go over well and should probably be restricted. If we all sat down a made a list of things that we think in our opinion don’t belong in Rust conferences, I’m sure we’d come up with a long list, and most of us would disagree on the particulars. A great way to get around that problem is to not insert our political perferences into trademark policy documents. I don’t understand what the thought process was here. On the one hand, the Rust Foundation supposedly values inclusivity and diversity, and on the other hand they dictate from on high in matters not related to Rust at all, assuming that their opinion matters more, imposing their will on all Rustaceans across the globe in all jurisdictions. I fear for the future of Rust if this is the direction the Foundation will take. Where does this lead? Will a future version prohibit the use of the Rust toolchain in the furtherance of activities that the politically active Foundation members find objectionable? Will speakers who privately hold beliefs that are out of step with the prevailing fashions on social media be banned from Rust conferences? Will we freedom-loving Rust users be forced to fork the project and make Frust, the free Rust, and litigate the question of fair use with the Rust Foundation lawyers while we swagger, armed and maskless, around our renegade conferences breathing the free air (which smells suspiciously of cigar smoke and scotch)? That’s not a future I want any part in (I wouldn’t mind the scotch part but the rest is downright dystopian). I could rant at length about the overall tone of the policy document and other particular sections, however this has gone on much longer than I intended already, and I think I’ve addressed the main points of contention. I hope my comments are taken in the spirit in which they are intended, and that my strong opinions on this matter are not mistaken for disrespect or abuse. I urge you to reconsider this entire policy framework and whatever legal analysis informed its development. I’m sure it’s possible to protect the Rust trademarks without the significant harm to the community cohesion and goodwill that this proposed policy will inflict. Sincerely, </blockquote> I really hope they back away from this authoritarian direction. It’s been clear since the early days that the Rust team tend towards the politically progressive in a fashion more overt than I think is appropriate, but up to this point I was able chalk that up to political differences because it didn’t significantly impact my use of, and enjoyment of, Rust (there was that time I had to do some work because they wanted to virtue signal and changed a warning from blacklisted_keywords</code> to disallowed_keywords</code> or something like that, but I muttered under my breath and moved on). Unfortunately this goes well beyond virtue-signaling, and crosses over into activism using the Rust Foundation’s trusted role in the Rust community to impose a degree of control over the community that is completely inappropriate for an open source project like Rust. Letter to my US Senators regarding Ukraine 2022-03-03T00:00:00+00:00 Both of my US Senators, Scott and Rubio, do not allow their constituents to write messages longer than 3000 character (for Rubio) or 2000 characters (for Scott). Presumably it’s not a good use of staffer time sifting through long missives from bothersome constituents. If either of them have a regular email address one can write to, I haven’t found it. So I have no choice but to post the full letter here, and cut it down to a size that is more convenient for Senate staffers to digest. The full text of the letter follows: Senator: I have temporarily resided in Kyiv, Ukraine for the last three and a half years, although my home is in Miami. I was lucky to get out of Ukraine days before the war escalated. I’m now safe in Tbilisi trying to arrange accommodations for the women and children we’re able to get out of the country. I left behind 48 colleagues with whom I have worked for the last four years, some of whom are now serving in the Ukrainian Armed Forces or Territorial Defense fighting desperately to save their homeland. I’m glad that this war has finally served as a wake-up call to the slumbering European and US defense establishment, and that military aid to Ukraine is flowing. But I’m writing to you today to tell you, we are not doing enough for Ukraine, and if we allow Ukraine to fall to Putin the consequences to the US and to Europe will be dire. As of today, as far as I am aware, we are not providing anti-missile systems to the Ukrainian forces, nor anti-ship missiles, and the Europeans have inexplicably declined to provide surplus Soviet-era fighter aircraft to the Ukrainian Air Force. NATO has categorically refused to impose a no-fly zone anywhere over Ukraine, and is doing nothing to deter Belarus from sending forces into Ukraine other than a sternly worded letter and the promise of export restrictions. Even the removal of Russian banks from SWIFT hasn’t fully taken effect and will not until mid-March. All of this while Ukrainian cities burn and the Russian aggressor advances. The fear of “escalation” seems ridiculous. What is this full-scale invasion of Ukraine but an escalation? Will we be afraid to escalate when Putin has taken Ukraine, consolidated his forces, and moves next onto the Baltics? Because he will by then be stronger, will have learned from his many mistakes in this invasion, and will be that much more of a threat to Europe. Are we so afraid of escalation that a nuclear-armed madman can do what he likes, prey on whomever he likes, while we refuse to do that which is well within our power but might upset him further? The United States will decide in the next few days, whether there will be an independent and democratic Ukraine on NATO’s border, or a militarized vassal state of Putin’s providing a base for operations against Europe. Already my colleagues are writing me from their basements and bomb shelters, asking me why won’t NATO close the skies to the Russian invader? Why won’t the US provide anti-missile defenses? Why won’t Europe provide fighter aircraft? What is this “escalation” you are so afraid of, while we are being bombed today? How am I to answer their pleas? How can I explain that the United States of America, which promised Ukraine territorial integrity in exchange for surrendering her nuclear arsenal in the Budapest Memorandum, won’t save Ukraine against the aggressor she has always feared, because is might make Putin angry and he might hurt us next? What do I tell my Ukrainian fiance as her countrymen are dying on the battlefield and in churches and kindergartens and hospitals and their own homes? Are we going to repeat the mistakes of the past, yet again? Will we again ignore a conflict in Europe because it doesn’t threaten our borders, only to see it metastasize into a world war which this time around could very likely be a nuclear war? If we do, history will judge us harshly, but not so harsh as the Ukrainian people we would be abandoning in their hour of greatest need. I urge you to take immediate action to exert as much pressure as you can on the Biden administration to do what needs to be done and do it now. I can think of no higher priority for the long-term well-being of Floridians, our nation, and indeed the free world. I and my Ukrainian colleagues eagerly await your response. Sincerely, Adam Nelson </blockquote> It’s a testament to the feelings of despair and helplessness I am feeling in this moment that I would even bother to write to my senators. I may as well sacrifice a chicken to Gaia or light a candle in a chapel. I suppose I just don’t want to live with myself knowing I did absolutely nothing, so instead I make an entirely symbolic gesture. As much as it sucks to feel this helpless, at least I’m feeling helpless from the safety and comfort of Tbilisi, and not huddled in a basement in Kyiv waiting for the shelling to stop… Thoughts on Rust after six months 2019-02-22T00:00:00+00:00 Disclaimer</h1> I’m writing this post as an honest and frank description of my experience coding in Rust these last six months. It’s just, like, my opinion, man</a>. Maybe you would have a completely different experience. Maybe I’m doing it all wrong and should be regarded with a blend of pity and contempt. Maybe I’m a Russian troll sowing discord as part of some inscrutable geopolitical long game. I leave it to you to decide for yourself. Background</h1> First off I should admit this is not my first time with Rust. I tried Rust back in August 2015, and I did so in a public repo so you can see for yourself</a> the ensuing debacle. My final commit message in that repo was thus: Came to the conclusion that Rust’s ownership model is too restrictive to be enjoyable to use (see src/wtf.rs) </blockquote> I don’t remember what the specific problem was, but I know Rust was a different language back then, and I was certainly a different programmer. I can see from the code I wrote then that I was not making enough effort to understand the mechanics of the borrow checker, and thus I wrote non-idiomatic Rust code and fought constantly against the compiler. I know many other programmers have had this same experience, and I think anyone who approaches Rust expecting to write C++ or Java-like code without understanding Rust idioms is bound to have a bad time. The tl;dr is: Rust is a very poor language for anyone who wants to write (C++/Java/etc) code. If you want to program in Rust, you need to write Rust code. Tautology? You’d be surprised… Present attempt</h1> For the last six months or so I’ve been working on a few Rust projects. None of them are in public repos yet, but they will all eventually be open sourced under the dual MIT/Apache 2.0 license favored by the Rust community. The details of my projects aren’t particularly important, but only to say that they are non-trivial libraries in which high (and consistent) performance and concurrency are critical. Given Rust’s reputation it should be a perfect fit. How not to learn Rust</h1> I alluded above to the fact that if one wants to learn Rust, one must be open to a very different way of approaching some problems. I can virtually guarantee that anyone with prior programming experience in any modern statically typed language will initially find themselves fighting the Rust compiler for every inch of ground. It will be frustrating, and not at all productive. I think noted Rust programmer and author George Bernard Shaw said it best: I learned long ago, never to wrestle with the Rust compiler. You get dirty, and besides, the compiler likes it. George Bernard Shaw</li> </ul> </blockquote> How to learn Rust</h1> My advice to anyone who is genuinely curious about Rust and genuinely interested in trying it out, is to approach it with humility and an open mind. First read the book</a>, and then try to write something of your own. If you’re anything like me, you’ll fail in a cascade of messages about missing lifetimes and types without a size and traits not being objects. It can be frustrating at times, but you should try to cultivate the right mindset: many smart and experienced programmers enjoy using Rust every day, they can’t all be masochistic imbeciles, maybe there’s something I’m missing, and it will be interesting to figure out what it is. I should also point out that the Rust community on Stack Overflow is very active and very helpful. If you get stuck they can help, though most likely someone else already asked the exact same question and the help is only a search away. My only complaint about the available resources for learning Rust is that they are very poorly SEO’d. I very often get results from older editions of The Book, from mailing lists and RFCs that are ancient and no longer reflect modern Rust, or even worse RFCs or GitHub issues discussing some cool new future feature of Rust that isn’t ready yet and may never land. Though I’m an avid user of DDG</a>, I found myself having to prefix all my Rust searches with !g</code> because Google seemed to do a much better job of pulling relevant results. How to fight the compiler</h1> The only way to get smarter is by playing a smarter opponent. The Rust compiler is a smarter opponent. Fundamentals of Chess</del> Rust Programming 1883</li> </ul> </blockquote> Rust reminds me a lot of Scala in that the compiler is fanatically pedantic, seemingly incapable of the slightest bit of imagination. If anything Rust is worse in this regard, due to the incorporation of lifetimes into the type system. Over the last six months I’ve come to expect a mighty struggle for every successful build, and in fact I’ve come to relish it. I can’t emphasize enough the importance of mindset, particularly early on. The Rust compiler is an incredible achievement, not only in the use of borrow checking for provably-correct elimination of an entire category of bugs, but even more so for the astonishing helpfulness of most of its error messages. While it’s sometimes frustrating trying to understand how to fix some of the errors it reports, I’m consistently amazed by the humane quality of the compiler output. Take advantage of this achievement and learn from the compiler. As I gained experience with its particular brand of pedantry, I found myself anticipating its objections, and structuring my code differently. Not just differently; better. More idiomatically Rust, yes, but also, subjectively, better code. I heard this time and again from other programmers new to Rust: the Rust compiler forced me to become a better programmer. This was my experience as well. I’ve lost count of how many times I’ve written code that was obviously correct, only to have it soundly rejected by the compiler. In the ensuing fusillade of profanity in which the virtue of the compiler’s mother is rudely impugned, I’d then come to the humble realization that it’s correct, and that what I was trying to do has some subtle edge case or failure mode I didn’t see. After I’d meekly change the code or add an explicit lifetime or change the scope or whatever I need to do, I’d be forced to admit the result is simply better code. rustfmt</code> is fantastic</h1> For most of my programming career, I’ve been fanatical</del> fairly opinionated about code style. I long wished there were some tool that could be run on (other people’s) code so that it would look right</del> like my code. I tried a few tools over the years but I always got bogged down in litigating specific details of this style or that style and never actually got anything done. A few years ago, while coming to the conclusion that Go is a language that simply rubs me the wrong way, I had occassion to use gofmt</code>. Easily the best feature of the Go ecosystem. There are no configuration options, no ability for your team to bikeshed or litigate on which indentation style is best. There’s only one way, and it’s the gofmt</code> way. That imperious arrogance is part of why I really, really came to dislike Go, but I won’t hesitate to give praise where it’s warranted: gofmt</code> is the right answer. Thankfully there’s rustfmt</code> for Rust and it’s similarly great. Though it has more configurability than gofmt</code>, it seems to be generally accepted within the Rust community that the defaults are the right answer. This is even built in to the official Rust plugin for Vim, so rustfmt</code> formatting can be automatically applied on save. Because rustfmt</code> must first run the code through a much less thorough and complete version of a compilation cycle, I’ve noticed that even without looking at the compiler output I can tell when there’s an error somewhere if on save my code isn’t slightly reformatted by rustfmt</code>. Seriously, be careful with rustfmt</code>. If you use it for a little while and then have to go back to a language without a universally accepted beautifier, you are likely to seriously regret some of your life choices… clippy</code> is great too</h1> clippy</code> is a linter for Rust. While the rustc</code> compiler compiles the code, and rustfmt</code> handles formatting, clippy</code> analyzes working, compiled code for all manner of possible improvements. It’s not on be default, but I strongly suggest you install it and configure it to fail the build on any violations of the checks that are turned on by default. I haven’t experimented with turning on the more pedantic checks, because I’m sure they’re called “pedantic” for a reason. The great thing about clippy</code> for a beginner is it finds and points out all sorts of small mistakes that you otherwise might not notice. For example, maybe you are using unwrap_or</code> when unwrap_or_else</code> would be sufficient, or you tried to Box</code> a Vec</code> which is just a wasted allocation. As hard as it is to write Rust code that will actually compile, there’s still plenty of room for mistakes that get past the compiler only to be caught by clippy</code>. cargo</code> and rustup</code> make dependencies effortless</h1> Though I don’t intend to write a post bashing Go, I can’t help but take another swipe here. Go’s dependency management story was absurdly awful when I tried it (in fairness that was years ago maybe it sucks less now). So bad that the highest praise I could think of at the time was “it’s better than C++”. Ouch. I mention this only by way of contrast to Rust’s dependency story. It is honestly not much different in ergonomics than npm</code> for Javascript or bundler</code> for Ruby. Well, except that Rust is a strongly-typed, compiled, low-overhead systems language! It’s such a joy to find the crate you want, add it to Cargo.toml</code>, then cargo just does what it needs to do. “What it needs to do” usually means resolving a long dependency graph, downloading the code for all the new transitive dependencies, compiling them, and resolving your use</code> statements to the right libraries. Honestly it’s still impressive to me how well cargo</code> works. I have very quickly come to take it for granted. Coming as I did most recently from Scala, I can compare cargo</code> most directly to sbt</code>. There is no comparison. sbt</code> is a radioactive dumpster fire of pathological over-engineering (does the s</code> still stand for “simple”??), and the Maven/Ivy dependency scheme is…well, it’s exactly what one expects of the Java ecosystem, I can’t think of harsher criticism than that. cargo</code> is a breath of fresh air, after which I finally realized just how much I hate working with sbt</code>. Don’t be afraid of nightly</h1> Though Rust is surprisingly mature for such a young language, it’s still a very quickly moving target. There’s a new stable release seemingly every week, though in fact I think it’s more like every six or eight weeks. Getting the new hotness requires only a rustup update</code>, and in the stable</code> channel nothing has ever broken for me after an update. In 2018 the new “2018 edition” of Rust was released, which was the first edition since “2015”. The 2018 edition added some new functionality that wasn’t entirely backwards-compatible, but a tool called rustfix</code> handled the changes automatically for me. That said, the Rust nightly</code> channel is an entirely different beast. As the name implies, it is built nightly, and is where all of the new, experimental, unstable, and also the most cool features live. In my case, I’m writing a lot of async I/O code for which the forthcoming async/await</code> syntax is critical. This is not stable, it’s only available in nightly</code>, which means I have to build my code against nightly</code> rust. This isn’t ideal, but nor is it a deal breaker. It just requires a bit more understanding of what is going on. For someone just learning Rust my advice is to stay on stable</code>, but if you run into something that requires nightly</code> don’t automatically assume it’s too unstable and broken for you to work with. It’s easy enough to rustup install</code> nightly, and equally easy to go back to the warm maternal embrace of stable</code> if nightly</code> is too dark and full of terrors for you. All the cool kids are doing it</h1> I’m not one to be a slave to fashion, however I’m surprised how often I’ll start using a tool only to discover that it’s written in Rust. Here’s a partial list of fantastic open-source tools built in Rust: alacritty</code></a> - My favorite terminal emulator</li> ripgrep</code></a> - grep</code> alternative that’s absurdly fast</li> bat</code></a> - A smarter faster cat</code> that does syntax highlighting</li> fd</code></a> - Stupidly fast fd</code> goes great with fzf</code></li> actix-web</code></a> - A Rust web app framework with very impressive TechEmpower benchmark results</li> LanguageClient-neovim</code></a> - The Vim plugin I use to add LSP support to Vim</li> stratisd</code></a> - RedHat’s new storage management system for RHEL, replacing Btrfs with something that presumably won’t eat your data</li> </ul> Summary</h1> How would I summarize my experience with Rust six months in? Thusly: ❤️ Don’t get me wrong, Rust still has rough edges. I often run into something that doesn’t work, for which the answer is “oh yeah that should work, once (INSERT RFC HERE) lands, which BTW might not ever happen but in any case won’t be soon”. Sometimes the compiler’s pedantry still aggravates me. Sometimes I find the unit testing support to be primitive. And it’s definitely not the language which allows for the fastest prototyping of ideas, though it’s definitely a contender for the language which produces the fastest production-ready implementation. But all languages have their limitations. After venting one’s frustration about some limitation or bug or design decision, the important question to ask, and to answer, is “would I rather be using another language”? For me, now, the answer to that most important question is always “no!”. In fact, I haven’t enjoyed working in a language this much since I started to play with Scala back around 2012. I expect that Rust’s popularity will only continue to grow, and that more members of Go projects will look longingly at Rust projects and wonder what might have been… I am officially a Ukrainian resident! 2018-12-01T00:00:00+00:00 Official Ukraine Residency Permit Obtaining this residency permit was not complicated, and the entire process start to finish was well under three months. It probably could be done in two if one was highly motivated. With this permit I can live and work in Ukraine for three years. Critically, this also enables me to open a Ukrainian bank account and use Ukrainian payment systems. </figcaption> </figure> I am as of today officially granted temporary residency in Ukraine, for a period of three years (subject of course to good behavior). I’ve written before on the progress towards this moment: Form Ukrainian LLC and tax ID (this was done for me by my lawyer there was nothing to write about)</li> Obtaining Ukrainian work permit</a></li> Obtaining Ukrainian type “D” visa in Krakow</a></li> Applying for residency in the fancy modern passport service center</a></li> </ul> I applied for this residency card about two weeks ago. Yesterday my lawyer texted me to say the card was ready and we had an appointment today to pick it up. Careful readers will note that today is a Saturday! It doesn’t matter, this service center works on the weekends too. Good luck getting a weekend appointment at a DMV office in the US. When we arrived we checked in at the front desk and were given a number which was to be called in about ten minutes. My lawyer and I sat at the coffee bar (yes there’s a coffee bar in the passport service center), and we must have lost track of time because one of the employees at the center came over to find us to tell us our number had been called. Again, try getting away with that in a US government office. We walked up to the window which had called our number, and I was handed the card and a document to sign confirming receipt. And that was it! In and out in maybe 15 minutes. I was shocked. Overall the experience has been a breeze. There are admittedly some bureaucratic hurdles to overcome, but almost all of them were handled by the Ukrainian law firm I hired to help with the process. Every step I’ve been personally involved with has been straightforward and modern. There is still one step left before I’m done with this initial process: I must open a Ukrainian bank account using my new residency status. More about that in a few days. The Azov Sea Incident 2018-11-26T00:00:00+00:00 By now the major Western media outlets will have reported on the incident in the Azov Sea in which Russian forces fired upon and captured three Ukrainian Navy craft conducting FONOPS in what is apparently now disputed waters. Today the president of Ukraine has asked the Ukrainian parliament to approve his request to impose martial law, ostensibly as a precaution to strengthen Ukraine’s defensive posture against a possible Russian escalation. It’s not clear yet if this will be approved or what it means for those of us who live in Ukraine. I don’t have much of anything to add to the analyses already swirling in the media. I don’t know what it means for me or for my Ukrainian friends, but I do have some thoughts about the situation from my perspective as an interested third party. I’ve also heard from many of my family members who are trying to understand what’s going on through the haze of media hysterics. First, it’s easy to forget in the West but war with Russia isn’t some theoretical possibility: Ukraine is currently at war with the eastern region of the country which has attempted to secede from Ukraine in a 2014 uprising that was facilitated, equipped, and in some cases even fought by Russian operators, though Russia firmly denies any involvement. Despite a formal ceasefire in 2015, the fighting continues to this day with both sides exchanging artillery, small arms, and rocket fire. Ukrainian Army and rebel forces continue to die on that front almost daily. So it’s not a question of “what if Ukraine and Russia go to war” but “what if the war escalates from border skirmishes to an all-out invasion?” I don’t know the answer to that question. When Russian forces invaded Georgia in 2008 they did so quickly and decisively, even bombing the capital Tbilisi until the Georgian government relented. I don’t think Russia wants more Ukrainian territory, I think they want to have a weak and Russia-aligned state on their border. Ironically, ever since the events of 2014, Ukraine has moved decisively to the west, with serious talks of joining NATO and the EU; I suspect any escalation of the fighting would be limited to reversing this trend and getting Russia-friendly political figures in power. It must be said that Ukraine’s military capabilities are vastly inferior to the forces Russia brings to bear. If Putin were willing to commit enough forces, I’m sure the Russian military could defeat the entire Ukrainian armed forces within a month. But so what? As both the Soviet and NATO armies have learned in Afghanistan, vast military superiority can only kill one’s enemies, not pacify them. Russia has no need of Ukrainian territory, and an outright invasion and occupation of Ukraine would only deepen Russian economic and political isolation, to say nothing of the inevitable bloody insurgency which would surely follow. For this reason, I’m not worried about Russian tanks rolling down Khreschatik street any time soon. My fear for Ukraine is that they play into the hands of their enemy and end up doing more harm to themselves. They’ve made some progress towards modernization and Westernization, but if they lose momentum or if opportunistic Ukrainian politicians take advantage of the situation to enrich themselves, the Ukrainian people will suffer. There is so much potential in this country, unrealized after decades of systematically corrupt and incompetent leadership, arguably at least since the time of the Tsar and perhaps before. As a foreigner it’s hard to get a clear picture of the current political situation, but my sense is that this movement towards US and EU sphere of influence over the last five years is less a matter of principle and more the present leadership engaging in pragmatic realpolitik in the hopes of holding power. It wouldn’t take much of a nudge from the Russian side to shift this balance in their favor, which is what led to the Yanukovich presidency and the 2014 revolution. I submitted my application for temporary residency, and it wasn't a Soviet nightmare! 2018-11-15T00:00:00+00:00 Brand new, modern, efficient Passport Center in Ukraine (photo by Bogdan Gandziuk</a>) I had heard many horror stories from expats and Ukrainians about the nightmarish Soviet-style bureaucracy involved in obtaining immigration documents. Imagine my surprise when I learned that Ukraine has just within the last month started to offer foreigners the use of their new, modernized passport centers. I was pleasantly surprised by the comfort, ease, and efficiency of the experience. There were no long miserable queues, no petty bribes, no uncaring bureaucrats. There was even a coffee bar! </figcaption> </figure> Nearly two weeks ago I obtained my Ukraine type “D” visa“</a>, which was the second to the last step to obtaining temporary residency. The final step is to apply for a temporary residence permit</a> at the relevant Ukrainian government office, with several documents: Passport with “D” visa and border entry stamp</li> Original Ukraine work permit</li> Copies of proof of private Ukrainian health insurance policy</li> Employment agreement with company which is sponsoring residency (in my case this is a Ukrainian company I formed specifically for this purpose, though to make the process easier my Ukrainian lawyer is a director)</li> Residency application form</li> Application fee</li> </ul> I had been warned to anticipate a nightmarish Soviet-style bureaucracy, in which one shuffles through multiple slow-moving queues and hopes that the unpleasant bureaucrats will be satisfied with your documents or will not be too unreasonable in demanding bribes. Imagine my surprise when I went to the Ukrainian Passport Center</a> here in Kyiv, located in a downtown shopping mall. It was well lit, staffed with young and surprisingly pleasant workers (who spoke English!), and used a take-a-number system so there were no long hours spent holding one’s place in a line. My Ukrainian lawyer even made an appointment online in advance, to minimize wait time. As it happens, there was a mistake in my employment agreement document. Or rather, some legal terminology was not exactly to their liking. Normally this would mean yet another bureaucratic nightmare, but to my surprise the officer (a girl not more than 25; Ukraine seems to have purged much of their old civil service) gave us the office’s email address and allowed my lawyer to correct the document and email it to her, after which she printed it out for us to sign. Try doing that at a DMV in the US! Because of this problem with the document it took about three hours to complete the process. Had the document been correct, the entire process soup to nuts would probably have been under 45 minutes. Considering that the conclusion of this process is a permit that allows a foreigner to live and work in Ukraine for three years, I’d say that’s astonishing. Another surprise for me is that this center was overwhelmingly dealing with foreigners. Ukrainians go to this same office to get their passports issued or renewed, but there were easily 10 foreigners for every Ukrainian. But, strangely, no Europeans or Americans. As far as I could tell from the languages spoken, most of the applicants were from India, Pakistan, Arabic-speaking countries, and Africa. I’m told these groups come here because they can study in Ukraine’s top universities for much less than the alternatives, though I’m sure some of the applicants where coming here to work or start businesses. I bet this lack of Western representation won’t last. There’s just too much potential here to keep it a secret much longer. Issuance of the permit takes up to three weeks. When it’s ready, the office will send a text message to my Ukrainian phone number, in English, and I’ll make an appointment to come back and obtain the actual permit card. Legally that is all that is required to live and work here, but in practice I will also need a Ukrainian bank account, about which more later. First Kiev snowfall of the 2018 season 2018-11-14T00:00:00+00:00 Today I awoke to a very cold fall morning. At least, cold by my standards. -3° C probably isn’t a big deal to the cold-hardened locals here, but I came from a warm climate and struggle to adapt to this new reality. Imagine my surprise then when I glanced at my phone and saw the threat of “wintry mix” in the next hour or two. It seems the Ukrainian Meteorological Service doesn’t play around; snow started in earnest right on time. The fresh snow on slippery stone sidewalks combined with my fancy leather shoes (actual treads are for proles, apparently) made for a rather slippery walk to the office. I managed to make slow but steady progress by walking carefully and staring at the ground in front of me. Of course, this being Ukraine, I was quickly overtaken by numerous Ukrainian women stalking past in substantial heels and a catlike ability to stay upright no matter the conditions. To add insult to injury, the city government has restricted retail alcohol sales after 11PM</a> I got my Ukraine long term visa! 2018-11-05T00:00:00+00:00 Freshly issued Ukrainian "D" visa. Obtaining this type "D" visa was much easier than expected, and opens the door to temporary residency in Ukraine. </figcaption> </figure> About a month ago I obtained my Ukraine work permit</a>, which was to be the final step before obtaining legal residency in Ukraine. But there was some confusion on the part of my immigration lawyer, as a result of which it turned out I needed to go to a Ukrainian consulate outside Ukraine and obtain a long-term “D” visa, prior to obtaining residency. I chose to go to the Ukrainian consulate in Krakow, Poland, both because it’s very close (1.5 hour flight from KBP) and because I’d been to Krakow before and really liked the city. I had prepared all necessary documents with my lawyer when I was still in Kiev. I then went to the Ukraine Ministry of Foreign Affairs e-Visa</a> site, and filled out the application online. Once that was done I was able to make an appointment, also online, at the Krakow consulate. In addition to a printed copy of the visa application, I also brought: A copy of my Ukraine work permit</li> A notarized proof of health insurance in Ukraine, which my lawyer obtained for me</li> A notarized photo copy of the same</li> A copy of my most recent US bank statement (I didn’t need to notarize it)</li> Two passport photos</li> My US passport</li> </ul> The process was simple. The consular staff spoke English, and were obviously very familiar with the process. Once I handed over the documents, they gave me the address of a Bank Pekao branch about a mile from the consulate, where I had to go pay the USD 184 application fee. It was a pleasant fall day in Krakow so I just walked leisurely there. One mistake I made was not bringing US dollars with me; I assumed it would be possible to pay by credit card, but it was not. Fortunately I had EUR 200 in cash, which the bank converted into dollars for a small fee. If I had it to do over again I’d be sure to bring US dollars. After paying the visa fee at the bank, they provided me with a receipt as proof that I had made the payment. I took this back to the consulate, and gave them my passport as well. That was at around 10:30 AM. They told me to come back at 2PM and my visa would be ready. I was astonished! I had planned to stay in Krakow five days, and hoped that would be enough time. I had no idea it would be so quick. I went and killed time in the city for four hours (not a difficult task in Krakow), and by the time I returned to the consulate my visa was ready, pasted into one of the visa pages of my passport. Strangely, they did not place my photo on the visa itself, so I’m not sure why they needed the two passport photos. All in all the entire process was straightforward and modern. I have been very impressed with the Ukrainian bureaucracy so far; compared to the horror stories I’ve heard from expats who’ve gone before me this has been an effortless experience. I would say the biggest problem I had was how to entertain myself for four more days until my flight back to Kiev! Adventures in De-Googling (Part 2) - ProtonMail 2018-10-15T00:00:00+00:00 In the last episode of Adventures in De-Goggling</a>, I laid out the principles behind my desire to reduce the amount of intrusive Google privacy violations in my life. The next step was to try to migrate one Google account to ProtonMail</a> and see how it went. This post is the result of that next step. TL;DR</h1> The quick summary of this post is: imapsync</code> is a great tool</li> ProtonMail isn’t the right solution for me.</li> </ul> Read on for the gory details… Background</h1> One of my oldest domains has been in continuous operation by me since 1998. For security purposes let’s just call this one graybeard.org</code>. Over the years graybeard.org</code> has used many different email solutions, most of which were self-hosted and managed by me until Google came along. For at least the last several years it’s been using Google’s hosted service for email (this is called “G Suite” now but initially it was a free tier of something called “Google Apps for Domains”, and being grandfathered into this old scheme I’ve never paid anything to Google for hosting all of my email). On this particular domain I don’t use any other G Suite features, not even contacts or calendars. So this one should be a great candidate for migration to ProtonMail. I have a Visionary account with ProtonMail, paid anonymously with cryptocurrency. This entitles me to all of the premium ProtonMail features, including the one that is required in order to perform this migration: the ProtonMail Bridge</a>. At the time of this writing the Linux version of the bridge was in closed beta, so I had to submit a request to be granted access, but that request was granted promptly. I already added graybeard.org</code> to ProtonMail’s configuration as a custom domain, although of course I didn’t actually update the DNS MX</code> records until the migration was confirmed. The Plan</h1> I’m going to use imapsync</code> </a> to perform the migration by simply reading all of the messages from the Gmail IMAP servers, and re-creating them on the IMAP server exposed by the ProtonMail bridge. This isn’t as efficient as I would have liked, but it seems to be the best available option at the time of this writing. Though ProtonMail now have an import/export app in beta</a>, this was not available when I first performed this migration. I can’t comment on the quality of this tool, maybe it’s rock solid but I doubt it’s more robust than imapsync</code>. I’ll do one big migration first, while Google are still hosting the graybeard.org</code> mail. Once it completes successfully and my spot checks in ProtonMail leave me satisfied that the migration properly handled all metadata and attachments and such, I’ll update the MX records to switch over to ProtonMail for incoming email, then re-run the migration to pick up whatever messages came into Google in the meantime. Fortunately there are only two users on graybeard.org</code> in total, so coordinating this migration will be easy. If I had more than a handful of users this would need to be done more carefully, but thankfully that’s not my problem. Initial Setup</h1> I don’t want to write a post that duplicates the existing documentation for ProtonMail, ProtonMail Bridge, and imapsync</code>. However I do want to make a few notes about the initial setup which might not be obvious. ProtonMail Bridge</h2> This approach requires you have ProtonMail Bridge installed and running on the same system that will be running imapsync</code>. If you don’t have a paid ProtonMail plan, you’re out of luck. The ProtonMail docs</a> cover setting up the bridge in great detail, so read all about it there. Suffice to say that I had the bridge set up and working, which I verified by using it with Thunderbird. I also used “Switch to split addresses” mode in the Bridge, because I want each address (anelson@graybeard.org</code>, anelson@domain2.com</code>, etc) to be presented via IMAP as its own account. I suggest you do this also, as there is essentially no support for switching between multiple ProtonMail user accounts the way that Google’s apps allow you to do. When the Bridge is configured as I suggest, it will generate and show in the UI a separate password for each email address, so that IMAP clients must be configured to log in to the Bridge with one login per email address, as if these were separate accounts. In ProtonMail itself they are not separate, but the Bridge presents this illusion to IMAP clients, in our case imapsync</code>. Google Security Config</h2> Google’s default security settings are strict enough that it’s probably not possible to perform the migration; in any case I couldn’t figure it out. You’ll have to disable the more advanced security settings in order for this to work. As long as you have strong passwords which aren’t shared with any other sites there shouldn’t be a significant risk in doing this. I logged into my Google account, went to My Account and then Sign-in and Security, all the way at the end of the page is an option “Allow less secure apps”. This needs to be enabled. imapsync</code></h2> Reading the man pages for imapsync</code> inspires great confidence. The tool has the feel of an instrument which has been refined over many long years of in-the-trenches use, with flags for all manner of edge cases. I found perusing the imapsync</code> Gmail FAQ</a> to be quite useful as preparation for this migration. You’d be wise to do the same. Understanding Folders and Labels in ProtonMail</h1> Unfortunately ProtonMail Bridge has a strange way of exposing the structure of its folders and labels. In the root of the IMAP tree it’s not possible to create any folders. New folders go under the Folders/</code> folder and labels under the Labels/</code> folder. Messages moved into one of the Labels/</code> folders are not moved there but labeled with that label, while messages moved to a Folders/</code> subfolder are moved to that folder. It’s stupid and I can’t understand why they would take this approach. This requires the use of complicated regex trickery to map properly. In Gmail, the folder Inbox</code> represents the inbox, and other folders under the [Gmail]</code> folder correspond to actual folders. All top-level folders other than [Gmail]/</code> and Inbox</code> are actually labels. So our challenge is to tell imapsync</code> how to map the labels to “folders” under Labels/</code> in the ProtonMail bridge, while leaving the real GMail folders alone. Fortunately imapsync</code> is flexible enough to support this via its regextrans2</code> option, as you’ll see below. For now just understand that this conceptual difference exists, particularly when you’re trying to navigate your ProtonMail email via an IMAP client like Thunderbird. Running the migration</h1> I opened a terminal window on my Arch Linux system to run these commands. I would have preferred to use a VPS for better persistence, but it wasn’t obvious how to run the ProtonMail Bridge headless. It should go without saying that you must not interrupt this process, by turning off your computer or letting it go to sleep. The actual incantation to make imapsync</code> work looks like this: imapsync -gmail1 --user1 anelson@graybeard.org \ --host2 127.0.0.1 --user2 anelson@graybeard.org --password2 BRIDGE_PASSWORD_HERE \ --port2 1143 \ --regextrans2 's/^((?!INBOX|\[Gmail\]).+)$/Labels\/$1/' \ --regextrans2 's/^\[Gmail\]\/Starred$/Labels\/Starred/' \ --regextrans2 's/^\[Gmail\]\/Important/Labels\/Important/' \ --regextrans2 's/^\[Gmail\]\/Drafts/Labels\/Drafts/' \ --exclude '^\[Mailbox]\/.+$' </code></pre> This will prompt on STDIN for the Google account’s password. You can avoid that by passing it on the command line with --password1</code>, but I didn’t do that because it is foolish to put credentials on the CLI. It’s ok for --password2</code> (the ProtonMail Bridge password) because that’s only used on this local system to connect to the ProtonMail Bridge. You can optionally omit --password2</code> and be prompted for both passwords each time you run the command. Using the --gmail1</code> option automatically configures imapsync</code> to use Google’s IMAP servers as the input. This saves a lot of duplication, and importantly also throttles IMAP operations to one message per second. Google apparently rate-limit their IMAP interface so slamming it too fast will get your IP banned, and that’s not fun for anyone. Unfortunately this means the migration is slow. How slow? I have about 3GB of email, almost 60,000 individual messages, and it took three days to run. Your mileage may vary. Assuming everything works, after a very long time the migration will finish. Remember that I initiated this migration while Google was still the mail handler for graybeard.org</code> according to the MX records. This means that I received about three days’ worth of mail while the migration is running. Fortunately the imapsync</code> command is idempotent; it can be run repeatedly it won’t create a duplicate copy of already-migrated messages. So, once this migration finished and I spot-checked a few messages to ensure they migrated properly, I switched the MX records over to ProtonMail and ran the migration again. It took another three days, after which I had every last one of my emails migrated to ProtonMail. Post-migration experience</h1> As I write this I’ve had about a month of experience with ProtonMail as the host of record for one of my domains. As much as I really want ProtonMail to succeed, and as much as I support their philosophical stance on privacy, frankly I’ve had a pretty shitty experience overall, and won’t be migrating any more domains to ProtonMail. I don’t want this to turn into a rant, but here’s a quick list of issues I’ve run into. If you’re considering migrating to ProtonMail, don’t let this dissuade you, but do make sure you understand each of these issues and be prepared to deal with them if they matter to you. ProtonMail Bridge Sucks</h2> According to their own FAQ</a>: On macOS, we have tested the Bridge on Apple Mail, Thunderbird, and Outlook 2011/2016. On Windows, we have tested the Bridge on Thunderbird and Outlook 2010/2013/2016. Every client implements the IMAP standard slightly differently, so we cannot make any guarantees about how the Bridge will behave on clients other than the ones listed. </blockquote> I thought that was just the usual caveat from a cautious software engineer. I mean, of course they can’t guarantee it will work with other clients. But in fact what this means is that it pretty much will not work with other clients. For example, MailSpring doesn’t work at all. On Linux, you are stuck with Thunderbird. If you like Thunderbird then I guess that’s not a problem for you, but despise it and can’t bring myself to use it. Even if you like Thunderbird, you still should be prepared for random hangs or crashes of the Bridge, and often Thunderbird operations will time out. I can’t speak to the quality of the bridge on Windows or macOS but on Linux it’s rubbish. Mobile and Web don’t sync</h2> If you archive a message in the Web interface, you’ll still see it on the Android client, and vice versa. Sometimes I see messages on one device that I already archived on another. It’s maddening. No multi-user support</h2> Google’s apps and web interface all work well with multiple user accounts. You can set up multiple logins, and easily switch between them in the UI. In the GMail mobile app you can see unified list of all messages across all your logins. ProtonMail can’t do that. You literally have to log out of one account and into another. It’s useless. No calendaring</h2> Recall that I chose graybeard.org</code> precisely because I don’t use the Google Calendar there. But I do rely daily on Google Calendar on another Google account, and as a result I cannot migrate that account to ProtonMail. They claim this is in the backlog but I am tired of waiting. Sluggish, glitchy web interface</h2> Because of all the problems with the Bridge, I use the web interface on my desktop and laptop systems. It’s…not fun. On my 2018 XPS 13 it’s quite sluggish, presumably due to all the asymmetric key crypto they’re running in Javascript. It often spins up the fan, and the keyboard shortcuts are not consistently responsive. In particular, a common workflow for me is to multi-select several messages in the inbox using keyboard shortcuts, then archive them. The “archive” shortcut it a</code>, not e</code> which Google has drilled into my muscle memory, but even after I remember that and press a</code>, it often doesn’t actually archive, or archives all but one of the messages, or archives but after I’ve given up waiting and move the cursor to click the “Archive” button. Perhaps I’ve just been spoiled by the Gmail web interface, but ProtonMail’s feels like going back in time to a much less pleasant era. It’s not exactly RoundCube-level bad, but I curse it every day. Conclusion</h1> ProtonMail as it exists at the end of 2018 is not robust enough to take the place of Google for my email and calendaring needs. I’ll migrate another domain over to Fastmail in the hopes that’s a better result. Bonus conclusion: despite being written in Perl (!!!), imapsync</code> is great. I heartily recommend it for all your IMAP migration needs. I've been in Kiev for a month already 2018-10-11T00:00:00+00:00 It’s hard to believe, but I’ve been in Kiev now for exactly one month. Time as flown by, too fast for my taste in fact. It seems like so little has been accomplished so far, but in fact thinking back look at what’s been done: Form Ukraine LLC</li> Obtain Ukraine tax ID</li> Obtain Ukraine work permit</li> Find rental apartment (haven’t moved yet though)</li> </ul> It’s hard to describe just how quickly one adapts to a new place, even as foreign as this one is to Americans. A month ago I was standing on the balcony in my big, modern, fancy apartment, looking down at the city I was about to leave. At the time, it seemed like I would surely miss my place, my car, my city, all the conveniences of home. Strangely, one month hence, I never even think about my old home. I don’t long for any of it, and I certainly don’t want to go back. I’ve adapted well enough to my life here that I’m comfortable, and I feel fortunate to have this opportunity to build something new in Ukraine. Another surprise for me is just how quickly I got used to being unemployed. The burdens of my former job were always foremost on my mind, either looming on the horizon as obligations I had to perform, or anxieties anticipating future calamities. That was almost two months ago, and now it seems silly to think about. Why did I care so much about such trivialities? Why mark time doing a job I didn’t like, just because the money was good and I could work from home? In almost every story of a professional quitting or being fired from their high-stakes job, the protagonist inevitably experiences the same epiphany, but for some reason experiencing it myself takes me by surprise. My Ukraine work permit is already issued! 2018-10-05T00:00:00+00:00 Ukrainian work permit, hot off the presses</figcaption> </figure> My immigration lawyer texted me this morning with a pleasant surprise: Less than two weeks after the initial work permit application was filed, my Ukraine work permit has been issued! I’m shocked at how fast this process is progressing. I keep waiting for this much-discussed Ukrainian inefficiency and bureaucracy to bite, but thus far it’s been an orderly, straightforward, and coherent process. Adventures in De-Googgling (Part 1) 2018-09-05T00:00:00+00:00 I have been an enthusiastic user of Google’s products for well over a decade. I eagerly gobbled up Gmail beta invite codes during the closed beta to make sure I could reserve all of my favorite 1337</code> hacker usernames (ah, the dubious tastes of youth). When GSuite launched (back in my day we called it “Google Apps for Domains”) I took advantage of the free tier to get Gmail-powered email and calendaring on my own domains. Having hosted my own mail servers and crappy webmail instances for many years, switching to Gmail and keeping my own branded domain names seemed at the time like just a click or two away from total Nirvana. Fast forward 10 years to our present dystopian free-market surveillance state. It’s become clear that we struck a devil’s bargain with Google, enthusiastically feeding them huge volumes of seemingly-irrelevant data so they can monetize us like billions of golden-fleeced sheep. I don’t know what my “spirit animal” would be, but I’m damn sure it’s not a sheep, gilded or otherwise. Time to cast off the shackles of ad-tech dark patterns and rediscover the freedoms our ancestors enjoyed all those many</del> few years ago. Vive la Resistance!</h1> Getting out of a relationship spanning over ten years is never easy. When that relationship is with the email provider which has borne witness to nearly every aspect of your life, your hopes, your dreams, your triumphs, and your crushing defeats, getting out becomes particularly difficult. It’s natural for one’s resolve to weaken, to start to rationalize sticking it out, to make excuses for bad behavior and exaggerate the good times you may have had. But enough is enough. We’ve made excuses for too long. The line must be drawn here. Exit Strategy</h1> When I began to research alternatives, I was dismayed to discover that Google have done an excellent job of making their tools much more convenient and easy than competing solutions. Getting out from under the Google Panopticon is going to require some adjustments, some new workflows, and, yes, some rough edges. I have five domains operating on GSuite, plus a few @gmail.com</code> addresses I still use for legacy purposes. I use Google for email and I’m utterly dependent upon Google’s contacts and calendar features. I use the Gmail app on my Android phone, and I take advantage of the seamless sync between my various devices. This isn’t going to be quick or easy, and it will need to be an incremental migration. To start with, I’ve decided to migrate my oldest domain name, which I’ve had since 1998. This contains an archive of most of the mail I’ve received over that time, but this is an address I use primarily for commercial purposes (e-commerce logins, dev account logins, etc). Thus it’s probably the largest mailbox I have to migrate, but also the lowest stakes in terms of needing to achieve a perfect migration. What’s more, I don’t actually use the Calendar or Contacts features on this GSuite domain, so I can focus specifically on migrating email, and see how that goes. Where to?</h1> There are two possible providers on my short list: Fastmail</a></li> ProtonMail</a></li> </ul> Both have strong reputations, are foreign to the US jurisdiction (though Fastmail servers are in the US), and pursue monetization strategies that do not involve ads or privacy violations. I like the idea of ProtonMail better because they guarantee privacy not just from mass government surveillance and data brokers, but from themselves. All mail is stored encrypted using an OpenPGP key pair which is generated on the browser and not available to ProtonMail engineers. While using ProtonMail still requires trusting the ProtonMail leadership and engineers not to backdoor their clients or web interface, at least there are some reasons to trust ProtonMail. Not only is their business model predicated upon being cryptographically hardened against surveillance by both corporations and governments, but even if that’s all a scam and they secretly cooperate with the NSA and the Five Eyes that’s still better than the behaviors that Google has publicly and explicitly admitted to, to say nothing of the undisclosed chicanery we don’t know about. Something about ProtonMail also appeals to the 16 year old cipherpunk in me, who came of age at a time when The Man was trying to ban PGP and force us all to use key-escrow crypto and put Clipper chips in TVs. At the time, merely downloading PGP and sending an encrypted email felt like a powerful gesture of defiance, as if I were wielding some futuristic weapon against which the combined might of the world’s superpower was powerless. By the time I was 17 I had a real job and needed to communicate with people for practical purposes, so the allure of PGP and web-of-trust and decentralized cryptographic security gave way to the more practical appeal of other people actually reading my messages. ProtonMail promises to bring back some of that thrill of defiance, at least to the extent I correspond with other ProtonMail users. That’s not to cast any aspersions on Fastmail, which I’ve used in the past and would not hesitate to use again. Their Austrailian leadership clearly seem to get privacy, and they’ve blogged in the past about their stance on law enforcement inquiries</a> and they make a convincing argument for the strength of their privacy model. They’re also a much more mature company and have a much richer feature set (like, for example, a working Calendar feature). I plan to migrate my first domain over to ProtonMail to see how it goes. Depending upon the result, I will decide how to proceed from there. Onward to Victory!</h1> Next step is to set up a ProtonMail account and figure out how to migrate the first Gmail account over. Start of a new Chapter 2018-08-31T00:00:00+00:00 For the first time in over a decade, I’m free of all professional obligations and must answer the question “what next?”.

Onward to Victory!</h1>
Next step is to set up a ProtonMail account and figure out how to migrate the first Gmail account over.</p>

Creative Articulation

Late Summer 2025 GenAI Tooling Review

Tools I’ve Discarded</h2>

Tools I’m Still Using Daily</h2>
These were all in the year-end tool review, although my use of them has evolved somewhat since then, they’re still daily drivers.</p>

With, By, or For - Three ways I use LLMs as a software engineer

My take on the American foreign policy shift on Ukraine

2024 Year-end GenAI Tooling Review

GenAI Tooling I’m Using Daily</h2>

Generating OpenGraph image cards for my Zola static blog with Cloudflare Functions and Rust

I'm leaving Ukraine (for the winter at least)

What's in my air raid bug-out bag?

Quick and dirty hack to automate tedious expense report task

The obvious solution that doesn’t work.</h2>
At first I thought this would be extremely trivial. I just dragged all of the PDF receipts over into Claude’s web UI and…</p>
</p>
But I have more than 5 PDFs! FML!</p>

My Air Raid Response Protocols

What daily life is like for an expat in Kyiv in August 2024

Calling SOAP APIs from Rust

I am a permanent resident of Ukraine, finally!

My public comments on the Rust Foundation Proposed Trademark Policy

Letter to my US Senators regarding Ukraine

Thoughts on Rust after six months

I am officially a Ukrainian resident!

The Azov Sea Incident

I submitted my application for temporary residency, and it wasn't a Soviet nightmare!

First Kiev snowfall of the 2018 season

I got my Ukraine long term visa!

Adventures in De-Googling (Part 2) - ProtonMail

I've been in Kiev for a month already

My Ukraine work permit is already issued!

Adventures in De-Googgling (Part 1)

Start of a new Chapter

Creative Articulation

Late Summer 2025 GenAI Tooling Review

Tools I’ve Discarded</h2>

Tools I’m Still Using Daily</h2> These were all in the year-end tool review, although my use of them has evolved somewhat since then, they’re still daily drivers.</p>

With, By, or For - Three ways I use LLMs as a software engineer

tl;dr - Three ways I use LLMs as an SWE</h2> I break down the ways I use LLMs as an SWE into three categories, in the order in which I discovered them:</p>

My take on the American foreign policy shift on Ukraine

2024 Year-end GenAI Tooling Review

GenAI Tooling I’m Using Daily</h2>

Generating OpenGraph image cards for my Zola static blog with Cloudflare Functions and Rust

I'm leaving Ukraine (for the winter at least)

What's in my air raid bug-out bag?

Quick and dirty hack to automate tedious expense report task

The obvious solution that doesn’t work.</h2> At first I thought this would be extremely trivial. I just dragged all of the PDF receipts over into Claude’s web UI and…</p> </p> But I have more than 5 PDFs! FML!</p>

My Air Raid Response Protocols

What daily life is like for an expat in Kyiv in August 2024

Ukrainian Losses</h2> Of course, the human cost of the war is incalculable.</p>

Visible Reminders of War</h2> Even in the center of the capital, which Russia never occupied, there are many reminders of the destruction they have wrought, and the armed struggle of the Ukrainians to remain free.</p>

Calling SOAP APIs from Rust

I am a permanent resident of Ukraine, finally!

My public comments on the Rust Foundation Proposed Trademark Policy

Letter to my US Senators regarding Ukraine

Thoughts on Rust after six months

How to fight the compiler</h1> The only way to get smarter is by playing a smarter opponent. The Rust compiler is a smarter opponent.</p>

I am officially a Ukrainian resident!

The Azov Sea Incident

I submitted my application for temporary residency, and it wasn't a Soviet nightmare!

First Kiev snowfall of the 2018 season

I got my Ukraine long term visa!

Adventures in De-Googling (Part 2) - ProtonMail

TL;DR</h1> The quick summary of this post is:</p> imapsync</code> is a great tool</li>

I've been in Kiev for a month already

My Ukraine work permit is already issued!

Adventures in De-Googgling (Part 1)

Start of a new Chapter

Tools I’m Still Using Daily</h2>
These were all in the year-end tool review, although my use of them has evolved somewhat since then, they’re still daily drivers.</p>

The obvious solution that doesn’t work.</h2>
At first I thought this would be extremely trivial. I just dragged all of the PDF receipts over into Claude’s web UI and…</p>
</p>
But I have more than 5 PDFs! FML!</p>