THE REGEX KING
Jeff Sisson's blog (email me)
Queens hydro power
Running a small weather station presents many opportunities to indulge in a particular fantasy of how software and data should work: for the small cost of a few sensors and a rented server, I can record weather data that has a very specific terroir…it comes from a specific and real place in Queens! I can store that data and do anything with it forever, without worrying about some provider or API going down or going out of business.
I’ve come to treat this vision of data provenance as a creative constraint: I generally only put weather data on Big Boy Weather where I know where it comes from, or had a hand in producing it. This constraint can indeed be quite limiting: many types of weather forecast depend on radars or high altitude balloon readings that I don’t have the means to produce myself, and so Big Boy Weather is unlikely to ever provide accurate 5-day weather forecasts or track hurricanes.
There’s been a recent development in Queens for which I’m making an exception to this general rule: the Champlain Hudson Power Express is a long range power line connecting Canadian hydro power from dams in Québec that was recently connected to the ConEd power substation on the north coast of Astoria, Queens (about 2 miles or so away from Big Boy Weather). It’s normally not possible to say for certain whether or not any given electron was produced by a particular energy source like the Champlain Hudson Power Express: energy producers send electricity into a large grid that moves electrons to wherever they can be consumed right now. So it’s typically tricky to answer questions about how much of NYC’s energy needs are provided by any given non-fossil-fuel energy source: even where we build wind/solar/hydro/nuclear/battery energy, the energy produced tends to be diffused into the energy grid in a way that makes it hard to track except where you’re looking across the entire energy grid of New York.
This new Queens hydro connection is different: the power station it connects to in Astoria is close to the geographic center of New York City, and so we can reasonably assume all of those hydro power electrons are being used within the city. This electron (& data) Queens-locality fits nicely with the broader ethos of Big Boy Weather: I want to know where the data is coming from geographically, and here I can! And so I wrote a script that scrapes total NYC energy consumption along-side Queens hydro energy generation from the NY state energy grid operator every 5 minutes. Those data are rendered as a tiny chart with the last two hour’s of energy use in brown and Queens hydro energy production in blue. Here’s the live graph:
As of writing, there’s actually no Queens hydro energy being produced, and so the chart is all brown: the power line has been turned on and off over the course of July/August 2026 due to ongoing technical issues. The line is meant to deliver a stable 1250 megawatts worth of hydro power, which on a low energy day would provide around 15% of NYC’s energy usage (pretty good!). Over the past weeks, I’ve instead observed 600mW, 100mW, and 0mW at different times of day; I assume they’re doing a lot of flipping on and off the big lightswitch at some point between Queens and Québec. As with other Big Boy Weather offerings, I’m recording this temporal data in the event that this fluctuation becomes interesting over time, and also to possibly try to correlate changes in temperature/humidity with NYC-wide energy usage generally.
“Mack the Knife” — Louis Armstrong (1956)
I was listening to an archival radio broadcast from the late Phil Schaap about the swing jazz band man Tiny Bradshaw. In full Phil Schaap discursive mode, he briefly detoured to describe a recording that Louis Armstrong has made of “Mack the Knife” with Lotte Lenya, the singer who’d made this song famous originally in the 1930’s as part of Threepenny Opera. Mack the Knife is a stone cold classic; I knew Louis Armstrong had recorded it, but hadn’t quite appreciated that it was his recording that put reinserted it onto the cultural agenda, and that he’d recorded a version with Lotte Lenya!
From this essay on the occasion of the Lotte x Louis recording being added the Library of Congress:
[Mack the Knife] had the makings of a hit single, but [producer George Avakian] was unable to persuade any of Columbia’s artists to play his hunch. Dave Brubeck, Erroll Garner, and Gerry Mulligan all turned him down flat, finding the simple tune of “Moritat” to be too repetitious.
…not too repetitious for Louis, though! His immediate reaction is “LFG”:
“Oh, I’m going to love doing this!” he told Avakian. “I knew cats like this in New Orleans. Every one of them, they’d stick a knife into you without blinking an eye! ‘Mack the Knife’! Let’s go!”
The recording he made with Lotte Lenya wasn’t itself released immediately, and Phil Schaap noted that she doesn’t quite know how to swing her lines. This version is as cute as you think it might be:
talk2html
I like a live event where the format is people riffing, and try to attend these when I can. It’s also remarkable that we live in an era with unprecedented access to recordings of these types of live events: I’m always seeing or being sent links to a video recording of a presentation, a panel discussion, or a video version of a podcast.
Unfortunately, the practice of watching or listening to recordings of a talking event has never worked for me: I don’t have a car commute, I don’t use headphones on the train or the bike, and I’m not able to listen to recordings of people talking while I’m on the clock at work. When I do have time at home available for durational media, I’m trying to listen to music, catch a basketball game, watch a movie, or watch braindead TV, in that order.
I do, however, have plenty of other time to read, and this has become even more true as my kid has learned to read and we’ve unlocked “everyone in the family is reading at the same time”. I’d always wondered whether I might be able to catch more of these live event recordings if I could timeshift them from durational media to written media.
To gather some data on this hypothesis, I made a command line tool called talk2html. This tool wraps the excellent yt-dlp command line tool to download videos from YouTube, and post-process them as a filmstrip of images from the video with text presented along-side the recording’s images.
talk2html.I’m not yet sure if “reading” a talk/panel/discussion/podcast instead of watching it loses something in translation. I just tried “reading” this great conversation between Barbara Fields and Ta-Nehisi Coates; it was cool to be able to read the whole thing where I almost certainly would not have otherwise had time to watch it at all. I suspect this tool will help me to move recordings like these off of my “to watch” list.
But I still found myself jumping to specific parts of the recording where the automatic transcript wasn’t good, or where it was clear that some meaning was being lost as written vs. as spoken. Which is maybe just to say: I do not recommend talk2html for use by grindset guys who are looking to try to pound media (written or durational) like nutraceuticals in a functional smoothie.
On the other hand, I did use talk2html to “read” a conference talk about local-first software, which I won’t name. That talk turned out to be kind of low information and vaporware to my taste, and I was glad to not have spent an hour with it.
Interview: Julien Minet - OpenArdenneMap
Great interview with the maintainer of a region-specific OpenStreetMap style called “OpenArdenneMap”, that’s been optimized for hiking in a specific region in the world (the Ardennes) while still being usable and forkable for use elsewhere.
I found this interview while researching tools I could use to make a print map of the neighborhood for my son (I ended up using QGIS). I’m inspired by this idea of region-specific cartography and site-specific digital tools:
To come back to OpenArdenneMap, as its name suggests, it’s designed for one region in particular: the Ardennes. A friend once asked me to produce a map of Brittany using OpenArdenneMap and then complained that the sea didn’t appear. Of course, there’s no sea in the Ardennes. Far from any chauvinism or nationalism, the idea is to create a rendering specific to a region, with its own characteristics and particular objects, and of course this can evolve over time. For example, there is a specific rendering in OpenArdenneMap for Christmas tree plantations, because they cover a large area in the Ardennes, perhaps 10% of the agricultural area. But this is a recent development, and it’s possible that in a few decades there won’t be any left.
One disadvantage of web maps is that we are witnessing a standardisation of styles, with styles that can be designed for a Californian environment and then applied in Europe: this makes no sense. Each region has its own geographical richness, the result of the evolution of the natural landscapes inhabited by the human communities who have lived there. And so each region is entitled to its own styles, which highlight its characteristics.
Utopian XHTML at the SEC
I’ve been working on a project that was riffed into existence by friends John, Olov, and Nora at Grand Morelos: it’s called Labor Leverage. It’s loosely based on workshops John and our coworker Stacy Cowley have run previously, where they share with other unions how to research employer finances in order to build power at the bargaining table. Big public companies are required to report data to the SEC as part of that institution’s purpose in stabilizing public markets, and it turns out many of the same data are useful to reveal how your employer and its executives actually spend its money. Profits, stock buybacks, executive salaries and stock compensation, all available in SEC data made available to the public!
The format of this data is interesting: companies upload their data to the SEC’s EDGAR (Electronic Data Gathering, Analysis, and Retrieval) system. The API is beautiful, and is barely even an API at all: it’s “just” HTML files, uploaded at predictable paths. Specifically, it’s XHTML, the older and crankier subset of today’s HTML that was meant to validate against a strict XML syntax. Companies embed snippets of another XML flavor, iXBRL (eXtensible Business Reporting Language) inside of the HTML, so that you can have a report that looks like a webpage with tabular profit and loss tables, buried within which are machine-readable XML tags describing key accounting figures as structured data.
Many of the tags encode dry, Generally Accepted Accounting Principles (GAAP) data like us-gaap:NetIncomeLoss, us-gaap:StockRepurchasedDuringPeriodValue, or us-gaap:CashCashEquivalentsRestrictedCashAndRestrictedCashEquivalents. Maybe more interesting are the data that companies are required to report by the government, but which don’t use machine-readible XML tags because they are not legally required to do so. These include data that’s absolutely relevant to unions like the one I belong to, such as the number of total employees, or the CEO Pay Ratio disclosures required by law following the 2008 financial crisis. These data are written as prose in the report, and so extracting them from the text requires the use of cruder text matching techniques. It’s tempting to picture a regulatory regime where financial data relevant to workers also had legally-mandated, machine-readable XML tags!
Compared to the freewheeling, single page app oriented ecosystem of today’s HTML5, reading iXBRL-flavored XHTML is like discovering an exotic dialect of a language that’s been cut off from its parent for so long that it’s had to invent grammars and styles of its own. <FONT> is everywhere. All elements have 20-30 inline style definitions in lieu of CSS. When I was scraping this data, I was surprised to find that that all the 10,000 or so companies clock in with 29GB worth of HTML. The reason the dataset is that large is because each document tends to have extreme HTML energy: the biggest document—a DEF-14A filing from Vaso Corporation— weighs in at 36MB, in part because there are long sections where every single character is wrapped in a distinct font tag (!).
I remember writing HTML in the era where the trend was towards making sure your HTML passed strict XML validation, like these documents do; there were hundreds of tools that would feed your HTML thru a validator and scold you for your HTML sins. At some point things trended away from XML generally, and away from HTML validation specifically, but it’s kind of beautiful that the backstop of a powerful government agency has left an oasis where XML validation and truly semantic XML elements still reign. Matt Levine recently wrote about how there are even companies whose entire business is to “EDGARize” companies' HTML:
If you are an SEC lawyer — if you look at this system from a reasonable level of abstraction — then the way Edgar works is (1) a company sends its material nonpublic information to Edgar and (2) Edgar immediately makes it public. Information passes through the system for an infinitesimal time; the filing system does not hold on to a bunch of secret corporate information in its servers. Edgar is a publication interface, not a database of company secrets.
But of course if you are an Edgar typesetter, you do not look at this system from that level of abstraction. You deal with the actual system, which operates at a not-at-all-infinitesimal time scale. You are like “man, this stuff sure takes a long time to typeset.” And there are trades there.
It’s absolutely someone’s dream to run a business helping companies to write XML-compliant HTML, with a sideline making millions from illegal stock market information you’ve read in the XML-compliant HTML!