THE REGEX KING

jeff sisson's blog (email me)

Same temp map

15 Aug 2023

An experience I’m guaranteed to have on any given summer: it’s 90°F indoors and probably some nasty level of humidity, and I’m stewing in the living room wondering how many other people are in the exact same situation right now. I added a little feature called “same-temp” to the Big Boy Weather station that answers this. It’ll show you everywhere else (in the Contental United States) it’s currently 90°F:

Contental United States of 90°F

…or everywhere else it’s currently 60°F:

Contental United States of 60°F

…or wherever it’s it’s currently 72°F:

Contental United States of 72°F

The maps update a couple of times an hour, and use data produced by the National Weather Service to create the National Forecast with a file format called “GRIB2”. GRIB2 files are cool: they’re about 50MB a piece and they describe meteorological data like temperature or rainfall, with instructions for how to project the data onto specific points on the earth. It’s a funky file format…every other horizontal line of “latitude” changes direction from east/west to west/east…and temperature data is in degrees kelvin, which always makes me think of hell…

I like working with file formats like GRIB2 because there’s something pure about something that’s “just a file”! Even though GRIB2 files are “used by the big guys that make the operational weather forecasts”, I can be a small guy using GRIB2 to write my own software to make these little maps of every place in the Contintental United States that shares a given temperature. I keep a sqlite database with a list of all of the cities on OpenStreetMap for exactly this occasion, and so the map will show a handful of place names that share whichever same temp you’re looking at, too. If you click on one of the same-temp links, you can dial the HTML slider and cruise the full temperature range. It can be fun (or disturbing?) to realize other specific places you’ve never been to where someone else may be sweating (or shivering) as you are right now…


Big Boy Bike Directions

26 May 2023

I’m convinced that there is a perfect bike route between any two points in New York City. When you ride the same way over and over, these perfect routes kind of just reveal themselves naturally. You randomly encounter and memorize which streets have the best timed signals, the fewest double parkers, the roomiest shoulders.

It’s harder to find a perfect route when you’re going somewhere you’ve never biked before. Most corporate transit apps happily give you bike directions, but the routes they recommend are sometimes bad. I’ve always wanted bike directions app that felt like a friend’s inner monologue of their preferred route, and I built Big Boy Bike Directions to scratch this itch.

Costing

Big Boy Bike Directions modifies the excellent open source routing software Valhalla, tweaking its bike directions algorithm to suit my needs. Valhalla’s core metaphor is the idea of “costing”: every street segment or intersection is assigned a number called a cost. A cost will always include the time it takes to bike a given distance, but may also reflect other factors like the available bike infrastructure or the speed limit. Compute the sequence of streets with the lowest cost and you’ve found your bike route.

Big Boy Bike Directions exposes all of the street-level factors that determine a street segment’s cost as knobs that you can tune to your biking preferences. Here are some of the specific costing factors that are made prominent in Big Boy Bike Directions to help suggest routes that match how you like to bike:

One-way streets

one way street knob

I always prefer taking a single lane one-way street, where one is available in my direction of travel. Cars behave a tiny bit more predictably on a one way street because there’s only one way everyone can go. The “One Way Streets” knob exposes this as a preference.

Separated bike paths

bike path knob

I can’t stand riding in traffic with cars, and will go out of my way to avoid it if there are grade-separated bike paths available. But corporate bike directions apps don’t always seem to make this distinction, or if they do it sometimes seems to be superseded by a preference for shorter distances. The “Separated bike paths” knob will treat bike lanes that are physically separated from car traffic in a class above other types of bike infrastructure. Dialing up this knob to the max will produce bike routes that are sometimes less efficient, but always more relaxing.

Avoiding cars

avoid cars knob

Valhalla’s routing already included in a use_roads factor that I’m calling “cars: hate em / love em”. It accentuates the costing difference between low-intensity and high-intensity streets when you dial up the knob. I’m actually not sure why anyone wouldn’t want to avoid cars always, but I’ve left this as a knob because there are some high speed bike freaks out there and Big Boy Bike Directions respects the reality that we all take some flavor of calculated risk every day.

Bus lanes

Sometime in the 2010’s, Giovanni passed along some street wisdom I’ve never forgotten: the painted Dean Street bike lane always seems like it’s the optimal east-west bike route in the Atlantic Ave. corridor, but it’s actually stressful to bike on because it overlaps with the B65 bus route and bus drivers have to weave in and out of traffic as they make their stops. The nearby St. Marks Ave, which has no bike lane but is quite wide, is much more relaxing to bike on by comparison.

Big Boy Bike directions uses bus route data to impose an additional penalty to bus routes, comparable to designated truck routes.

Avoided roads

Sometimes you just don’t want to bike on a particular street, for rational or irrational reasons. I’d be content if I never biked on the Greenpoint Avenue bridge over Newtown Creek again — I can’t stand riding alongside the trucks!

Big Boy Bike Directions has an “Avoided Roads” feature that allows you to eliminate specific road segments like these from consideration in bike routing by drawing them on the map. Its directions will always avoid those street segments afterwards.

avoided streets

I'm content to take hugely roundabout routes like this one to avoid streets like the Greenpoint Avenue bridge I don't like to bike on.

It’s not worldwide and it runs on one of my computers

I’ve built Big Boy Bike Directions for the long haul. In particular, I’m running Valhalla for routing, and hosting Protomaps tiles for vector tile rendering. I had to stop myself from building a DIY geocoder for the time being, and asked the kind folks at Geocode Earth for help with an API key.

Because of these and other constraints, Big Boy Bike Directions intentionally only works in a handful of cities where I know people who bike: New York City, Minneapolis, Santa Cruz, and Providence for now. But I’m very interested in hearing from anyone who uses this to bike, and open to adding other cities. Email me if you use it, find any bugs, or discover relaxing bike routes!


Flip phone review

09 Dec 2021

I was inspired by Laurel’s phones post, and want to describe my flip phone rig.

Left to right: really old LG flip phone, hibernating Alcatel Go Flip V, active Sunbeam Orchid. It's too bad they get bigger over time!

My current phone

I currently use a Sunbeam Orchid, which is a flip phone running on a fork of Android called “BasicOS” that doesn’t have a browser or apps.

I like the philosophy of the company — they’re mennonites making specific technology choices — but I’ve actually never had a phone with a browser or apps before, so a lot of the thoughtful software constraints of this phone go over my head.

Because I don’t use apps, I don’t need a data plan, and so having a phone is pretty cheap (around 30 bucks a month). I’m at my computer most of the day, and have an iPad, so I’m app literate but I don’t choose or need to bring apps with me when I leave the house.

The one “spec” I genuinely care about is battery life. Unfortunately, the Orchid doesn’t fare well here: I only get about a day and a half on a charge. My previous flip phone (an Alcatel Go Flip V) had a battery that would last about 7 days or so. The one I used before that lasted around 2 weeks. I don’t generally agree with the sentiment that technology has gotten worse during my lifetime, but having been largely free from a charger until relatively recently, it sucks to re-enter a charger tethered lifestyle. Sunbeam is at least transparent about this mostly being the fault of VoLTE radios. We’ll see how long I last with the Orchid’s smartphone-esque battery life; I might just return to my older Alcatel flip phone, even though its software for things like group texting is really bad compared to the Orchid.

How I get around

Most of my friends bought a smartphone sometime in the past ten years, but there are a couple of holdouts (Josh and Nobu) and sometimes we compare notes on how you can make it work. Here are a number of useful patterns that have supported my flip phone lifestyle:

  • When I’m going somewhere, I’ll try to study directions beforehand. It’s relatively easy to memorize major transit or bike arterials, but the trick I’ve learned for remembering someone’s house address or apartment number is to write a temporary little song with it in the lyrics (something like this). This works really well! If a route is super torturous, or I’ve got multiple stops, I’ll take notes before-hand on a tiny piece of paper.
  • When I’m lost, there’s usually a physical map nearby. Many subway stations have detailed bus maps, but by far the best tool for unexpected wayfinding is a Citibike kiosk. Even if you’re not a Citibike user, the kiosk lets you use a little map showing where you are, and you can zoom and drag it and see street names and everything. The LinkNYC kiosks are also sometimes useful when you just need to Google something nearby.
  • Because my phone’s battery life is usually pretty long, the fear of being “out of battery” largely goes away, and so the worst case scenario when I’m out and about is that I have to figure out who might be sitting near the internet and can help me look something up. I usually call Kathryn when this happens; it sort of feels like a quotidian version of when they call an “operator” in The Matrix.
  • I sometimes end up in a situation where some kind of gatekeeper has made an assumption that everyone is expected to have a smartphone in order to do something. This happened most recently at my dentist, where they wanted me to sign a waiver on a website to get my teeth cleaned. It’s oddly freeing to show them your phone and say “I can’t do that”; sometimes this works, and they’ll happily let you use some paper-based alternative instead. When that doesn’t work, my hail mary strategy is to ask to use the gatekeeper’s smartphone. Something about the personal phone boundary is so sacrosanct that often a gatekeeper will blink and just let you do whatever it is you were trying to do without requring whatever website or app they wanted you to use (which is probably how it should be anyways!)

sms2tweet

One of the great pleasures of having a phone without a data plan used to be that I could tweet stupid stuff out in the field, untethered from the feedback loop of Twitter’s website, by texting an official Twitter phone number which is 40404. In a textbook case of one person ruining something fun for everybody, Twitter shut down tweeting via SMS in 2019 when its former CEO Jack Dorsey was hacked.

I’d always imagined it would be pretty easy to rebuild a simple tweet via SMS app myself, but it was perpetually one of these software projects that never quite justified the effort whenever I sat down to pre-write the code in my head. And so I just never built it. But when I got my latest flip phone I decided it was time to treat myself to this capability again, and I wrote a piece of software called sms2tweet that has reinstated write-only tweeting from my phone. It’s fun to truly shout into the void this way!

To my fellow flip phone freaks

I love hearing from other people who use a flip phone. At this point I think I know about 4 or 5. But if you’re reading this and use one: email me!


What is Haberman?

05 Aug 2019

About a year ago, while looking up directions, I noticed that a prominent new place name had been added to the map in western Queens:

haberman-google

I’d never heard of “Haberman” before. The name of the neighborhood that people who live here would recognize is Maspeth (which you can see up-and-to-the-right of Haberman). Is Haberman even a real neighborhood? Why did Google put this giant Haberman label on the map?

“Haberman” past & present

If you Google “haberman”, you will quickly discover that Haberman was the name of a former Long Island Railroad station that stopped running trains in 1998. It was located on the aptly-named Rust Street. The station was originally placed there to service workers at “Haberman’s National Enameling and Stamping Company”, a factory located nearby in the early 20th century.

Haberman station in 1998 when it closed, via ARRT'S ARRCHIVES

Towards the end of its life as a station, Haberman “had no stationhouses, sheds and very few signs — the stations were just clearings at the tracks, which were unelectrified”, almost like a vaporware station. The train line and station were shut down in 1998 due to low ridership (at this time, supposedly there were only 2 or 3 daily riders!).

But knowing that the label on the map draws from this rich piece of railroad trivia still doesn’t answer the question of why a big fat label was put on the map in 2019!

At some point, I had to check Haberman out. Is it a real place? Are there Haberman locals, cafes, bars, etc? Its parent neighborhood Maspeth is a mostly industrial neighborhood in Queens, with some small residential pockets at the edges. In general in New York City, you run into very few people who live in Maspeth.

Luckily, an opportunity presented itself: we had bought a rug made in Turkey, and because it was bulky & from abroad, UPS saw fit to deliver it to the UPS regional package waystation rather than our house. This waystation happens to be located deep in the heart of Haberman! FedEx’s package waystation is also located nearby, which makes “cavernous package purgatory” a kind of cottage industry in Haberman.

kathryn in haberman

I was hoping to see some sign of overzealous NYC real estate agents trying to make a neighborhood like Haberman “happen”. I had read a little about how Google / Apple / Microsoft / Facebook sometimes purchase neighborhood boundary data from data brokers for liminal neighborhoods like this. In one case, this lead to the erasure of the name of a historically Black neighborhood called “Fruit Belt” in Buffalo; “Medical Campus” was the name that replaced it on Google’s and other maps, ultimately determined by the data broker Pitney Bowes. This is like boring, evil, real life SEO!

But walking around Haberman, it was clear that this didn’t seem to be the case here. The area mostly felt like a rare 100% industrial neighborhood in New York City. There aren’t many sidewalks, and many of them are borderline unwalkable (via double parking from NYPD or DOT workers in nearby fleet repair shops) or unmaintained. Lots of produce warehouses, factories, repair shops.

Names on the land

Returning to Google to try to find answers about why the fake name “Haberman” was placed there, I noticed a trend: a lot of other websites made reference to Haberman. Their unifying charecteristic was that they all looked programatically generated; sort of like how all lyrics websites feel like tiny tweaks to the same template, all of these websites were some variation on fake-local data (“Find local businesses in Haberman!”, “Haberman Bed Bug Exterminators”, etc.). This suggested that they were all using the same piece of data to generate their websites…

One of these sites gave a clue: the “GNIS ID for Haberman” is 972582. GNIS is the Geographic Names Information System, run by the U.S. Geological Survey. It’s a gazetteer, a dictionary of place names. The entry for Haberman notes several interesting pieces of information:

  • it’s a “Populated Place”, a category defined by the GNIS as “Place or area with clustered or scattered buildings and a permanent human population (city, settlement, town, village). A populated place is usually not incorporated and by definition has no legal boundaries. However, a populated place may have a corresponding “civil” record, the legal boundaries of which may or may not coincide with the perceived populated place. Distinct from Census and Civil classes.
  • it was added to the GNIS via the “U.S. Geological Survey. Geographic Names Phase I data compilation (1976-1981)”
  • specifically, it was added on January 23, 1980

From further reading about the GNIS, I learned that many place names were originally added by government employees manually transcribing the names from paper maps or surveys that the USGS maintains…which made me want to find that map, and understand what “Haberman” looked like on January 23, 1980!

Luckily, the USGS has a wonderful and free viewer for all of the historical maps in their collection called “TopoView” (check it out here). You can filter by date, and I looked at all maps within the area of “Haberman” that predated 1980. The latest map the USGS had from before 1980 is called “Brooklyn 1967” and looks like this — there’s Haberman!!!!!

The "Brooklyn 1967" map that the Haberman GNIS name is drawn from (viewable here). You can see labels for Haberman, Maspeth and Penny Bridge

Something stood out to me immediately: the letter spacing for “Haberman” is subtly different from the label for “Maspeth”.

Compare these former train station labels:

Bushwick Junction

Haberman

Penny Bridge

…with these neighborhood labels, where the letters are more spaced out:

Maspeth

Sunnyside

In GNIS, if you look up Maspeth or Sunnyside, they are also classified as a “Populated Place”; this designation makes sense to me, as they remain places where people live and do human activity! But looking at the former train stations, some of these are also classified as populated places — Haberman and Bushwick Junction — while others are classified as a “Locale” — Penny Bridge (a Locale is defined as “Place at which there is or was human activity; it does not include populated places, mines, and dams (battlefield, crossroad, camp, farm, ghost town, landing, railroad siding, ranch, ruins, site, station, windmill)"). There was clearly some inconsistency in how these were classified.

Here’s my theory

Back in the dark winter hours of January 23, 1980, a government employee was tasked with transcribing place names from the “Brooklyn 1967” map into the Geographic Names Information System. Maybe they get tripped up by the typography…or maybe they’ve never been to New York, and wouldn’t know that this place name that appears to be placed like a neighborhood is just a lowly train station…at any rate, Haberman was entered into the literal public record as a “Populated Place”.

Fast forward to the present day, when corporate overlords like Google and Apple compete for their maps' accuracy, buying or acquiring mapping data at great cost. A data set like the Geographic Names Information System probably serves as a useful baseline for their maps in the US: it’s hand-curated, and (I assume) generally quite accurate. Probably when their own proprietary neighborhood data appears thin in a given area, they fall back to showing any “Populated Place” names as a best-guess for a likely neighborhood name. But because this is done with no editorial oversight, this analog mistake from 1980 lives on in the devices of literally millions of Google Maps users as a 100% fake neighborhood.

Know anything further about Haberman? Got hot Haberman tips? Are you a Google Maps or USGS employee? Email me!

And long live Haberman!!!!!

Update 8/7/2019: Google erased Haberman!

After this percolated on the internet a bit, Google appears to have quietly erased the Haberman map label:

haberman no longer

Booooo! Apple and Microsoft still have it, though:

Long live Haberman!!!


sunrise sunset photos

12 Jul 2019

I wanted to rig up some way to automatically take a photo of every sunrise and sunset. New York City has spectacular sunsets, but I’ve always felt weird about taking photos of them because I experience an “observer effect” where I feel like the sunset doesn’t hit you as hard if you’re busy documenting it. Knowing that a camera will be capturing the sunset automatically is ne way to enjoy it without a lens while still knowing that if I want to return to savor it later, I can.

(I’d always assumed that part of reason we have good sunsets here is the poor air quality, but it turns out this isn’t true! Check out this comprehensive tour thru why clean skies lead to better sunsets from Stephen F. Corfidi at NOAA.)

To take the sunrise/sunset photos, I bought a Raspberry Pi Zero W, which is a cheap, small linux computer that has a camera attached and is small enough that it can be mounted to a window with suction cups.

The Raspberry Pi makes it easy to take a photo on demand with a command line utility called raspistill. But I needed a way to take the photos at precisely sunrise and sunset. I’d always heard of the terms “civil twilight” or “nautical twilight”, and Wikipedia has a great illustration of the differences between these different types of light that occur at sunrise and sunset; the main difference between these is the angle of the sun. For my purposes, I wanted to take a photo when the top of the sun was at the horizon line.

To make this happen, I found a clever open source command line program called sunwait. You give it your latitude/longitude, whether you’re interested in sunrise or sunset, and what kind of twilight you’re interested in (e.g. daylight | civil | nautical | astronomical) and it very simply waits from now until the correct moment occurs. This makes it suitable for use in a program that runs on a schedule (e.g. cron) — you can run your program at the same time every day and sunwait will wait to do whatever you want to do afterwards until the correct sunrise/sunset moment.

Here’s what my script that I use to take the photo looks like:

# this halts the script until the sunrise/sunset event has happened,
# at the "daylight" setting, which is when the top of the sun is at
# zero degrees
/home/pi/sunwait-master/sunwait wait $SUNTIME daylight $LATITUDE $LONGITUDE;
# this takes the picture, with a watermark showing the date/time
/usr/bin/raspistill --rotation 270 --height 2000 \
	-ae 90,0xff,0x808000 --annotate "$(date)" \
	--timeout 4000 --nopreview --output $FILENAME;
# ...then I upload the jpeg to my web server automatically

And here’s an example of a sunrise on a day that started out cloudy and became dreamy:

In the longer term, I want to try to figure out how to use these images to divine further data about the current weather conditions (e.g. cloud identification), but for now I’m completely content looking at the day’s sunrise & sunset! You see them update daily near the bottom of the page on Big Boy Weather.


More Regex King posts:

New York State election data quest

Origins of big boy chat

elvis-tools

present weather sensor

hello world

working hard or hardly working?

non-lethal sentences

Good random numbers

NYC LLC

yes(1)

Legalese

Rip Van Winkle logout

Allen's addendum to International Art English

Go to any level / jump higher / stay bigger / live forever

Speech Acts

Selections from O'Reilly's Regular Expressions Pt. 1