Patrick Weaver: Blog

Building a Web Page That Edits Itself

Patrick Weaver — Mon, 13 Apr 2026 00:00:00 +0000

The original vision for the world wide web was a read/write medium. A few years ago I thought, “what if an HTML file could update itself?”

When I first started making goofy web pages many years ago, the index.html file felt magical. It felt like the file was the website, and a few .html files in a directory were a little community of pages. Over time, working with things like PHP and WordPress, then Django, Rails, and Node.js, “web pages” faded into the background, even my small personal websites felt like systems that just generated ephemeral files by necessity.

After moving my personal website (this one!) to 11ty in 2020, and inspired by ideas like the “file-first” approach of Omar Rizwan’s TabFS and the “read/write web” revitalization of XH’s mmm.page and Beaker Browser, I had the idea: “A web page that can update itself!”

I quickly threw together a few getElementById() and .innerHTMLs and had a working prototype of an index.html file that could update its own DOM via the UI, and then save a copy of itself to a local file.

Next, realizing that I needed a way to reach beyond the UI in the browser to update things like the page title, or global styles, I added a control panel to update metadata.

After adding controls for global styles like the <body> width I realized I needed to rethink the UX if I wanted the project to be easy to use. For example, shrinking the <body> width also squished the control panel to the same width, after all, it was just another element in the <body>.

width has been set to 200px." style="max-width: 100%; width: 450px; margin: 0 auto;">

I had started the prototype with the classic trio of index.html, script.js and style.css, but after a few commits switched to a single file with <script> and <style> tags within the HTML. Coming back to the project a few months after that initial prototype, the code was getting to be quite repetitive and hard to follow. It was hard to keep track of logic, there were a lot of event listeners. I decided to refactor all of the JavaScript into modules using Vite as a bundler, and followed that with another refactor in TypeScript.

One early design choice was that when switching an element like an <h1> or <p> into editing mode, I would hide the original element and insert a <textarea> with the original element’s content. At first this made things like canceling the edit easier because I could just un-hide the “real” element, but I needed lots of custom logic for saving the change for different types of elements.

The TypeScript refactor had helped me finally pin down some of these bugs and I realized that it would be easier to use contenteditable="true" to edit the “real” element in-place. With this change, I was able to avoid any synchronization on save, and simplify the editing UI to just a set of buttons that I inserted below the element in the DOM. I was still able to restore the original content after a “cancel” action by storing it in data- attributes.

My original intention was to add a few more element types, things like <ul> and <strong> and <em>, but I decided it had been long enough and I should finish up with my current feature set of:

Text elements: <h1> through <h6>, and <p>
Images: <img> (these are stored as Data URLs to avoid breaking out of the one file)
Links: Selections of text, or images can be wrapped in <a>

I cleaned up my prototype CSS, added some fun early-2000s button styling, made a quick attempt to make the concept work on mobile (it’s not great), and at the suggestion of an early Recurse Center playtester, moved the control panel to the right-hand side on desktop.

You can try it out now at doodles.patrickweaver.net/one-pager. The editing UI only displays under two conditions, the file is opened locally with a file:/// protocol, or being viewed at the specific URL above on my “Doodles” subdomain. Anywhere else a web page made with One Pager will not be editable, until you download the index.html! Below is an example web page I made about my 24 Hour kid-appropriate-non-kids-songs playlist.

If you make any web pages with One Pager, let me know, and I will add it to the Example Gallery. Good luck building some HTML!

IRL HTML Is the Place For Your Most Hand-Written Websites

Patrick Weaver — Wed, 10 Jul 2024 00:00:00 +0000

Last year before HTML Energy's HTML Day in San Francisco I had the idea to iterate on some of my previous projects like Whiteboard Email, and make both an app that would do OCR on HTML, and a website to host truly “hand-written” HTML. That became IRL HTML and it was good enough to make some very simple websites!

The published website and the hand written source code, we realized that some letters were more recognizable as upper case, some as lower case.

The first prototype of IRL HTML used Tesseract.js which is only reliable for typewritten text. Given my success with album covers I tried the Google Vision API, which worked well enough for some (carefully) hand-written HTML, but still made a lot of errors, which might prevent the page from rendering at all.

A lot of the projects I’ve done over the years have used computer vision in creative ways, and I’ve always been searching for more reliable ways to have computers read hand-written text. At a Recurse Center meetup in SF in January 2024 someone suggested that the ChatGPT API might be capable of it. We made a simple website on IRL HTML to commemorate the occasion.

To get ready for the 2024 HTML Day I decided to update IRL HTML using both the ChatGPT and Anthropic Claude APIs. Both were able to read my hand-written HTML code surprisingly well! They do sometimes output text that isn’t really there (notice the <ul> tags were turned into <p> tags in the example below), and sometimes it adds some description of the output, which for this case isn’t useful. One benefit of the multi-modal nature of the LLMs is that I can inform the model that the output is supposed to be HTML, I can add instructions to fix any minor syntax errors. I could probably improve the accuracy even more by combining OCR methods, or making multiple requests to the LLMs.

The prompt I’m using for both LLM APIs is:

This is an image of a piece of paper with HTML code on it. If there are any syntax errors, fix them with the most likely valid HTML. Respond with just the HTML code property formatted, not wrapped in markdown or any description of what is in the response.

There is probably some room to iterate on the prompt for better output, but the cycle of testing (especially with different hand-written code) is long and not free, so I may stick with what is working reasonably well.

More complex hand-written source code, and the webiste (with a picture of itself)

Overall, though I’m skeptical of the utility of LLMs more generally, it’s exciting to more easily and reliably make the kinds of quirky computer vision projects I’ve been dreaming of for a long time. I’d be interested in smaller handwriting OCR specific models that I might be able to run locally someday!

Unravelling an iMessage URL Parsing Mystery

Patrick Weaver — Mon, 05 Feb 2024 00:00:00 +0000

I have long been a fan of frontend-only websites, usually single-page apps, that are able to persist state in sharable URLs despite not having a database or a backend. One of the more complex projects I used this technique for was my Crossword Puzzle Editor, which stores an entire puzzle, both clues and correct answers in the URL hash.

However, when I composed a small puzzle and shared it with a friend via iMessage I was greeted with a link to the app with empty state above a long string of random characters in a blue bubble:

I tried again with a smaller puzzle and got the same kind of broken link, but it couldn’t just be the length of the URL because surely all of the tracking URLs used in every marketing email wouldn’t break like this. I tried one of a similar length, it sent flawlessly and rendered as a clean link preview, not the jumble of query string params it really was. I looked closer and noticed that the tracking URL had one visible difference from my crossword URL: it had “-” characters every so often.

I wrote a quick function to insert intermittent “-”s in the URLs, and remove them on parsing the state from the URL, then tried sending the original crossword. It worked!

At the time I couldn’t find anyone else discussing the issue online, but my guess is that this quirk of iMessage URL parsing is due to security concerns with parsing text. A significant proportion of the zero day vulnerabilities in iOS are exploited through iMessage, and I would guess that a lot of the techniques used to create them are encoding malicious code in Base64 strings, just like I am with non malicious data.

I would guess that because “-” is not a valid Base64 character, the parser used by iMessage can review each section separately. I finished up my crossword app adding “-” as the 64th character in every URL hash (I didn’t think about the length being serendipitous with the Base64 encoding until now). In the back of my mind I was curious about how long the sections of Base64 encoded text could be but didn’t look any further into it.

A few years passed and I saw the following post on Mastodon:

It was time for my weird Base64 URL tidbit to come to the rescue! I replied with the tip about breaking up with “-”s and the original poster replied that it had worked for them as well!

After being reminded about the technique I was curious to figure out some of the details like, how long exactly could the sections be, and what other characters would work as separators. Using the technique I built a small app that would store 10 RGB values in the URL (this example has a rainbow), but would also let the user control how frequent the “-” characters were, or replace them with other characters not used in Base64.

Here is an example with the same 10 colors as the link above, but with a “+” every 10th character:

https://doodles.patrickweaver.net/imessage-url-state-example/#eyIxIjp7I+nIiOiIyMj+MiLCJnIjo+iMjIiLCJi+IjoiMTU3I+n0sIjIiOn+siciI6MjE+wLCJnIjoi+MzEiLCJiI+joiODcifS+wiMyI6eyJ+yIjoiMjA1+IiwiZyI6N+zMsImIiOi+IzNiJ9LCI+0Ijp7InIi+OiIyNTUiL+CJnIjoiMT+k3IiwiYiI+6MzZ9LCI1+Ijp7InIiO+iIyNTIiLC+JnIjoiMjQ+wIiwiYiI6+IjExOSJ9L+CI2Ijp7In+IiOiI2OSI+sImciOiIx+NzkiLCJiI+jowfSwiNy+I6eyJyIjo+iNjEiLCJn+IjoiMTA0I+iwiYiI6Mj+MyfSwiOCI+6eyJyIjoi+MTI3IiwiZ+yI6IjkwIi+wiYiI6IjI+0MiJ9LCI5+Ijp7InIiO+iIxOTAiLC+JnIjoiMTQ+4IiwiYiI6+IjIyOCJ9L+CIxMCI6ey+JyIjoiMjE+0IiwiZyI6+IjgxIiwiY+iI6IjE5NC+J9fQ==

Or one with a “*” every 300th character (there is only one):

https://doodles.patrickweaver.net/imessage-url-state-example/#eyIxIjp7InIiOiIyMjMiLCJnIjoiMjIiLCJiIjoiMTU3In0sIjIiOnsiciI6MjEwLCJnIjoiMzEiLCJiIjoiODcifSwiMyI6eyJyIjoiMjA1IiwiZyI6NzMsImIiOiIzNiJ9LCI0Ijp7InIiOiIyNTUiLCJnIjoiMTk3IiwiYiI6MzZ9LCI1Ijp7InIiOiIyNTIiLCJnIjoiMjQwIiwiYiI6IjExOSJ9LCI2Ijp7InIiOiI2OSIsImciOiIxNzkiLCJiIjowfSwiNyI6eyJyIjoiNjEiLCJnIjoiMTA0Iiw*iYiI6MjMyfSwiOCI6eyJyIjoiMTI3IiwiZyI6IjkwIiwiYiI6IjI0MiJ9LCI5Ijp7InIiOiIxOTAiLCJnIjoiMTQ4IiwiYiI6IjIyOCJ9LCIxMCI6eyJyIjoiMjE0IiwiZyI6IjgxIiwiYiI6IjE5NCJ9fQ==

300 character long sections of Base64 was longer that I expected would work, but as soon as I increased the “distance” between the separator characters to 302, which is a “-” (or “+”...) every 303rd character the links started to break.

Interestingly, the limit was 301 character long sections of Base64 on both iMessage and Signal!

A working link (301 distance) above a broken link (302 distance) in Signal

Now that I know the exact cut off of 301 characters I’ve been able to find one other blog post, but I’m still surprised that this isn’t more widely known.

Try out the app yourself or experiment with other platforms, I would love to learn more about the details behind this! This link without a hash will randomly generate 10 colors (and then store them in the URL hash): doodles.patrickweaver.net/imessage-url-state-example.

A Blog Post With Every HTML Element

Patrick Weaver — Thu, 03 Aug 2023 00:00:00 +0000

After learning a little bit more about web accessibility last year I had been exploring some of the less common HTML elements, and making changes to this website, like wrapping the text of the posts on this blog in <article> tags and adding a <main> tag in the website’s layout templates (this website is built using Eleventy).

I had previously done some work to make sure that <figure> and <figcaption> elements were layed out nicely for images with associated captions, and I had been impressed with various Recurser’s implementation of footnotes or sidenotes₁, and have been thinking it would be interesting to see what other interesting layouts were possible with just HTML.

I could, element by element, continue to add support (mostly by making CSS updates for each element to fit in with the rest of my style choices) as I came across specific needs for them, but not one to shy away from an exhaustive exploration, I decided to write this post and attempt to use every element.

A goal of the post, was to avoid delaying other future posts with CSS updates on a previously unused element, but in reality it took a year and a half to make all the updates for just this post! I am using the MDN Web Docs list of HTML elements as a reference which has more than 100 tags divided into a few categories, which I will also use in this post. Many of the tags like <html> don’t make sense to include in the text of a blog post, but if you’re viewing this post on patrickweaver.net, then every one of the elements is used somewhere on this page.

The Elements

Main Root

<html>

I didn’t have to make any changes to the <html> tag for this post, but one thing I don’t always remember to include is the lang property (in this case lang="en").

Document Metadata

I wasn’t familiar with the <base> tag before writing this post, though I’ve now added one with relative links to my layout templates. This caused a few issues with things like local development, and relative links, though they were easily resolved. The rest of the metadata tags are familiar and were already here.

Sectioning Root

<body>

Like the Document metadata tags, in the layout, though it was interesting to read the documentation and learn about attributes, like onbeforeprint that provides functionality I’ve used more hacky methods to accomplish previously, and onblur, whose primary utility seems to be annoying popups.

Content Sectioning

Headers and Document Organization

When I first looked at the list I assumed that <address> would be designed exclusively for mailing addresses, but was surprised to see that it can be used for email addresses, and even links. I updated the email address on the About page of site site, but I’ll add an <address> below also:

Email Patrick:

hello.patrickw@gmail.com

As I mentioned above, I updated the blog post page template to use the <article> tag, but reading the documentation, I’m now wondering if it would fit on every page of the site.

I’m now using <aside> elements, which represent, a portion of a document whose content is only indirectly related to the document’s main content, to wrap around my footnotes at the bottom of this page (though I may try to style them as sidenotes in the future). The current design of this website doesn’t have a <footer>, but I’ve added one to this blog post, and while I had a <header> element on the page header previously, I now know that more than one is acceptable so there is one around the header section of each blog posts as well.

Heading elements

As part of the style update and cleanup of this site that inspired this post I realized I was often using headers with incorrect hierarchy, which I cleaned up. Though I never reached 6 levels of headers, the closest I got was <h3> in some of the blog posts with multiple sections and subsections.

How to use top level headings

It is a little bit unclear how to use <h1> tags in a post like this from just the MDN documentation. This tip from the W3C suggests different implementations for pages with standalone, or collection content, which would make it difficult to use the same templates and styles for different pages on this site.

How I decided to handle it

Before writing this post I had updated the site to use <h1> for the name of the site (my name) at the top, <h2> for the section name, for this page, “Blog”, and <h3> for the title of what would be the <article> on a page. However, after reading the documentation and the tip above, I decided to update the hierarchy and use <h1> for different things on different pages, and use classes for styles, which is probably more in line with the separation of concerns of HTML and CSS (which means this paragraph is below an <h6>).

Sometime between when I started this experiment in early 2022 and when I published it in Summer 2023 <hgroup> and <search> were added to the MDN documentation (which I realize is not the official spec). I’ve added an <hgroup> around the heading of this section, with a subtitle <p> element. <search> is a semantic element that indicates that an input can be used for search, not for search results (with the exception of quick results that populate within a form ). While a <search> element that contains a <form> works with just HTML on a website that can generate search results on a server, because this website is statically generated the example below requires JavaScript. I haven’t seen a <search> element anywhere else, neither MDN’s header search, or google.com use it in 2023.

The <main> element was one of the initial curiosities that led me down the path of reading about and implementing every element, though it wasn’t until I read through the MDN list that I added a <nav> element around the menu at the top of this page. I’ve added <section> elements to this post, but I’m not sure how often I will use them elsewhere. One reason is that it makes it harder to mix and match HTML and markdown with visually clear nesting in the document where I am writing this post.

Text Content

I had previously used <blockquote> for embedding tweets into blog posts, but for this post I decided to add styles for standalone quotes. I didn’t know about the <blockquote> cite property or <cite> element until reading the docs more closely:

cite

A URL that designates a source document or message for the information quoted. This attribute is intended to point to information explaining the context or the reference for the quote.

MDN on <blockquote>

<dd>, <dl>, and <dt> are elements that, after finding out about them when first looking into more obscure HTML elements, I was very surprised to have not known about sooner. Although, it’s unclear from the documentation whether lists like the links on the current version of my portfolio page should use <dl> (I’m currently using <ul>). I am curious how <ol> and <ul> became part of almost every “Intro to HTML” class, but <dl> is relatively obscure. It’s also strange that <ol> and <ul> have default margins, but for <dl> the margin is on <dd>.

HTML list elements

dl: Description list
ol: Ordered list
ul: Unordered list

<div> has acquired a bad reputation on the modern web due to overuse, so I was surprised to be using relatively few <div> elements, though once I started using elements that I wanted to style together, I ended up with around 20. It shows how unnecessary most <div>s probably are, though this site has minimal generic “sections”.

Among the first more obscure elements that I added to this site were <figcaption> and <figure>. I was adding some blog posts that had originally been published on Medium, and wanted to add captions below images in a web-semantically correct way. Though after reading the documentation, which says <figure> can be used for, image, illustration, diagram, code snippet, etc., I realized that there are a lot of places where I currently have code blocks that I could be using them.

I have always liked <hr> elements a lot, but I’m never sure when to use them. The documentation says, While it may still be displayed as a horizontal rule in visual browsers, this element is now defined in semantic terms, rather than presentational terms, so if you wish to draw a horizontal line, you should do so using appropriate CSS which made me revisit the <hr> styles on this site and I decided to include an emoji in an hr:after rule, though I should check how that works on a screen reader.

The items in the (inline styled) lists of elements on this page are of course <li> elements, though I had never looked at the documentation until now. It’s interesting that the same <li> element is used in both <ol> and <ul> lists, with quirks like The value attribute has no meaning for unordered lists, but is not used to wrap the <dt> and <dd> elements in a <dl>.

I had not encountered <menu> before writing this post, and I was initially surprised that it survived to HTML 5 (while <menuitem> didn’t) because modern browsers treat it as essentially a <ul>. Researching further on Wikipedia I read:

MENU existed in HTML Tags, and was standardized in HTML 2.0; deprecated in HTML 4.0 Transitional; invalid in HTML 4.0 Strict; then redefined in HTML5, but removed in HTML 5.2,

and now I don’t know what to think, but here’s a <menu>:

<ol> and <ul> are some of the HTML elements I used in my earliest web pages, and more recently I try to use <ul> with CSS in places that are semantically lists, but might not be styled like a traditional list.

I was surprised not to find more guidance on using <p> tags on the MDN docs, something I wonder often when adding non long form text to a website is, “is this really a paragraph?” But, it seems like as far as HTML is concerned, if it’s not a heading, then probably!

Before reading the documentation I had really only considered using <pre> tags along with <code> tags for code blocks, but the example used on MDN is used to show how <pre> can display meaningful whitespace for things like:

  ┏┓
  ┃┃╱╲ in
  ┃╱╱╲╲ this
  ╱╱╭╮╲╲house
  ▔▏┗┛▕▔ we
  ╱▔▔▔▔▔▔▔▔▔▔╲
write semantic HTML
  ╱╱┏┳┓╭╮┏┳┓ ╲╲
  ▔▏┗┻┛┃┃┗┻┛▕▔

Inline Text Semantics

As with other very common tags I was curious to get to the documentation for <a> to see if there was anything I was unaware of, or had been using incorrectly. I was surprised to discover that <a href="#">link</a> links to the top of the page after years as just using it as a placeholder when I didn’t know the URL yet.

<abbr> seems likely to be the least used tag, especially because the MDN documentation doesn’t make a great case for it the purpose of this element is purely for the convenience of the author.

The distinction between <b> and <strong>, as the docs have it, is not what I had previously thought (that <strong> was the HTML5 replacement for <b>). I think the distinction is more nuanced and overlapping than the technical nature of the documentation is really able to convey. The way I will probably explain it to other people in the future is that, for the most part, you should use <b> for single words (or compound words), and <strong> for whole sentences or phrases. This is more a rule of thumb and bypasses the actual distinction. The documentation reads:

The <strong> element is for content that is of greater importance, while the <b> element is used to draw attention to text without indicating that it’s more important.

MDN on “<b> vs. <strong>”

But I can’t think of many instances where I would want to draw attention to a word, where that word is not more important than other words. The recommendation seems to be from a reality where, for the most part, all words are of equal importance, but deserve different amounts of attention, which doesn’t seem very common.

I’ll probably think of <b> from now on as the HTML tag with a similar effect as the quotation marks used for “emphasis” you might see on signage that can cause intergenerational confusion.

<bdi> is a tag for a concept that I hadn’t considered until now, text whose direction might (or might not) differ from the direction of the surrounding text. The examples on the documentation are all names, but could stand in for any user inputted text. In the future I will probably try to wrap any tags, like the ones controlled by the input below, that might contain arbitrary user inputted unicode characters in a <bdi> tag just to be safe.

Hello, , thanks for reading!

(عَبْد الرَّحْمٰن بِن إِبْرَاهِيم المُنِيف)

<bdo> I will probably use less often because I don’t work with RTL languages often, but it is still good to know how to handle small amounts of RTL text, like this link to the page for HTML on the Farsi Wikipedia: اچ‌تی‌ام‌ال.

<br> is interesting to me, because for a time (a long time ago) it seemed to be as misused as <div> has been recently, but, as CSS has gotten more robust that is probably less often the case; though I wonder if it is still taught early in HTML classes. It’s interesting that the example use case in the docs is poetry because it makes me wonder how the grey area between a paragraph and a standalone line was considered in creating the HTML spec. It also seems like a <pre> tag might be a better fit in some cases.

<cite> and <q> are interesting because they try to add HTML semantic elements for meaning that is also conveyed by visible punctuation in most languages. It’s the kind of redundancy that underscores repetitiousness of working with computers (the default style for cite uses italics while q adds quotation marks not in the text.).

I’ve clearly used many <code> elements in this post so far, but reading the documentation made me wonder whether the recently released Markdown support in Google Docs uses <code> for text, but on inspecting the HTML of a Doc, I remembered that <canvas> is used to render the document now.

Occasionally the MDN documentation examples are difficult to mentally translate to real-world use cases. <data> is one of these, where the only examples show a <data> element with a value property with product IDs wrapped around product names In that case the IDs seem either, user facing, in which case it would probably be better to display them to the user, or non user facing, in which case, I’m not sure who the “data” is for. Interestingly, <data> doesn’t seem to appear in the W3C HTML5 specification.

<dfn> is another strange element because it seems like it is misnamed, as it wraps, not the definition of a term, but instead that term within the definition. It does only seem to have been part of the never officially adopted HTML 2.0 proposal, and it exemplifies the academic nature of the early web. The HTML Tags: Past, Present, Proposed page on martinrinehart.com defines HTML 2.0 as There never was an HTML 2.0 standard, but these all shaped browser development in the late 20th century with references to several RFC documents.

<em> and <i> bring up similar usage questions as <b> and <strong>, but in my opinion with even more nuance, especially, as the documentation notes, with other more specific tags like <cite> that also by default italicize text in most browsers. The rule of thumb from above also for the most part fits here, <em> for single words, and <i> for sentences or phrases, though the examples in the documentation seem to mostly use <i> for drawing attention to potential confusion, as in, The word the is an article.

<kbd> is another tag that makes me wonder about the conceptual boundaries of the usage of the tag. It is intended for specifying keys on a computer keyboard, for example: to type the < character used for (the non escaped) version of the tags in this post, I press Shift + , (the styles here on <kbd> are applied through custom CSS). But I’m curious if it would also be appropriate to put a <kbd> around something like Right click (in this case I used <i> instead).

<mark> is interesting because it suggests a 2-way authoring web that was originally envisioned, but failed to come to fruition, with usage notes like, Think of this like using a highlighter pen in a book to mark passages that you find of interest. The yellow here is the default style in all major browsers.

<rp>, <rt>, and <ruby> all relate to rendering “ruby” or “agate” fonts, which are the smallest legible text used in print. They are used in HTML to, provide pronunciation, translation, or transliteration information for East Asian typography. Because I don’t read any East Asian languages, I’ll use the same example as the MDN docs do below. Interestingly, <rp> is used to hide parentheses characters, which are included in the source. It’s surprising to me that there is an element to hide these characters in a very rare instance, but we still rely on CSS to hide content visually (but still show it to screen readers).

漢 (kan) 字 (ji)

<s> is the strikethrough element, which should be used to indicate text that is not accurate or relevant, but was previously. Along with <menu> it is one of two elements (or according to MDN 3 with <u>) that were deprecated in HTML 4.01, and un-deprecated and redefined in HTML5. The redefinition changes the use of the tag from presentational, which should now be achieved with CSS, to relevant to the context of the text. However, some screen readers don’t announce the strikethrough, which seems potentially confusing. I’ve added the CSS recommended by the MDN docs to my website, so the strikethrough should be announced here.

~~HTML 3.0: <s> is used for presentation~~
HTML 5: <s> is used for removed text

<samp> is another element I wasn’t aware of before reading the MDN docs. It is used for rendering the output of a computer program, which I had previously just used <code> for (I just updated my Raspberry Pi blog post to use <samp>). Another example is below:

A “Hello, World” program in JavaScript:


console.log("Hello, World")

Hello, World

<small> is used to render smaller text, which is something I would usually have done with CSS. The MDN docs don’t provide much clarity about which strategy to use: Authors are encouraged to use their best judgement when determining whether to use <small> or CSS.. I will probably continue to use CSS for the most part since the effect seems to be presentational for the most part.

<span> is a very familiar element, but reading the documentation made me wonder for the first time why HTML and CSS allow contradictory element styles like a

<div style="display: inline" />

or a <span style="display: block" />. Though the <div> breaks the parent <p> tag, while the <span> does not.

<sub> (used for footnotes here) and <sup> (used in the equation below) are more elements I probably have used the CSS implementations of previously (though it is appropriate for presentation only super or subscript.)

<time> seems like a helpful element for creating semantic HTML, but even Google doesn’t seem to use it in search results. I’ve updated the dates on my blog posts to use it.

<u> has probably my favorite quote from the MDN docs, Most of the time, you actually don’t want to use <u>. Their strongest recommendation seems to be indicating spellling errors.

<var> is a way to indicate semantically that a string is a variable for either math or programming. For example, rendering the pythagorean theorem uses both <var> and <sup>: a² + b² = c².

<wbr> was also a new element until now, but it solves a problem I have hit many times before, breaking long “words” at certain points, specifically URLs. If I put a <wbr> element at clear break points in this URL: https://www.patrickweaver.net/pages/averylongurlwithoutanypunctuationbutstillmadeupofwords/ then it will break at clear points at different screen widths, which is is more readable compared to the same link without the <wbr> elements:

https://www.patrickweaver.net/pages/averylongurlwithoutanypunctuationbutstillmadeupofwords/

Though including hyphen characters in the URL also creates clear breakpoints in modern browsers. I did have to wrap the URL above in a special scrollable <p> to prevent it from breaking the layout of the rest of the page.

Image and Multimedia

<area> and <map> are elements that I hadn’t been familiar with previously, even though I had made a few image map type websites in the days before CSS 3. It seems like an indication of how seldom they are used these days that the tools for debugging the boundaries of <area> elements are hard to use. One <area> border at a time will display while using tab focus, but styling the elements does not work (unless there is a display hack I couldn’t figure out). It’s also somewhat strange that <area> is essentially an <a> with a shape.

Here’s a <map> (with <area>s on each tag linking to MDN) of a handwritten HTML document I made for June 3rd’s HTML Day Freewrite in SF.

<audio> is one of the classic multimedia HTML 5 tags so I’ve used it before, but thinking about it now, it’s surprising I don’t see them more often in the 2020s. I would guess that there are more in use than I thought, for example, the Bandcamp player uses a hidden <audio> element even though the UI is a custom mix of <a> and <div>s inside a <table> oddly enough.

A recording of me reading the paragraph above. (Excuse my scratchy late night voice)

<img> is of course one of the first elements I used, but I was curious what properties could be used with it that I might not have heard of. srcset, used for specifying multiple sizes of the same image to load at different screen resolutions, was one of those. I’ve created a demo below, which goes against the spirit of srcset.

The image below has 3 images provided to its srcset property:

A yellow background and a monospace font that is 300 pixels wide and should render when the page is less than 520 pixels wide
A blue background and a serifed font that should render when the page is between 520 and 800 pixels wide
A green background and a script font that should render when the page is greater than 800 pixels wide.

tag that loads different images at different sizes" id="image-with-srcset">

I have tested it, and it does work, but it takes a lot of forethought to make sure that you will be able to load the correct image. Things like the browser’s cache, and scaled displays will change the behavior. If you want to see the effect you will probably need to load the page in a private window on a non scaled or “retina” display.

It’s interesting that while both <audio> and <video> are less widely used than <img> even after their introduction in HTML 5, the distribution of videos online seems to be centralized, mostly on YouTube, while audio is more decentralized, with people serving individual files when distributing things like podcasts, even though the audio player that is often used is not on the web.

Download the video

A screen recording of my dev setup while writing the paragraph above.

Select below to control if the video above is shown in a <video>, <embed>, or <object> element:

<video>

<embed>

<object>

One reason for this may be that there are still some quirks with <video> elements. For example, there is a <figcaption> accompanying the video above, but when I initially tried creating the element with a self closing tag it did not render. Also, as I’m writing this, Safari does not support the video at all, likely because of the development server I am using not supporting the “Range” request header. I am curious to find out whether the hosted version of the site (on GitHub pages) will support playing the video in Safari. I thought that I might have a similar issue with <track>, but that just turned out to be me not setting up my Eleventy build correctly for .vtt files.

Embedded Content

It is somewhat surprising that <embed> and <object> have not been deprecated since most of their uses have been superseded by specific tags like <video> and <iframe>. Maybe eventually they will be, or they will just live on in the name of backwards compatibility.

Speaking of <iframe> it’s also interesting that the very 90s name “iframe” (“inline frame”, not “iPod Frame” or something, and I was surprised to learn does render display: inline) has survived while <frame> was deprecated in HTML 5. I thought that <iframe> might be a way to force the <img> srcset example above to render the smaller images on a high DPI display, but even at 150px wide on my device the large image still renders:

An embed of a page with just the srcset example from above at 150px wide.

<picture> is a tag that is, with <source> actually intended for layout tricks like the srcset experiment I tried above, rendering different versions of an image in different situations. The image below should show an icon of a computer when used with a mouse/trackpad, or an icon of a phone when used with a touch screen (using the pointer media query to determine), and should fall back to an image of a red circle with a line through it when neither media query is appropriate. The responsive design developer tools in your browser should trick it, though interestingly, not when inspecting an element.

<portal> is one of the more mysterious sounding elements, and seems intended for the iPad-style link previews that I’ve seen implemented on some websites on hover, but also come with SPA-like performance benefits. It unfortunately is still an experimental feature and isn’t (as of 2023) enabled by default in any browsers. It seems that it used to be an available experimental flag in Chrome, but today in 2023 it is not available at chrome://flags. I’ll include one below in case it works for future readers! I did try it in an very old Chromium version I still had installed. <portal> doesn’t seem to be part of the HTML 5 Spec, I’m not sure why it’s included in the MDN documentation.

A <portal> element that should display the “Links” page on patrickweaver.net, if the feature is ever implemented in browsers.

SVG and MathML

This is an interesting section of the documentation because it pairs one of the most ubiquitous elements with one of the most obscure. <svg> elements are everywhere, especially that tools like Figma and Sketch have made them easy to prototype and export. Pairing them with <math> in the documentation evokes a more artisanal, hand-crafted <svg> that is relatively uncommon these days. I’ve gone hand-crafted here and drawn some <svg> circles.

An svg that shows overlapping circles each with a different color of the rainbow.

<math> is really a wrapper element for other non HTML elements from the MathML namespace, so I guess I don’t have to include every possible child element here. I’ll stick to something simple: $2 + 2 = 5$

Scripting

I did a deep dive into <canvas>, specifically drawing crisp lines in 2020 while I was at Recurse Center. One thing I wasn’t familiar with before reading the MDN docs though, was that there is a maximum size for a <canvas> element in each browser (though all modern browsers it is about 32 thousand pixels in each dimension). Below is a re-implementation of <marquee> with <canvas> and <script> (and <noscript>).

Start or Stop the fake <marquee>:

There were a lot of tags that I expected to feel a little out of date when I started this post, but <script> wasn’t one of them, but now that I have written most of the post, and have created a few <script> tags, I’m realizing how little I use them now. For most of the code that I write, even though most of it is JavaScript, all of the “scripting” gets put into on <script> tag by build tools and it ends up feeling more like boilerplate than a “markup” tag.

On the other hand, for most of the interactive websites I’ve made, I’ve rarely included a <noscript> tag with the exception of those minified and compiled single page apps where the user would likely get a blank page if it weren’t for the <noscript> tag (and even then, probably less often than I should have, though I’ve made sure to include one with every <script> tag here).

Demarcating Edits

Another set of new to me elements ~~I had never come across before~~ on reading the MDN documentation, <del> and <ins> seem to be intended for ~~creating~~ use in a word processor. Reading about them led to yet another round of, “~~I wonder if these are used in rendering Google Docs?~~ Oh wait, it’s all <canvas>.”

Table Content

It’s not a new insight that <table> elements were overused for layout purposes on the early web, but an irony that I’m only realizing now is that recently, as the web has gotten more and more populated by data, that <table> elements have become rarely used, probably mostly due to their clunky default design, but probably also because one main goal of creating a web UI for the data that is otherwise probably stored in database tables, is to create a different view of the same data.

I don’t have any tabular data as this is an exclusively document based blog post, but below I’ve added a table with some custom CSS and “data” on emoji that is likely only accurate for the Apple emoji set in 2023. I wasn’t familiar with <col> and <colgroup> before, but I don’t know that there are many cases where I would use them rather than a more custom, non <table> design.

A table with a custom CSS
Number	1	2	3	4	5	6
Emoji	1️⃣	2️⃣	3️⃣	4️⃣	5️⃣	6️⃣
Emoji with leaves	🥭	🌱	🍂	🍀	🪴
Emoji with fruit	🥭	🍒	🫐
Emoji with slices	🍰	🥪		🥒
Totals	4	4	3	3	2	1

Forms

I have usually used <form> and it’s associated elements when submitting the form would redirect to a new page, not for in-page user interaction, but either is valid. I would guess the form elements are probably the most important for accessibility so I will likely refer to the documentation whenever implementing one in the future.

Fill out this form and scroll down to get an emoji surprise below!

Your emoji:

Interactive Elements

The <details> and <summary> combination is probably the most common, “You don’t need JavaScript to…” element (another popular blog post making this point prompted me to finally finish this one). Making this kind of expand/collapse interactive view was one of the first things I used JavaScript for (it was probably jQuery). I remember it being somewhat confusing at the time (2006?), so it’s funny to me to see it so easy with just a couple HTML tags.

Speculation about <details>

I do wonder if <details> and <summary> were being considered for the HTML spec today, if they would be withdrawn in favor of a similar effect using CSS.

I wasn’t aware of the HTML only <dialog> before using it here. It requires JavaScript to be useful beyond just its stylistic impact, but it’s an interesting way to create interactions that are more custom than alert(), but still simple, because while alert() is styled by the browser <dialog> can be styled with CSS.

       🤠
     🌴🌴🌴
   🌴  🌴  🌴
   👇 🌴 🌴 👇
　 　 🌴 🌴
　 　 🌴 🌴
　 　 👢 👢
  the palm sheriff

Web Components

Web components had been on a (long) list of “things I should learn more about” for a while, so I was glad to get to this section of the MDN documentation. I was surprised that JavaScript (and Class syntax at that) was necessary to get web components to render, but after reading more about the Shadow DOM, it makes sense, because HTML has no built in way to encapsulate styles, though it does seem to be somewhat similar conceptually to <iframe> elements.

I’ve used web components below to create a table of contents for this post below, though I’m not making the best use of them. One thing I couldn’t quickly figure out how to do, when compared to components from frontend frameworks, is pass in properties that are themselves used as properties of elements in the child template, so in the list below I have to pass a full <a> element into each child (which renders the <li>), to be able to set the correct href property. Someday after finishing this post I will read more about web components and maybe figure out something I’m missing now.

Obsolete and Deprecated Elements

There are of course a few deprecated HTML elements, so why not include them here at the end.

Everyone’s favorite is of course,

<marquee>

Hello!

Many of the elements, like <big>, <acronym>, <font> (used above), <nobr> and <strike> are examples of layout related functionality that has been moved to CSS rather than ~~have HTML handle it~~. The narrow box below has an example of <nobr>:

This sentence is wrapped in a <nobr>.

This sentence is wrapped in a <p>.

This sentence is wrapped in a <p> with style="white-space: nowrap;".

There are some other strange ones as well though. I couldn’t implement <content> and <shadow> here because they were never fully implemented part of the web components spec. I’ll add some below anyway though.

<shadow> and <content> elements (that won’t render).

And <dir> is a very webserver version of <ul> for listing directories. Since my Eleventy static site generator does use directories here’s a list:

Some of deprecated elements won’t render without some extra work, for example <frame> and <frameset> are designed to be used instead of a <body>, for I guess some kind of collage web page made up of other pages. I’ve ironically only been able to use them inside of an <iframe> below.

An embed of a page with <frameset> and <frame> elements that itself embeds the Links page of this site.

I expected that <image> wouldn’t work since the MDN documentation says:

This does not appear to have been part of any formal specification. It was mentioned in HTML+ Discussion Document - Dave Raggett, November 8, 1993 (Section 5.9 - Images), but was not part of the first revision of HyperText Markup Language Specification - 2.0 in 1994.

But it does seem to work in some browsers in 2023, though this might be just a fallback for potential confusion with <img>.

An icon of a fax machine

One of the more interesting browser compatibility stories of the deprecated elements is <menuitem>, which supposedly was partially supported in Firefox versions 8 - 84, but I couldn’t get an example to work in any of the older versions I installed locally (even with editing about:config). The examples imply that the <menu> (of type=context) and child <menuitem> elements shouldn’t render until the element whose contextmenu property is right clicked, but as you can see below, the items always render:

Radio Button 1 Radio Button 2

It’s surprising that <noembed>, essentially a <noscript> style fallback, was deprecated while <embed> was kept in the spec. Here’s my <embed> from above, but this time with a <noembed> child. There is also <noframes> which I included above with the <frame> element.

An embed of a screen recording of my dev setup while writing the paragraph above.

An <embed> with a <noembed> tag child.

It’s unclear why <param> ever existed instead of just using attributes on the <object> element. When I started this blog post it was listed in the “Embedding Content” section, but by the time I finished it was deprecated. Here is one below:

An <object> with a <param> tag child.

<plaintext> is another weird one, and notable among the deprecated elements in that it works. It is another one I am going to have to wrap in an <iframe> because otherwise the rest of the page would be rendered as the source code:

An embed of a page with a <plaintext> element.

However, <xmp> and <tt>, used in a similar way, have closing tags

so they can be used here for displaying source code or text in a monospace fontas it would be shown on a teletype without an <iframe>.

A few of the deprecated elements have just been replaced by newer additions to the spec. <rb> and <rtc> were used for displaying pronunciation of East Asian characters, but it seems like the spec was simplified. Here is the example from MDN:

馬(mǎ)來(lái)西(xī)亞(yà)(Malaysia)

Other Elements

I consulted a few other lists of HTML tags after realizing that the  tag wasn’t in my list from MDN (there’s one below this paragraph)

An older list from MDN in 2013 has elements like <blink>, and for accessibility reasons, almost all browsers currently ignore this element. I’ll add one hidden below if you can find a very old browser that supports it:

Also on the list is <spacer>, which doesn’t work in modern browsers and,

would now be done

with CSS.

Then there’s others like <bgsound> (2013 docs) that seems to have never been implemented (except maybe in Netscape?), but alludes to a vision for a much more multimedia enabled web documents, and <isindex> (2013 docs) which seems like it is from a world where JavaScript doesn’t exist.

Another attempt at <xmp>, <tt>, <code> was

listing

I’m surprised it seems to still work!

And of course <applet>, but I’m not even going to try with that one.

Footnotes

Which I am just now discovering there is no native HTML implementation for!
↑ Return to post

Alphabetic Internet Time: A Time Zone for the Internet

Patrick Weaver — Thu, 18 Aug 2022 00:00:00 +0000

In the Fall of 2020 I participated in a 12 week batch at the Recurse Center. While I had a lot of previous experience working remotely, both before and during the pandemic, RC was the first time where I was communicating with people in more than 1 or 2 other time zones. I now work at a fully remote company, mostly spread over the 4(ish) U.S. time zones, but with a few people permanently or temporarily in other places.

Communicating across time zones

RC when I was there (mostly I would guess because their software was designed and configured for in person retreats in NYC) operated for the most part in Eastern Time, although I see from my limited participation in the community as an alum that this may have shifted, especially thanks to Zulip’s time-zone-relative timestamp support. RC had transitioned to hosting Recursers remotely a few months before my batch, which is how they continue to operate now in Summer 2022. At my current job we often default to either Eastern Time or Pacific Time, but some people will also just use their local time and let others figure it out.

Out of politeness and miscalculation-anxiety reducing redundancy I often will use multiple time zones, for example when proposing a meeting I might say, “Let’s meet at 🗽 12:30 PM ET / 🏔 10:30 AM MT / 🌁 9:30 AM PT,” (I’m still searching for a good Central Time emoji, 🛣 🌽 🌪 are the best I’ve got), but time zone based confusion often takes more thinking than it should.

(Side note, it bewilders me that Slack still doesn’t have something similar to Zulip’s “you write in your time zone, they read in their time zone” timestamp support in 2022. 🙃)

Time zones are pretty confusing, DST makes them worse

Something that probably doesn’t cause very much confusion for most other people, but I can’t help but consistently notice is that most people exclusively use ”EST/CST/MST/PST” throughout the year, even during the Spring, Summer, and Fall, when most of the U.S. is observing Daylight Saving Time. This linguistic pattern (which I assume is somewhat caused by “EST” seeming like an abbreviation for “Eastern”) makes me especially curious what the commonly used abbreviations will be if the U.S. switches to permanent DST, and how people in Arizona, which mostly does not observe DST understand someone in Denver saying, “Let’s meet at 10:30 MST on August 14.” I would imagine them thinking something like, ”😫.”

Even Google search seems to get this wrong as searching “current time MST” at 8:39 PM EDT in NYC in August shows ”6:39 Mountain Time (MT)”, though I also wonder what the same search would show in Arizona.

A time zone for the internet

All of this leads to my Alphabetic Internet Time (AIT), which I consider “A Great Idea That Will Never Work.” AIT is essentially UTC, but with the letters A through X of the (English) alphabet replacing the hour digits. Midnight UTC is A:00, Noon UTC is M:00, 1:34 PM UTC is N:34. Just like UTC, these times are not local, S:00 is the same moment everywhere (18:00 UTC). For someone in New York in August 2022 observing EDT, it is 2:00 PM, for the same person in January 2023 S:00 is 1:00 PM. For someone in San Francisco in August, S:00 is 11:00 AM.

This may all sound confusing, but it’s mostly because AIT is not really useful at all for knowing what time it is somewhere else. The main goal of AIT is reducing the amount of time you spend thinking about what time it is somewhere else (though it is likely still polite to make sure you’re not trying to schedule something in the middle of someone else’s night). AIT starts to make sense when you think about people in different time zones communicating about something that will happen online. As someone who works normal business hours in NYC, my work day goes from N:00 (9 AM EDT) to V:00 (5 PM EDT), with lunch usually at Q:30 (though during the Winter it would be O:00 to W:00). If I’m planning with someone in San Francisco who also works normal business hours (Q:00 - A:00) and we want at least an hour, we could easily see that any time between R:00 and V:00 would work, and we would both know what time that was for us.

If AIT were to work it wouldn’t rely on people manually calculating the current time, they would need clocks that display AIT like this reference implementation I made: doodles.patrickweaver.net/ait/

A reference implementation of an AIT clock at: doodles.patrickweaver.net/ait/

Why not just use UTC?

People who are already used to using UTC might look at AIT and think that it’s just a confusing extra layer on top of a system that works pretty well, but in my opinion, moving whole communities of people to UTC would likely be very difficult because doing so asks them to do the same mental math as coordinating between time zones, but often with larger numbers.

Even after changing my phone to use a 24 hour clock for weeks while spending time in countries where that is common I still found myself translating 16:00 to “16 - 2 - 10 = 4 PM” in my head. My guess is that for most people it would be easier to convert between two separate concepts, their local time with numbers, and AIT with letters, but I may be wrong. The other advantage is clarity, even if the known best practice is to use UTC, a message like “Let’s meet at 15:30” can still be unclear, potentially UTC, the sender’s local time, or your own!

Does anything like this exist?

The closest thing to AIT that currently exists is the Military Time Zones which also use the alphabet to refer to time zones, but in a way that doesn’t make much sense. Like AIT the time zones move East from UTC with UTC+1 being “Alfa”, UTC+2 “Bravo”, and continuing through the NATO phonetic alphabet (though skipping “Juliett”) to UTC+12, “Mike”. However, rather than continue through the alphabet at UTC-11, “November” is back near where we started as UTC-1 (and continues moving West from there).

The military time zone names do seem useful, and are possibly too similar to AIT for my idea to catch on, but out of sequence order make the concept seem too confusing for anyone using it for much beyond just “time zone names.”

One interesting (and further confusing thing) about the military time zone names is that “Mike”, UTC+12 and “Yankee”, UTC-12 are the same time, but on different dates because the border straddles the international date line. It’s quirks like this that make me think that any improvement on time zone related communication is likely to fail.

Reasons why AIT will never work

As good an idea as I think this is, I have no hopes of AIT catching on mostly because I think that most people wouldn’t be able to get over the ridiculousness of saying “M:45” (and not being worried people would think they’re talking about a bus). But beyond the silliness, it’s still Daylight Saving Time that makes communicating across time zones unlikely to be improved on.

If your own time zone stayed the same relative to AIT year round I think there would be a chance that, with tools like an AIT display next to your local time, or calendar integrations, that it would work for a lot of people who frequently communicate with people in other time zones, but having to switch recurring times back and forth by one letter twice a year seems like friction that would dampen most people’s enthusiasm.

Bringing dates into the picture also seems like it would complicate things. Even though A:15 is 5:15 PM PDT in San Francisco, if we put a date on it, it would have to be “tomorrow” for the person in SF to match UTC. I don’t think that this would add any additional confusion to communicating with people whose time zone is in the next day relative to your own, but a new system like AIT would likely get blamed for the inherent awkwardness of time zones.

But who knows! Weirder ideas have caught on in the past, like Daylight Saving Time! Or, maybe we can all just join the UTC+8 “Hotel” time zone since almost 25% of the world population already lives in it!

Aug 18, 2022 B:14, Brooklyn, NY

How to Set Up a Raspberry Pi as a Home Web Server

Patrick Weaver — Mon, 03 May 2021 00:00:00 +0000

Like most of the world, I’ve been spending a lot of time at home lately. Over the winter I was experimenting with building my own web-based digital tools, like an app to track articles and books I was reading. I initially thought I would need to buy hosting space to have the app accessible, but realized I mostly needed to use it when I was at home anyway, so a URL that only existed on my local network work also work.

I have done a few projects with Raspberry Pis before, most notably a computer vision “record player”, but always found myself looking up the same things like how to set up Wi-Fi and SSH access. I seemed to need to do the same handful of things every time, but they were spread across various documentation and how-tos. This time while setting up the Pi web server I took notes so I would have it all in one place.

To follow along with these steps you will need a second computer to set up the SD card (though you could probably start with a preformatted SD card also), and any kind of Raspberry Pi. I have run web servers on Pi Zeros before, though it can be a challenge to install newer versions of Node.js on ARMv6 based Pis (Zero and the original Raspberry Pi), and sometimes when building a large client app I’ve had to move to a faster Pi from a Pi Zero.

SD Card Setup

1. Install the OS

Raspberry Pi recently released their own tool for formatting SD cards, Raspberry Pi Imager. It makes the process a lot easier than it had been previously. The first step is picking an OS to format the card with. Any of the Debian-based OSes should work, though the smallest download is Raspberry Pi OS Lite, which is all you need if you won’t be connecting a display. After that, select the SD card you want to install onto (there will probably only be one choice), and click “Write”.

Step 1 Summary

1. Install the OS: Use Raspberry Pi Imager tool to write the OS to the SD card source

2. Set up Wi-Fi

The RPi Imager tool will likely eject the SD card for you when it’s finished writing, but you want to do two more things before booting up the Pi. You may need to remove the SD card and re-insert it into your computer to see the “Boot” filesystem. First, to enable Wi-Fi you will need to create a file with your network credentials. If you will be plugging the Pi into an ethernet cable you can skip this step.

Create a file called wpa_supplicant.conf in the root directory of the SD card. Open the file in a text editor and paste in the following:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=<2 LETTER ISO 3166-1 COUNTRY CODE>

network={
    ssid="<WI-FI SSID>"
    psk="<WI-FI PASSWORD>"
}

Update the country line with your ISO 3166 country code (for the United States this will be US without quotes). Update the ssid and psk lines of the network section to your Wi-Fi network SSID and password, with quotes. Read more about setting up Wi-Fi here.

My wpa_supplicant.conf might look like this:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US

network={
    ssid="patricknet"
    psk="myverysecurepassword"
}

Note on Text Encoding

Setting up a Raspberry Pi headless: “Depending on the OS and editor you are creating this on, the file could have incorrect newlines or the wrong file extension so make sure you use an editor that accounts for this. Linux expects the line feed (LF) newline character.”

Learn More

Scott Hanselman on CR/LF

Step 2 Summary

2. Set up Wi-Fi: Create a wpa_supplicant.conf file in the root directory of the SD card, and populate it with your Wi-Fi credentials to enable Wi-Fi on boot. source

3. Set up SSH

By default SSH access is disabled on a Raspberry Pi, the usual way of enabling it is either through the GUI or raspi-config, but there is also a simple way to pre-setup on the SD card. Create an empty file called ssh (without a file extension) in the root directory of the SD card, this will enable SSH on boot.

Step 3 Summary

3. Set up SSH: Create an empty ssh file in the root directory of the SD card to enable SSH access on boot. source

Raspberry Pi Setup

You are now ready to boot the Raspberry Pi, insert the SD card and power up the Pi. After a minute or two of boot time it should automatically connect to your network.

4. Connect via SSH

In a terminal on a second computer and connect to the Raspberry Pi via SSH. The default username and password will be pi and raspberry, and the default hostname will be raspberrypi. If your local network supports the .local TLD you may be able to connect using:

ssh pi@raspberrypi.local

Otherwise you will need to look up the IP address of the Pi in your network admin tools and connect using the IP address:

ssh pi@192.168.XXX.XXX

You can also skip this step and open the Terminal app on the Pi if using a display and keyboard connected to the Pi.

If you have previously connected via SSH to a Raspberry Pi with the default hostname on the computer you are using you may see a message about the remote host identification changing. To successfully connect via SSH you will need to update the known_hosts file to remove the key stored for the other Raspberry Pi.

On Linux or macOS the known_hosts file will be at /Users/[YOUR USERNAME]/.ssh/known_hosts. Find the line in this file that starts with either raspberrypi.local or the Raspberry Pi’s IP address and remove it.

Step 4 Summary

4. Connect via SSH: Connect to the Raspberry Pi via ssh: ssh pi@raspberrypi.local or ssh pi@192.168.XXX.XXX (and enter the default password raspberry). You may need to remove a previous raspberrypi.local from /Users/USERNAME/.ssh/known_hosts.

5. Change the Password

After connecting to the Raspberry Pi via SSH the login message will suggest that you change the default password:

SSH is enabled and the default password for the 'pi' user has not been changed. > This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.

Following the instructions you can type passwd which will prompt you first for your current password (raspberry), then for a new password twice.

Step 5 Summary

5. Change the password: Run passwd to have the system prompt you for a new, more secure password.

6. Change the Hostname

The default hostname of your Raspberry Pi is raspberrypi. You may want to change this for a few reasons. If you have more than one Raspberry Pi on your network this will let you tell them apart, and if your network supports the .local TLD you can use the hostname as the URL to your app.

You will need to update two files to change your hostname. First open /etc/hostname and change the first and only line in the file (currently raspberrypi) to your new hostname. Then, open /etc/hosts and update the the line with raspberrypi to your new hostname. Finally, reboot your Pi for the changes to take effect. Remember, if you SSHed in as pi@raspberrypi.local you will need to use the new hostname instead.

Step 6 Summary

6. Change the Hostname: Open sudo nano /etc/hostname and change raspberrypi to your new hostname, then sudo nano /etc/hosts and change raspberrypi to your new hostname. Then reboot, sudo reboot now. source

7. Change the Prompt

I also like the change the default bash prompt whenever I am going to be SSHing into a remote computer so that I can tell the difference at a glance between a local and remote terminal window.

A simple change is just to prepend an emoji to the default prompt, which you can do by adding a line to ~/.bashrc:

echo 'PS1="🥧 ${PS1}"' >> ~/.bashrc

I like the following prompt which adds colors that are different than the ones that I use locally, and a pie emoji:

Mon May 03 23:41:09
pi@b: ~
🥧 $

Update your prompt by editing your ~/.bashrc file with sudo nano ~/.bashrc. The bash prompt is set with the PS1 variable, which has two versions in the default Raspberry Pi ~/.bashrc, one for when color_prompt is set to yes, another for otherwise. The default in Raspberry Pi OS is that color_prompt is set to yes, so feel free to overwrite only the first PS1 variable.

The value to set the prompt I showed above is (the sections with [\033 ... and \[$(tput... control turning on and off the colors):

PS1="\n\[\033[48;5;7m\] \d \t \[$(tput sgr0)\]\n\[$(tput bold)\]\[\033[38;5;0m\]\[\033[48;5;14m\] \u@\H: \[$(tput sgr0)\] \[\033[38;5;0m\]\[\033[48;5;15m\] \w \[$(tput sgr0)\]\n 🥧 \\$ "

Run source ~/.bashrc (or . ~/.bashrc) to reload your terminal and see your new prompt.

Step 7 Summary

7. Change the Prompt: Update the PS1 variable in ~/.bashrc to make it look different. Add a pie emoji with echo 'PS1="🥧 ${PS1}"' >> ~/.bashrc.

8. Install Git

You will probably want to use Git to manage the source code for your app. It is not installed by default on Raspberry Pi OS so you will need to install it now with apt:

sudo apt install git

Note: You may have to update by running sudo apt-get update in order for Git to install.

You may also want to install another text editor if you want something more customizable than nano.

Step 8 Summary

8. Install Git: Install git to manage your source code with sudo apt install git.

9. Install Nginx

You can now run a web server (with sudo) on port 80 and have it be available at your IP address or [YOUR_HOSTNAME].local, but that might require more configuration. It will be easier to manage your app if you install a web server/reverse proxy like Nginx. You can install Nginx with apt:

sudo apt install nginx

Once Nginx is installed you should be able to access the default Nginx webpage at your Raspberry Pi’s IP Address or [YOUR_HOSTNAME].local. Make sure to type in the http:// the first time if you are using the .local TLD to avoid triggering search in your browser.

You can now serve any static website from /var/www/html, if you look there you will see an index.html or index.nginx-debian.html file that is generating the current page.

Step 9 Summary

9. Install Nginx: Install with apt install nginx, then put a static webpage in /var/www/html.

10. Install nvm and npm

I make most of my web apps these days in JavaScript, there are lots of ways to install node and npm, but one easy way to manage versions is with nvm, which can be installed with the script below:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash

Then reload your terminal with . ~/.bashrc and install the current LTS version of node with nvm install lts.

Note: On an original Raspberry Pi, or a Raspberry Pi Zero, the ARMv6 chip is no longer supported by current versions of node. “Unofficial” ARMv6 builds of node are available at: unofficial-builds.nodejs.org. To install a version using an unofficial build use:

NVM_NODEJS_ORG_MIRROR=https://unofficial-builds.nodejs.org/download/release nvm install lts

Step 10 Summary

10. Install nvm and npm: curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash, . ~/.bashrc, nvm install lts.

11. Set Up Reverse Proxy

In order to access your app on your local network without a port number in the URL it will need to be accessible on port 80, but it will be easier to run your app on another port locally. This can be done with an Nginx reverse proxy.

Nginx configuration is done via server block files, you can delete the default enabled server block by running:

sudo rm /etc/nginx/sites-enabled/default

Next, make your own empty server block in the sites-available directory: . The [YOUR SITE] part of the path can be anything, but it makes sense to give it the name of your app.

sudo touch /etc/nginx/sites-available/[YOUR SITE].conf

You can now edit your server block file:

sudo nano /etc/nginx/sites-available/[YOUR SITE].conf

The server block below will proxy processes running on port 8080 to port 80 (but you can substitute any port on the proxy_pass line). Make sure to replace the [HOSTNAME] sections with your Raspberry Pi’s hostname (or omit this line if accessing via IP Address):

server {
    listen 80;
    server_name www.[HOSTNAME].local [HOSTNAME].local;

    location / {
       proxy_pass http://127.0.0.1:8080;
    }
}

Then, link your Nginx config file to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/[YOUR-SITE].conf /etc/nginx/sites-enabled/[YOUR-SITE].conf

Test your Nginx config with sudo nginx -t. You should see a success confirmation message:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

To enable the configuration reload the Nginx config:

sudo systemctl reload nginx

The last step is cloning your project to the Raspberry Pi and running it on port 8080, your app should now be available at [HOSTNAME].local!

Step 11 Summary

11. Set Up Reverse Proxy: Remove default: sudo rm /etc/nginx/sites-enabled/default, Create config file: sudo touch /etc/nginx/sites-available/[YOUR SITE].conf, Edit config file (see above): sudo nano /etc/nginx/sites-available/[YOUR SITE].conf, Link to sites-enabled: sudo ln -s /etc/nginx/sites-available/[YOUR-SITE].conf /etc/nginx/sites-enabled/[YOUR-SITE].conf, Reload Nginx: sudo systemctl reload nginx. source

All Steps:

Install the OS: Use Raspberry Pi Imager tool to write the OS to the SD card source
Set up Wi-Fi: Create a wpa_supplicant.conf file in the root directory of the SD card, and populate it with your Wi-Fi credentials to enable Wi-Fi on boot. source
Set up SSH: Create an empty ssh file in the root directory of the SD card to enable SSH access on boot. source
Connect via SSH: Connect to the Raspberry Pi via ssh: ssh pi@raspberrypi.local or ssh pi@192.168.XXX.XXX (and enter the default password raspberry). You may need to remove a previous raspberrypi.local from /Users/USERNAME/.ssh/known_hosts.
Change the password: Run passwd to have the system prompt you for a new, more secure password.
Change the Hostname: Open sudo nano /etc/hostname and change raspberrypi to your new hostname, then sudo nano /etc/hosts and change raspberrypi to your new hostname. Then reboot, sudo reboot now. source
Change the Prompt: Update the PS1 variable in ~/.bashrc to make it look different. Add a pie emoji with echo 'PS1="🥧 ${PS1}"' >> ~/.bashrc.
Install Git: Install git to manage your source code with sudo apt install git.
Install Nginx: Install with apt install nginx, then put a static webpage in /var/www/html.
Install nvm and npm: curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash, . ~/.bashrc, nvm install lts.
Set Up Reverse Proxy: Remove default: sudo rm /etc/nginx/sites-enabled/default, Create config file: sudo touch /etc/nginx/sites-available/[YOUR SITE].conf, Edit config file (see above): sudo nano /etc/nginx/sites-available/[YOUR SITE].conf, Link to sites-enabled: sudo ln -s /etc/nginx/sites-available/[YOUR-SITE].conf /etc/nginx/sites-enabled/[YOUR-SITE].conf, Reload Nginx: sudo systemctl reload nginx. source

Participating in a Remote Batch at the Recurse Center

Patrick Weaver — Mon, 25 Jan 2021 16:33:26 +0000

Participating in a batch at the Recurse Center is something that I’ve thought on-and-off about doing since I found out about it (then called Hacker School) sometime between 2013 and 2015, but it just never seemed like the right time to leave my job and its subsidized health insurance until I was laid off (what turned out to be) early on in the pandemic. I had as recently as January 2020 been thinking, “maybe now is the right time to finally quit and do it,” and though I’m relieved I was able to get a few more months of savings and health insurance into 2020, participating in RC was the perfect change of pace and a motivating bridge between employment and the job search.

Screenshot from recurse.com

I had been working from home at my old job from mid March to when I left in mid July, but I didn’t realize until my first day at RC, that even though I had frequently seen friends and coworkers on video calls and in outdoor settings for the past 6 months, it was rare that I had the chance to speak to anyone I hadn’t previously met since March 2020.

There are a few different pieces to the experience of a remote batch at RC. Virtual RC is a fun game-like experience where each participant (and visiting alumni) has an avatar on the map. One corner of the map is more organized, and designed with a layout intended to mimic the physical RC space in Brooklyn, but the rest of the map is open for creative world building or experiments (there is even a Virual RC API now!). Each of the “rooms” in the structured corner of the map has a persistant video call link in it, and you can see which other participants are in the room on a call. Since I’ve started my new job I’ve missed being able to see who is already in a zoom call before deciding to join an optional meeting! Participants are encouaged to create a desk, where you can set an emoji status and leave a short note describing what you are up to that day. There are also audio rooms, which are great for more informal, or group conversations, and I’m only realizing now how much easier it is to start a call when you don’t have to open a new link and wait for a video call service to connect. A lot of the day-to-day conversation at remote RC happens via text on Zulip (though this was the case for in-person RC also). It’s easy to reach out for help or to look for a pairing buddy on Zuip, and one of the most active streams (like channels in other chat apps) is Checkins where in-batch Recursers regularly post about what they’re working on.

Hanging out on top of the RC logo in Virutal RC

I came into RC with a few goals ranging form ambitious (real-time NYC subway map or self-hosted social network prototype), to the kinds of things I often made when I had a few hours to spare (SwiftUI proof of concept checklist app). I had spent the weeks leading up to my batch doing what I wanted to think of as a mini RC-like experience re-learning Swift and using SwiftUI for the first time. I had hoped to finish the 100 Days of SwiftUI tutorials in time for day 1 at RC but ended up being about a week too slow.

I was surprised at the 9-5ness of the events and chatter in remote RC, though one of the most exciting things about doing a remote batch was being able to meet and collaborate with people from around the world (I counted at least 8 timezones in my batch), most events occurred bewteen 9am and 5pm Eastern Time. I would guess that this means that the average RC participant is spending less time in RC, than when in an in-person batch, which depending on your goals and ability to focus when something potentially more fun is going on, could be a benefit or a drawback to the remote RC experience.

After speeding through the SwiftUI tutorials in my first week (and getting kind of tired of working in an unfamiliar language), I decided to spend most of my time working on my real-time NYC subway map project that I had first prototyped in early 2020. I eventually was able to get a mostly working prototype of the full system, coincidentally on almost the exact day that the MTA released their own version. I thought that the subway map would be one of a few medium sized projects that I would be able to spend 2 to 3 weeks on during my RC time, but I had already spent more than half of my 12 week batch on it. I had done a few other one-off projects like an abstract “video game” based on the 10 Print pattern and a few explorations into techincal topics like the browser JavaScript canvas API, but I decided that I should try to find a real medium-sized project for my last few weeks at RC.

It’s hard to draw crisp lines on the JS canvas!

I felt like I had mostly been working though a big challenge with technologies that were pretty familiar so far (JavaScript and the Leaflet.js mapping library), so I wanted to find a project that I could use to learn something new, but not spend all my time on implementation details like I had been doing learning SwiftUI. I had created a static website on the doodles subdomain of my website that I wanted to keep filling out, so a useful constraint was to make something on the web that wouldn’t need a backend (the doodles website is static and built with Eleventy , as are this website, and most of the doodles)). In the last year at my job, and through collaborating with other RC participants during my batch I had gotten pretty quick with prototyping things with React, but I hadn’t used either React Hooks, or TypeScript for anything. I decided to try learning both and created a web-based Crossword Puzzle composer.

A crossword puzzle with multiple kids of symmetries

The crossword puzzle app was a fun mix of small challenges and interesting design decisions. One of the first things that stood out to me was that I had never really thought about the algorithm that is used to number crossword puzzle answers on the board, the kind of thing that is immediately understandable, but more complex that you would expect it to be when represented with a programming language. React Hooks were immediately very useful since most of the components for the app needed a sprinkling of state, but not too much, and TypeScript was helpful, especially in the context of RC because I paired with a few people on certain self-contained parts of the app like the symmetry for square toggling, and the defined types help people quickly pick up what the code we were writing would do. My favorite part of the crossword puzzle app is that to store a composed puzzle without a backend, I encode the whole puzzle state into the URL (here is a small RC themed puzzle). I even discovered seemingly documented nowhere else bug in iMessage when prototyping this feature. When I first sent a friend a link to a puzzle as a test, iMessage wouldn’t parse it as a link, so they just got a wall of what looked like random characters. I looked around at other long tracking links I’d received in my emails and noticed they frequently had hyphen characters every so often. I added hyphens to my urls and all of a sudden iMessage could parse them as links!

A failed attempt to send a crossword

During the second half of my RC batch I also interviewed with a lot of companies. To help with this I attended a daily group for pairing on software job interview style questions. Coming back to these types of problems a few times a week was really helpful in being able to think through them quickly and I noticed a lot of progress in my ability to complete these kinds of challenges the more I practiced and interviewed.

When I first applied to RC I had been feeling nervous about committing such a long period of time to not working before (hopefully!) starting a new job, I considered just doing a 1 week mini batch, or a 6 week half-batch, but the 12 weeks went by way more quikly than I had anticipated, and I was fortunately able to hit my idealistic goal of starting a new job in the new calendar year! The 12 weeks helped me feel free to dive into weird corners of the problems I wanted to work on like drawing parallel lines on a map (lines were a big theme in my projects), without worrying that it would keep me from “finishing” my projects (author’s note: I still didn’t “finish” most of them). In the spirit of “never graduate” I hope to someday participate in another batch at the physical RC space, but in the meantime, anyone who like me has been sitting on the fence on applying to RC for years should take the opportunity to participate remotely and start the programming project you’ve always been meaning to get to!

An early attempt to draw MTA track geography data as parallel tracks on a map

Making a Real-Time NYC Subway Map with Real Weird NYC Subway Data

Patrick Weaver — Wed, 21 Oct 2020 00:00:00 +0000

Earlier this week the NYC MTA released a new digital-first map. The Curbed exclusive that announced its release accurately portrays it as a strange child of both the 1972 map design by Massimo Vignelli and the current “paper” map. One feature of the new map (though it’s harder than it should be to notice at first) is real-time visualizations of each train in the system.

I’ve been working on a similar concept, starting in February 2020, on which progress stalled once I stopped riding the subway regularly in March. But, when I started my batch at Recurse Center I decided to pick up the project again. My inspiration for the map was the large TV screens that the MTA has installed in stations over the last few years, which frustratingly display the “paper” version of the map.

Subway station TV (This is not a good photo, but it’s hard to take a picture of a screen underground)

Over the past few weeks at RC the subway map has been my main focus, which is longer than I expected the project to take (and though I have a prototype, I wouldn’t say I’m close to “done”). A big factor in the time the project has taken is some of the quirks in working with MTA data, which based on some of the bugs I’ve seen in the "official" version I’d say the team working on that version had to grapple with as well. I’m not sure I will ever finish this project to a state that would look great on a subway station tv, or be useful, but I do want to point out The Weekendest by Sunny Ng (who also made goodservice.io), which is a great take on the concept, and handles some of the challenges of this kind of project much better than the MTA map does.

For a long time the NYC Subway was almost completely lacking in real-time data. For many years the only line that had even countdown clocks in stations was the L, which seems to be the line the MTA tries out new technology on, likely because it never shares tracks with any other line. Over the last 5 years the MTA has slowly installed countdown clocks in every station, and made the data that powers the countdown clocks available on their website, in apps, and as data online.

Inspired and frustrated by the “paper” maps on the tv screens, I first became interested in working with MTA data in 2018, but I initially started working with bus data, I think for two reasons: the first was because at the time the API key for working with bus data was easier to obtain than for subway data, the second because my morning commute at the time usually started with the bus (The MTA, earlier this year, has fortunately updated the system for obtaining an API key for subway data). I made a small prototype of an iPhone app that would show real-time bus data, but got distracted by learning the Swift programming language and abandoned the project without building functionality beyond what the MTA already provided on their bustime website.

That MTA Bustime website was the other inspiration for what became my map idea. Though you could only view one route at a time (and the functionality is not available on phone-size devices), the Bustime website showed, in addition to countdowns for each stop, the physical location of each bus on a map. This leads us to the first weird thing about working with NYC Subway data, unlike real-time bus data, the subway data does not contain the latitude and longitude data for each train that would make it easy to show them on a map.

Real-Time Transit Data

Transit data for most transit systems is available in formats called GTFS (General Transit Feed Specification) and GTFS Realtime, which were developed by Google (makes you wonder what the “G” originally stood for), but are now widely used. A GTFS file is, “a collection of at least six, and up to 13 CSV files (with extension .txt) contained within a .zip file.” and “The GTFS Realtime data exchange format is based on Protocol Buffers” (which are “Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data”).

The GTFS Realtime feeds are available through 9 different API endpoints from the MTA. These 9 endpoints roughly correspond to the line colors, with Shuttles combined with trains they share tracks or stations with, and the 1/2/3 and 4/5/6 sharing one endpoint. This list of separate endpoints is another challenge with working with the entirety of the MTA data.

I have exclusively worked with the NYC MTA’s GTFS Realtime feeds through the npm module maintained by Google. It is very possible that some of the challenges I’ve encountered are due to trying to squeeze the “extensible mechanism for serializing structured data” into JSON. Each API response is mostly composed of an array of "Feed Entity" objects like these, but there are a few quirks to working with this data (some maybe because of the JSON conversion).

Each item in the array has an id property, but unfortunately these ids do not consistently refer to the same train between each update, it’s best to ignore it.
The array consists of pairs of objects that either have a tripUpdate property or a vehicle property. Each of these have a sub-property called tripId that allows you to unite the pairs, but there are also some that don’t have a corresponding item (usually these represent trips that recently ended or haven’t yet begun).
The data mixes together HH:MM:SS timestamps for data about when a train’s trip started, and Unix timestamps for data about the current time (according to the API) and when a train will arrive at a station (the API provides both arrival and departure times but as far as I have seen they are always identical).
tripUpdate items show information about the stops a train will make in the future (stopTimeUpdates) and vehicle items show information about the current status of the train, but the first stopTimeUpdate is usually in the past.

I had never heard of Protocol Buffers before starting this project, so I was excited to learn more about them while reading through Designing Data Intensive Applications with fellow Recursers. In the book Martin Kleppmann notes that a, "curious detail of Protocol Buffers is that it does not have a list or array datatype, but instead has a repeated marker for fields (which is a third option alongside required and optional)." This could be the reason for the strange organization of the tripUpdate and vehicle properties.

Calculating Train Locations

The subway real-time API doesn’t have latitude and longitude data because it is designed to feed data to countdown clock style applications that show when the train will be at a specific station. One of the earliest features that I built into the real-time map was a way to translate these station-by-station countdown clocks into an approximation of the location of each train. My first attempt at this was to just show a list of stations and display an icon for a train between the names of the station it had been at previously and the station it was approaching.

A first prototype

The next step was plotting the stations on a map. To start off, as with the diagram version, I just placed each train at the midpoint between the station it was traveling from and the station it was traveling towards.

A goal I had for the project was not just to show real-time train locations, but to animate them as they moved around the map. To determine how long I should expect a train to take to travel between each station I logged updates from the MTA API for a few hours and noted both the average time for a pair of stations, and the longest time I had seen for the pair. I’m still experimenting a little bit with what values to use as the baseline, but from looking at the logged numbers there does seem to be an expected amount of time for most stations.

G: {
   G22: { N: { avg: 60, max: 71 }, S: null },
   G24: { N: { avg: 123, max: 180 }, S: { avg: 60, max: 106 } },
   G26: { N: { avg: 75, max: 90 }, S: { avg: 108, max: 180 } },
   G28: { N: { avg: 128, max: 180 }, S: { avg: 78, max: 90 } },
   G29: { N: { avg: 60, max: 76 }, S: { avg: 139, max: 180 } },
   G30: { N: { avg: 51, max: 74 }, S: { avg: 70, max: 90 } },
   G31: { N: { avg: 60, max: 90 }, S: { avg: 58, max: 69 } },
   G32: { N: { avg: 66, max: 87 }, S: { avg: 53, max: 84 } },
   G33: { N: { avg: 50, max: 66 }, S: { avg: 67, max: 84 } },
   G34: { N: { avg: 58, max: 90 }, S: { avg: 54, max: 66 } },
   G35: { N: { avg: 68, max: 86 }, S: { avg: 50, max: 81 } },
   G36: { N: { avg: 81, max: 161 }, S: { avg: 59, max: 71 } },
   A42: { N: { avg: 70, max: 177 }, S: { avg: 87, max: 157 } },
   F20: { N: { avg: 68, max: 90 }, S: { avg: 86, max: 165 } },
   F21: { N: { avg: 72, max: 120 }, S: { avg: 76, max: 120 } },
   F22: { N: { avg: 62, max: 90 }, S: { avg: 84, max: 120 } },
   F23: { N: { avg: 90, max: 120 }, S: { avg: 88, max: 150 } },
   F24: { N: { avg: 101, max: 120 }, S: { avg: 67, max: 84 } },
   F25: { N: { avg: 139, max: 180 }, S: { avg: 71, max: 90 } },
   F26: { N: { avg: 120, max: 120 }, S: { avg: 109, max: 150 } },
   F27: { N: null, S: { avg: 81, max: 120 } },
 }

Average and max wait times in seconds for stops on the G line.

Trip Id	Trip Start Time	Trip Date	Route	Stop1 Arrival	Stop1 Id	Stop2 Arrival	Stop2 Id	Seconds
073476_G..N	12:14:46	20200818	G	1597768631	G33N	1597768692	G32N	61
074600_G..N	12:26:00	20200818	G	1597769103	G33N	1597769172	G32N	69
075000_G..N	12:30:00	20200818	G	1597769531	G33N	1597769596	G32N	65
076501_G..N	12:45:01	20200818	G	1597770333	G33N	1597770396	G32N	63
077700_G..N	12:57:00	20200818	G	1597771043	G33N	1597771104	G32N	61
078403_G..N	13:04:02	20200818	G	1597771443	G33N	1597771524	G32N	81
079600_G..N	13:16:00	20200818	G	1597772051	G33N	1597772112	G32N	61
080550_G..N	13:25:30	20200818	G	1597772711	G33N	1597772776	G32N	65

Logged travel Times between the Bedford - Nostrand stop and the Myrtle - Willoughby stop on the G train

Secret Stations

One thing I discovered while logging updates from the MTA API was that it contained secret stations! The MTA provides a list of all of the stations in the system with data like latitude and longitude. Each station has an ID (see Stop1 and Stop2 id in the diagram above and called "GTFS Stop ID" in the list). The stop IDs are a letter and 2 numbers, with the letter often corresponding to the line it serves (or used to historically), and the numbers mostly occurring in sequence. but some trains would have planned "stops" at stations that weren’t in the list! My best guess is that these stations are something station-like in the MTA’s infrastructure, which usually appear near the end of a line.

"H17" is between Howard Beach/JFK Airport and Broad Channel on the A and Rockaway Shuttle and is likely where the Shuttle trains go to turn around.
"H19" is before the Broad Channel stop on the A and Rockaway Shuttle and may also be related to Shuttle turnaround?
"H18" and "H05" are between Broad Channel and Beach 67 St on the A and Rockaway Shuttle, which may have to do with tracks that run between Beach 67 and Beach 90 Sts.
"A29" is between Penn Station and 14th St on the A, C, and E, which is strange because the C and E stop at 23rd St., which is also between those stations, but has the ID "A30".
"A39" is between Fulton St. and High St. on the A and C, which might have something to do with the track stubs on the Brooklyn side (one of which is the NYC Transit museum).
"A58" is between Grant Av. and 80th St. on the A, which, is where the A train emerges from a tunnel to run on elevated tracks.
"A62" is between Rockaway Blvd. and 104th St. on the A and probably has something to do with the merging between the 3 versions of the A at Rockaway Blvd.
"R60" is between Queensboro Plaza and Lexington Ave/59th St. on the N, R, and W. My guess is that this has something to do with the N/W and R tracks merging before going into a tunnel.
"R65" is between the Whitehall St. and Court St. stops on the R, and could also be related to the same track stubs as "A39".
"B24" is between Bay 50th St and Coney Island on the D, and is probably the MTA Coney Island Yard.

Drawing the Static Map

I wasn’t quite satisfied with the angular paths that drawing lines directly between stations created, and I was fortunately able to find Sunny Ng’s advice on extracting shape arrays from the non real-time GTFS MTA data. Using these shape arrays I could draw route maps with smooth curves, and even animate trains along those curves. But one of the things that makes a subway map a subway map is seeing lines that run on the same tracks as parallel lines. I also wanted to double these lines and show Northbound and Southbound trains on separate tracks (something that the new MTA map fails to do).

After trying to approach the parallel lines problem geometrically I was pointed in the right direction by a fellow RC participant and was able to draw great looking lines by treating the Latitude/Longitude points in the shape arrays as vectors (More on this in my interactive slides on this problem and more on the challenge of drawing nice train lines from the Transit app).

Similar sections of my map and the MTA map

A Prototype

My map project is available on GitHub, bugs and all. I did most of the work on the map using only the G train API endpoint. This was a helpful limitation when I was first experimenting with what was possible using the data, but may have led to more bugs because of the slight differences in the data available for each set of lines.

A common complaint about the new official real-time map is that it seems to use as much computer power as it can. My map isn’t much better because it is doing all of the geographic calculations in the user’s browser, my guess is that the MTA’s map is also. One update I might take on over the next week and a half as my time at RC winds down is moving these calculations to a server, and sending only train position changes to the map visualization. This may also help with the bug my current prototype exhibits where leaving and coming back to the tab a few minutes later will cause trains to fly around the map without regard for the lines or stations.

The MTA data is weird because it’s created by a system that could never have anticipated the kind of systems that now try to contain it. Overall, working with and working around the weirdness in the data has been challenging, but a great reminder that the most interesting real-world problems are often hard to jam into our brittle computer systems, and that’s probably a good thing.

Week 1 at Recurse Center: Two Approaches to Learning

Patrick Weaver — Sun, 16 Aug 2020 22:22:13 +0000

I just finished my first week as part of the Fall 1 '20 batch at Recurse Center. I tried to split my time between being social and building skills that I want to use for future projects. The social aspect of RC is interesting because this batch is being conducted remotely.

My batch at RC started exactly 5 months after my last day in an office, and something I had been thinking about over the last few months, is that I haven’t met anyone new since early March when NYC shut down because of the pandemic. It’s been very refreshing to meet people again, even if it’s via video calls and chat. RC has created an online representation of their physical space we call "Virtual RC". Each of us have avatars we can move around the space, and there are permanent links to video call rooms that we can pop into for events or impromptu conversations. The Virtual RC experience pairs well with group chat organized into streams on different topics.

My avatar hanging out in the "Shannon" room at Virtual RC

Other than social events I spent my first week at RC focused on two projects. The first was going through the 100 Days of SwiftUI tutorials (at many more than one a day), which I had started before the batch. The second was reviewing the real time NYC Subway map project I first started in January.

I wish I had been able to finish the SwiftUI tutorials before the batch (my original goal), because it didn’t feel as useful to share what I was working on with other RC participants as I was re-building apps that someone else had designed, but I finished the last app on Friday and I’m excited to be able to use SwiftUI to quickly prototype projects for my phone.

When I first started the subway map project in early 2020 I was excited about finishing it quickly, but in March when I also stopped taking the subway I lost enthusiasm for the project. Starting at RC has helped rekindle the excitement I had for it, I think partly because even though the batch is remote and there are participants from around the world, there is also a strong NYC contingent.

An interesting aspect of building a map from MTA data is that the data structures that the MTA publishes are designed for building countdown clock style apps. I would imagine that the MTA calculates this per-station data from the location of each train, but in order to build an app that is more focused on location than time, I need to reverse engineer the location data from the distance in time each train from the next station. There are also other weird quirks, like train trip start times being provided in "HH:MM:SS" format, but station arrival times being provided in Unix time format.

{
  "id": "000001G",
  "tripUpdate": {
    "trip": {
      "tripId": "126481_G..N",
      "startTime": "21:04:49",
      "startDate": "20200815",
      "routeId": "G"
    },
    "stopTimeUpdate": [
      {
        "arrival": { "time": "1597541836" },
        "departure": { "time": "1597541836" },
        "stopId": "G28N"
      },
      {
        "arrival": { "time": "1597541904" },
        "departure": { "time": "1597541904" },
        "stopId": "G26N"
      }

      /* More stations below in real data */
    ]
  }
}

Example MTA Data

Splitting my programming time between learning SwiftUI and trying to wrap my head around the subway map again has felt like two different approaches to learning. In the past I have often jumped into building my own projects with a new tool or technology as soon as I feel like I had learned enough to do so, but I’ve often realized later that I spent a lot of time figuring out answers that I might have gotten to more quickly by completing a more thorough overview before starting.

The subway map is one example of that type of project, but for good reason. It’s a project that when I started, I wasn’t sure was possible. It was inspired by a similar (but much simpler) map in the underground MUNI stations in San Francisco. The MUNI maps I remember from 15 years ago only showed trains in "the tunnel," which although many lines run through, has a single trunk, that lines branch out from after going above ground (newer maps as seen below seem to show the whole system). Since these MUNI maps had existed since at least the early 00s I figured if one hadn’t been made (I’ve since found NYC maps that have been made) for NY there must be a technical reason (it may just be that because until relatively recently per station data wasn’t available in NYC).

@MBTA @sfmta_muni @d_tribe @universalhub closer zoom. Incredibly useful. pic.twitter.com/jqHuGyZrJR
— Ari Ofsevit (@ofsevit) April 13, 2016

A photo of the real time MUNI map in a tweet by @ofsevit

In order to gauge the feasibility of the project I wanted to get started quickly, throwing a map together and adding some markers for stations and individual trains at fixed times. Because I had limited experience working with transit data (which is provided in a very complex format), or maps, this left me with both the sense that the project is possible, and a big spaghetti code mess.

The first prototype of the subway map app

I started looking into SwiftUI because I’m interested in making my own home screen widgets once I update my phone to iOS 14, and these widgets are built using the framework. Knowing I would have time to experiment during my batch at RC, and wanting to avoid my usual new tool mess phenomenon, I prioritized giving myself a good understanding of how SwiftUI works, what it can and can’t do, and a general review of Swift. Once my batch started I found it more and more difficult to keep going through the tutorials instead of jumping into an original idea, but throughout the week, spending time figuring out what I was even thinking reviewing the code for the subway map project re-motivated me to get through it.

In the first week at RC the return to a daily routine has also put me back in the frame of mind I spent a lot of my last job in, where I would come up with ideas for about 3 apps a day that would solve tiny problems. One example is, after a particular session of an RC event for pair programming on software job interview style questions where we selected a problem that ended up being very difficult in the language we chose to work in, I thought that maybe I should create a mini app for the group where we could rate problems for each other. A lot of these ideas that are generated through trying to solve small problems in my routine end up being apps that organize data into text boxes, which are often not very interesting, so I’m conflicted on whether or not to spend time following these threads that will likely continue to appear.

A few other things that are slightly more interesting, or at least broadly useful that I want to get done during my time at RC are:

The real time subway map
A simple checklist app with a home screen widget made with SwiftUI
A tool to convert an export from a Blogger blog to a static site generator
A prototype of a self-hosted social network profile that can interface with other people’s self-hosted profiles
A second iteration of my handwritten email sending project
Something that takes ideas similar to Dynamicland (or the Living Room project at RC) into the remote world we’re living in.

Overall I’m very excited to be spending time at RC, and I hope that it gives me time to explore ideas with weird corners and not settle for solving small simple problems quickly.

How to Download an Image from a Google Doc

Patrick Weaver — Thu, 16 Jul 2020 00:00:00 +0000

For some reason Google hasn’t built in a way for you to download images in Google docs! There are workarounds to get those image files like using Google Keep, or downloading your whole doc as a .zip file, but these have always felt like too many steps.

And this is something that people really want!

As Steve Krouse points out here, it is possible to get the real URL of the image in your doc (but confusingly, as soon as you click on the image to select it the URL becomes obfuscated!).

I also noticed the URLs in the source, and decided to make an easy way to access it. The one trick ended up being, because clicking on the image made it disappear, finding a way to tell the code which image you wanted!

I looked through some JavaScript documentation and realized I could use the mouseover event to detect when someone was hovering over the image. Unfortunately this means that it won’t work on a touchscreen device, but I’m guessing that most people who want to download an image are on a traditional computer.

How does it work?

I needed a way to run my code on any Google Doc, there’s probably a way to make Google Doc or Chrome extension to do this, but since I was asking people to run code in their potentially private docs I wanted to make the code as short and open source as possible.

I decided that the best way to do this was a bookmarklet. If you’re unfamiliar with bookmarklets, they’re bookmarks (usually placed in your bookmarks toolbar (Cmd-Shift-B to toggle this on and off on a Mac), that instead of navigating to a webpage, run JavaScript when you click them.

Great! Tell me how to do it!

To get started you’ll have to "install" the bookmarklet. This is easy to do, and just means dragging a button into your bookmarks toolbar. I’ve hosted it here.

The instructions are simple!

Drag one of the bookmarklets (see the embed above) to your bookmarks toolbar. The text displayed is what will be show on the toolbar:
Then, when you’re on a Google Doc, click the bookmarklet, then hover over an image embedded in the doc. Depending on your browser settings it will either download immediately, or open the actual image in a new tab.

Great my problems are solved forever!

No guarantees that this will work long term, a quick look at the source code for any Google Doc will show that they’re very complex! I wouldn’t be surprised if Google changes the way these URLs work in the future, but this tool has worked for 6 months so maybe not!

Long term I hope that they build in a way for people to download their images, but for now I hope this is helpful!