FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

Your feed appears to be encoded as "utf-8", but your server is reporting "US-ASCII" [help]
line 27, column 0: content should not contain loading attribute (11 occurrences) [help]
```
    <img loading="auto" src="./img/cover.jpeg" alt="Book Cover">
```

line 343, column 0: content should not contain iframe tag [help]

<div class="responsive-container"><iframe width="560" height="315" src="http ...

line 351, column 0: Non-html tag: picture (25 occurrences) [help]
```
    <picture>
```

line 906, column 0: content should not contain script tag (2 occurrences) [help]

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Sublime Tron Legacy ...

line 1023, column 0: style attribute contains potentially dangerous content: grid-template-columns (2 occurrences) [help]
```
<div style='display: grid; grid-template-columns: 1fr 1fr; justify-items: ce ...
```
line 1025, column 0: content should not contain widht attribute [help]
```
    <img widht='199.5' height='250' src='./aws.jpg'/>
```
line 1488, column 0: content should not contain controls attribute (5 occurrences) [help]
```
  <video controls width="100%" preload="metadata">
```
line 1488, column 0: content should not contain preload attribute (5 occurrences) [help]
```
  <video controls width="100%" preload="metadata">
```

Source: https://bret.io/feed.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>bret.io</title>
<id>https://bret.io/feed.xml</id>
<updated>2024-02-13T18:24:49.707Z</updated>
<link rel="self" type="application/atom+xml" href="https://bret.io/feed.xml"/>
<link rel="alternate" type="application/json" href="https://bret.io/feed.json"/>
<link rel="alternate" type="text/html" href="https://bret.io"/>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<generator uri="https://github.com/bcomnes/jsonfeed-to-atom#readme" version="1.2.5">jsonfeed-to-atom</generator>
<rights>© 2024 Bret Comnes</rights>
<subtitle>A running log of announcements, projects and accomplishments.</subtitle>
<entry>
<id>https://bret.io/blog/2024/the-art-of-doing-science-and-engineering/#2024-02-13T18:24:49.707Z</id>
<title>The Art of Doing Science and Engineering</title>
<updated>2024-02-13T18:24:49.707Z</updated>
<published>2024-02-13T18:24:49.707Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[<figure>
<a href="./img/cover.jpeg">
<img loading="auto" src="./img/cover.jpeg" alt="Book Cover">
</a>
<figcaption>The art of doing science and engineering : Learning to Learn by Richard W. Hamming</figcaption>
</figure>
Richard Hamming’s “The Art of Doing Science and Engineering” is a book capturing the lessons he taught in a course he gave at the U.S. Navy Postgraduate School in Monterey, CA.
He characterizes what he was trying to teach was “style” of thinking in science and engineering.
Having a physics degree myself, and also finding myself periodically ruminating on the agony of professional software development and hoping to find some overlap between my professional field and Hamming’s life experience, I gave it a read.
The book is filled with nuggets of wisdom and illustrates a career of move-the-needle science and engineering at Bell Labs.
I didn’t personally find much value in many of the algebraic walk-through’s of various topics like information theory, but learning about how Hamming discovered error correcting codes definitely was interesting and worth a read.
The highlight of book comes in the second half where he includes interesting stories, analogies and observations on nearly every page. Below are my highlights I pulled while reading.
<h2 id="on-what-makes-good-design" tabindex="-1">On What Makes Good Design</h2>
<blockquote>
That brings up another point, which is now well recognized in software for computers but which applies to hardware too. Things change so fast that part of the system design problem is that the system will be constantly upgraded in ways you do not now know in any detail! Flexibility must be part of the modern design of things and processes. Flexibility built into the design means not only will you be better able to handle the changes which will come after installation, but it also contributes to your own work as the small changes which inevitably arise both in the later stages of design and in the field installation of the system…
Thus rule two:
Part of systems engineering design is to prepare for changes so they can be gracefully made and still not degrade the other parts.
– p.367
</blockquote>
This quote is my favorite out of the entire book.
It feels like a constant fight in software engineering between the impulse to lock down runtime versions, specific dependency versions, and other environmental factors versus developing software in such a way that accommodates wide variance in all of these different component factors.
Both approaches argue reliability and flexibility, however which approach actually tests for it?
In my experience, the tighter the runtime dependency specifications, the faster fragility spreads, and it’s satisfying to hear Hamming’s experience echo this observation. Sadly though, his observation that those writing software will universally understand this simply hasn’t held up.
<blockquote>
Good design protects you from the need for too many highly accurate components in the system. But such design principals are still, to this date, ill understood and need to be researched extensively. Not that good designers do not understand this intuitively, merely it is not easily incorporated into the design methods you were thought in school.
Good minds are still need in spite of all the computing tools we have developed. The best mind will be the one who gets the principle into the design methods taught so it will be automatically available for lesser minds!.
– p.268
</blockquote>
Here Hamming is describing H.S. Black’s feedback circuit’s tolerance for low accuracy components as what constitutes good design. I agree! Technology that works at any scale, made out of commodity parts with minimal runtime requirements tends to be what is most useful across the longest amount of time.
<h2 id="on-committees" tabindex="-1">On Committees</h2>
<blockquote>
Committee decisions, which tend to diffuse responsibility, are seldom the best in practice—most of the time they represent a compromise which has none of the virtues of any path and tends to end in mediocrity.
– p.274
</blockquote>
I appreciated his observations on committees, and their tendency to launder responsibility.
They serve a purpose, but its important to understand their nature.
<h2 id="on-data-and-observation" tabindex="-1">On Data and Observation</h2>
<blockquote>
The Hawthorne effect strongly suggests the proper teaching method will always to be in a state of experimental change, and it hardly matters just what is done; all that matters is both the professor and the students believe in the change.
– p.288
</blockquote>
<blockquote>
It has been my experience, as well as the experience of many others who have looked, that data is generally much less accurate than it is advertised to be. This is not a trivial point—we depend on initial data for many decisions, as well as for the input data for simulations which result in decisions.
– p.345
</blockquote>
<blockquote>
Averages are meaningful for homogeneous groups (homogeneous with respect to the actions that may later be taken), but for diverse groups averages are often meaningless. As earlier remarked, the average adult has one breast and one testicle, but that does not represent the average person in our society.
– p.356
</blockquote>
<blockquote>
You may think the title means that if you measure accurately you will get an accurate measurement, and if not then not, but it refers to a much more subtle thing—the way you choose to measure things controls to a large extent what happens. I repeat the story Eddington told about the fishermen who went fishing with a net. They examined the size of the fish they caught and concluded there was a minimum size to the fish in the sea. The instrument you use clearly affects what you see.
– p.373
</blockquote>
Intuitively I think many people who attempt to measure anything understand that their approach reflects in the results to some degree.
I hadn’t heard of the <a href="https://en.wikipedia.org/wiki/Hawthorne_effect">Hawthorne effect</a> before, but intuitively it makes sense.
People with an idea on how to improve something implement their idea and it works, because they want it to work and allow the effects to be fully effective.
Someone else is prescribed this idea or brought into the fold where the idea is implemented and the benefits of the idea evaporate.
I’ve long suspected that in the context of professional software development, where highly unscrutinized benchmarks and soft data are the norm, people start with an opinion or theory and work back to data that supports it.
Could it just be that people need to believe that working in a certain way is necessary for them to work optimally? Could it be “data” is often just a work function used to out maneuver competing ideas?
Anyway, just another thing to factor for when data is plopped in your lap.
<h2 id="on-theory" tabindex="-1">On Theory</h2>
<blockquote>
Moral: there need not be a unique form of a theory to account for a body of observations; instead, two rather different-looking theories can agree on all the predicted details. You cannot go from a body of data to a unique theory! I noted this in the last chapter.
–p.314
</blockquote>
<blockquote>
Heisenberg derived the uncertainty principle that conjugate variables, meaning Fourier transforms, obeyed a condition in which the product of the uncertainties of the two had to exceed a fixed number, involving Planck’s constant. I earlier commented, Chapter 17, this is a theorem in Fourier transforms-any linear theory must have a corresponding uncertainty principle, but among physicists it is still widely regarded as a physical effect from nature rather than a mathematical effect of the model.
–p.316
</blockquote>
I appreciate Hamming suggesting that some of our understanding of physical reality could be a byproduct of the model being used to describe it.
It’s not exactly examined closely in undergraduate or graduate quantum mechanics, and I find it interesting Hamming, who’s clearly highly intuitive with modeling, also raises this question.
<h2 id="predictions" tabindex="-1">Predictions</h2>
<blockquote>
Let me now turn to predictions of the immediate future. It is fairly clear that in time “drop lines” from the street to the house (they may actually be buried, but will probably still be called “drop lines”) will be fiber optics. Once a fiber-optic wire is installed, then potentially you have available almost all the information you could possibly want, including TV and radio, and possibly newspaper articles selected according to your interest profile (you pay the printing bill which occurs in your own house). There would be no need for separate information channels most of the time. At your end of the fiber there are one or more digital filters. Which channel you want, the phone, radio, or TV, can be selected by you much as you do now, and the channel is determined by the numbers put into the digital filter-thus the same filter can be multipurpose, if you wish. You will need one filter for each channel you wish to use at the same time (though it is possible a single time-sharing filter would be available) and each filter would be of the same standard design. Alternately, the filters may come with the particular equipment you buy.
– p.284-285
</blockquote>
Here Hamming is predicting the internet. He got very close, and it’s interesting to think that these signals would all just be piped to your house in a bundle you you pay for a filter to unlock access to the ones you want. Hey Cable TV worked that for a long time!
<h2 id="on-leadership" tabindex="-1">On Leadership</h2>
<blockquote>
But a lot of evidence on what enabled people to make big contributions points to the conclusion that a famous prof was a terrible lecturer and the students had to work hard to learn it for themselves! I again suggest a rule:
What you learn from others you can use to follow;
What you learn for yourself you can use to lead.
– p.292
</blockquote>
Learn by doing, not by following.
<blockquote>
What you did to become successful is likely to be counterproductive when applied at a later date.
– p.342
</blockquote>
It’s easy to blame changing trends in software development for the disgustingly short half-life of knowledge regarding development patterns and tools, but I think it’s probably just the nature of knowledge based work.
Operating by yourself may be effective and work well, but its not a recipe for success at any given moment in time.
<blockquote>
A man was examining the construction of a cathedral. He asked a stonemason what he was doing chipping the stones, and the mason replied, “I am making stones.” He asked a stone carver what he was doing; “I am carving a gargoyle.” And so it went; each person said in detail what they were doing. Finally he came to an old woman who was sweeping the ground. She said, “I am helping build a cathedral.”
If, on the average campus, you asked a sample of professors what they were going to do in the next class hour, you would hear they were going to “teach partial fractions,” “show how to find the moments of a normal distribution,” “explain Young’s modulus and how to measure it,” etc. I doubt you would often hear a professor say, “I am going to educate the students and prepare them for their future careers.”
This myopic view is the chief characteristic of a bureaucrat. To rise to the top you should have the larger view—at least when you get there.
– p.360
</blockquote>
Software bureaucrats aplenty. Really easy to fall into this role.
<blockquote>
I must come to the topic of “selling” new ideas. You must master three things to do this (Chapter 5):
<ol>
<li>Giving formal presentations,</li>
<li>Producing written reports, and</li>
<li>Mastering the art of informal presentations as they happen to occur.</li>
</ol>
All three are essential—you must learn to sell your ideas, not by propaganda, but by force of clear presentation. I am sorry to have to point this out; many scientists and others think good ideas will win out automatically and need not be carefully presented. They are wrong;
– p.396
</blockquote>
One thing I regret over the last 10 years of my career is not writing down more insights I have learned through experience.
Ideas simply don’t transmit if they aren’t written down or put into some consumable format like video or audio.
Nearly every annoying tool or developer trend you are forced to use is in play because it communicated the idea through blogs, videos and conference talks.
And those who watched echoed these messages.
<h2 id="on-experts" tabindex="-1">On Experts</h2>
<blockquote>
An expert is one who knows everything about nothing; a generalist knows nothing about everything.
In an argument between a specialist and a generalist, the expert usually wins by simply (1) using unintelligible jargon, and (2) citing their specialist results, which are often completely irrelevant to the discussion. The expert is, therefore, a potent factor to be reckoned with in our society. Since experts both are necessary and also at times do great harm in blocking significant progress, they need to be examined closely. All too often the expert misunderstands the problem at hand, but the generalist cannot carry though their side to completion. The person who thinks they understand the problem and does not is usually more of a curse (blockage) than the person who knows they do not understand the problem.
– p.333
</blockquote>
Understand when you are generalist and a specialist.
<blockquote>
Experts, in looking at something new, always bring their expertise with them, as well as their particular way of looking at things. Whatever does not fit into their frame of reference is dismissed, not seen, or forced to fit into their beliefs. Thus really new ideas seldom arise from the experts in the field. You cannot blame them too much, since it is more economical to try the old, successful ways before trying to find new ways of looking and thinking.
If an expert says something can be done he is probably correct, but if he says it is impossible then consider getting another opinion.
– p.336
</blockquote>
Anyone wading into a technical field will encounter experts at every turn.
They have valuable information, but they are also going to give you dated, myopic advice (gatekeeping?).
I like Hamming’s framing here and it reflects my experience when weighing expert opinion.
<blockquote>
In some respects the expert is the curse of our society, with their assurance they know everything, and without the decent humility to consider they might be wrong. Where the question looms so important, I suggested to you long ago to use in an argument, “What would you accept as evidence you are wrong?” Ask yourself regularly, “Why do I believe whatever I do?” Especially in the areas where you are so sure you know, the area of the paradigms of your field.
– p.340
</blockquote>
I love this exercise. It will also drive you crazy. Tread carefully.
<blockquote>
Systems engineering is indeed a fascinating profession, but one which is hard to practice. There is a great need for real systems engineers, as well as perhaps a greater need to get rid of those who merely talk a good story but cannot play the game effectively.
– p.372
</blockquote>
Controversial, harsh, but true.
<h2 id="the-binding" tabindex="-1">The Binding</h2>
The last thing I want to recognize is the beautiful cloth resin binding and quality printing of the book. Bravo Stripe Press for still producing beautiful artifacts at affordable pricing in the age of print on demand.
]]></content>
<link rel="alternate" href="https://bret.io/blog/2024/the-art-of-doing-science-and-engineering/"/>
</entry>
<entry>
<id>https://bret.io/blog/2024/async-neocities-bin/#2024-01-15T23:55:33.582Z</id>
<title>async-neocities has a bin</title>
<updated>2024-01-15T23:55:33.582Z</updated>
<published>2024-01-15T23:55:33.582Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[<a href="https://github.com/bcomnes/async-neocities"><code>async-neocities</code></a> <a href="https://github.com/bcomnes/async-neocities/releases/tag/v3.0.0">v3.0.0</a> is now available and introduces a CLI.
<pre><code class="hljs language-console">Usage: async-neocities [options]
Example: async-neocities --src public
--help, -h print help text
--src, -s The directory to deploy to neocities (default: "public")
--cleanup, -c Destructively clean up orphaned files on neocities
--protect, -p String to minimatch files which will never be cleaned up
--status Print auth status of current working directory
--print-key Print api-key status of current working directory
--clear-key Remove the currently associated API key
--force-auth Force re-authorization of current working directory
async-neocities (v3.0.0)
</code></pre>
When you run it, you will see something similar to this:
<pre><code class="hljs language-console">> async-neocities --src public
Found siteName in config: bret
API Key found for bret
Starting inspecting stage...
Finished inspecting stage.
Starting diffing stage...
Finished diffing stage.
Skipping applying stage.
Deployed to Neocities in 743ms:
Uploaded 0 files
Orphaned 0 files
Skipped 244 files
0 protected files
</code></pre>
<a href="https://github.com/bcomnes/async-neocities"><code>async-neocities</code></a> was previously available as a GitHub Action called <a href="https://github.com/marketplace/actions/deploy-to-neocities">deploy-to-neocities</a>. This Action API remains available, however the CLI offers a local-first workflow that was not previously offered.
<h2 id="local-first-deploys" tabindex="-1">Local First Deploys</h2>
Now that <code>async-neocities</code> is available as a CLI, you can easily configure it as an <code>npm</code> script and run it locally when you want to push changes to <a href="https://neocities.org">neocities</a> without relying on GitHub Actions.
It also works great in Actions with side benefit of deploys working exactly the same way in both local and remote environments.
Here is a quick example of that:
<ul>
<li>Install <code>async-neocities@^3.0.0</code> to your project’s <code>package.json</code>.</li>
<li>Set up a <code>package.json</code> deploy script:<pre><code class="hljs language-json"> "scripts": {
"build": "npm run clean && run-p build:*",
"build:tb": "top-bun",
"clean": "rm -rf public && mkdir -p public",
"deploy": "run-s build deploy:*",
"deploy:async-neocities": "async-neocities --src public --cleanup"
},
</code></pre>
</li>
<li>Run a deploy once locally to set up the <code>deploy-to-neocities.json</code> config file. Example config contents:<pre><code class="hljs language-json">{"siteName":"bret"}
</code></pre>
</li>
<li>Run deploys locally with <code>npm run deploy</code>.</li>
<li>Configure your CI to run <code>npm run deploy</code> and configure the token secret.<pre><code class="hljs language-yaml">name: Deploy to neociteis
on:
push:
branches:
- master
env:
node-version: 21
FORCE_COLOR: 2
concurrency: # prevent concurrent deploys doing starnge things
group: deploy-to-neocities
cancel-in-progress: true
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Create LFS file list
run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
- name: Restore LFS cache
uses: actions/cache@v3
id: lfs-cache
with:
path: .git/lfs
key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}-v1
- name: Git LFS Pull
run: git lfs pull
- name: Use Node.js
uses: actions/setup-node@v4
with:
node-version: ${{env.node-version}}
- run: npm i
- run: npm run deploy
env:
NEOCITIES_API_TOKEN: ${{ secrets.NEOCITIES_API_TOKEN }}
</code></pre>
</li>
</ul>
The <code>async-neocities</code> CLI re-uses the same ENV name as <code>deploy-to-neocities</code> action so migrating to the CLI requires no additional changes to the Actions environment secrets.
<h2 id="clis-vs-actions" tabindex="-1">CLIs vs Actions</h2>
This prompts some questions regarding when are CLIs and when are actions most appropriate. Lets compare the two:
<h3 id="clis" tabindex="-1">CLIs</h3>
<ul>
<li>Pro: Local deploys</li>
<li>Pro: Easily re-usable in CI as well</li>
<li>Con: Requires Node.js, but this is not a problem when already using Node.js</li>
</ul>
<h2 id="actions" tabindex="-1">Actions</h2>
<ul>
<li>Pro: Works great with Non-node ecosystems without requiring a <code>package.json</code> or <code>node_modules</code></li>
<li>Con: Only runs in CI environments</li>
</ul>
<h2 id="conclusion" tabindex="-1">Conclusion</h2>
In addition to the CLI, <code>async-neocities</code> migrates to full Node.js <code>esm</code> and internally enables <code>ts-in-js</code> though the types were far to dynamic to export full type support with the time I had available.
With respect to an implementation plan going forward regarding CLIs vs actions, I’ve summarized my thoughts below:
Implement core functionality as a re-usable library.
Exposing a CLI makes that library an interactive tool that provides a local first workflow and is equally useful in CI.
Exposing the library in an action further opens up the library to a wider language ecosystem which would otherwise ignore the library due to foreign ecosystem ergonomic overhead.
The action is simpler to implement than a CLI but the CLI offers a superior experience within the implemented language ecosystem.
]]></content>
<link rel="alternate" href="https://bret.io/blog/2024/async-neocities-bin/"/>
</entry>
<entry>
<id>https://bret.io/blog/2023/reorganized/#2023-12-02T20:49:41.713Z</id>
<title>Reorganized</title>
<updated>2023-12-02T20:49:41.713Z</updated>
<published>2023-12-02T20:49:41.713Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[Behold, a mildly redesigned and reorganized landing page:
<img src="./img/screenshot.png" alt="screenshot of the new website">
It’s still not great, but it should make it easier to keep it up to date going forward.
It has 3 sections:
<ul>
<li><a href="/#">Featured Projects</a>: important projects of note.</li>
<li><a href="/#recent-posts">Recent Posts</a>: now that this site has proper blog support, I can highlight recent posts on the landing page.</li>
<li><a href="/#open-source">Open Source</a>: interesting and notable projects that have found some use and that I still maintain. This section now includes a bunch of open source work from the past year that I’ve never had time to write about.</li>
<li><a href="/#past-projects">Past projects</a>: inactive projects that are not longer active, but still interesting enough to share.</li>
</ul>
I removed a bunch of older inactive projects and links and stashed them in a <a href="/projects/previous-projects/">project</a>.
Additionally, the edit button in the page footer now takes you to the correct page in GitHub for editing, so if you ever see a typo, feel free to send in a fix!
Finally, the <a href="/about/">about</a> page includes a live dump of the dependencies that were used to build the website.
]]></content>
<link rel="alternate" href="https://bret.io/blog/2023/reorganized/"/>
</entry>
<entry>
<id>https://bret.io/blog/2023/reintroducing-top-bun/#2023-11-23T15:14:54.910Z</id>
<title>Reintroducing top-bun 7 🥐</title>
<updated>2023-11-23T15:14:54.910Z</updated>
<published>2023-11-23T15:14:54.910Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[After some unexpected weekends of downtime looking after sick toddlers, I’m happy to re-introduce <a href="https://github.com/bcomnes/top-bun"><code>top-bun</code> v7</a>.
Re-introduce? Well, you may remember <a href="https://github.com/bcomnes/top-bun/tree/v6.0.0"><code>@siteup/cli</code></a>, a spiritual offshoot of <a href="https://github.com/ungoldman/sitedown"><code>sitedown</code></a>, the static site generator that turned a directory of markdown into a website.
<h2 id="tb-v7" tabindex="-1">Whats new with <code>top-bun</code> v7?</h2>
Let’s dive into the new feature, changes and additions in <code>top-bun</code> 7.
<h3 id="rename-to-top-bun" tabindex="-1">Rename to <code>top-bun</code></h3>
<code>@siteup/cli</code> is now <code>top-bun</code>.
As noted above, <code>@siteup/cli</code> was a name hack because I didn’t snag the bare <code>npm</code> name when it was available, and someone else had the genius idea of taking the same name. Hey it happens.
I described the project to Chat-GPT and it recommended the following gems:
<ul>
<li><code>quick-brick</code></li>
<li><code>web-erect</code></li>
</ul>
OK Chat-GPT, pretty good, I laughed, but I’m not naming this <code>web-erect</code>.
The kids have a recent obsession with <a href="https://wallaceandgromit.com">Wallace & Gromit</a> and we watched a lot of episodes while she was sick. Also I’ve really been enjoying 🥖 <a href="https://breadcrum.net">bread themes</a> so I decided to name it after Wallace & Gromit’s bakery “Top Bun” in their hit movie <a href="https://en.wikipedia.org/wiki/A_Matter_of_Loaf_and_Death">“A Matter of Loaf and Death”</a>.
<div class="responsive-container"><iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/zXBmZLmfQZ4?si=cGvWktfUU2xnqHrM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe></div>
<h3 id="a-docs-website" tabindex="-1">A Docs Website</h3>
<code>top-bun</code> now builds it’s own repo into a docs website. It’s slightly better than the GitHub README.md view, so go check it out! It even has a real domain name so you know its for real.
<ul>
<li>🌎 <a href="https://top-bun.org">top-bun.org</a></li>
</ul>
<figure>
<a href="./img/docs-site.png">
<picture>

<img loading="auto" src="./img/docs-site.png" alt="Screenshot of the docs site">
</picture>
</a>
<figcaption>top-bun builds itself into its own docs website in an act dubbed "dogfooding".</figcaption>
</figure>
<h3 id="css-bundling-is-now-handled-by-esbuild" tabindex="-1"><code>css</code> bundling is now handled by <code>esbuild</code></h3>
<code>esbuild</code> is an amazing tool. <code>postcss</code> is a useful tool, but its slow and hard to keep up with. In <code>top-bun</code>, <code>css</code> bundling is now handled by <a href="https://esbuild.github.io/content-types/#css"><code>esbuild</code></a>.
<code>css</code> bundling is now faster and less fragile, and still supports many of the same transforms that <code>siteup</code> had before. <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_nesting/Using_CSS_nesting">CSS nesting</a> is now supported in every modern browser so we don’t even need a transform for that. Some basic transforms and prefixes are auto-applied by setting a relatively modern browser target.
<code>esbuild</code> doesn’t support import chunking on css yet though, so each <code>css</code> entrypoint becomes its own bundle. If <code>esbuild</code> ever gets this optimization, so will <code>top-bun</code>. In the meantime, <code>global.css</code>, <code>style.css</code> and now <code>layout.style.css</code> give you ample room to generally optimize your scoped css loading by hand. It’s simpler and has less moving parts!
<figure>
<a href="./img/esbuild-css.png">
<picture>

<img loading="auto" src="./img/esbuild-css.png" alt="Screenshot of esbuild css docs">
</picture>
</a>
<figcaption>The esbuild css docs are worth a cruise through.</figcaption>
</figure>
<h3 id="multi-layout-support" tabindex="-1">Multi-layout support</h3>
You can now have more than one layout!
In prior releases, you could only have a single <code>root</code> layout that you customized on a per-page basis with variables.
Now you can have as many layouts as you need.
They can even nest.
Check out this example of a nested layout from this website. It’s named <code>article.layout.js</code> and imports the <code>root.layout.js</code>. It wraps the <code>children</code> and then passes the results to <code>root.layout.js</code>.
<pre><code class="hljs language-js">// article.layout.js
import { html } from 'uhtml-isomorphic'
import { sep } from 'node:path'
import { breadcrumb } from '../components/breadcrumb/index.js'
import defaultRootLayout from './root.layout.js'
export default function articleLayout (args) {
const { children, ...rest } = args
const vars = args.vars
const pathSegments = args.page.path.split(sep)
const wrappedChildren = html`
${breadcrumb({ pathSegments })}
<article class="article-layout h-entry" itemscope itemtype="http://schema.org/NewsArticle">
<header class="article-header">
<h1 class="p-name article-title" itemprop="headline">${vars.title}</h1>
<div class="metadata">
<address class="author-info" itemprop="author" itemscope itemtype="http://schema.org/Person">
${vars.authorImgUrl
? html`<img height="40" width="40" src="${vars.authorImgUrl}" alt="${vars.authorImgAlt}" class="u-photo" itemprop="image">`
: null
}
${vars.authorName && vars.authorUrl
? html`
<a href="${vars.authorUrl}" class="p-author h-card" itemprop="url">
<span itemprop="name">${vars.authorName}</span>
</a>`
: null
}
</address>
${vars.publishDate
? html`
<time class="published-date dt-published" itemprop="datePublished" datetime="${vars.publishDate}">
<a href="#" class="u-url">
${(new Date(vars.publishDate)).toLocaleString()}
</a>
</time>`
: null
}
${vars.updatedDate
? html`<time class="dt-updated" itemprop="dateModified" datetime="${vars.updatedDate}">Updated ${(new Date(vars.updatedDate)).toLocaleString()}</time>`
: null
}
</div>
</header>
<section class="e-content" itemprop="articleBody">
${typeof children === 'string'
? html([children])
: children /* Support both uhtml and string children. Optional. */
}
</section>
<!--
<footer>
Footer notes or related info here...
</footer>
-->
</article>
${breadcrumb({ pathSegments })}
`
return defaultRootLayout({ children: wrappedChildren, ...rest })
}
</code></pre>
<h3 id="layout-styles-and-js-bundles" tabindex="-1">Layout styles and js bundles</h3>
With multi-layout support, it made sense to introduce two more style and js bundle types:
<ul>
<li>Layout styles: styles that load on every page that uses a layout</li>
<li>Layout bundles: js bundles that load on every page that uses a layout</li>
</ul>
Prior the <code>global.css</code> and <code>global.client.js</code> bundles served this need.
<figure>
<a href="./img/layout-bundles.png">
<picture>

<img loading="auto" src="./img/layout-bundles.png" alt="Screenshot of layout bundles in an editor">
</picture>
</a>
<figcaption>Layouts introduce a new asset scope in the form of layout clients and style.</figcaption>
</figure>
<h3 id="layouts-and-global-assets-live-anywhere" tabindex="-1">Layouts and Global Assets live anywhere</h3>
Layouts, and <code>global.client.js</code>, etc used to have to live at the root of the project <code>src</code> directory. This made it simple to find them when building, and eliminated duplicate singleton errors, but the root of websites is already crowded. It was easy enough to find these things anywhere, so now you can organize these special files in any way you like. I’ve been using:
<ul>
<li><code>layouts</code>: A folder full of layouts</li>
<li><code>globals</code>: A folder full of the globally scoped files</li>
</ul>
<figure>
<a href="./img/anywhere.png">
<picture>

<img loading="auto" src="./img/anywhere.png" alt="Screenshot of globals and layouts living anywhere in an editor">
</picture>
</a>
<figcaption>You are free to organize globals and layouts wherever you want now. The globals and layouts folders are great choice!</figcaption>
</figure>
<h3 id="template-files" tabindex="-1">Template files</h3>
Given the <code>top-bun</code> variable cascade system, and not all website files are html, it made sense to include a templating system for generating any kind of file from the <code>global.vars.js</code> variable set. This lets you generate random website “sidefiles” from your site variables.
It works great for generating RSS feeds for websites built with <code>top-bun</code>. Here is the template file that generates the RSS feed for this website:
<pre><code class="hljs language-js">import pMap from 'p-map'
import jsonfeedToAtom from 'jsonfeed-to-atom'
/**
* @template T
* @typedef {import('@siteup/cli').TemplateAsyncIterator<T>} TemplateAsyncIterator
*/
/** @type {TemplateAsyncIterator<{
* siteName: string,
* siteDescription: string,
* siteUrl: string,
* authorName: string,
* authorUrl: string,
* authorImgUrl: string
* layout: string,
* publishDate: string
* title: string
* }>} */
export default async function * feedsTemplate ({
vars: {
siteName,
siteDescription,
siteUrl,
authorName,
authorUrl,
authorImgUrl
},
pages
}) {
const blogPosts = pages
.filter(page => ['article', 'book-review'].includes(page.vars.layout) && page.vars.published !== false)
.sort((a, b) => new Date(b.vars.publishDate) - new Date(a.vars.publishDate))
.slice(0, 10)
const jsonFeed = {
version: 'https://jsonfeed.org/version/1',
title: siteName,
home_page_url: siteUrl,
feed_url: `${siteUrl}/feed.json`,
description: siteDescription,
author: {
name: authorName,
url: authorUrl,
avatar: authorImgUrl
},
items: await pMap(blogPosts, async (page) => {
return {
date_published: page.vars.publishDate,
title: page.vars.title,
url: `${siteUrl}/${page.pageInfo.path}/`,
id: `${siteUrl}/${page.pageInfo.path}/#${page.vars.publishDate}`,
content_html: await page.renderInnerPage({ pages })
}
}, { concurrency: 4 })
}
yield {
content: JSON.stringify(jsonFeed, null, ' '),
outputName: 'feed.json'
}
const atom = jsonfeedToAtom(jsonFeed)
yield {
content: atom,
outputName: 'feed.xml'
}
yield {
content: atom,
outputName: 'atom.xml'
}
}
</code></pre>
<h3 id="page-introspection" tabindex="-1">Page Introspection</h3>
Pages, Layouts and Templates can now introspect every other page in the <code>top-bun</code> build.
You can now easily implement any of the following:
<ul>
<li>RSS feeds</li>
<li>Auto-populating index feeds</li>
<li>Tag systems</li>
</ul>
Pages, Layouts and Templates receive a <code>pages</code> array that includes <a href="https://github.com/bcomnes/top-bun/blob/e83ea560f86dabcda68b0f701c8bfa770aba91ed/lib/build-pages/page-data.js#L25">PageData</a> instances for every page in the build. Variables are already pre-resolved, so you can easily filter, sort and target various pages in the build.
<h3 id="full-type-support" tabindex="-1">Full Type Support</h3>
<code>top-bun</code> now has full type support. It’s achieved with <a href="https://github.com/voxpelli/types-in-js"><code>types-in-js</code></a> and it took a ton of time and effort.
The results are nice, but I’m not sure the juice was worth the squeeze. <code>top-bun</code> was working really well before types. Adding types required solidifying a lot of trivial details to make the type-checker happy. I don’t even think a single runtime bug was solved. It did help clarify some of the more complex types that had developed over the first 2 years of development though.
The biggest improvement provided here is that the following types are now exported from <code>top-bun</code>:
<pre><code class="hljs language-ts">LayoutFunction<T>
PostVarsFunction<T>
PageFunction<T>
TemplateFunction<T>
TemplateAsyncIterator<T>
</code></pre>
You can use these to get some helpful auto-complete in LSP supported editors.
<h4 id="types-in-js-not-typescript" tabindex="-1"><code>types-in-js</code> not Typescript</h4>
This was the first major dive I did into a project with <a href="https://github.com/voxpelli/types-in-js"><code>types-in-js</code></a> support.
My overall conclusions are:
<ul>
<li><code>types-in-js</code> provides a superior development experience to developing in <code>.ts</code> by eliminating the development loop build step.</li>
<li><a href="https://jsdoc.app">JSDoc</a> and <code>types-in-js</code> are in conflict with each other. <code>types-in-js</code> should win, its better than JSDoc in almost every way (but you still use both).</li>
<li>Most of the JSDoc auto-generating documentation ecosystem doesn’t support <code>types-in-js</code>. Find something that consumes the generated types instead of consuming the JSDoc blocs.</li>
<li>There are a number of rough edges around importing types.</li>
<li>The final api documentation features are nice.</li>
<li>The internal types feel good, but were mostly a waste of time being introduced after the fact.</li>
<li>I wish there was a way to strictly introduce types on just the public interfaces and work back, but this is challenging because many details come from deep within the program.</li>
<li>Typescript puts upper limits on the dynamism normally allowed with JS. It’s a superset syntax, that forces a subset of language functionality.</li>
<li><code>@ts-ignore</code> liberally. Take a pass or two to remove some later.</li>
</ul>
<h3 id="handlebars-support-in-md-and-html" tabindex="-1">Handlebars support in <code>md</code> and <code>html</code></h3>
Previously, only <code>js</code> pages had access to the variable cascade inside of the page itself. Now <code>html</code> and <code>md</code> pages can access these variables with <a href="https://handlebarsjs.com">handlebars</a> placeholders.
<pre><code class="hljs language-markdown">## My markdown page
Hey this is a markdown page for {{ vars.siteName }} that uses handlebars templates.
</code></pre>
<h3 id="locally-shipped-default-styles" tabindex="-1">Locally shipped default styles</h3>
Previously, if you opted for the default layout, it would import <a href="http://mine-css.neocities.org">mine.css</a> from <a href="https://unpkg.com">unpkg</a>. This worked, but went against the design goal of making <code>top-bun</code> sites as reliable as possible (shipping all final assets to the dest folder).
Now when you build with the default layout, the default stylesheet (and theme picker js code) is built out into your <code>dest</code> folder.
<h3 id="built-in-browser-sync" tabindex="-1">Built-in <code>browser-sync</code></h3>
<code>@siteup/cli</code> previously didn’t ship with a development server, meaning you had to run one in parallel when developing. This step is now eliminated now that <code>top-bun</code> ships <a href="https://browsersync.io"><code>browser-sync</code></a>. <code>browser-sync</code> is one of the best Node.js development servers out there and offers a bunch of really helpful dev tools built right in, including scroll position sync so testing across devices is actually enjoyable.
If you aren’t familiar with browser-sync, here are some screenshots of fun feature:
<figure class="borderless">
<a href="./img/bs-remote-debugging.png">
<picture>

<img loading="auto" src="./img/bs-remote-debugging.png" alt="Screenshot of Browser Sync debugging">
</picture>
</a>
<figcaption>BrowserSync remote debugging</figcaption>
</figure>
<figure class="borderless">
<a href="./img/bs-grid.png">
<picture>

<img loading="auto" src="./img/bs-grid.png" alt="Screenshot of Browser Sync grid">
</picture>
</a>
<figcaption>BrowserSync Grid overlay</figcaption>
</figure>
<figure class="borderless">
<a href="./img/bs-outline.png">
<picture>

<img loading="auto" src="./img/bs-outline.png" alt="Screenshot of Browser Sync css outline">
</picture>
</a>
<figcaption>BrowserSync CSS outline</figcaption>
</figure>
<figure class="borderless">
<a href="./img/bs-depth.png">
<picture>

<img loading="auto" src="./img/bs-depth.png" alt="Screenshot of Browser Sync css depth">
</picture>
</a>
<figcaption>BrowserSync CSS depth</figcaption>
</figure>
<figure class="borderless">
<a href="./img/bs-depth.png">
<picture>

<img loading="auto" src="./img/bs-depth.png" alt="Screenshot of Browser Sync css depth">
</picture>
</a>
<figcaption>BrowserSync CSS depth</figcaption>
</figure>
<figure class="borderless">
<a href="./img/bs-network-throttle.png">
<picture>

<img loading="auto" src="./img/bs-network-throttle.png" alt="Screenshot of Browser Sync network throttle">
</picture>
</a>
<figcaption>BrowserSync Network throttle</figcaption>
</figure>
<h3 id="top-bun-eject" tabindex="-1"><code>top-bun</code> eject</h3>
<code>top-bun</code> now includes an <code>--eject</code> flag, that will write out the default layout, style, client and dependencies into your <code>src</code> folder and update <code>package.json</code>. This lets you easily get started with customizing default layouts and styles when you decide you need more control.
<pre><code class="hljs language-console">√ default-layout % npx top-bun --eject
top-bun eject actions:
- Write src/layouts/root.layout.mjs
- Write src/globals/global.css
- Write src/globals/global.client.mjs
- Add mine.css@^9.0.1 to package.json
- Add uhtml-isomorphic@^2.0.0 to package.json
- Add highlight.js@^11.9.0 to package.json
Continue? (Y/n) y
Done ejecting files!
</code></pre>
The default layout is always supported, and its of course safe to rely on that.
<h3 id="improved-log-output-%F0%9F%AA%B5" tabindex="-1">Improved log output 🪵</h3>
The logging has been improved quite a bit. Here is an example log output from building this blog:
<pre><code class="hljs language-console">> top-bun --watch
Initial JS, CSS and Page Build Complete
bret.io/src => bret.io/public
├─┬ projects
│ ├─┬ websockets
│ │ └── README.md: projects/websockets/index.html
│ ├─┬ tron-legacy-2021
│ │ └── README.md: projects/tron-legacy-2021/index.html
│ ├─┬ package-automation
│ │ └── README.md: projects/package-automation/index.html
│ └── page.js: projects/index.html
├─┬ jobs
│ ├─┬ netlify
│ │ └── README.md: jobs/netlify/index.html
│ ├─┬ littlstar
│ │ └── README.md: jobs/littlstar/index.html
│ ├── page.js: jobs/index.html
│ ├── zhealth.md: jobs/zhealth.html
│ ├── psu.md: jobs/psu.html
│ └── landrover.md: jobs/landrover.html
├─┬ cv
│ ├── README.md: cv/index.html
│ └── style.css: cv/style-IDZIRKYR.css
├─┬ blog
│ ├─┬ 2023
│ │ ├─┬ reintroducing-top-bun
│ │ │ ├── README.md: blog/2023/reintroducing-top-bun/index.html
│ │ │ └── style.css: blog/2023/reintroducing-top-bun/style-E2RTO5OB.css
│ │ ├─┬ hello-world-again
│ │ │ └── README.md: blog/2023/hello-world-again/index.html
│ │ └── page.js: blog/2023/index.html
│ ├── page.js: blog/index.html
│ └── style.css: blog/style-NDOJ4YGB.css
├─┬ layouts
│ ├── root.layout.js: root
│ ├── blog-index.layout.js: blog-index
│ ├── blog-index.layout.css: layouts/blog-index.layout-PSZNH2YW.css
│ ├── blog-auto-index.layout.js: blog-auto-index
│ ├── blog-auto-index.layout.css: layouts/blog-auto-index.layout-2BVSCYSS.css
│ ├── article.layout.js: article
│ └── article.layout.css: layouts/article.layout-MI62V7ZK.css
├── globalStyle: globals/global-OO6KZ4MS.css
├── globalClient: globals/global.client-HTTIO47Y.js
├── globalVars: global.vars.js
├── README.md: index.html
├── style.css: style-E5WP7SNI.css
├── booklist.md: booklist.html
├── about.md: about.html
├── manifest.json.template.js: manifest.json
├── feeds.template.js: feed.json
├── feeds.template.js-1: feed.xml
└── feeds.template.js-2: atom.xml
[Browsersync] Access URLs:
--------------------------------------
Local: http://localhost:3000
External: http://192.168.0.187:3000
--------------------------------------
UI: http://localhost:3001
UI External: http://localhost:3001
--------------------------------------
[Browsersync] Serving files from: /Users/bret/Developer/bret.io/public
Copy watcher ready
</code></pre>
<h3 id="support-for-mjs-and-cjs-file-extensions" tabindex="-1">Support for <code>mjs</code> and <code>cjs</code> file extensions</h3>
You can now name your page, template, vars, and layout files with the <code>mjs</code> or <code>cjs</code> file extensions. Sometimes this is a necessary evil. In general, set your <code>type</code> in your <code>package.json</code> correctly and stick with <code>.js</code>.
<h2 id="what%E2%80%99s-next-for-top-bun" tabindex="-1">What’s next for <code>top-bun</code></h2>
The current plan is to keep sitting on this feature set for a while. But I have some ideas:
<ul>
<li>Tell the world about it?</li>
<li>Server routes with <a href="https://fastify.dev">fastify</a> (or expose a plugin to slot into an existing server)?</li>
<li>Deeper integration with <a href="https://ghub.io/uhtml"><code>uhtml</code></a>?</li>
<li>More web-component examples? <code>top-bun</code> is already one of the best environments for implementing sites that use web-components. Page bundles are a perfect place to register components!</li>
<li>Comparisons with other tools in the “enterprise-js” ecosystem?</li>
</ul>
If you try out <code>top-bun</code>, I would love to hear about your experience. Do you like it? Do you hate it? <a href="https://github.com/bcomnes/top-bun/discussions">Open an discussion item.</a> or reach out privately.
<h2 id="history" tabindex="-1">History of <code>top-bun</code></h2>
OK, now time for the story behind <code>top-bun</code> aka <code>@siteup/cli</code>.
I ran some experiments with orthogonal tool composition a few years ago. I realized I could build sophisticated module based websites by composing various tools together by simply running them in parallel.
What does this idea look like? See this snippet of a <code>package.json</code>:
<pre><code class="hljs language-json"> { "scripts": {
"build": "npm run clean && run-p build:*",
"build:css": "postcss src/index.css -o public/bundle.css",
"build:md": "sitedown src -b public -l src/layout.html",
"build:feed": "generate-feed src/log --dest public && cp public/feed.xml public/atom.xml",
"build:static": "cpx 'src/**/*.{png,svg,jpg,jpeg,pdf,mp4,mp3,js,json,gif}' public",
"build:icon": "gravatar-favicons --config favicon-config.js",
"watch": "npm run clean && run-p watch:* build:static",
"watch:css": "run-s 'build:css -- --watch'",
"watch:serve": "browser-sync start --server 'public' --files 'public'",
"watch:md": "npm run build:md -- -w",
"watch:feed": "run-s build:feed",
"watch:static": "npm run build:static -- --watch",
"watch:icon": "run-s build:icon",
"clean": "rimraf public && mkdirp public",
"start": "npm run watch"
}
}
</code></pre>
<ul>
<li>I have <a href="https://postcss.org"><code>postcss</code></a> building css bundles, enabling an <code>@import</code> based workflow for css, as well as providing various transforms I found useful.</li>
<li>The markdown is built with <a href="https://github.com/ungoldman/sitedown"><code>sitedown</code></a>.</li>
<li>I wrote a tool to generate a RSS feed from markdown called <a href="https://github.com/bcomnes/generate-feed"><code>generate-feed</code></a></li>
<li>I generate favicons from a gravatar identifier with <a href="https://github.com/bcomnes/gravatar-favicons"><code>gravatar-favicons</code></a>.</li>
<li><code>js</code> bundling could easily be added in here with <a href="https://esbuild.github.io"><code>esbuild</code></a> or <a href="https://ghub.io/rollup">rollup</a>.</li>
<li>Steps are grouped into <code>build</code> and <code>watch</code> prefixes, and managed with <a href="https://github.com/bcomnes/npm-run-all2">npm-run-all2</a> which provides shortcuts to running these tasks in parallel.</li>
</ul>
I successfully implemented this pattern across 4-5 different websites I manged. It work beautifully. Some of the things I liked about it:
<ul>
<li>✅ Tools could be swapped out easily.</li>
<li>✅ I could experiment individual ideas on individual sites, without forcing them on every other project.</li>
<li>✅ All of the tools were versioned gated, and I could update them as I had time.</li>
<li>✅ It worked very well with <a href="/projects/package-automation/">release automation</a>.</li>
</ul>
But it had a few drawbacks:
<ul>
<li>❌ I liked the pattern so much, that I wanted to add websites to more projects, but setting this all up was a hassle.</li>
<li>❌ When I discovered a change I liked, re-implementing the change across multiple projects become tedious.</li>
<li>⚠️ Orthogonal tool composition helps keep this arrangement clean and fast, but some steps really do benefit from the awareness of the results of other steps.</li>
</ul>
So I decided to roll up all of the common patterns into a single tool that included the discoveries of this process.
<h2 id="%40siteup%2Fcli-extended-sitedown" tabindex="-1"><code>@siteup/cli</code> extended <code>sitedown</code></h2>
Because it was clear <code>sitedown</code> provided the core structure of this pattern (making the website part), I extended the idea in the project <code>@siteup/cli</code>. Here are some of the initial features that project shipped with:
<ul>
<li>Variable cascade + frontmatter: You could define global and page variables used to customize parts of the site layout. This lets you set the title, or other parts of the <code><head></code> tag easily from the <a href="https://jekyllrb.com/docs/front-matter/">frontmatter</a> section of markdown pages.</li>
<li>A <code>js</code> layout: This let you write a simple JS program that receives the variable cascade and child content of the page as a function argument, and return the contents of the page after applying any kind of template tool available in JS.</li>
<li><code>html</code> pages: <a href="https://commonmark.org">CommonMark</a> supports <code>html</code> in markdown, but it also has some funny rules that makes it more picky about how the <code>html</code> is used. I wanted a way to access the full power of <code>html</code> without the overhead of markdown, and this page type unlocked that.</li>
<li><code>js</code> pages: Since I enjoy writing JavaScript, I also wanted a way to support generating pages using any templating system and data source imaginable so I also added support to generate pages from the return value of a JS function.</li>
<li>JavaScript bundling: I wanted to make it super easy to include client side JavaScript, so it included some conventions for setting that up quickly and easily.</li>
<li>CSS bundling: Instead of adding the complexity of css-modules or other transform-based scoped css solution, <code>siteup</code> opted to make it super easy to add a global <code>css</code> bundle, and page scoped <code>css</code> bundles, which both supported a <code>postcss</code> <code>@import</code> workflow and provided a few common transforms to make working with <code>css</code> tolerable (nesting, prefixing etc).</li>
<li>Static files: Including static files was such a common pattern that I also included the ability, to copy over static files without any additional configuration. I believe <code>sitedown</code> also added this feature subsequently.</li>
<li>Asset co-location: I wanted a way to co-locate assets, code and anything near where it was depended upon. It is such a natural way to organize complex collections, and so many tools break this, or require special locations for special files. Let me put it wherever I want in <code>src</code>!</li>
</ul>
After sitting on the idea of <code>siteup</code> for over a year, by the time I published it to <code>npm</code>, the name was taken, so I used the npm org name hack to get a similar name <code>@siteup/cli</code>. SILLY NPM!
I enjoyed how <code>@siteup/cli</code> came out, and have been using it for 2 year now. Thank you of course to <a href="https://ungoldman.com">ungoldman</a> for laying the foundation of most of these tools and patterns. Onward and upward to <code>top-bun</code>!
<img src="./img/contribs.png" alt="contribution graph">
]]></content>
<link rel="alternate" href="https://bret.io/blog/2023/reintroducing-top-bun/"/>
</entry>
<entry>
<id>https://bret.io/blog/2023/hello-world-again/#2023-08-30T18:06:24.000Z</id>
<title>Hello world (again) 🌎</title>
<updated>2023-08-30T18:06:24.000Z</updated>
<published>2023-08-30T18:06:24.000Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[<a href="./sunset.jpeg"><img src="./sunset.jpeg" alt="sunset"></a>
This blog has been super quiet lately, sorry about that.
I had some grand plans for finishing up my website tools to more seamlessly
support blogging.
The other day around 3AM, I woke up and realized that the tools aren’t stopping me
from writing, I am.
Also my silently implemented policy about not writing about ‘future plans and ideas before they are ready’ was implemented far to strictly.
It is in fact a good thing to write about in-progress ideas and projects slightly out into the future.
This is realistic, interesting, and avoids the juvenile trap of spilling ideas in front of the world only to see you never realize them.
So here I am writing a blog post again.
Anyway, no promises, but it is my goal to write various ahem opinions, thoughts and ideas more often because nothing ever happens unless you write it down.
<h2 id="some-basic-updates" tabindex="-1">Some basic updates</h2>
Here are some updates from the last couple years that didn’t make it onto this site before.
<ul>
<li>I’m back in <a href="https://www.openstreetmap.org/#map=13/40.8455/-124.0532">California</a>.</li>
<li>I moved a bunch (homes, jobs). I bought a home and then sold it. I don’t recommend it!</li>
<li>I have a son and a daughter and am married. I recommend this!</li>
<li>I started <a href="https://hifiwi.fi">HifiWi.fi</a></li>
<li>The first product is <a href="https://breadcrum.net/">Breadcrum</a>, a bookmarking service with textual and media archiving super powers.</li>
<li>HifiWi has subsumed operation of <a href="https://gumcast.com/">Gumcast</a>, a tool to allow podcasting with GumRoad. Without advertising, it has picked up 100s of users organically from search results.</li>
<li>I had a near 2 year tenure at <a href="https://socket.dev">Socket Inc</a> dabbling in GitHub apps and npm security concerns.</li>
<li>I’m now working at <a href="https://socketsupply.co">socketsupply.co</a> on a runtime and P2P full time again. Really happy to be back in this space again.</li>
<li>I still work remote and intend to continue working remote.</li>
<li>A bunch of my projects are running on my own website/app builder too called <a href="https://github.com/bcomnes/top-bun">top-bun</a>. It’s maybe a 3rd of the way done, but has been useful for about 90% of my needs. I would like to write up a proper blog post about it someday.</li>
<li><a href="https://github.com/bcomnes/deploy-to-neocities/">bcomnes/deploy-to-neocities</a> has 300+ public users deploying websites from GitHub actions to Neocities.</li>
<li>My fork <a href="https://github.com/bcomnes/npm-run-all2">bcomnes/npm-run-all2</a> (co-maintained by <a href="https://voxpelli.com">voxpelli.com</a>) has picked up 1000+ public dependents.</li>
<li>Now that twitter (ahem x) has completely abandoned its charter as an open web website, you can find me posting on <a href="https://fosstodon.org/@bcomnes">@bcomnes@fosstodon.org</a> and <a href="https://bsky.app/profile/bret.io">@bret.io on bsky</a>. I would like to make this website authoritative though for me posts though.</li>
<li>I’m very lucky to have a new office that is a 1 minute walk from home:</li>
</ul>
<a href="./office.jpeg"><img src="./office.jpeg" alt="pic of the office"></a>
]]></content>
<link rel="alternate" href="https://bret.io/blog/2023/hello-world-again/"/>
</entry>
<entry>
<id>https://bret.io/projects/tron-legacy-2021/#2021-08-05T23:09:46.781Z</id>
<title>Tron Legacy 2021</title>
<updated>2021-08-05T23:09:46.781Z</updated>
<published>2021-08-05T23:09:46.781Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[I updated the Sublime Text <a href="https://packagecontrol.io/packages/Tron%20Color%20Scheme"><code>Tron Color Scheme</code></a> today after a few weeks of reworking it for the recent <a href="https://www.sublimetext.com/blog/articles/sublime-text-4">release of Sublime Text 4</a>.
The <a href="https://github.com/bcomnes/sublime-tron-color-scheme/releases/tag/v2.0.0"><code>2.0.0</code> release</a> converts the older <a href="https://www.sublimetext.com/docs/color_schemes_tmtheme.html"><code>.tmTheme</code></a> format into the Sublime specific theme format.
Overall the new Sublime theme format (<a href="https://www.sublimetext.com/docs/color_schemes.html"><code>.sublime-color-scheme</code></a>) is a big improvement, largely due to its simple JSON structure and its variables support.
JSON is, despite the common arguments against it, super readable and easily read and written by humans.
The variable support makes the process of making a theme a whole lot more automatic, since you no longer have to find and replace colors all over the place.
The biggest problem I ran into was poor in-line color highlighting when working with colors, so I ended up using a VSCode plugin called <a href="https://marketplace.visualstudio.com/items?itemName=naumovs.color-highlight"><code>Color Highlight</code></a> in a separate window.
Sublime has a great plugin also called <a href="https://packagecontrol.io/packages/Color%20Highlight"><code>Color Highlight</code></a> that usually works well, but not in this case.
The Sublime <code>Color Highlight</code> variant actually does temporary modifications to color schemes, which seriously gets in the way when working on color scheme files.
The rewrite is based it off of the new <a href="https://www.youtube.com/watch?v=_HoltQwvF2o">Mariana Theme</a> that ships with <abbr title="Sublime Text 4">ST4</abbr>, so the theme should have support for most of the latest features in <abbr title="Sublime Text 4">ST4</abbr> though there are surely features that even Mariana missed.
Let me know if you know of any.
Here a few other points of consideration made during the rewrite:
<ul>
<li>Remove the other theme variants. A breaking change, but they were poorly maintained to begin with.</li>
<li>Renamed the theme file to <code>Tron Legacy 4 (Dark)</code>.</li>
<li><a href="https://github.com/bcomnes/sublime-tron-color-scheme/pull/8">A light theme is in order</a>, but its a non-trivial rewrite to support a light mode.</li>
</ul>
<figure class="borderless">
<img src="./js.png" alt="Tron Legacy 4 JS Syntax Example">
<figcaption>JS Syntax example</figcaption>
</figure>
<figure class="borderless">
<img src="./md.png" alt="Tron Legacy 4 Markdown Syntax Example">
<figcaption>Markdown Syntax example</figcaption>
</figure>
<figure class="borderless">
<img src="./py.png" alt="Tron Legacy 4 Python Syntax Example">
<figcaption>Python Syntax example</figcaption>
</figure>
<figure class="borderless">
<img src="./c.png" alt="Tron Legacy 4 C Syntax Example">
<figcaption>C Syntax example</figcaption>
</figure>
<figure class="borderless">
<img src="./diff.png" alt="Tron Legacy 4 diff Syntax Example">
<figcaption>diff Syntax example</figcaption>
</figure>
Here a few more relevant links and please let me know what you think if you try it out.
<ul>
<li><a href="https://github.com/bcomnes/sublime-tron-color-scheme">github.com/bcomnes/sublime-tron-color-scheme</a></li>
<li><a href="https://github.com/bcomnes/sublime-tron-color-scheme/releases/tag/v2.0.0">Release page</a></li>
</ul>
<h2 id="syndication" tabindex="-1">Syndication</h2>
<ul>
<li><a href="https://twitter.com/bcomnes/status/1423418998725742602" rel="syndication" class="u-syndication">twitter thread</a></li>
</ul>
<blockquote class="twitter-tweet">Sublime Tron Legacy color scheme fully updated for <a href="https://twitter.com/sublimehq?ref_src=twsrc%5Etfw">@sublimehq</a> Text 4. Full syntax support, lots of other small improvements. Also it supports 'glow' text✌️ <a href="https://t.co/vShbGThgDF">pic.twitter.com/vShbGThgDF</a>— 🌌🌵🛸Bret🏜👨‍👩‍👧🚙 (@bcomnes) <a href="https://twitter.com/bcomnes/status/1423418998725742602?ref_src=twsrc%5Etfw">August 5, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<date>2021-08-05T23:09:46.781Z</date>
]]></content>
<link rel="alternate" href="https://bret.io/projects/tron-legacy-2021/"/>
</entry>
<entry>
<id>https://bret.io/jobs/littlstar/#2021-07-16T19:20:44.161Z</id>
<title>Littlstar Portfolio</title>
<updated>2021-07-16T19:20:44.161Z</updated>
<published>2021-07-16T19:20:44.161Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[<img src="./littlstar-logo.svg" height=100 width=100 style='float: left; margin-right: 1em;'>
After a short sabbatical in Denmark at <a href="https://archive.ph/m8Igr">Hyperdivision</a>, I joined <a href="https://littlstar.info">Littlstar</a>.
Here is a quick overview of some of the more interesting projects I worked on.
I joined Littlstar during a transitional period of the company and help them transition from a VR video platform to a video on demand and live streaming platform, and now an NFT sales and auction platform.
<h2 id="nycpf-vr-platform" tabindex="-1">NYCPF VR Platform</h2>
<figure class="borderless">
<img src="./nycpf.png" alt="NYCPF-XR screenshot">
<figcaption>NYCPF used a local process talking over websocket RPC to a webapp running in a local browser.</figcaption>
</figure>
My first project was picking up on an agency style project developing a novel VR reality training platform for <a href="https://www.nycpolicefoundation.org">NYCPF</a>, powered by a custom <a href="https://github.com/holepunchto/hypercore">hypercore</a> p2p file sharing network, delivering in-house developed unity VR scenarios.
These scenarios could then be brought to various locations like schools or events where NYCPF could guide participants through various law enforcement scenarios with different outcomes based on participant choices within the simulation.
By utilizing a p2p and offline first design approach, we were able to deliver an incredibly flexible and robust delivery platform that had all sorts of difficult features to develop for traditional file distribution platforms such as:
<ul>
<li>Automated offline re-distribution of simulation data to a fleet of devices.</li>
<li>Bittorrent style inverted bandwidth characteristics (more demand = more availability).</li>
<li>Extremely high performance content distribution speeds on commodity hardware.</li>
<li>WAN and LAN peering between devices.</li>
<li>Centrally controlled editorial over available content, with eventual-consistency properties to offline or air-gaped clients.</li>
<li>Cross platform codebase targeting Windows, MacOS and Linux. The dev team utilized all 3 platforms during development, interchangeably.</li>
</ul>
While the project was built on the backs of giants like the <a href="https://hypercore-protocol.org">Hypercore protocol</a>, as well as the amazing work of my <a href="https://github.com/orgs/little-core-labs/people">colleague</a>, I contributed in a number of areas to move the project forward during my time contracting on it.
<ul>
<li>Take ownership of the preexisting React application.</li>
<li>Simplify and maintain a mix of internal open source and closed source packages.</li>
<li>Simplify React app state layer by eliminating the need for Redux in favor of built in React state solutions.</li>
<li>Implement offline mode via Service workers in conjunction with Hypercore.</li>
<li>Packaging and delivery tasks.</li>
<li>Improved progress UI/UX.</li>
<li>Contribute to native packaging tools build with <a href="https://github.com/vercel/pkg">pkg</a> and <a href="https://github.com/little-core-labs/tiny-module-compiler">tiny-module-compiler</a>.</li>
</ul>
<img src="./tmc.png" alt="Tiny Module Compiler">
Some of the discrete software packages that resulted from this project are described below.
<h3 id="secure-rpc-protocol" tabindex="-1"><a href="https://github.com/little-core-labs/secure-rpc-protocol"><code>secure-rpc-protocol</code></a></h3>
Secure <a href="https://github.com/little-core-labs/rpc-protocol">rpc-protocol</a> over any duplex socket using <a href="https://github.com/emilbayes/noise-peer">noise-protocol</a>.
This was a refactor of an existing RPC over websocket solution the project was using.
It improved upon the previous secure RPC already used by switching to using the <a href="http://noiseprotocol.org">noise protocol</a> which implements well understood handshake patterns that can be shared and audited between projects, rather than relying on a novel implementation at the RPC layer.
It also decoupled the RPC protocol from the underlying socket being used, so that the RPC system could be used over any other channels we might want in the future.
<figure class="borderless">
<img src="./secure-rpc-protocol.png" alt="Secure RPC Protocol">
<figcaption><a href="https://github.com/little-core-labs/secure-rpc-protocol">Secure RPC protocol</a> uses the noise protocol for encryption, and works over any duplex socket.</figcaption>
</figure>
<h3 id="async-folder-walker" tabindex="-1"><a href="https://github.com/bcomnes/async-folder-walker"><code>async-folder-walker</code></a></h3>
An async generator that walks files.
This project was a refactor of an existing project called <a href="https://ghub.io/folder-walker">folder-walker</a> implementing a high performance folder walk algorithm using a more modern async generator API.
<figure>
<img src="./afw.jpeg" alt="Async folder walker">
<figcaption><a href="https://github.com/bcomnes/async-folder-walker">Async folder walker</a> provides a modern api to folder and filer walking of a directory.</figcaption>
</figure>
<h3 id="unpacker-with-progress" tabindex="-1"><a href="https://github.com/little-core-labs/unpacker-with-progress"><code>unpacker-with-progress</code></a></h3>
<code>unpacker-with-progress</code> is a specialized package that unpacks archives, and provides a progress api in order to provide UI feedback.
One of the deficiencies with the NYCPF project when I started was lack of UI feedback during the extraction process.
VR files are very large, and are delivered compressed to the clients.
After the download is complete, the next step to processing the data is unpacking the content.
This step did not provide any sort of progress feedback to the user because the underlying unpacking libraries did not expose this information, or only exposed some of the information needed to display a progress bar.
This library implemented support for unpacking the various archive formats the project required, and also added an API providing uniform unpacking progress info that could be used in the UI during unpacking tasks.
<figure class="borderless">
<img src="./uwp.png" alt="Unpacker with progress">
<figcaption><a href="https://github.com/little-core-labs/unpacker-with-progress">unpacker-with-progress</a> is a consistency layer on top of a few unpacking libraries, with an emphasis on progress reporting for use in UI applications.</figcaption>
</figure>
<h3 id="hchacha20" tabindex="-1"><a href="https://github.com/little-core-labs/hchacha20"><code>hchacha20</code></a></h3>
One of the more interesting side projects I worked on was porting over some of the <a href="https://doc.libsodium.org">libsodium</a> primitives to <a href="https://github.com/sodium-friends/sodium-javascript">sodium-javascript</a>.
I utilized a technique I learned about at Hyperdivision where one can write web assembly by hand in the <a href="https://developer.mozilla.org/en-US/docs/WebAssembly/Understanding_the_text_format">WAT format</a>, providing a much wider set of data types, providing the type guarantees needed to write effective crypto.
While the WAT was written for HChaCha20, the effort was quite laborious and it kicked off a debate as to whether it would be better to just wrap <a href="https://github.com/jedisct1/libsodium.js">libsodium-js</a> (the official libsodium js port) in a wrapper that provided the <a href="https://github.com/sodium-friends/sodium-universal">sodium-universal</a> API. This was achieved by another group in <a href="https://github.com/geut/sodium-javascript-plus">geut/sodium-javascript-plus</a> which successfully ran hypercores in the browser using that wrapper.
Ultimately, this effort was scrapped, determining that noise peer connections in the browser are redundant to webRTC encryption and https sockets.
It was a fun and interesting project none the less.
<figure class="borderless">
<img src="./hchacha.png" alt="Some HChacha WAT Code">
<figcaption>A peek into the world of WASM via WAT.</figcaption>
</figure>
<h3 id="reconnecting-sockets" tabindex="-1">Reconnecting sockets</h3>
We were having some state transition bugs between the webapp and the local server process, where the app could get into strange indeterminate states.
Both had independent reconnect logic wrapped up with application code, and it added a lot of chaos to understanding how each process was behaving when things went wrong (especially around sleep cycles on machines).
I implemented a generic reconnecting state machine that could accept any type of socket, and we were able to reduce the number of state transition bugs we had.
<ul>
<li><a href="https://github.com/little-core-labs/reconnecting-socket"><code>little-core-labs/reconnecting-socket</code></a></li>
<li><a href="https://github.com/little-core-labs/reconnecting-simple-websocket"><code>little-core-labs/reconnecting-simple-websocket</code></a></li>
</ul>
<figure class="borderless">
<img src="./rsw.png" alt="Reconnecting simple websocket">
</figure>
<h2 id="little-core-labs" tabindex="-1">Little Core Labs</h2>
<img src="./lcl.png" width=100 height=100 style='float: left; margin-right: 1em;'>
After working on various agency projects at Littlstar, we formed a separate organization to start a fresh rewrite of the technology stack.
My high level contributions:
<ul>
<li>Terraform running in Github actions</li>
<li>Provisioning AWS infrastructure.</li>
<li>Helping come up with how all of this stuff will work.</li>
</ul>
This was an amazing founders-style opportunity to help rethink and re-implement years of work that had developed at Littlstar prior to joining.
Effectively starting from 0, we rethought the entire technology pipeline, from operations, to infrastructure, to deployment, resulting in something really nice, modern, minimal, low maintenance and malleable.
<ul>
<li><a href="https://github.com/little-core-labs">Little Core Labs Github</a></li>
<li><a href="https://twitter.com/littlecorelabs">Little Core Labs Twitter</a></li>
</ul>
<figure>
<img src="./overview.jpg" alt="A snapshot of our cloudcraft">
<figcaption>
A screencap of our cloudformation diagram.
</figcaption>
</figure>
<h2 id="terraform-ops" tabindex="-1">Terraform ops</h2>
A culmination of ingesting the <a href="https://amzn.to/3er6lBB">“Terraform: Up & Running”</a> and <a href="https://amzn.to/3rbNakx">“AWS Certified Solutions Architect”</a> books, as well as building off existing organizational experience with AWS, I helped research and design an operations plan using <a href="https://www.terraform.io">Terraform</a> and <a href="https://github.com/features/actions">GitHub Actions</a>.
<div style='display: grid; grid-template-columns: 1fr 1fr; justify-items: center;'>
<div>
<img widht='199.5' height='250' src='./aws.jpg'/>
</div>
<div>
<img width='190.5' height='250' src='./terraform.jpg'/>
</div>
</div>
This arrangement has proven powerful and flexible.
While it isn’t perfect, it has been effective and reliable and cheap, despite its imperfections in relation to some of the more esoteric edge cases of Terraform.
A quick overview of how its arrange:
<ul>
<li>We have a global terraform repository with segmented terraform files for various services.</li>
<li>The <code>ops</code> global repo runs terraform in a bootstrapped GitHub actions environment.</li>
<li>We can create service level repos from the <code>ops</code> terraform repo, that in turn contain their own Terraform files specific to that service.</li>
<li>One of the benefits was that the GitHub environment worked along side a local environment, due to the use of AWS secrets manager and GitHub actions secrets (all managed in Terraform), so debugging was easy and flexible.</li>
</ul>
<h2 id="github-actions" tabindex="-1">Github actions</h2>
<img width='140' height='140' src='./actions.svg' alt='Actions logo' style='float: left; margin-right: 1em;' />
One of the drawbacks of rolling our own Terraform CI infrastructure was that we had to tackle many small edge cases inside the GitHub actions environment.
It was nice to learn about the various types of custom GitHub actions one can write, as well as expand that knowlege to the rest of the org, but it also ate up a number of days focusing on DevOps problems specific to our CI environment.
Here are some of the problems I helped solve in the actions environment.
<ul>
<li><a href="https://github.com/little-core-labs/install-terraform">install-terraform</a> - Install terraform at a specific version.</li>
<li><a href="https://github.com/little-core-labs/netrc-creds">netrc-creds</a> - Set up <a href="https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html">netrc</a> credentials.</li>
<li><a href="https://github.com/little-core-labs/get-git-tag">get-git-tag</a> - Easily get a git tag.</li>
</ul>
<img src="./netrc.png" alt="netrc-creds">
<h2 id="sdk-js" tabindex="-1">sdk-js</h2>
I helped lay the framework for the initial version of <code>sdk-js</code>, the Little Core Labs unified library used to talk to the various back-end services at Little Core Labs.
One of underlying design goals was to solve for the <a href="https://nodejs.org/api/esm.html">newly introduced native ESM features in node</a>, in such a way that the package could be consumed directly in the browser, natively as ESM in node, but also work in dual CJS/ESM environments like Next.js.
While this did add some extra overhead to the project, it serves as a design pattern we can pull from in the future, as well as a provide a highly compatible but modern API client.
I also extracted out this dual package pattern into a reusable template.
<ul>
<li><a href="https://github.com/bcomnes/esm-template">bcomnes/esm-template</a></li>
</ul>
<img src="./sdk.png" alt="sdk-js">
<h2 id="rad.live" tabindex="-1"><a href="https://rad.live">rad.live</a></h2>
<img src="./rad.jpg" alt="Rad.live">
I was the principal engineer on the new <a href="https://rad.live">Rad.live</a> website.
I established and implemented the tech stack, aiming to take a relatively conservative take on a code base that would maximize the readability and clarity for a Dev team that would eventually grow in size.
A high level, the application is simply an app written with:
<ul>
<li><a href="https://nextjs.org">Next.js</a> - React with a framework to provide standard bundling, SSR and routing features.</li>
<li><a href="https://swr.vercel.app">SWR</a> - Declarative data <a href="https://reactjs.org/docs/hooks-intro.html">‘hooks’</a>, minimizing/eliminating complex state management throughout the app.</li>
<li><a href="https://github.com/little-core-labs/gqlr">GQLR</a> - A simple GraphQL client tuned to ‘just work’.</li>
<li><a href="https://github.com/vercel/styled-jsx">styled-jsx</a> - A simple and understandable CSS-In-JS framework that minimizes orthogonal layout CSS.</li>
</ul>
From a development perspective, it was important to have testing and automation from the beginning.
For this we used:
<ul>
<li>GitHub actions to run CI on every interaction.</li>
<li>Organizational standard linting that supported the unique needs of React.</li>
<li><a href="https://storybook.js.org">Storybook</a> for data/state independent component testing (Using the new and improved storybook API)</li>
<li>Automated component testing, with a minimum of ‘does it render’ testing utilizing the storybook stories.</li>
<li>Release automation.</li>
</ul>
Overall, the codebase has had two other seasoned developers (one familiar and one new at React) jump in and find it productive based on individual testimony.
Gathering feedback from those thatI work with on technical decisions that I make is an important feedback look I always try to incorporate when green fielding projects.
Additionally, it has been a relatively malleable code base that is easy to add MVP features to and is in a great position to grow.
<h3 id="vision.css" tabindex="-1"><code>vision.css</code></h3>
<img src="./vision.jpg" alt="vision.css">
I implemented a custom design system in tandem with our design team.
This project has worked out well, and has so far avoided ‘css lockout’, where only one developer can effectively make dramatic changes to an app layout due to an undefined and overly general orthogonal ‘global style sheet’.
The way this was achieved was by focusing on a simple global CSS style sheet that implements the base HTML elements in accordance with the design system created by the design team.
While this does result in a few element variants that are based on a global style class name, they remain in the theme of only styling ‘built in’ html elements, so there is little question what might be found in the global style sheet, and what needs to be a scoped css style.
Some of the features we used for vision.css
<ul>
<li>Standard CSS syntax with a selection of a few ‘yet-to-be-released’ css syntax features.</li>
<li>A <a href="https://postcss.org">postcss</a> build pipeline.</li>
<li>Heavy use of <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/Using_CSS_custom_properties">CSS variables</a>.</li>
<li>A <a href="https://little-core-labs.github.io/vision.css/">website</a> that contains a style-guide and preview of the various styled elements.</li>
</ul>
<h3 id="eslint-config-12core" tabindex="-1"><a href="https://github.com/little-core-labs/eslint-config-12core"><code>eslint-config-12core</code></a></h3>
<img src="./eslint.png" alt="eslint-config-12core">
Linting is the “spell check” of code, but its hard to agree on what rules to follow.
Like most things, having a standard set of rules that is good-enough is always better than no rules, and usually better than an unpredictable and unmaintainable collection of project-unique rules.
I put together a shared ESLint config that was flexible enough for most projects so far at Little Core Labs based on the ‘StandardJS’ ruleset, but remained flexible enough to modify unique org requirements.
Additionally, I’ve implemented it across many of the projects in the Github Org.
<ul>
<li><a href="https://github.com/little-core-labs/eslint-config-12core"><code>eslint-config-12core</code></a></li>
</ul>
<h3 id="gqlr" tabindex="-1"><a href="https://github.com/little-core-labs/gqlr"><code>gqlr</code></a></h3>
<img src="./gqlr.png" alt="gqlr">
<code>gqlr</code> is a simplified fork of <a href="https://github.com/prisma-labs/graphql-request">graphql-request</a>.
This relatively simple wrapper around the JS fetch API has a gaggle of upstream maintainers with various needs that don’t really match our needs.
The fork simplified and reduced code redundancy, improved maintainability through automation, fixed bugs and weird edge cases and dramatically improved errors and error handling at the correct level of the tech stack.
These changes would have unlikely been accepted upstream, so by forking we are able to gain the value out of open source resources, while still being able to finely tune them for our needs, as well as offer those changes back to the world.
<ul>
<li><a href="https://github.com/little-core-labs/gqlr"><code>gqlr</code></a></li>
</ul>
<h3 id="local-storage-proxy" tabindex="-1"><a href="https://github.com/bcomnes/local-storage-proxy"><code>local-storage-proxy</code></a></h3>
<img src="./lsp.gif" alt="">
A configuration solution that allows for persistent overrides stored in <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage">local storage</a>, including cache busting capabilities. Implemented with a recursive <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy">JS proxy</a> to simulate native object interactions over a window.localstorage interface.
<ul>
<li><a href="https://github.com/bcomnes/local-storage-proxy"><code>local-storage-proxy</code></a></li>
</ul>
<h3 id="community-maintenance" tabindex="-1">Community maintenance</h3>
We ended up taking on maintainence of a few other packages, providing fixes and improvements where the original authors seem to have left off.
<ul>
<li><a href="https://github.com/little-core-labs/date-input-polyfill">little-core-labs/date-input-polyfill</a> - Polyfill for HTML5 date input element.</li>
<li><a href="https://github.com/little-core-labs/chromafi">little-core-labs/chromafi</a></li>
</ul>
<h3 id="video-platform" tabindex="-1">Video platform</h3>
Here are some snapshots of the video platform we launched.
<img src="./channels.jpg" alt="">
<img src="./player.jpg" alt="">
<h3 id="nft-auction-platform" tabindex="-1">NFT Auction Platform</h3>
Here are some screenshots of the NFT auction platform I helped build.
The UI was fully responsive and updated on the fly to new results, thanks to the powers of SWR.
<img src="./auction.jpg" alt="">
<h3 id="marketing-pages" tabindex="-1">Marketing pages</h3>
I did a few marketing pages as well.
<img src="./marketing-rad.jpg" alt="">
<img src="./marketing-nft.jpg" alt="">
<img src="./marketing-onnit.jpg" alt="">
<img src="./marketing-ps.jpg" alt="">
<h2 id="conclusion" tabindex="-1">Conclusion</h2>
While this isn’t everything I did at Littlstar, it captures many of the projects I enjoyed working on, and can hopefully provide some insights into my skills, interests and experiences from the past year.
]]></content>
<link rel="alternate" href="https://bret.io/jobs/littlstar/"/>
</entry>
<entry>
<id>https://bret.io/projects/package-automation/#2020-09-29T17:50:58.562Z</id>
<title>Fully Automated Luxury Space Age Package Maintenance</title>
<updated>2020-09-29T17:50:58.562Z</updated>
<published>2020-09-29T17:50:58.562Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[tldr; The full package maintenance life cycle should be automated and can be broken down into the following levels of automation sophistication:
<ul>
<li><a href="#level-0"><abbr title="git and Github">Level 0</abbr></a>: <code>git</code> + Github</li>
<li><a href="#level-1"><abbr title="Automated Tests + CI">Level 1</abbr></a>: Automated tests, running in <abbr title="Continuous Integration">CI</abbr></li>
<li><a href="#level-2"><abbr title="Dependency Bots">Level 2</abbr></a>: Use a dependency bot to keep on top of maintenance chores</li>
<li><a href="#level-3"><abbr title="Automated Changelog and Release Consistency Scripts">Level 3</abbr></a>: Automated changelog and release consistency scripts</li>
<li><a href="#level-4"><abbr title="Publishing Bots">Level 4</abbr></a>: Human triggered, bot-run version cutting and publishing</li>
<li><a href="#level-5"><abbr title="Project Generation">Level 5</abbr></a>: Capture common automations into a boiler plate generator</li>
</ul>
These solutions focus on Node.js + npm packages, automated on Github Actions, but the underlying principals are general to any language or automation platform.
<h2 id="background" tabindex="-1">Background</h2>
Maintaining lots of software packages is burdensome.
Scaling open source package maintenance beyond a single contributor who understands the release life cycle is challenging.
Long <code>CONTRIBUTING.md</code> files are often the goto solution, but are easily overlooked.
<figure>
<img src="./shrek-swamp.jpg" alt="Shrek swamp">
<figcaption>A pre-automation package is a lot like the shrek swamp. Fixing things requires a slog through the mud, and its constantly at risk for overgrowth. (<a href="https://shrek.fandom.com/wiki/Swamp">Img Source</a>)</figcaption>
</figure>
In the end, automating the package life cycle so that it can maintain itself, is the only way to realistically scale a large set of packages in a maintainable way.
<figure>
<img src="./forerunner_structure.jpg" alt="Forerunne Archetecture">
<figcaption>A fully automated luxury space age package maintains itself, re-activating after years of abandonment to operate the same as it did the day of its creation. This is about as much value I can get out of this silly analogy. (<a href="https://lortarkam.wordpress.com/2017/04/12/how-should-the-forerunners-really-look/">Img Source</a>)</figcaption>
</figure>
For a long time I didn’t seek out automation solutions for package maintenance beyond a few simple solutions like testing and <abbr title="Continuous Integration">CI</abbr>.
Instead I had a lengthly ritual that looked approximately like this:
<pre><code class="hljs language-console"># 🔮
git checkout -b my-cool-branch
# do some work
# update tests
# update docs
npm run test
git commit -am 'Describe the changes'
git push -u
hub browse
# do the PR process
# merge the PR
git checkout master
git pull
git branch --delete my-cool-branch
# hand edit changelog
git add CHANGELOG.md
git commit -m 'CHANGELOG'
npm version {major,minor,patch}
git push && git push --follow-tags
npx gh-release
npm publish
# 😅
</code></pre>
It was ritual, a muscle memory.
Over the years, I’ve managed to automate away large amount of raw labor to various bots, tools and platforms that tend to build on one another and are often usable in isolation or adopted one at a time.
I’ve broken various tools and opportunities for automation into levels, with each level building on the complexity contained in the level below.
<h2 id="level-0%3A-git-and-github" tabindex="-1"><abbr title="git and Github">Level 0</abbr>: <code>git</code> and Github</h2>
You are already automating your packages to a large extent by your use of <a href="https://git-scm.com"><code>git</code></a>.
<code>git</code> automates the process of working on code across multiple computers and collaborating on it with other people, and <a href="https://github.com">Github</a> is the central platform to coordinate and distribute that code.
<figure>
<img src="./git.gif" alt="The old git header">
<figcaption>The old git mascot knew what was up.</figcaption>
</figure>
If you are new to programming or learning <code>git</code>, its helpful to understand you are learning a tool used to automate the process by which you can cooperatively work on code with other people and bots.
<figure class="borderless">
<picture>
<source srcset="./fork-dark.png" media="(prefers-color-scheme: dark)">
<img src="./fork-light.png" alt="Screenshot of Fork.app">
</picture>
<figcaption>Use the <a href="https://git-fork.com">tree view</a> Luke!</figcaption>
</figure>
This isn’t an article about <code>git</code> though, so I won’t dive more into that.
<h2 id="level-1%3A-automated-tests-%2B-ci" tabindex="-1"><abbr title="Automated Tests + CI">Level 1</abbr>: Automated Tests + <abbr title="Continuous Integration">CI</abbr></h2>
There is no debate.
Software isn’t “done” until it has tests.
The orthodox position is that you shouldn’t be allowed to write code until the tests are written (<a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a>).
No matter the methodology, you are automating a verification process of the package that you would normally have to perform by hand.
<h3 id="test-runners" tabindex="-1">Test runners</h3>
These are my preferred test runners for Node.js:
<ul>
<li><a href="https://ghub.io/tape">tape</a>
<ul>
<li>No “test harness”.</li>
<li>Simple, reliable. Just a library, with a CLI that supports <a href="https://ghub.io/glob">globbing</a>.</li>
<li>It’s done. No surprises.</li>
</ul>
</li>
<li><a href="https://ghub.io/tap">tap</a>
<ul>
<li>Similar to tape.</li>
<li>Test runner defaults to improved default test globbing (e.g. <code>foo.test.js</code>).</li>
<li>async/await support</li>
<li>Built in <a href="https://github.com/istanbuljs/nyc">NYC</a> support.</li>
<li>Bad native ESM support.</li>
<li>Still room for evolution.</li>
</ul>
</li>
</ul>
<figure class="borderless">
<img src="./tap.png" alt="Screenshot of tap code">
<figcaption>tap offers some nice test runner features and better async support.</figcaption>
</figure>
<h3 id="additional-tests" tabindex="-1">Additional Tests</h3>
Unit tests that you run with a test runner are not the only type of test though.
There are lots of other easy tests you can throw at your package testing step that provide a ton of value:
<ul>
<li><a href="https://en.wikipedia.org/wiki/Lint_(software)">Linting</a> (<a href="https://standardjs.com"><code>standard</code></a>, <a href="https://eslint.org"><code>eslint</code></a> etc)
<ul>
<li>Enforce code style and catch errors! Basically spell check on code. This should really be the very first test on every package, since it helps you write correct code.</li>
</ul>
</li>
<li><a href="https://github.com/dependency-check-team/dependency-check"><code>dependency-check</code></a>
<ul>
<li>Verify dependencies listed in <code>package.json</code> matches usage in the actual code.</li>
<li>This test fails if it finds packages in <code>package.json</code> that are no longer used in code or if it finds pacakges in use in the code that are not listed in <code>package.json</code> which would cause a runtime error.</li>
</ul>
</li>
<li>Running a build
<ul>
<li>Simply make <code>npm run build</code> part of your test life cycle.</li>
<li>If your package has some sort of build step, simply running a build and checking if it runs without error is a very helpful test. It’s also simple to set up.</li>
</ul>
</li>
<li>Code Coverage metrics
<ul>
<li><a href="https://github.com/istanbuljs/nyc">nyc</a> is the defacto test coverage tool (built into tap).</li>
<li><a href="https://github.com/bcoe/c8">c8</a> is a native V8 coverage tool, but is lest robust than nyc.</li>
<li>These don’t need to be pass/fail in terms of coverage going up or down (but they can be).</li>
<li>These metrics help you validate the tests you wrote are actually running the parts of the code you expect (or don’t expect).</li>
<li>Platform integration for ingesting these metrics are available, but usually not worth it. Simply having console output is sufficient most of the time.
<ul>
<li><a href="https://coveralls.io">coveralls.io</a> is a nice indie service that provides coverage metrics reporting.</li>
</ul>
</li>
</ul>
</li>
</ul>
<figure class="borderless">
<picture>
<source srcset="./coveralls-dark.png" media="(prefers-color-scheme: dark)">
<img src="./coveralls-light.png" alt="Screenshot of Coveralls">
</picture>
<figcaption>Platforms like coveralls provide nice coverage tracking, but the cost of setting up integrations makes them optional.</figcaption>
</figure>
<h3 id="managing-complex-test-scripts" tabindex="-1">Managing complex test scripts</h3>
Running multiple tests under <code>npm test</code> can result in a long, difficult to maintain <code>test</code> script. Install <a href="https://github.com/bcomnes/npm-run-all2"><code>npm-run-all2</code></a> as a <code>devDependency</code> to break each <code>package.json</code> <code>test</code> command into its own sub-script, and then use the globbing feature to run them all in parallel (<code>run-p</code>) or in series (<code>run-s</code>):
<pre><code class="hljs language-json">{
"scripts": {
"test": "run-s test:*",
"test:deps": "dependency-check . --no-dev --no-peer",
"test:standard": "standard",
"test:tap": "tap"
}
}
</code></pre>
When testing locally, and an individual test is failing, you can bypass the other tests and run just the failing test:
<pre><code class="hljs language-console"># run just the dep tests
npm run test:deps
</code></pre>
<a href="https://github.com/bcomnes/npm-run-all2"><code>npm-run-all2</code></a> is a fantastic tool to keep your <a href="https://docs.npmjs.com/misc/scripts"><code>npm run</code></a> scripts manageable.
This builds on the fantastic and 2014 classic Keith Cirkel blog post <a href="https://www.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool/">How to Use npm as a Build Tool</a>.
<h3 id="actually-automating-the-tests" tabindex="-1">Actually automating the tests</h3>
While its obvious that writing automated tests is a form of automation, its still very common to see projects not take the actual step of automating the run step of the tests by hooking them up to a <abbr title="Continuous Integration">CI</abbr> system that runs the tests on every interaction with the code on Github.
Services like <a href="https://travis-ci.org">TravisCI</a> have been available for FREE for years, and there is literally no valid excuse not to have this set up.
Although <a href="https://travis-ci.org">TravisCI</a> has served many projects well over the years, <a href="https://github.com/features/actions">Github Actions</a> is a newer and platform native solution that many projects are now using. Despite the confusing name, Github Actions is primarily a <abbr title="Continuous Integration">CI</abbr> service.
<figure class="borderless">
<picture>
<source srcset="./actions-dark.png" media="(prefers-color-scheme: dark)">
<img src="./actions-light.png" alt="Screenshot of Github actions">
</picture>
<figcaption>Github actions has its own oddities, but overall its proven to be an extremely flexible and useful CI service, built right into the platform you are using if you are doing any kind of open source.</figcaption>
</figure>
Create the following action file in your package repo and push it up to turn on <abbr title="Continuous Integration">CI</abbr>.
<pre><code class="hljs language-yaml"># .github/workflows/tests.yml
name: tests
on: [push]
jobs:
test:
runs-on: $
strategy:
matrix:
os: [ubuntu-latest]
node: [14]
steps:
- uses: actions/checkout@v2.3.2
- name: Use Node.js $
uses: actions/setup-node@v2.1.1
with:
node-version: $
- run: npm i
- run: npm test
</code></pre>
For more information on Action syntax and directives, see:
<ul>
<li><a href="https://docs.github.com/en/actions">Actions Docs</a></li>
<li><a href="https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions">Actions context reference</a></li>
<li><a href="https://docs.github.com/en/actions/learn-github-actions/managing-complex-workflows">Actions examples</a></li>
</ul>
<h3 id="automated-checks" tabindex="-1">Automated Checks</h3>
Once you have a test suite set up, running in <abbr title="Continuous Integration">CI</abbr>, any pull request to your package features the results of the “Checks API”.
Various tests and integrations will post their results on every change to the pull request in the form of “running”, “pass” or “fail”.
The benefit to the checks status in the pull request UI is, depending on the quality and robustness of your test suite, you can have some amount of confidence that you can safely merge the proposed changes, while still having things work the way you expect, including the newly proposed changes.
No matter the reliability of the test suite, it is still important to read and review the code.
<figure class="borderless">
<picture>
<source srcset="./checks-dark.png" media="(prefers-color-scheme: dark)">
<img src="./checks-light.png" alt="Screenshot of Checks API UI">
</picture>
</figure>
<h2 id="level-2%3A-dependency-bots" tabindex="-1"><abbr title="Dependency Bots">Level 2</abbr>: Dependency Bots</h2>
Your package has dependences. Be it your test runner, or other packages imported or required into your package. They help provide valuable function with little upfront cost.
Dependencies form the foundation that your package is built upon. But that foundation is made of shifting sands⏳.
Dependencies have their own dependencies, which all have to slowly morph and change with the underlying platform and dependency changes.
Like a garden, if you don’t tend to the weeds and periodically water it with dependency updates, the plants will die.
With <code>npm</code>, you normally update dependencies by grabbing the latest copy of the code and <a href="https://docs.npmjs.com/cli-commands/outdated.html">checking for outdated</a> packages:
<pre><code class="hljs language-console">git checkout master
git pull
rm -rf node_modules
npm i
npm outdated
</code></pre>
<figure class="borderless">
<picture>
<source srcset="./outdated-dark.png" media="(prefers-color-scheme: dark)">
<img src="./outdated-light.png" alt="Screenshot of npm outdated output">
</picture>
<figcaption><a href="https://docs.npmjs.com/cli-commands/outdated.html"><code>npm outdated</code></a> will give you a list of your dependencies that have fallen behind their semver range and updates that are available outside of their semver range.</figcaption>
</figure>
Checking for updates any time you work on the package is not a bad strategy, but it becomes tiresome, and can present a large amount of maintenance work, unrelated to the prompting task at hand, if left to go a long time. A good package doesn’t need to change much, so it may rarely ever be revisited and rot indefinitely.
Enter dependency bots.
A dependency bot monitors your package repositories for dependency updates.
When a dependency update is found, it automatically creates a <abbr title="Pull Request">PR</abbr> with the new version.
If you have <abbr title="Automated Tests + CI">Level 1</abbr> automation setup, this <abbr title="Pull Request">PR</abbr> will run your tests with the updated version of the dependency.
The results will (mostly) inform you if its safe to apply the update, and also give you a button to press to apply the change.
No typing or console required! 🚽🤳
<abbr title="Automated Tests + CI">Level 1</abbr> automation isn’t required to use a dependency bot, but you won’t have any way to automatically validate the change, so they are much less useful in that case.
<h3 id="dependabot" tabindex="-1">Dependabot</h3>
<img height="275" src="./dependabot.svg">
Github now has a dependency bot built in called <a href="https://docs.github.com/en/github/administering-a-repository/enabling-and-disabling-version-updates">dependabot</a>.
To turn it on, create the following file in your packages Github repo:
<pre><code class="hljs language-yaml"># .github/dependabot.yml
version: 2
updates:
# Enable version updates for npm
- package-ecosystem: "npm"
# Look for `package.json` and `lock` files in the `root` directory
directory: "/"
# Check the npm registry for updates every day (weekdays)
schedule:
interval: "daily"
# Enable updates to github actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
</code></pre>
This enables updates for npm and github actions. It offers other ecosystems as well. See the <a href="https://docs.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates">dependabot docs</a> for more info.
<h3 id="in-range-breaking-change-%F0%9F%9A%A8" tabindex="-1">In-range breaking change 🚨</h3>
Before dependabot, there was a now-shut-down service called <a href="https://greenkeeper.io">Greenkeeper.io</a> which provided a very similar service. It offered a very interesting feature which I’m still not sure if dependabot has yet.
<figure class="borderless">
<picture>
<source srcset="./in-range-dark.png" media="(prefers-color-scheme: dark)">
<img src="./in-range-light.png" alt="Screenshot of Greenkeepers in-range breaking change PRs">
</picture>
<figcaption>Greenkeeper would run continuous checks for every new dependency in an and out of your packages semver range, proactively identifying breaking changes that could sneak in.</figcaption>
</figure>
It would run tests every time a dependency in your package was updated, in and out of <abbr title="Semantic Version">semver</abbr> range.
For in range updates that passed, nothing would happen.
For in range updates that failed, it would open a <abbr title="Pull Request">PR</abbr> alerting you that one of your dependencies inadvertently released a breaking change as a non-breaking change.
This was a fantastic feature, and really demonstrated the heights that automated tests, <abbr title="Continuous Integration">CI</abbr>, an ecosystem that fully utilized <abbr title="Semantic Version">semver</abbr> and dependency bots could achieve together.
Sadly, I haven’t seen other services or languages quite reach these heights of automation sophistication (many ecosystems even lack any kind of <code>major</code> version gate rage), but perhaps as awareness increases of these possibilities more people will demand it.
There is a lot more room for innovation in this space. It would be great to get get periodic reports regarding the health of downstream dependency chains (e.g. if you are depending on a project that is slowly rotting and not maintaining its deps).
As of now, dependabot seems to be recovering from a <a href="https://twitter.com/bcomnes/status/1303421464910192641">post acquisition engineering fallout</a>, but I hope that they can get these kinds of features back into reality sooner than later.
<h3 id="linter-bots%3F" tabindex="-1">Linter bots?</h3>
There are bots out there that can send in automated code changes source off lint tests and other code analysis tools. While this is ‘cool’, these tasks are better served at the testing level.
Automated code changes for failing lint tests should really just be apart of the human development cycle with whatever IDE or editor they use.
Still, the bot layer is open to experimentation, so go forth an experiment all you want, though note that external service integrations have a heavy integration cost usually. 🤖
<h2 id="level-3%3A-automated-changelog-and-release-consistency-scripts" tabindex="-1"><abbr title="Automated Changelog and Release Consistency Scripts">Level 3</abbr>: Automated Changelog and Release Consistency Scripts</h2>
Quick recap:
<ul>
<li><a href="#level-0"><abbr title="git and Github">Level 0</abbr></a>: we’ve automated collaboration and are receiving patch PRs from humans.</li>
<li><a href="#level-1"><abbr title="Automated Tests + CI">Level 1</abbr></a>: We automatically validate code with a test suite that runs on every <abbr title="Pull Request">PR</abbr>.</li>
<li><a href="#level-2"><abbr title="Dependency Bots">Level 2</abbr></a>: We have a bot sending in patches for tedious dependency management tasks.</li>
</ul>
That means our package is going to morph and change with time (hopefully not too much though). We need a way way to communicate that clearly to downstream dependents, be that us, someone else on the team or a large base of <a href="https://www.hanselman.com/blog/DarkMatterDevelopersTheUnseen99.aspx">dark-matter developers</a>.
The way to do this is with a CHANGELOG.md file. Or release notes in a Github release page. Or ideally both. <a href="https://keepachangelog.com/en/1.0.0/">keepachangelog.com</a> offers a good overview on the correct format a CHANGELOG.md should follow.
<figure class="borderless">
<picture>
<source srcset="./changelog-dark.png" media="(prefers-color-scheme: dark)">
<img src="./changelog-light.png" alt="Screenshot of various changelog specs">
</picture>
<figcaption><a href="https://keepachangelog.com">keepachangelog.com</a>, <a href="https://www.conventionalcommits.org/en/v1.0.0/">Conventional Commits</a> and <a href="https://semantic-release.gitbook.io/semantic-release/">Semantic Release</a> are all conventions and systems for documenting changes to a project.</figcaption>
</figure>
This is a tedious process. If you work with other people, they might not be as motivated as you to handcraft an artisan CHANGELOG file. In my experience, the handcrafted, artisan CHANGELOG is too much work and easy to forget about. Also, I haven’t found a good linter tool to enforce its maintenance.
<h3 id="auto-changelog" tabindex="-1"><a href="https://ghub.io/auto-changelog"><code>auto-changelog</code></a></h3>
<a href="https://ghub.io/auto-changelog"><code>auto-changelog</code></a> is a tool that takes your git history and generates a CHANGELOG that is almost-just-as-good as the artisan handcrafted one. Hooking this tool into your package’s <a href="https://docs.npmjs.com/misc/scripts"><code>version</code> life cycle</a> enforces that it is run when a new version is generated with <code>npm version {major,minor,patch}</code>.
While <a href="https://keepachangelog.com/en/1.0.0/">keepachangelog.com</a> advocates for the handcrafted version, and discourages ‘git commit dumps’, as long as you are halfway concious of your <code>git</code> commit logs (as you should be), the <code>auto-changelog</code> output is generally still useful.
You can even follow <a href="https://www.conventionalcommits.org/en/v1.0.0/">conventionalcommits.org</a> if you want an even more structured git log.
Automating <code>auto-changelog</code> to run during <code>npm version</code><a href="#fn1" id="fnref1">[1]</a> is easy.
Install it as a <code>devDependency</code> and set up the following script in <code>package.json</code>:
<pre><code class="hljs language-json">{
"scripts": {
"version": "auto-changelog -p --template keepachangelog auto-changelog --breaking-pattern 'BREAKING CHANGE:' && git add CHANGELOG.md"
}
}
</code></pre>
The <code>version</code> script is a <code>npm run</code> lifecycle script that runs after the <code>package.json</code> version is bumped, but before the git commit with the change are created. Kind of a mouthful, but with nice results.
<figure class="borderless">
<picture>
<source srcset="./changelog-example-dark.png" media="(prefers-color-scheme: dark)">
<img src="./changelog-example-light.png" alt="Screenshot of an auto-changelog changelog.">
</picture>
<figcaption><a href="https://ghub.io/auto-changelog"><code>auto-changelog</code></a> generates satisfactory changelogs. The consistency it provides exceeds the value a hand written changelog can provide due to its inconsistent nature.</figcaption>
</figure>
<h3 id="publishing-consistency-scripts" tabindex="-1">Publishing consistency scripts</h3>
Ok, so we have local changes merged in, and we’ve created a new version of the module with an automated changelog generated as part of the <code>npm version</code> commit. Time to get this all pushed out and published! By hand we could do:
<pre><code class="hljs language-console">git push --follow-tags
# copy contents of changelog
# create a new Github release on the new tag with the changelog contents
npm publish
</code></pre>
But that is tedious.
And what happens when your colleague forgets to push the git commit/tag to Github and just publishes to <code>npm</code>?
Or more likely, they just forget to create the Github release, creating inconsistency in the release process.
The solution is to automate all of this!
Use the <a href="https://docs.npmjs.com/misc/scripts"><code>prepublishOnly</code></a> hook to run all of these tasks automatically before publishing to npm via <code>npm publish</code>.
Incorporate a tool like <a href="https://ghub.io/gh-release"><code>gh-release</code></a> to create a Github release page for the new tag with the contents of your freshly minted <a href="https://ghub.io/auto-changelog"><code>auto-changelog</code></a>.
<pre><code class="hljs language-json">{
"scripts": {
"prepublishOnly": "git push --follow-tags && gh-release -y"
}
}
</code></pre>
<figure>
<video controls width="100%" preload="metadata">
<source src="./gh-release-demo.mp4#t=0.5" type="video/mp4">
</video>
<figcaption><a href="https://ghub.io/gh-release"><code>gh-release</code></a> makes it easy to create Github releases from a CHANGELOG.md.</figcaption>
</figure>
The result of this is our release process is returned to the lowest common denominator of process dictated by npm:
<pre><code class="hljs language-console">npm version {major,minor,patch}
npm publish
</code></pre>
But we still get all of these results, completely automated:
<ul>
<li>Automated changelog updating via <a href="https://ghub.io/auto-changelog"><code>auto-changelog</code></a></li>
<li>Automated version bumping and creating a new version <code>git</code> commit+tag.</li>
<li>Automated pushing to Github (including new tags)</li>
<li>Automated Github release creation via <a href="https://ghub.io/gh-release"><code>gh-release</code></a>, with the new contents of CHANGELOG</li>
<li>Published assets to <code>npm</code>.</li>
</ul>
<h3 id="all-together" tabindex="-1">All together</h3>
Those two run scripts together:
<pre><code class="hljs language-json">{
"scripts": {
"version": "auto-changelog -p --template keepachangelog auto-changelog --breaking-pattern 'BREAKING CHANGE:' && git add CHANGELOG.md",
"prepublishOnly": "git push --follow-tags && gh-release -y"
}
}
</code></pre>
<h4 id="extending-with-a-build-step" tabindex="-1">Extending with a build step</h4>
Some packages have builds steps. No problem, these are easily incorporated into the above flow:
<pre><code class="hljs language-json">{
"scripts": {
"build": "do some build command here",
"prepare": "npm run build",
"version": "run-s prepare version:*",
"version:changelog": "auto-changelog -p --template keepachangelog auto-changelog --breaking-pattern 'BREAKING CHANGE:'",
"version:git": "git add CHANGELOG.md dist",
"prepublishOnly": "git push --follow-tags && gh-release -y"
}
}
</code></pre>
Since version becomes a bit more complex, we can break it down into pieces with <a href="https://github.com/bcomnes/npm-run-all2"><code>npm-run-all2</code></a> as we did in the testing step. We ensure we run fresh builds on development install (<code>prepare</code>), and also when we <code>version</code>. We capture any updated build outputs in <code>git</code> during the version step by staging the <code>dist</code> folder (or whatever else you want to capture in your <code>git</code> version commit).
This pattern was documented well by <a href="http://twitter.com/swyx">@swyx</a>: <a href="https://dev.to/swyx/semi-automatic-npm-and-github-releases-with-gh-release-and-auto-changelog-4b5a">Semi-Automatic npm and GitHub Releases with <code>gh-release</code> and <code>auto-changelog</code></a>.
<h2 id="level-4%3A-publishing-bots-%F0%9F%A4%96" tabindex="-1"><abbr title="Publishing Bots">Level 4</abbr>: Publishing Bots 🤖</h2>
We now have a process for fully managing the maintenance and release cycle of our package, but we are still left to pull down any changes from our Github repo and run these release commands, as simple as they are now.
You can’t really do this on your phone (easily) and someone else on the project still might manage to not run <code>npm version</code> and just hand-bump the version number for some reason, bypassing all our wonderful automation.
What would be cool is if we could kick of a special <abbr title="Continuous Integration">CI</abbr> program that would run <code>npm version && npm publish</code> for us, at the push of a button.
It turns out Github-Actions has a feature now called <a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#workflow_dispatch"><code>workflow_dispatch</code></a>, which lets you press a button on the repos actions page on GitHub and trigger a <abbr title="Continuous Integration">CI</abbr> flow with some input.
<figure class="borderless">
<picture>
<source srcset="./workflow-dispatch-dark.png" media="(prefers-color-scheme: dark)">
<img src="./workflow-dispatch-light.png" alt="Screenshot of workflow dispatch UI">
</picture>
<figcaption><a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#workflow_dispatch"><code>workflow_dispatch</code></a> actions lets you trigger an action from your browser, with simple textual inputs. Use it as a simple shared deployment environment.</figcaption>
</figure>
Implementing <a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#workflow_dispatch"><code>workflow_dispatch</code></a> is easy: create a new action workflow file with the following contents:
<pre><code class="hljs language-yaml"># .github/workflows/release.yml
name: npm version && npm publish
on:
workflow_dispatch:
inputs:
newversion:
description: 'npm version {major,minor,patch}'
required: true
env:
node_version: 14
jobs:
version_and_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.3.2
with:
# fetch full history so things like auto-changelog work properly
fetch-depth: 0
- name: Use Node.js $
uses: actions/setup-node@v2.1.1
with:
node-version: $
# setting a registry enables the NODE_AUTH_TOKEN env variable where we can set an npm token. REQUIRED
registry-url: 'https://registry.npmjs.org'
- run: npm i
- run: npm test
- run: git config --global user.email "user@email.com"
- run: git config --global user.name " $"
- run: npm version $
- run: npm publish
env:
GH_RELEASE_GITHUB_API_TOKEN: $
NODE_AUTH_TOKEN: $
</code></pre>
Next, <a href="https://docs.npmjs.com/creating-and-viewing-authentication-tokens">generate an npm token</a> with publishing rights.
Then set that token as a <a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets">repo secret</a> called <code>NPM_TOKEN</code>.
<figure class="borderless">
<picture>
<source srcset="./secrets-dark.png" media="(prefers-color-scheme: dark)">
<img src="./secrets-light.png" alt="Screenshot of Github secrets">
</picture>
<figcaption><a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets"><code>GitHub</code></a> secrets allows you to securely store tokens for use in your GitHub Actions runs. It's not bulletproof, bit its pretty good.</figcaption>
</figure>
Now you can visit the actions tab on the repo, select the <code>npm version && npm publish</code> action, and press run, passing in either <code>major</code>, <code>minor</code>, or <code>patch</code> as the input, and a GitHub action will kick off running our <abbr title="Automated Changelog and Release Consistency Scripts">Level 3</abbr> version and release automations along with publishing a release to npm and GitHub.
Note: Its recommended that you <code>.gitignore</code> <code>package-lock.json</code> files, otherwise they end up in a library source, where it provides little benefit and lots of drawbacks.
I created a small actions called <a href="https://github.com/marketplace/actions/npm-bump"><code>npm-bump</code></a> which can clean up some of the above action boilerplate:
<pre><code class="hljs language-yaml">name: Version and Release
on:
workflow_dispatch:
inputs:
newversion:
description: 'npm version {major,minor,patch}'
required: true
env:
node_version: 14
jobs:
version_and_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.3.2
with:
# fetch full history so things like auto-changelog work properly
fetch-depth: 0
- name: Use Node.js $
uses: actions/setup-node@v2.1.1
with:
node-version: $
# setting a registry enables the NODE_AUTH_TOKEN env variable where we can set an npm token. REQUIRED
registry-url: 'https://registry.npmjs.org'
- run: npm i
- run: npm test
- name: npm version && npm publish
uses: bcomnes/npm-bump@v2.0.1
with:
git_email: bcomnes@gmail.com
git_username: $
newversion: $
github_token: $ # built in actions token. Passed tp gh-release if in use.
npm_token: $ # user set secret token generated at npm
</code></pre>
<figure class="borderless">
<picture>
<source srcset="./npm-bump-dark.png" media="(prefers-color-scheme: dark)">
<img src="./npm-bump-light.png" alt="Screenshot of npm-bump in the marketplace.">
</picture>
<figcaption><a href="https://github.com/marketplace/actions/npm-bump"><code>npm-bump</code></a> helps cut down on some of the npm version and release GitHub action boilerplate YAML. Is it better? Not sure!</figcaption>
</figure>
So this is great! You can maintain packages by merging automatically generated pull requests, run your tests on them to ensure package validity, and when you are ready, fully release the package, with a CHANGELOG entry, all from the push of a button on your cell phone. Fully Automated Luxury Package Space Maintenance. 🛰🚽🤳
<h2 id="level-5%3A-project-generation" tabindex="-1"><abbr title="Project Generation">Level 5</abbr>: Project generation</h2>
What is the best way to manage all of these independent pieces? A template! Or, a template repo.
You mean things like <a href="https://yeoman.io">yeoman</a>? Maybe, though that tool is largely used to ‘scaffold’ <a href="https://github.com/facebook/create-react-app">massive amounts of web framework boilerplate</a> and is a complex ecosystem.
Something simpler will be more constrained and easier to maintain over time. <a href="https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/creating-a-template-repository">Github repo templates</a> and <a href="https://ghub.io/create-project"><code>create-project</code></a> are good choices.
<h3 id="github-template-repos" tabindex="-1">Github Template Repos</h3>
Github offers a very simple solution called <a href="https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/creating-a-template-repository">template repos</a>. You take any repo on GitHub, go into its settings page and designate it as a template repo. You can create a new repo from a template repo page with the click of a button, or select it from a drop down in the github <a href="https://github.com/new">create repo wizard</a>.
<figure class="borderless">
<picture>
<source srcset="./gh-template-dark.png" media="(prefers-color-scheme: dark)">
<img src="./gh-template-light.png" alt="Screenshot of creating a repo from a template.">
</picture>
<figcaption>Repos that are designated as templates show up in the new repo UI.</figcaption>
</figure>
The only issue, is that you then have to go through and modify all the repo specific parameters by hand.
Better than nothing!
But we can do better.
<h3 id="create-project-repos" tabindex="-1"><a href="https://ghub.io/create-project"><code>create-project</code></a> repos</h3>
<a href="https://ghub.io/create-project"><code>create-project</code></a> is a simple CLI tool (by <a href="https://github.com/mafintosh">@mafintosh</a>) that works similar to Github template repos, except it has a `` system that lets you insert values when spawning off a project repo. You can designate your <a href="https://ghub.io/create-project"><code>create-project</code></a> template repos to also be a Github template repo, and create new projects either way you feel like it.
<figure>
<video controls width="100%" preload="metadata">
<source src="./create-project.mp4#t=0.5" type="video/mp4">
</video>
<figcaption><a href="https://ghub.io/create-project"><code>create-project</code></a> lets you spawn a new project from a git repo and inject values into special <code></code> blocks.</figcaption>
</figure>
Here are some of my personal template repos:
<ul>
<li><a href="https://github.com/bcomnes/templates/tree/5fd64ef3e919d5e9e0d64bec6fd176c2f746cf1a">bcomnes/templates</a> - simple template for Node.js CJS code.</li>
<li><a href="https://github.com/bcomnes/esm-template">bcomnes/esm-template</a> - template that includes complexity to publish an ESM module to npm with CJS fall back support. A topic for a future blog post!</li>
<li><a href="https://github.com/bcomnes/go-template">bcomnes/go-template</a> - a <a href="https://golang.org">Go</a> package boilerplate template that includes versioned dev tooling support. <code>create-project</code> doesn’t need to only manage boilerplate for Node.js projects. Maybe a go tool would be better for this, but it does show the flexibility of the tools.</li>
</ul>
<h2 id="what-about-docs%3F" tabindex="-1">What about docs?</h2>
There are various solutions for generating docs from comments that live close to the code.
<ul>
<li><a href="https://jsdoc.app">jsdoc</a></li>
<li><a href="https://typedoc.org">typedoc</a></li>
<li><a href="https://documentation.js.org">documentation.js</a></li>
<li><a href="https://medium.com/@trukrs/type-safe-javascript-with-jsdoc-7a2a63209b76">Type Safe JavaScript with JSDoc</a></li>
<li><a href="https://voxpelli.com/2019/10/use-type-script-3-7-to-generate/">Pelle Wessman on JSDoc-comments</a></li>
<li><a href="https://doc.deno.land/builtin/stable">deno doc</a></li>
</ul>
I haven’t found a solution that satisfies my needs well enough to use one generally.
Its hard to exceed the quality docs written by hand for and by humans.
If you have a good solution please let me know!
<figure class="borderless">
<picture>
<source srcset="./deno-doc-dark.png" media="(prefers-color-scheme: dark)">
<img src="./deno-doc-light.png" alt="Screenshot of deno doc.">
</picture>
<figcaption><a href="https://doc.deno.land">deno doc</a> is beautiful and keeps code and docs tightly coupled. But is the human consumed result better? Or does it just lead to cutting corners in exchange for static type checking? Time will tell!</figcaption>
</figure>
<h2 id="all-together-now" tabindex="-1">All together now</h2>
That was a lot to cover.
If you want to see a complete level 0 through level 5 example, check out my <code>create-template</code> <a href="https://github.com/bcomnes/templates/tree/5fd64ef3e919d5e9e0d64bec6fd176c2f746cf1a">template repo</a> snapshotted to the latest commit of the time of publishing.
<figure>
<video controls width="100%" preload="metadata">
<source src="./publish-flow.mp4#t=0.5" type="video/mp4">
</video>
<figcaption>Here we can see the various parts of level 0-4 automation in action together. Changes are made to the git history, the module is versioned and published by a bot. It includes a changelog and was automatically validated by tests.</figcaption>
</figure>
<h2 id="final-thoughts" tabindex="-1">Final thoughts</h2>
This collection is written in the context of the Node.js programming system, however the class of tools discussed apply to every other language ecosystem and these automation levels could serve as a framework for assessing the maturity of automation capabilities of other programming language systems.
Hopefully they can can provide some insights into the capabilities and common practices around modern JavaScript development for those unfamiliar with this ecosystem.
Additionally, this documents my personal suite of tools and processes that I have developed to automate package maintenance, and is by no means normative. Modification and experimentation is always encouraged.
There are many subtle layers to the Node.js programming system, and this just covers the maintenance automation layer that can exist around a package.
Much more could be said about the versioned development tooling, standardized scripting hooks, diamond dependency problem solutions, localized dependencies, upstream package hacking/debugging conveniences and local package linking. An even deeper dive could be made on the overlap these patterns have (and don’t) have in other JS runtimes like <a href="https://deno.com">Deno</a> which standardizes a lot around <abbr title="Automated Tests + CI">Level 1</abbr>, or even other languages like Go or Rust.
If you enjoyed this article, have suggestions or feedback, or think I’m full of it, follow me on twitter (<a href="https://twitter.com/bcomnes">@bcomnes</a>) and feel free to hop in the the accompanying thread. I would love to hear your thoughts, ideas and examples! Also subscribe to my <a href="http://bret-dk.local:3000/feed.xml">RSS</a>/<a href="http://bret-dk.local:3000/feed.json">JSON</a> Feed in your favorite RSS reader.
<blockquote class="twitter-tweet">"Fully Automated Luxury Space Age Package Maintenance" I wrote up how tedious package maintenance tasks can be fully automated. Hope someone enjoys!<a href="https://t.co/fvYIu2Wq0r">https://t.co/fvYIu2Wq0r</a> <a href="https://t.co/q220LTax8X">pic.twitter.com/q220LTax8X</a>— 🌌🌵🛸Bret🏜👨‍👩‍👧🚙 (@bcomnes) <a href="https://twitter.com/bcomnes/status/1311034520305569800?ref_src=twsrc%5Etfw">September 29, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h3 id="syndications" tabindex="-1">Syndications</h3>
<ul>
<li><a href="https://www.reddit.com/r/node/comments/j27pbt/fully_automated_luxury_space_age_package/" rel="syndication" class="u-syndication">/r/node</a></li>
<li><a href="https://github.community/t/fully-automated-luxury-space-age-package-maintenance-with-github-actions/135103" rel="syndication" class="u-syndication">Github Community Forum</a></li>
<li><a href="https://twitter.com/bcomnes/status/1311034520305569800" rel="syndication" class="u-syndication">twitter thread</a></li>
</ul>
<h3 id="see-also" tabindex="-1">See also</h3>
<ul>
<li><a href="https://www.swyx.io/release-automation">swyx.io/release-automation</a></li>
</ul>
<hr class="footnotes-sep">
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item">Just as a refresher, <code>npm version</code> is a command that bumps the version in <code>package.json</code>, creates a commit with that change titled <code>0.0.0</code>, then tags it <code>v0.0.0</code>. See <a href="https://docs.npmjs.com/misc/scripts">npm</a> docs for more info. <a href="#fnref1" class="footnote-backref">↩︎</a>
</li>
</ol>
</section>
]]></content>
<link rel="alternate" href="https://bret.io/projects/package-automation/"/>
</entry>
<entry>
<id>https://bret.io/jobs/netlify/#2020-01-16T22:19:35.001Z</id>
<title>Netlify Portfolio</title>
<updated>2020-01-16T22:19:35.001Z</updated>
<published>2020-01-16T22:19:35.001Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[I was lucky to be able to contribute to many features and experiences that affected <a href="https://www.netlify.com">Netlify</a>’s massive user base. Here are some examples of things that I worked on. If this kind of work looks interesting to you, Netlify is a fantastic team to join: <a href="https://www.netlify.com/careers/">netlify.com/careers</a>. Questions and comments can be sent via email or <a href="https://twitter.com/bcomnes/status/1111324728277368832">twitter</a>.
<h2 id="platform" tabindex="-1">Platform</h2>
After working on the Product team for slightly over a year, I switched to working on Netlify’s platform team. The team has a strong DevOps focus and maintains a redundant, highly available multi-cloud infrastructure on which Netlify and and all of its services run. My focus on the team is to maintain, develop, scale and improve various critical services and libraries. Below are some examples of larger projects I worked on.
<h3 id="buildbot" tabindex="-1">Buildbot</h3>
One of my primary responsibilities upon joining the team is to maintain the Buildbot that all customer site builds run on. It is partially open-source so customers can explore the container in a more freeform matter locally.
<ul>
<li class="lang docker"><a href="https://github.com/netlify/build-image">netlify/build-image</a> - Docker file which defines the container builds run in.</li>
<li class="lang go"><a href="https://pkg.go.dev/github.com/netlify/open-api/go">netlify/open-api/go</a> - Go open-api client used in the bot.</li>
</ul>
<a href="./buildbot.png"><img src="./buildbot.png" alt="screenshot of buildbot logs"></a>
<h4 id="selectable-build-images" tabindex="-1"><a href="https://www.netlify.com/blog/2019/03/14/a-more-flexible-build-architecture-with-updated-linux/">Selectable build images</a></h4>
One of the first feature additions I launched for the Buidlbot was selectable build images. This project required adding the concept of additional build images to the API and UI and to develop an upgrade path allowing users to migrate their websites to the new build image image wile also allowing them to roll back to the old image if they needed more time to accommodate the update.
Additionally, I performed intake on a number of user contributed build-image additions and merged other various potential breaking changes within a development window before releasing. I also helped develop the changes to the Ruby on Rails API, additions to the React UI, as well as write the user documentation. It was widely cross cutting project.
<a href="https://www.netlify.com/blog/2019/03/14/a-more-flexible-build-architecture-with-updated-linux/"><img src="./build-image-blog.png" alt="screenshot of buildbot image blogpost"></a>
<a href="./image-selection.png"><img src="./image-selection.png" alt="screenshot of buildbot settings"></a>
<h3 id="open-api" tabindex="-1"><a href="https://open-api.netlify.com">Open API</a></h3>
I help maintain and further develop Netlify’s Open-API (aka Swagger) API definition, website and surrounding client ecosystem. While open-api can be cumbersome, it has been a decent way to synchronize projects written in different language ecosystems in a unified way.
<ul>
<li class="lang go"><a href="https://github.com/netlify/open-api">netlify/open-api</a> - API definition and Go client</li>
<li class="lang js"><a href="https://github.com/netlify/js-client">netlify/js-client</a> - Node.JS and Browser JS client</li>
</ul>
<a href="https://open-api.netlify.com"><img src="./open-api-web.png" alt="screenshot of the open-api website"></a>
<h3 id="other-platform-projects-i-work-on" tabindex="-1">Other platform projects I work on</h3>
<ul>
<li class="lang go"><a href="https://github.com/netlify/gotrue">netlify/gotrue</a> - Netlify's Identity service</li>
<li class="lang go"><a href="https://github.com/netlify/gocommerce">netlify/gocommerce</a> - Netlify's Commerce service</li>
<li class="lang js"><a href="https://github.com/netlify/zip-it-and-ship-it">netlify/zip-it-and-ship-it</a> - Netlify's Lambda function packaging algorithm</li>
</ul>
<h2 id="product" tabindex="-1">Product</h2>
I worked on Neltify’s Product team for a bit over a year and completed many successful user facing projects. Here are just a few examples:
<h3 id="cli" tabindex="-1"><a href="https://cli.netlify.com">CLI</a></h3>
I was primary author on Netlify’s current CLI codebase.
<ul>
<li class="lang js"><a href="https://github.com/netlify/cli">netlify/cli</a> - Netlify's extensible CLI</li>
<li class="lang js"><a href="https://github.com/netlify/js-client">netlify/js-client</a> - The API client used to make all API calls in the CLI</li>
<li class="lang js"><a href="https://github.com/netlify/cli-utils">netlify/cli-utils</a> - A common utility class for loading and saving configuration from a CLI command</li>
<li class="lang js"><a href="https://github.com/bcomnes/netlify-lambda-builder">bcomnes/netlify-lambda-builder</a> - I wrote a CLI that packaged Netlify functions in a convenient manner in order to communicate the idea clearly as a possible direction to take.</li>
</ul>
<figure>
<video controls width="100%" preload="metadata">
<source src="./netlify-cli.mp4#t=0.5" type="video/mp4">
</video>
<figcaption>Build image selection UI.</figcaption>
</figure>
<a href="https://www.netlify.com/blog/2018/09/10/netlify-cli-2.0-now-in-beta/"><img src="./cli-blog.png" alt="screenshot of the cli announcement blog post"></a>
<h3 id="jamstack-slides" tabindex="-1"><a href="https://brets-jamstack-pdx-slides.netlify.app/#0">JAMStack slides</a></h3>
I gave a talk on some ideas and concepts I came across working with Netlify’s platform and general JAMStack architecture at a local Portland JAMStack meetup.
<ul>
<li class="lang html"><a href="https://github.com/bcomnes/jamstack-pdx-slides">bcomnes/jamstack-pdx-slides</a></li>
</ul>
<a href="https://brets-jamstack-pdx-slides.netlify.app/#0"><img src="./jam-slides.png" alt="jamstack-slides"></a>
<h3 id="domains" tabindex="-1"><a href="https://www.netlify.com/blog/2018/06/19/buy-and-secure-a-custom-domain-through-netlify/">Domains</a></h3>
I led the Netlify Domains project which allowed users to to add an existing live domain during site setup, or buy the domain if it is available. This feature enabled users to deploy a website from a git repo to a real live domain name with automatic https in a matter of minutes and has resulted in a nice stream of ever increasing AAR for the company.
<a href="https://www.netlify.com/blog/2018/06/19/buy-and-secure-a-custom-domain-through-netlify/"><img src="./domains-blogpost.png" alt="domains blogpost"></a>
<h3 id="build-status-favicons" tabindex="-1"><a href="https://www.netlify.com/blog/2018/05/22/netlify-now-shows-your-deploy-status-on-its-favicon/">Build Status Favicons</a></h3>
I helped lead and implement build status favicons, so you can put a build log into a tab, and monitor status from the tab bar.
<a href="https://www.netlify.com/blog/2018/05/22/netlify-now-shows-your-deploy-status-on-its-favicon/"><img src="./build-icons.jpeg" alt="build status icons"></a>
<h3 id="lambda-functions" tabindex="-1"><a href="https://www.netlify.com/blog/2018/03/20/netlifys-aws-lambda-functions-bring-the-backend-to-your-frontend-workflow/">Lambda Functions</a></h3>
I implemented the application UI for Netlify’s Lambda functions and logging infrastructure, and have continued to help design and improve the developer ergonomics of the Lambda functions feature set.
<a href="https://www.netlify.com/blog/2018/03/20/netlifys-aws-lambda-functions-bring-the-backend-to-your-frontend-workflow/"><img src="./functions.png" alt="functions"></a>
<h3 id="identity-widget" tabindex="-1"><a href="https://identity.netlify.com">Identity Widget</a></h3>
I helped architect and implement Netlify’s Identity widget.
<a href="https://identity.netlify.com"><img src="./identity-widget.png" alt="identity widget"></a>
<h3 id="dashboard" tabindex="-1"><a href="https://www.netlify.com/blog/2017/08/22/introducing-site-dashboards/">Dashboard</a></h3>
I helped implement the UI for our site dashboard redesign.
<a href="https://www.netlify.com/blog/2017/08/22/introducing-site-dashboards/"><img src="./dashboard.png" alt="dashboard"></a>
<h3 id="security-audit-log" tabindex="-1"><a href="https://www.netlify.com/blog/2017/07/27/introducing-audit-log/">Security Audit Log</a></h3>
I led the project to specify and implement the Audit Log for teams and identity instances.
<a href="https://www.netlify.com/blog/2017/07/27/introducing-audit-log/"><img src="./audit-log.png" alt="audit log screenshot"></a>
<h3 id="split-testing" tabindex="-1"><a href="https://www.netlify.com/blog/2017/06/28/introducing-teams-new-features-and-an-update-to-our-plans/">Split Testing</a></h3>
I implemented the UI for Netlify’s split testing feature.
<video controls width="100%" preload="metadata">
<source src="./split-testing.mp4#t=0.5" type="video/mp4">
</video>
]]></content>
<link rel="alternate" href="https://bret.io/jobs/netlify/"/>
</entry>
<entry>
<id>https://bret.io/projects/websockets/#2019-10-29T18:50:05.140Z</id>
<title>Websockets</title>
<updated>2019-10-29T18:50:05.140Z</updated>
<published>2019-10-29T18:50:05.140Z</published>
<author>
<name>Bret Comnes</name>
<uri>https://bret.io</uri>
</author>
<content type="html"><![CDATA[Websockets are a fantastic and underutilized API. Here are some tools and experiments I built to make working with websockets a bit nicer.
<h2 id="universal-reconnecting-websocket" tabindex="-1"><a href="https://github.com/bcomnes/universal-reconnecting-websocket"><code>universal-reconnecting-websocket</code></a></h2>
<a href="https://github.com/bcomnes/universal-reconnecting-websocket/"><img src="urws.png" alt=""></a>
Universal reconnecting websocket is a thin websocket client wrapper on top of DOM <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSocket"><code>WebSocket</code></a>’s and Node.js’s <a href="https://github.com/websockets/ws">ws</a>. It provides the following features:
<ul>
<li>Automatic serializing of messages</li>
<li>Automatic reconnect, with customizable strategies</li>
<li>Binary support</li>
<li>Fully defined state machine</li>
<li>Useful connection state events</li>
<li>Message and error handling</li>
<li>Unified Node.js and Browser API</li>
</ul>
The idea behind this library was to bring the wrapper API that I desired for websockets to both Node.js and Browsers in a unified way. It turns out, that there are a bunch of subtle but incompatible API differences between the <code>ws</code> client and the one found in the browser, so ultimately, I was a bit disappointed with how this experiment turned out.
I was very happy with the browser side of the API. I think this part turned out great, and was what the surface API was designed against.
The Node.js API was essentially mixed on top of the API that was implemented against the DOM api. The <code>ws</code> client has a lot more options than Browser websockts, and a number of key differences around protocol and binary options, which would require one to write two sepearate clients for Node and Browsers which kind of defeated the purpose of a universal client. I did not realize this until after I wrote it.
The next step for this project would be to implement two specific <code>abstract</code> reconnecting sockets with the same API, but implemented and documented specifically to the socket type it is written around.
<h3 id="key-takeway" tabindex="-1">Key Takeway</h3>
Don’t try to write a “universal” module when wrapping <code>io</code> primitives. Instead, implement an <code>abstract</code> version of the api specific to the <code>io</code> primitives it uses. If, down the road, it is determined to be beneficial to have a single module that automatically chooses which underlying <code>abstract</code> implementation to use based on the environment, you can use the singular <code>abstract</code> implementations to achieve that. Universal modules should usually only be for algorithmic modules.
<h2 id="websocket-chat" tabindex="-1"><a href="https://github.com/bcomnes/websocket-chat"><code>websocket-chat</code></a></h2>
<a href="https://github.com/bcomnes/websocket-chat"><img src="./websocket-chat.png" alt=""></a>
Websocket chat is a prototype grade chat server using <a href="https://github.com/websockets/ws"><code>ws</code></a>. It implements a few demonstration features:
<ul>
<li>Keepalive Heartbeat</li>
<li>Message broadcast</li>
<li>Echo server</li>
<li>Chat room messages</li>
<li>Nickname support</li>
<li>Websocket server testing</li>
</ul>
It’s not much but it acts as a handy reference on how to implement a naive chat server with limited features. It is hosted on Heroku and is completely separate from the client, other than the protocol assumptions.
<h2 id="websocket-chat-client" tabindex="-1"><a href="https://github.com/bcomnes/websocket-chat-client"><code>websocket-chat-client</code></a></h2>
<a href="https://websocket-chat-client.netlify.com"><img src="./websocket-chat-client.jpg" alt="Screenshot of websocket client UI"></a>
<ul>
<li>🌎<a href="https://websocket-chat-client.netlify.com">Live Demo</a></li>
<li>🛠<a href="https://github.com/bcomnes/websocket-chat-client">Code</a></li>
</ul>
This is a prototype grade example of a real time chat app that uses <code>universal-reconnecting-websocket</code> and connects to the <code>websocket-chat</code> server. The application is completely static, and can connect to any arbitrary <code>websocket-chat</code> server, demonstrating a <a href="https://jamstack.org">JAMStack</a> architecture. Keeping the client and server decoupled ensures you have to take care of all protocol assumptions up front which would help ensure you could implement other clients against the same server (for example, a native mobile app client).
<h2 id="dom-event-handler" tabindex="-1"><a href="https://github.com/bcomnes/dom-event-handler"><code>dom-event-handler</code></a></h2>
<a href="https://github.com/bcomnes/dom-event-handler"><img src="./dom-event-handler.png" alt=""></a>
This is module of a <a href="https://webreflection.medium.com/dom-handleevent-a-cross-platform-standard-since-year-2000-5bf17287fd38">WebReflection article</a> discussing the ancient and often forgotten detail of the <a href="https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener">EventLister</a> api. Unfortunately the author of that article would probably disapprove of this module due to lack of semicolons. C’est la vie. (I still have mad respect for your work Andrea!)
Essentially: it lets you write class methods and use them as event handler functions, without binding. This has some performance implications if you have many event handler functions, and arguably some ergonomic gains.
These are two functionally equivalent implementations:
<pre><code class="hljs language-js">const DOMEventHandler = require("dom-event-handler")
class MyWSController extends SomeOtherClass {
constructor () {
this.ws = new WebSocket('ws://localhost:8080')
this.handler = new DOMEventHandler(this, this.ws)
}
// These methods handle the websocket events
onmessage (ev) {}
onopen (ev) {}
onerror (ev) {}
onclose (ev) {}
}
</code></pre>
vs…
<pre><code class="hljs language-js">const ws = new WebSocket('ws://localhost:8080')
class VerboseWSController {
constructor () {
this.foo = 'bar'
this.onmessage = this.onmessage.bind(this)
this.onopen = this.onopen.bind(this)s
this.onerror = this.onerror.bind(this)
this.onclose = this.onclose.bind(this)
}
onmessage (ev) {}
onopen (ev) {}
onerror (ev) {}
onclose (ev) {}
}
const c = new VerboseWSController()
ws.addEventListener('message', c.onmessage)
ws.addEventListener('open', c.onopen)
ws.addEventListener('error', c.onerror)
ws.addEventListener('close', c.onclose)
</code></pre>
Isn’t that nice?
<h2 id="node-event-handler" tabindex="-1"><a href="https://github.com/bcomnes/node-event-handler"><code>node-event-handler</code></a></h2>
<a href="https://github.com/bcomnes/node-event-handler"><img src="./node-event-handler.png" alt=""></a>
When implementing <code>universal-reconnecting-websocket</code>, it was assumed that the <a href="https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener">EventLister</a> api could be mocked for the <code>ws</code> client events. It turns out, this wasn’t easy.
Node.js and DOM event systems are just too different. Here some some challenges in making them overlap:
<ul>
<li>Node.js events don’t support the <code>handleEvent</code> method, like DOM events.</li>
<li>The shape of the event data objects are totally different in both systems.</li>
<li>DOM events are a large API and difficult to simulate in Node.js.</li>
</ul>
These differences broguht me to the following conclusions.
<ul>
<li>When Node.js was the hot new thing, it was in vogue to implement Node compatible API layers for the browser. It was usually straight forward, but inevitably a userspace solution. This had a lot of advantages (like nice and simple APIs that worked effectively) and little drawbacks other than a bit of extra bundle size and lack of a standards authority dictating how things should work. <a href="http://browserify.org"><code>browserify</code></a> used this strategey to great effect, and it still works great today.</li>
<li>As Node.js aged, and its opponents slowly regained power to push back against its influence, and as Node.js’s innovations slowly sublimated into implemented, yet incompatible “standards”, it became fashionable to implement DOM apis compatible for node. The sudden interest in <a href="https://github.com/node-fetch/node-fetch">node-fetch</a> is testament to this trend, despite many <a href="https://github.com/search?q=repo%3Anode-fetch%2Fnode-fetch+clone+&type=issues">bugs</a>, and awkward differences between it and the real DOM API.</li>
<li>Porting Node.js APIs to the browser is easy, since they are fundamentally simple, userspace derived APIs.</li>
<li>Porting Browser APIs to Node in userspace is not easy, complex and error prone. Avoid doing it. See the key takeaways from <code>urws</code> above for the proper way to handle IO abstractions.</li>
<li>Node events are simpler to understand, but lack the handle event API found in the DOM. Maybe a userspace event system could accommodate this a bit better, and still remain simple and compatible with Node.js events.</li>
<li>DOM events are <a href="https://developer.mozilla.org/en-US/docs/Web/API/Event">complex</a> and way more difficult to comprehend, especially for newbies. The handle event method is a neat idea to avoid binding, but re-introducing the complexity of DOM events to node is not a good idea.</li>
</ul>
]]></content>
<link rel="alternate" href="https://bret.io/projects/websockets/"/>
</entry>
</feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid Atom 1.0" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=https%3A//bret.io/feed.xml"><img src="valid-atom.png" alt="[Valid Atom 1.0]" title="Validate my Atom 1.0 feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=https%3A//bret.io/feed.xml

Home · About · News · Docs · Terms