Richard Hamming’s “The Art of Doing Science and Engineering” is a book capturing the lessons he taught in a course he gave at the U.S. Navy Postgraduate School in Monterey, CA. He characterizes what he was trying to teach was “style” of thinking in science and engineering.

Having a physics degree myself, and also finding myself periodically ruminating on the agony of professional software development and hoping to find some overlap between my professional field and Hamming’s life experience, I gave it a read.

The book is filled with nuggets of wisdom and illustrates a career of move-the-needle science and engineering at Bell Labs. I didn’t personally find much value in many of the algebraic walk-through’s of various topics like information theory, but learning about how Hamming discovered error correcting codes definitely was interesting and worth a read.

The highlight of book comes in the second half where he includes interesting stories, analogies and observations on nearly every page. Below are my highlights I pulled while reading.

On What Makes Good Design

That brings up another point, which is now well recognized in software for computers but which applies to hardware too. Things change so fast that part of the system design problem is that the system will be constantly upgraded in ways you do not now know in any detail! Flexibility must be part of the modern design of things and processes. Flexibility built into the design means not only will you be better able to handle the changes which will come after installation, but it also contributes to your own work as the small changes which inevitably arise both in the later stages of design and in the field installation of the system…

Thus rule two:

Part of systems engineering design is to prepare for changes so they can be gracefully made and still not degrade the other parts.

– p.367

This quote is my favorite out of the entire book. It feels like a constant fight in software engineering between the impulse to lock down runtime versions, specific dependency versions, and other environmental factors versus developing software in such a way that accommodates wide variance in all of these different component factors. Both approaches argue reliability and flexibility, however which approach actually tests for it?

In my experience, the tighter the runtime dependency specifications, the faster fragility spreads, and it’s satisfying to hear Hamming’s experience echo this observation. Sadly though, his observation that those writing software will universally understand this simply hasn’t held up.

Good design protects you from the need for too many highly accurate components in the system. But such design principals are still, to this date, ill understood and need to be researched extensively. Not that good designers do not understand this intuitively, merely it is not easily incorporated into the design methods you were thought in school.

Good minds are still need in spite of all the computing tools we have developed. The best mind will be the one who gets the principle into the design methods taught so it will be automatically available for lesser minds!.

– p.268

Here Hamming is describing H.S. Black’s feedback circuit’s tolerance for low accuracy components as what constitutes good design. I agree! Technology that works at any scale, made out of commodity parts with minimal runtime requirements tends to be what is most useful across the longest amount of time.

On Committees

Committee decisions, which tend to diffuse responsibility, are seldom the best in practice—most of the time they represent a compromise which has none of the virtues of any path and tends to end in mediocrity.

– p.274

I appreciated his observations on committees, and their tendency to launder responsibility. They serve a purpose, but its important to understand their nature.

On Data and Observation

The Hawthorne effect strongly suggests the proper teaching method will always to be in a state of experimental change, and it hardly matters just what is done; all that matters is both the professor and the students believe in the change.

– p.288

It has been my experience, as well as the experience of many others who have looked, that data is generally much less accurate than it is advertised to be. This is not a trivial point—we depend on initial data for many decisions, as well as for the input data for simulations which result in decisions.

– p.345

Averages are meaningful for homogeneous groups (homogeneous with respect to the actions that may later be taken), but for diverse groups averages are often meaningless. As earlier remarked, the average adult has one breast and one testicle, but that does not represent the average person in our society.

– p.356

You may think the title means that if you measure accurately you will get an accurate measurement, and if not then not, but it refers to a much more subtle thing—the way you choose to measure things controls to a large extent what happens. I repeat the story Eddington told about the fishermen who went fishing with a net. They examined the size of the fish they caught and concluded there was a minimum size to the fish in the sea. The instrument you use clearly affects what you see.

– p.373

Intuitively I think many people who attempt to measure anything understand that their approach reflects in the results to some degree. I hadn’t heard of the Hawthorne effect before, but intuitively it makes sense.

People with an idea on how to improve something implement their idea and it works, because they want it to work and allow the effects to be fully effective. Someone else is prescribed this idea or brought into the fold where the idea is implemented and the benefits of the idea evaporate.

I’ve long suspected that in the context of professional software development, where highly unscrutinized benchmarks and soft data are the norm, people start with an opinion or theory and work back to data that supports it. Could it just be that people need to believe that working in a certain way is necessary for them to work optimally? Could it be “data” is often just a work function used to out maneuver competing ideas?

Anyway, just another thing to factor for when data is plopped in your lap.

On Theory

Moral: there need not be a unique form of a theory to account for a body of observations; instead, two rather different-looking theories can agree on all the predicted details. You cannot go from a body of data to a unique theory! I noted this in the last chapter.

–p.314

Heisenberg derived the uncertainty principle that conjugate variables, meaning Fourier transforms, obeyed a condition in which the product of the uncertainties of the two had to exceed a fixed number, involving Planck’s constant. I earlier commented, Chapter 17, this is a theorem in Fourier transforms-any linear theory must have a corresponding uncertainty principle, but among physicists it is still widely regarded as a physical effect from nature rather than a mathematical effect of the model.

–p.316

I appreciate Hamming suggesting that some of our understanding of physical reality could be a byproduct of the model being used to describe it. It’s not exactly examined closely in undergraduate or graduate quantum mechanics, and I find it interesting Hamming, who’s clearly highly intuitive with modeling, also raises this question.

Predictions

Let me now turn to predictions of the immediate future. It is fairly clear that in time “drop lines” from the street to the house (they may actually be buried, but will probably still be called “drop lines”) will be fiber optics. Once a fiber-optic wire is installed, then potentially you have available almost all the information you could possibly want, including TV and radio, and possibly newspaper articles selected according to your interest profile (you pay the printing bill which occurs in your own house). There would be no need for separate information channels most of the time. At your end of the fiber there are one or more digital filters. Which channel you want, the phone, radio, or TV, can be selected by you much as you do now, and the channel is determined by the numbers put into the digital filter-thus the same filter can be multipurpose, if you wish. You will need one filter for each channel you wish to use at the same time (though it is possible a single time-sharing filter would be available) and each filter would be of the same standard design. Alternately, the filters may come with the particular equipment you buy.

– p.284-285

Here Hamming is predicting the internet. He got very close, and it’s interesting to think that these signals would all just be piped to your house in a bundle you you pay for a filter to unlock access to the ones you want. Hey Cable TV worked that for a long time!

On Leadership

But a lot of evidence on what enabled people to make big contributions points to the conclusion that a famous prof was a terrible lecturer and the students had to work hard to learn it for themselves! I again suggest a rule:

What you learn from others you can use to follow;

What you learn for yourself you can use to lead.

– p.292

Learn by doing, not by following.

What you did to become successful is likely to be counterproductive when applied at a later date.

– p.342

It’s easy to blame changing trends in software development for the disgustingly short half-life of knowledge regarding development patterns and tools, but I think it’s probably just the nature of knowledge based work. Operating by yourself may be effective and work well, but its not a recipe for success at any given moment in time.

A man was examining the construction of a cathedral. He asked a stonemason what he was doing chipping the stones, and the mason replied, “I am making stones.” He asked a stone carver what he was doing; “I am carving a gargoyle.” And so it went; each person said in detail what they were doing. Finally he came to an old woman who was sweeping the ground. She said, “I am helping build a cathedral.” If, on the average campus, you asked a sample of professors what they were going to do in the next class hour, you would hear they were going to “teach partial fractions,” “show how to find the moments of a normal distribution,” “explain Young’s modulus and how to measure it,” etc. I doubt you would often hear a professor say, “I am going to educate the students and prepare them for their future careers.” This myopic view is the chief characteristic of a bureaucrat. To rise to the top you should have the larger view—at least when you get there.

– p.360

Software bureaucrats aplenty. Really easy to fall into this role.

I must come to the topic of “selling” new ideas. You must master three things to do this (Chapter 5):

1. Giving formal presentations,
2. Producing written reports, and
3. Mastering the art of informal presentations as they happen to occur.

All three are essential—you must learn to sell your ideas, not by propaganda, but by force of clear presentation. I am sorry to have to point this out; many scientists and others think good ideas will win out automatically and need not be carefully presented. They are wrong;

– p.396

One thing I regret over the last 10 years of my career is not writing down more insights I have learned through experience. Ideas simply don’t transmit if they aren’t written down or put into some consumable format like video or audio. Nearly every annoying tool or developer trend you are forced to use is in play because it communicated the idea through blogs, videos and conference talks. And those who watched echoed these messages.

On Experts

An expert is one who knows everything about nothing; a generalist knows nothing about everything.

In an argument between a specialist and a generalist, the expert usually wins by simply (1) using unintelligible jargon, and (2) citing their specialist results, which are often completely irrelevant to the discussion. The expert is, therefore, a potent factor to be reckoned with in our society. Since experts both are necessary and also at times do great harm in blocking significant progress, they need to be examined closely. All too often the expert misunderstands the problem at hand, but the generalist cannot carry though their side to completion. The person who thinks they understand the problem and does not is usually more of a curse (blockage) than the person who knows they do not understand the problem.

– p.333

Understand when you are generalist and a specialist.

Experts, in looking at something new, always bring their expertise with them, as well as their particular way of looking at things. Whatever does not fit into their frame of reference is dismissed, not seen, or forced to fit into their beliefs. Thus really new ideas seldom arise from the experts in the field. You cannot blame them too much, since it is more economical to try the old, successful ways before trying to find new ways of looking and thinking.

If an expert says something can be done he is probably correct, but if he says it is impossible then consider getting another opinion.

– p.336

Anyone wading into a technical field will encounter experts at every turn. They have valuable information, but they are also going to give you dated, myopic advice (gatekeeping?). I like Hamming’s framing here and it reflects my experience when weighing expert opinion.

In some respects the expert is the curse of our society, with their assurance they know everything, and without the decent humility to consider they might be wrong. Where the question looms so important, I suggested to you long ago to use in an argument, “What would you accept as evidence you are wrong?” Ask yourself regularly, “Why do I believe whatever I do?” Especially in the areas where you are so sure you know, the area of the paradigms of your field.

– p.340

I love this exercise. It will also drive you crazy. Tread carefully.

Systems engineering is indeed a fascinating profession, but one which is hard to practice. There is a great need for real systems engineers, as well as perhaps a greater need to get rid of those who merely talk a good story but cannot play the game effectively.

– p.372

Controversial, harsh, but true.

The Binding

The last thing I want to recognize is the beautiful cloth resin binding and quality printing of the book. Bravo Stripe Press for still producing beautiful artifacts at affordable pricing in the age of print on demand.

Usage: async-neocities [options]

    Example: async-neocities --src public

    --help, -h            print help text
    --src, -s             The directory to deploy to neocities (default: "public")
    --cleanup, -c         Destructively clean up orphaned files on neocities
    --protect, -p         String to minimatch files which will never be cleaned up
    --status              Print auth status of current working directory
    --print-key           Print api-key status of current working directory
    --clear-key           Remove the currently associated API key
    --force-auth          Force re-authorization of current working directory

async-neocities (v3.0.0)

When you run it, you will see something similar to this:

> async-neocities --src public

Found siteName in config: bret
API Key found for bret
Starting inspecting stage...
Finished inspecting stage.
Starting diffing stage...
Finished diffing stage.
Skipping applying stage.
Deployed to Neocities in 743ms:
    Uploaded 0 files
    Orphaned 0 files
    Skipped 244 files
    0 protected files

async-neocities was previously available as a GitHub Action called deploy-to-neocities. This Action API remains available, however the CLI offers a local-first workflow that was not previously offered.

Local First Deploys

Now that async-neocities is available as a CLI, you can easily configure it as an npm script and run it locally when you want to push changes to neocities without relying on GitHub Actions. It also works great in Actions with side benefit of deploys working exactly the same way in both local and remote environments.

Here is a quick example of that:

• Install async-neocities@^3.0.0 to your project’s package.json.
• Set up a package.json deploy script:

   "scripts": {
      "build": "npm run clean && run-p build:*",
      "build:tb": "top-bun",
      "clean": "rm -rf public && mkdir -p public",
      "deploy": "run-s build deploy:*",
      "deploy:async-neocities": "async-neocities --src public --cleanup"
    },
  

• Run a deploy once locally to set up the deploy-to-neocities.json config file. Example config contents:

  {"siteName":"bret"}
  

• Run deploys locally with npm run deploy.
• Configure your CI to run npm run deploy and configure the token secret.

  name: Deploy to neociteis
  
  on:
    push:
      branches:
        - master
  
  env:
    node-version: 21
    FORCE_COLOR: 2
  
  concurrency: # prevent concurrent deploys doing starnge things
    group: deploy-to-neocities
    cancel-in-progress: true
  
  jobs:
    deploy:
      runs-on: ubuntu-latest
  
      steps:
      - uses: actions/checkout@v4
      - name: Create LFS file list
        run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
      - name: Restore LFS cache
        uses: actions/cache@v3
        id: lfs-cache
        with:
          path: .git/lfs
          key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}-v1
      - name: Git LFS Pull
        run: git lfs pull
  
      - name: Use Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{env.node-version}}
      - run: npm i
      - run: npm run deploy
        env:
          NEOCITIES_API_TOKEN: ${{ secrets.NEOCITIES_API_TOKEN }}
  

The async-neocities CLI re-uses the same ENV name as deploy-to-neocities action so migrating to the CLI requires no additional changes to the Actions environment secrets.

CLIs vs Actions

This prompts some questions regarding when are CLIs and when are actions most appropriate. Lets compare the two:

CLIs

• Pro: Local deploys
• Pro: Easily re-usable in CI as well
• Con: Requires Node.js, but this is not a problem when already using Node.js

Actions

• Pro: Works great with Non-node ecosystems without requiring a package.json or node_modules
• Con: Only runs in CI environments

Conclusion

In addition to the CLI, async-neocities migrates to full Node.js esm and internally enables ts-in-js though the types were far to dynamic to export full type support with the time I had available.

With respect to an implementation plan going forward regarding CLIs vs actions, I’ve summarized my thoughts below:

Implement core functionality as a re-usable library. Exposing a CLI makes that library an interactive tool that provides a local first workflow and is equally useful in CI. Exposing the library in an action further opens up the library to a wider language ecosystem which would otherwise ignore the library due to foreign ecosystem ergonomic overhead. The action is simpler to implement than a CLI but the CLI offers a superior experience within the implemented language ecosystem.

Please check out my resume to see what I’ve been up to and enjoy some highlights below. If you think any of my qualities line up with your hiring needs, please let me know!

(Spare a click? Please boost! x.com, bsky, mastodon, linkedin)

Birds eye view: I have a variety of experience building software/SAAS infrastructure from the ground up and contributing to large existing codebase and orgs. I generate between 1500 and 3000+ contributions a year, some of which are captured for public scrutiny.

I work primarily in Javascript, and now Typescript for frontend and backend, but I also really love working in Go.

In addition to my professional work, I work on open source tools and products while my kids are napping on the weekend. My latest project is 🥖 breadcrum.net which lets you extract articles and media from around the web so that you can read/listen/watch them in your preferred podcast app and reader device. It’s free right now too! You should check it out.

I enjoy working within the GitHub open source model, and have publisher status on over 300 npm packages, some of which I even originated. There is a decent chance I could push code changes into your web projects!

One of my most recent package adoptions, npm-run-all2, grew from 1000 to over 11,000 dependents over the last few months. Incredible growth that hasn’t slowed down. I am pleased to contribute back a small portion of the work that I benefit from when building with the open source ecosystem.

I will be reaching out to a bunch of you individually, but in the meantime if you are seeing this, please reach out! I would love to reconnect to personally catch up and also chat about ideas for work.

It’s still not great, but it should make it easier to keep it up to date going forward.

It has 3 sections:

• Featured Projects: important projects of note.
• Recent Posts: now that this site has proper blog support, I can highlight recent posts on the landing page.
• Open Source: interesting and notable projects that have found some use and that I still maintain. This section now includes a bunch of open source work from the past year that I’ve never had time to write about.
• Past projects: inactive projects that are not longer active, but still interesting enough to share.

I removed a bunch of older inactive projects and links and stashed them in a project.

Additionally, the edit button in the page footer now takes you to the correct page in GitHub for editing, so if you ever see a typo, feel free to send in a fix!

Finally, the about page includes a live dump of the dependencies that were used to build the website.

• Frontend Testing with Socket
• Improved File Resolution

If you read them, let me know what you think!

Re-introduce? Well, you may remember @siteup/cli, a spiritual offshoot of sitedown, the static site generator that turned a directory of markdown into a website.

Whats new with top-bun v7?

Let’s dive into the new feature, changes and additions in top-bun 7.

Rename to top-bun

@siteup/cli is now top-bun. As noted above, @siteup/cli was a name hack because I didn’t snag the bare npm name when it was available, and someone else had the genius idea of taking the same name. Hey it happens.

I described the project to Chat-GPT and it recommended the following gems:

• quick-brick
• web-erect

OK Chat-GPT, pretty good, I laughed, but I’m not naming this web-erect.

The kids have a recent obsession with Wallace & Gromit and we watched a lot of episodes while she was sick. Also I’ve really been enjoying 🥖 bread themes so I decided to name it after Wallace & Gromit’s bakery “Top Bun” in their hit movie “A Matter of Loaf and Death”.

A Docs Website

top-bun now builds it’s own repo into a docs website. It’s slightly better than the GitHub README.md view, so go check it out! It even has a real domain name so you know its for real.

• 🌎 top-bun.org

css bundling is now handled by esbuild

esbuild is an amazing tool. postcss is a useful tool, but its slow and hard to keep up with. In top-bun, css bundling is now handled by esbuild. css bundling is now faster and less fragile, and still supports many of the same transforms that siteup had before. CSS nesting is now supported in every modern browser so we don’t even need a transform for that. Some basic transforms and prefixes are auto-applied by setting a relatively modern browser target.

esbuild doesn’t support import chunking on css yet though, so each css entrypoint becomes its own bundle. If esbuild ever gets this optimization, so will top-bun. In the meantime, global.css, style.css and now layout.style.css give you ample room to generally optimize your scoped css loading by hand. It’s simpler and has less moving parts!

Multi-layout support

You can now have more than one layout! In prior releases, you could only have a single root layout that you customized on a per-page basis with variables. Now you can have as many layouts as you need. They can even nest. Check out this example of a nested layout from this website. It’s named article.layout.js and imports the root.layout.js. It wraps the children and then passes the results to root.layout.js.

// article.layout.js

import { html } from 'uhtml-isomorphic'
import { sep } from 'node:path'
import { breadcrumb } from '../components/breadcrumb/index.js'

import defaultRootLayout from './root.layout.js'

export default function articleLayout (args) {
  const { children, ...rest } = args
  const vars = args.vars
  const pathSegments = args.page.path.split(sep)
  const wrappedChildren = html`
    ${breadcrumb({ pathSegments })}
    <article class="article-layout h-entry" itemscope itemtype="http://schema.org/NewsArticle">
      <header class="article-header">
        <h1 class="p-name article-title" itemprop="headline">${vars.title}</h1>
        <div class="metadata">
          <address class="author-info" itemprop="author" itemscope itemtype="http://schema.org/Person">
            ${vars.authorImgUrl
              ? html`<img height="40" width="40"  src="${vars.authorImgUrl}" alt="${vars.authorImgAlt}" class="u-photo" itemprop="image">`
              : null
            }
            ${vars.authorName && vars.authorUrl
              ? html`
                  <a href="${vars.authorUrl}" class="p-author h-card" itemprop="url">
                    <span itemprop="name">${vars.authorName}</span>
                  </a>`
              : null
            }
          </address>
          ${vars.publishDate
            ? html`
              <time class="published-date dt-published" itemprop="datePublished" datetime="${vars.publishDate}">
                <a href="#" class="u-url">
                  ${(new Date(vars.publishDate)).toLocaleString()}
                </a>
              </time>`
            : null
          }
          ${vars.updatedDate
            ? html`<time class="dt-updated" itemprop="dateModified" datetime="${vars.updatedDate}">Updated ${(new Date(vars.updatedDate)).toLocaleString()}</time>`
            : null
          }
        </div>
      </header>

      <section class="e-content" itemprop="articleBody">
        ${typeof children === 'string'
          ? html([children])
          : children /* Support both uhtml and string children. Optional. */
        }
      </section>

      <!--
        <footer>
            <p>Footer notes or related info here...</p>
        </footer>
      -->
    </article>
    ${breadcrumb({ pathSegments })}
  `

  return defaultRootLayout({ children: wrappedChildren, ...rest })
}

Layout styles and js bundles

With multi-layout support, it made sense to introduce two more style and js bundle types:

• Layout styles: styles that load on every page that uses a layout
• Layout bundles: js bundles that load on every page that uses a layout

Prior the global.css and global.client.js bundles served this need.

Layouts and Global Assets live anywhere

Layouts, and global.client.js, etc used to have to live at the root of the project src directory. This made it simple to find them when building, and eliminated duplicate singleton errors, but the root of websites is already crowded. It was easy enough to find these things anywhere, so now you can organize these special files in any way you like. I’ve been using:

• layouts: A folder full of layouts
• globals: A folder full of the globally scoped files

Template files

Given the top-bun variable cascade system, and not all website files are html, it made sense to include a templating system for generating any kind of file from the global.vars.js variable set. This lets you generate random website “sidefiles” from your site variables.

It works great for generating RSS feeds for websites built with top-bun. Here is the template file that generates the RSS feed for this website:

import pMap from 'p-map'
import jsonfeedToAtom from 'jsonfeed-to-atom'

/**
 * @template T
 * @typedef {import('@siteup/cli').TemplateAsyncIterator<T>} TemplateAsyncIterator
 */

/** @type {TemplateAsyncIterator<{
 *  siteName: string,
 *  siteDescription: string,
 *  siteUrl: string,
 *  authorName: string,
 *  authorUrl: string,
 *  authorImgUrl: string
 *  layout: string,
 *  publishDate: string
 *  title: string
 * }>} */
export default async function * feedsTemplate ({
  vars: {
    siteName,
    siteDescription,
    siteUrl,
    authorName,
    authorUrl,
    authorImgUrl
  },
  pages
}) {
  const blogPosts = pages
    .filter(page => ['article', 'book-review'].includes(page.vars.layout) && page.vars.published !== false)
    .sort((a, b) => new Date(b.vars.publishDate) - new Date(a.vars.publishDate))
    .slice(0, 10)

  const jsonFeed = {
    version: 'https://jsonfeed.org/version/1',
    title: siteName,
    home_page_url: siteUrl,
    feed_url: `${siteUrl}/feed.json`,
    description: siteDescription,
    author: {
      name: authorName,
      url: authorUrl,
      avatar: authorImgUrl
    },
    items: await pMap(blogPosts, async (page) => {
      return {
        date_published: page.vars.publishDate,
        title: page.vars.title,
        url: `${siteUrl}/${page.pageInfo.path}/`,
        id: `${siteUrl}/${page.pageInfo.path}/#${page.vars.publishDate}`,
        content_html: await page.renderInnerPage({ pages })
      }
    }, { concurrency: 4 })
  }

  yield {
    content: JSON.stringify(jsonFeed, null, '  '),
    outputName: 'feed.json'
  }

  const atom = jsonfeedToAtom(jsonFeed)

  yield {
    content: atom,
    outputName: 'feed.xml'
  }

  yield {
    content: atom,
    outputName: 'atom.xml'
  }
}

Page Introspection

Pages, Layouts and Templates can now introspect every other page in the top-bun build.

You can now easily implement any of the following:

• RSS feeds
• Auto-populating index feeds
• Tag systems

Pages, Layouts and Templates receive a pages array that includes PageData instances for every page in the build. Variables are already pre-resolved, so you can easily filter, sort and target various pages in the build.

Full Type Support

top-bun now has full type support. It’s achieved with types-in-js and it took a ton of time and effort.

The results are nice, but I’m not sure the juice was worth the squeeze. top-bun was working really well before types. Adding types required solidifying a lot of trivial details to make the type-checker happy. I don’t even think a single runtime bug was solved. It did help clarify some of the more complex types that had developed over the first 2 years of development though.

The biggest improvement provided here is that the following types are now exported from top-bun:

LayoutFunction<T>
PostVarsFunction<T>
PageFunction<T>
TemplateFunction<T>
TemplateAsyncIterator<T>

You can use these to get some helpful auto-complete in LSP supported editors.

types-in-js not Typescript

This was the first major dive I did into a project with types-in-js support. My overall conclusions are:

• types-in-js provides a superior development experience to developing in .ts by eliminating the development loop build step.
• JSDoc and types-in-js are in conflict with each other. types-in-js should win, its better than JSDoc in almost every way (but you still use both).
• Most of the JSDoc auto-generating documentation ecosystem doesn’t support types-in-js. Find something that consumes the generated types instead of consuming the JSDoc blocs.
• There are a number of rough edges around importing types.
• The final api documentation features are nice.
• The internal types feel good, but were mostly a waste of time being introduced after the fact.
• I wish there was a way to strictly introduce types on just the public interfaces and work back, but this is challenging because many details come from deep within the program.
• Typescript puts upper limits on the dynamism normally allowed with JS. It’s a superset syntax, that forces a subset of language functionality.
• @ts-ignore liberally. Take a pass or two to remove some later.

Handlebars support in md and html

Previously, only js pages had access to the variable cascade inside of the page itself. Now html and md pages can access these variables with handlebars placeholders.

## My markdown page

Hey this is a markdown page for {{ vars.siteName }} that uses handlebars templates.

Locally shipped default styles

Previously, if you opted for the default layout, it would import mine.css from unpkg. This worked, but went against the design goal of making top-bun sites as reliable as possible (shipping all final assets to the dest folder).

Now when you build with the default layout, the default stylesheet (and theme picker js code) is built out into your dest folder.

Built-in browser-sync

@siteup/cli previously didn’t ship with a development server, meaning you had to run one in parallel when developing. This step is now eliminated now that top-bun ships browser-sync. browser-sync is one of the best Node.js development servers out there and offers a bunch of really helpful dev tools built right in, including scroll position sync so testing across devices is actually enjoyable.

If you aren’t familiar with browser-sync, here are some screenshots of fun feature:

top-bun eject

top-bun now includes an --eject flag, that will write out the default layout, style, client and dependencies into your src folder and update package.json. This lets you easily get started with customizing default layouts and styles when you decide you need more control.

√ default-layout % npx top-bun --eject

top-bun eject actions:
  - Write src/layouts/root.layout.mjs
  - Write src/globals/global.css
  - Write src/globals/global.client.mjs
  - Add mine.css@^9.0.1 to package.json
  - Add uhtml-isomorphic@^2.0.0 to package.json
  - Add highlight.js@^11.9.0 to package.json

Continue? (Y/n) y
Done ejecting files!

The default layout is always supported, and its of course safe to rely on that.

Improved log output 🪵

The logging has been improved quite a bit. Here is an example log output from building this blog:

> top-bun --watch

Initial JS, CSS and Page Build Complete
bret.io/src => bret.io/public
├─┬ projects
│ ├─┬ websockets
│ │ └── README.md: projects/websockets/index.html
│ ├─┬ tron-legacy-2021
│ │ └── README.md: projects/tron-legacy-2021/index.html
│ ├─┬ package-automation
│ │ └── README.md: projects/package-automation/index.html
│ └── page.js: projects/index.html
├─┬ jobs
│ ├─┬ netlify
│ │ └── README.md: jobs/netlify/index.html
│ ├─┬ littlstar
│ │ └── README.md: jobs/littlstar/index.html
│ ├── page.js:      jobs/index.html
│ ├── zhealth.md:   jobs/zhealth.html
│ ├── psu.md:       jobs/psu.html
│ └── landrover.md: jobs/landrover.html
├─┬ cv
│ ├── README.md: cv/index.html
│ └── style.css: cv/style-IDZIRKYR.css
├─┬ blog
│ ├─┬ 2023
│ │ ├─┬ reintroducing-top-bun
│ │ │ ├── README.md: blog/2023/reintroducing-top-bun/index.html
│ │ │ └── style.css: blog/2023/reintroducing-top-bun/style-E2RTO5OB.css
│ │ ├─┬ hello-world-again
│ │ │ └── README.md: blog/2023/hello-world-again/index.html
│ │ └── page.js: blog/2023/index.html
│ ├── page.js:   blog/index.html
│ └── style.css: blog/style-NDOJ4YGB.css
├─┬ layouts
│ ├── root.layout.js:             root
│ ├── blog-index.layout.js:       blog-index
│ ├── blog-index.layout.css:      layouts/blog-index.layout-PSZNH2YW.css
│ ├── blog-auto-index.layout.js:  blog-auto-index
│ ├── blog-auto-index.layout.css: layouts/blog-auto-index.layout-2BVSCYSS.css
│ ├── article.layout.js:          article
│ └── article.layout.css:         layouts/article.layout-MI62V7ZK.css
├── globalStyle:               globals/global-OO6KZ4MS.css
├── globalClient:              globals/global.client-HTTIO47Y.js
├── globalVars:                global.vars.js
├── README.md:                 index.html
├── style.css:                 style-E5WP7SNI.css
├── booklist.md:               booklist.html
├── about.md:                  about.html
├── manifest.json.template.js: manifest.json
├── feeds.template.js:         feed.json
├── feeds.template.js-1:       feed.xml
└── feeds.template.js-2:       atom.xml

[Browsersync] Access URLs:
 --------------------------------------
       Local: http://localhost:3000
    External: http://192.168.0.187:3000
 --------------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 --------------------------------------
[Browsersync] Serving files from: /Users/bret/Developer/bret.io/public
Copy watcher ready

Support for mjs and cjs file extensions

You can now name your page, template, vars, and layout files with the mjs or cjs file extensions. Sometimes this is a necessary evil. In general, set your type in your package.json correctly and stick with .js.

What’s next for top-bun

The current plan is to keep sitting on this feature set for a while. But I have some ideas:

• Tell the world about it?
• Server routes with fastify (or expose a plugin to slot into an existing server)?
• Deeper integration with uhtml?
• More web-component examples? top-bun is already one of the best environments for implementing sites that use web-components. Page bundles are a perfect place to register components!
• Comparisons with other tools in the “enterprise-js” ecosystem?

If you try out top-bun, I would love to hear about your experience. Do you like it? Do you hate it? Open an discussion item. or reach out privately.

History of top-bun

OK, now time for the story behind top-bun aka @siteup/cli.

I ran some experiments with orthogonal tool composition a few years ago. I realized I could build sophisticated module based websites by composing various tools together by simply running them in parallel.

What does this idea look like? See this snippet of a package.json:

 { "scripts": {
    "build": "npm run clean && run-p build:*",
    "build:css": "postcss src/index.css -o public/bundle.css",
    "build:md": "sitedown src -b public -l src/layout.html",
    "build:feed": "generate-feed src/log --dest public && cp public/feed.xml public/atom.xml",
    "build:static": "cpx 'src/**/*.{png,svg,jpg,jpeg,pdf,mp4,mp3,js,json,gif}' public",
    "build:icon": "gravatar-favicons --config favicon-config.js",
    "watch": "npm run clean && run-p watch:* build:static",
    "watch:css": "run-s 'build:css -- --watch'",
    "watch:serve": "browser-sync start --server 'public' --files 'public'",
    "watch:md": "npm run build:md -- -w",
    "watch:feed": "run-s build:feed",
    "watch:static": "npm run build:static -- --watch",
    "watch:icon": "run-s build:icon",
    "clean": "rimraf public && mkdirp public",
    "start": "npm run watch"
  }
}

• I have postcss building css bundles, enabling an @import based workflow for css, as well as providing various transforms I found useful.
• The markdown is built with sitedown.
• I wrote a tool to generate a RSS feed from markdown called generate-feed
• I generate favicons from a gravatar identifier with gravatar-favicons.
• js bundling could easily be added in here with esbuild or rollup.
• Steps are grouped into build and watch prefixes, and managed with npm-run-all2 which provides shortcuts to running these tasks in parallel.

I successfully implemented this pattern across 4-5 different websites I manged. It work beautifully. Some of the things I liked about it:

• ✅ Tools could be swapped out easily.
• ✅ I could experiment individual ideas on individual sites, without forcing them on every other project.
• ✅ All of the tools were versioned gated, and I could update them as I had time.
• ✅ It worked very well with release automation.

But it had a few drawbacks:

• ❌ I liked the pattern so much, that I wanted to add websites to more projects, but setting this all up was a hassle.
• ❌ When I discovered a change I liked, re-implementing the change across multiple projects become tedious.
• ⚠️ Orthogonal tool composition helps keep this arrangement clean and fast, but some steps really do benefit from the awareness of the results of other steps.

So I decided to roll up all of the common patterns into a single tool that included the discoveries of this process.

@siteup/cli extended sitedown

Because it was clear sitedown provided the core structure of this pattern (making the website part), I extended the idea in the project @siteup/cli. Here are some of the initial features that project shipped with:

• Variable cascade + frontmatter: You could define global and page variables used to customize parts of the site layout. This lets you set the title, or other parts of the <head> tag easily from the frontmatter section of markdown pages.
• A js layout: This let you write a simple JS program that receives the variable cascade and child content of the page as a function argument, and return the contents of the page after applying any kind of template tool available in JS.
• html pages: CommonMark supports html in markdown, but it also has some funny rules that makes it more picky about how the html is used. I wanted a way to access the full power of html without the overhead of markdown, and this page type unlocked that.
• js pages: Since I enjoy writing JavaScript, I also wanted a way to support generating pages using any templating system and data source imaginable so I also added support to generate pages from the return value of a JS function.
• JavaScript bundling: I wanted to make it super easy to include client side JavaScript, so it included some conventions for setting that up quickly and easily.
• CSS bundling: Instead of adding the complexity of css-modules or other transform-based scoped css solution, siteup opted to make it super easy to add a global css bundle, and page scoped css bundles, which both supported a postcss @import workflow and provided a few common transforms to make working with css tolerable (nesting, prefixing etc).
• Static files: Including static files was such a common pattern that I also included the ability, to copy over static files without any additional configuration. I believe sitedown also added this feature subsequently.
• Asset co-location: I wanted a way to co-locate assets, code and anything near where it was depended upon. It is such a natural way to organize complex collections, and so many tools break this, or require special locations for special files. Let me put it wherever I want in src!

After sitting on the idea of siteup for over a year, by the time I published it to npm, the name was taken, so I used the npm org name hack to get a similar name @siteup/cli. SILLY NPM!

I enjoyed how @siteup/cli came out, and have been using it for 2 year now. Thank you of course to ungoldman for laying the foundation of most of these tools and patterns. Onward and upward to top-bun!

This blog has been super quiet lately, sorry about that. I had some grand plans for finishing up my website tools to more seamlessly support blogging.

The other day around 3AM, I woke up and realized that the tools aren’t stopping me from writing, I am. Also my silently implemented policy about not writing about ‘future plans and ideas before they are ready’ was implemented far to strictly. It is in fact a good thing to write about in-progress ideas and projects slightly out into the future. This is realistic, interesting, and avoids the juvenile trap of spilling ideas in front of the world only to see you never realize them. So here I am writing a blog post again.

Anyway, no promises, but it is my goal to write various ahem opinions, thoughts and ideas more often because nothing ever happens unless you write it down.

Some basic updates

Here are some updates from the last couple years that didn’t make it onto this site before.

• I’m back in California.
• I moved a bunch (homes, jobs). I bought a home and then sold it. I don’t recommend it!
• I have a son and a daughter and am married. I recommend this!
• I started HifiWi.fi
• The first product is Breadcrum, a bookmarking service with textual and media archiving super powers.
• HifiWi has subsumed operation of Gumcast, a tool to allow podcasting with GumRoad. Without advertising, it has picked up 100s of users organically from search results.
• I had a near 2 year tenure at Socket Inc dabbling in GitHub apps and npm security concerns.
• I’m now working at socketsupply.co on a runtime and P2P full time again. Really happy to be back in this space again.
• I still work remote and intend to continue working remote.
• A bunch of my projects are running on my own website/app builder too called top-bun. It’s maybe a 3rd of the way done, but has been useful for about 90% of my needs. I would like to write up a proper blog post about it someday.
• bcomnes/deploy-to-neocities has 300+ public users deploying websites from GitHub actions to Neocities.
• My fork bcomnes/npm-run-all2 (co-maintained by voxpelli.com) has picked up 1000+ public dependents.
• Now that twitter (ahem x) has completely abandoned its charter as an open web website, you can find me posting on @bcomnes@fosstodon.org and @bret.io on bsky. I would like to make this website authoritative though for me posts though.
• I’m very lucky to have a new office that is a 1 minute walk from home:

The 2.0.0 release converts the older .tmTheme format into the Sublime specific theme format. Overall the new Sublime theme format (.sublime-color-scheme) is a big improvement, largely due to its simple JSON structure and its variables support.

JSON is, despite the common arguments against it, super readable and easily read and written by humans. The variable support makes the process of making a theme a whole lot more automatic, since you no longer have to find and replace colors all over the place.

The biggest problem I ran into was poor in-line color highlighting when working with colors, so I ended up using a VSCode plugin called Color Highlight in a separate window. Sublime has a great plugin also called Color Highlight that usually works well, but not in this case. The Sublime Color Highlight variant actually does temporary modifications to color schemes, which seriously gets in the way when working on color scheme files.

The rewrite is based it off of the new Mariana Theme that ships with ST4, so the theme should have support for most of the latest features in ST4 though there are surely features that even Mariana missed. Let me know if you know of any.

Here a few other points of consideration made during the rewrite:

• Remove the other theme variants. A breaking change, but they were poorly maintained to begin with.
• Renamed the theme file to Tron Legacy 4 (Dark).
• A light theme is in order, but its a non-trivial rewrite to support a light mode.

Here a few more relevant links and please let me know what you think if you try it out.

• github.com/bcomnes/sublime-tron-color-scheme
• Release page

Syndication

• twitter thread

Sublime Tron Legacy color scheme fully updated for @sublimehq Text 4. Full syntax support, lots of other small improvements. Also it supports 'glow' text✌️ pic.twitter.com/vShbGThgDF

— 🌌🌵🛸Bret🏜👨‍👩‍👧🚙 (@bcomnes) August 5, 2021

2021-08-05T23:09:46.781Z

After a short sabbatical in Denmark at Hyperdivision, I joined Littlstar. Here is a quick overview of some of the more interesting projects I worked on. I joined Littlstar during a transitional period of the company and help them transition from a VR video platform to a video on demand and live streaming platform, and now an NFT sales and auction platform.

NYCPF VR Platform

My first project was picking up on an agency style project developing a novel VR reality training platform for NYCPF, powered by a custom hypercore p2p file sharing network, delivering in-house developed unity VR scenarios. These scenarios could then be brought to various locations like schools or events where NYCPF could guide participants through various law enforcement scenarios with different outcomes based on participant choices within the simulation.

By utilizing a p2p and offline first design approach, we were able to deliver an incredibly flexible and robust delivery platform that had all sorts of difficult features to develop for traditional file distribution platforms such as:

• Automated offline re-distribution of simulation data to a fleet of devices.
• Bittorrent style inverted bandwidth characteristics (more demand = more availability).
• Extremely high performance content distribution speeds on commodity hardware.
• WAN and LAN peering between devices.
• Centrally controlled editorial over available content, with eventual-consistency properties to offline or air-gaped clients.
• Cross platform codebase targeting Windows, MacOS and Linux. The dev team utilized all 3 platforms during development, interchangeably.

While the project was built on the backs of giants like the Hypercore protocol, as well as the amazing work of my colleague, I contributed in a number of areas to move the project forward during my time contracting on it.

• Take ownership of the preexisting React application.
• Simplify and maintain a mix of internal open source and closed source packages.
• Simplify React app state layer by eliminating the need for Redux in favor of built in React state solutions.
• Implement offline mode via Service workers in conjunction with Hypercore.
• Packaging and delivery tasks.
• Improved progress UI/UX.
• Contribute to native packaging tools build with pkg and tiny-module-compiler.

Some of the discrete software packages that resulted from this project are described below.

secure-rpc-protocol

Secure rpc-protocol over any duplex socket using noise-protocol. This was a refactor of an existing RPC over websocket solution the project was using. It improved upon the previous secure RPC already used by switching to using the noise protocol which implements well understood handshake patterns that can be shared and audited between projects, rather than relying on a novel implementation at the RPC layer. It also decoupled the RPC protocol from the underlying socket being used, so that the RPC system could be used over any other channels we might want in the future.

async-folder-walker

An async generator that walks files. This project was a refactor of an existing project called folder-walker implementing a high performance folder walk algorithm using a more modern async generator API.

unpacker-with-progress

unpacker-with-progress is a specialized package that unpacks archives, and provides a progress api in order to provide UI feedback. One of the deficiencies with the NYCPF project when I started was lack of UI feedback during the extraction process.

VR files are very large, and are delivered compressed to the clients. After the download is complete, the next step to processing the data is unpacking the content. This step did not provide any sort of progress feedback to the user because the underlying unpacking libraries did not expose this information, or only exposed some of the information needed to display a progress bar.

This library implemented support for unpacking the various archive formats the project required, and also added an API providing uniform unpacking progress info that could be used in the UI during unpacking tasks.

hchacha20

One of the more interesting side projects I worked on was porting over some of the libsodium primitives to sodium-javascript. I utilized a technique I learned about at Hyperdivision where one can write web assembly by hand in the WAT format, providing a much wider set of data types, providing the type guarantees needed to write effective crypto.

While the WAT was written for HChaCha20, the effort was quite laborious and it kicked off a debate as to whether it would be better to just wrap libsodium-js (the official libsodium js port) in a wrapper that provided the sodium-universal API. This was achieved by another group in geut/sodium-javascript-plus which successfully ran hypercores in the browser using that wrapper.

Ultimately, this effort was scrapped, determining that noise peer connections in the browser are redundant to webRTC encryption and https sockets. It was a fun and interesting project none the less.

Reconnecting sockets

We were having some state transition bugs between the webapp and the local server process, where the app could get into strange indeterminate states. Both had independent reconnect logic wrapped up with application code, and it added a lot of chaos to understanding how each process was behaving when things went wrong (especially around sleep cycles on machines).

I implemented a generic reconnecting state machine that could accept any type of socket, and we were able to reduce the number of state transition bugs we had.

• little-core-labs/reconnecting-socket
• little-core-labs/reconnecting-simple-websocket

Little Core Labs

After working on various agency projects at Littlstar, we formed a separate organization to start a fresh rewrite of the technology stack.

My high level contributions:

• Terraform running in Github actions
• Provisioning AWS infrastructure.
• Helping come up with how all of this stuff will work.

This was an amazing founders-style opportunity to help rethink and re-implement years of work that had developed at Littlstar prior to joining. Effectively starting from 0, we rethought the entire technology pipeline, from operations, to infrastructure, to deployment, resulting in something really nice, modern, minimal, low maintenance and malleable.

• Little Core Labs Github
• Little Core Labs Twitter

Terraform ops

A culmination of ingesting the “Terraform: Up & Running” and “AWS Certified Solutions Architect” books, as well as building off existing organizational experience with AWS, I helped research and design an operations plan using Terraform and GitHub Actions.

This arrangement has proven powerful and flexible. While it isn’t perfect, it has been effective and reliable and cheap, despite its imperfections in relation to some of the more esoteric edge cases of Terraform.

A quick overview of how its arrange:

• We have a global terraform repository with segmented terraform files for various services.
• The ops global repo runs terraform in a bootstrapped GitHub actions environment.
• We can create service level repos from the ops terraform repo, that in turn contain their own Terraform files specific to that service.
• One of the benefits was that the GitHub environment worked along side a local environment, due to the use of AWS secrets manager and GitHub actions secrets (all managed in Terraform), so debugging was easy and flexible.

Github actions

One of the drawbacks of rolling our own Terraform CI infrastructure was that we had to tackle many small edge cases inside the GitHub actions environment.

It was nice to learn about the various types of custom GitHub actions one can write, as well as expand that knowlege to the rest of the org, but it also ate up a number of days focusing on DevOps problems specific to our CI environment.

Here are some of the problems I helped solve in the actions environment.

• install-terraform - Install terraform at a specific version.
• netrc-creds - Set up netrc credentials.
• get-git-tag - Easily get a git tag.

sdk-js

I helped lay the framework for the initial version of sdk-js, the Little Core Labs unified library used to talk to the various back-end services at Little Core Labs.

One of underlying design goals was to solve for the newly introduced native ESM features in node, in such a way that the package could be consumed directly in the browser, natively as ESM in node, but also work in dual CJS/ESM environments like Next.js. While this did add some extra overhead to the project, it serves as a design pattern we can pull from in the future, as well as a provide a highly compatible but modern API client.

I also extracted out this dual package pattern into a reusable template.

• bcomnes/esm-template

rad.live

I was the principal engineer on the new Rad.live website. I established and implemented the tech stack, aiming to take a relatively conservative take on a code base that would maximize the readability and clarity for a Dev team that would eventually grow in size.

A high level, the application is simply an app written with:

• Next.js - React with a framework to provide standard bundling, SSR and routing features.
• SWR - Declarative data ‘hooks’, minimizing/eliminating complex state management throughout the app.
• GQLR - A simple GraphQL client tuned to ‘just work’.
• styled-jsx - A simple and understandable CSS-In-JS framework that minimizes orthogonal layout CSS.

From a development perspective, it was important to have testing and automation from the beginning. For this we used:

• GitHub actions to run CI on every interaction.
• Organizational standard linting that supported the unique needs of React.
• Storybook for data/state independent component testing (Using the new and improved storybook API)
• Automated component testing, with a minimum of ‘does it render’ testing utilizing the storybook stories.
• Release automation.

Overall, the codebase has had two other seasoned developers (one familiar and one new at React) jump in and find it productive based on individual testimony. Gathering feedback from those thatI work with on technical decisions that I make is an important feedback look I always try to incorporate when green fielding projects. Additionally, it has been a relatively malleable code base that is easy to add MVP features to and is in a great position to grow.

vision.css

I implemented a custom design system in tandem with our design team. This project has worked out well, and has so far avoided ‘css lockout’, where only one developer can effectively make dramatic changes to an app layout due to an undefined and overly general orthogonal ‘global style sheet’.

The way this was achieved was by focusing on a simple global CSS style sheet that implements the base HTML elements in accordance with the design system created by the design team. While this does result in a few element variants that are based on a global style class name, they remain in the theme of only styling ‘built in’ html elements, so there is little question what might be found in the global style sheet, and what needs to be a scoped css style.

Some of the features we used for vision.css

• Standard CSS syntax with a selection of a few ‘yet-to-be-released’ css syntax features.
• A postcss build pipeline.
• Heavy use of CSS variables.
• A website that contains a style-guide and preview of the various styled elements.

eslint-config-12core

Linting is the “spell check” of code, but its hard to agree on what rules to follow. Like most things, having a standard set of rules that is good-enough is always better than no rules, and usually better than an unpredictable and unmaintainable collection of project-unique rules.

I put together a shared ESLint config that was flexible enough for most projects so far at Little Core Labs based on the ‘StandardJS’ ruleset, but remained flexible enough to modify unique org requirements. Additionally, I’ve implemented it across many of the projects in the Github Org.

• eslint-config-12core

gqlr

gqlr is a simplified fork of graphql-request.

This relatively simple wrapper around the JS fetch API has a gaggle of upstream maintainers with various needs that don’t really match our needs.

The fork simplified and reduced code redundancy, improved maintainability through automation, fixed bugs and weird edge cases and dramatically improved errors and error handling at the correct level of the tech stack. These changes would have unlikely been accepted upstream, so by forking we are able to gain the value out of open source resources, while still being able to finely tune them for our needs, as well as offer those changes back to the world.

• gqlr

local-storage-proxy

A configuration solution that allows for persistent overrides stored in local storage, including cache busting capabilities. Implemented with a recursive JS proxy to simulate native object interactions over a window.localstorage interface.

• local-storage-proxy

Community maintenance

We ended up taking on maintainence of a few other packages, providing fixes and improvements where the original authors seem to have left off.

• little-core-labs/date-input-polyfill - Polyfill for HTML5 date input element.
• little-core-labs/chromafi

Video platform

Here are some snapshots of the video platform we launched.

NFT Auction Platform

Here are some screenshots of the NFT auction platform I helped build. The UI was fully responsive and updated on the fly to new results, thanks to the powers of SWR.

Marketing pages

I did a few marketing pages as well.

Conclusion

While this isn’t everything I did at Littlstar, it captures many of the projects I enjoyed working on, and can hopefully provide some insights into my skills, interests and experiences from the past year.