Micah's Blog

Conditional Logic with Zod + React Hook Form

Micah Engle-Eshleman — Thu, 16 Nov 2023 06:54:00 +0000

When I started at Redwood last year, it became clear that a big part of my job would involve building long and complex forms. Like most software, the easiest to use forms often mask a lot of complexity under the hood, and I knew from experience that schemas and type safety would be my friends. After some research I settled on:

React Hook Form for managing form state changes w/ minimal re-renders
Zod for writing form schemas and doing validation

Almost a year later, with quite a few forms in production, I'm still happy with this pairing. BUT, there were definitely some major gotchas along the way, which is why I'm writing this post.

A super simple form

Let's start with just a single email field. We use Zod to validate the email address and also generate a FormSchema type that React Hook Form uses to infer the field names & values.

import { useForm } from "react-hook-form";
import { z } from "zod";
import { zodResolver } from "@hookform/resolvers/zod";

// Setup Zod schema
const formSchema = z.object({
  email: z.string().email(),
});
type FormSchema = z.infer<typeof formSchema>;

export function Form() {
  // Setup React Hook Form validated by Zod
  const {
    register,
    handleSubmit,
    formState: { errors },
  } = useForm<FormSchema>({
    resolver: zodResolver(formSchema),
  });

  // data is guaranteed to match FormSchema 
  const onSubmitSuccess = (data: FormSchema) => {
    console.log("Valid submission:", data);
  };

  return (
    <form onSubmit={handleSubmit(onSubmitSuccess)}>
      <label>
        Email
        {/* If we were to mis-type "email", we'd get a Typescript error, yay! */}
        <input type="email" {...register("email")} />
        {errors.email && <span>{errors.email.message}</span>}
      </label>
      <button type="submit">Submit</button>
    </form>
  );
}

This example is a bit contrived, as we could accomplish all of this with plain HTML:

<input type="email" required />

But the benefit of Zod is that it allows you to apply custom validation to any field type and you're guaranteed to end up with the schema you expect upon submission (or an intelligible error message).

The importance of default values

There is one bug in our simple form above, and it's due to missing default values. According to FormSchema, email must always be a string, so we'd expect it to start out as an empty string. But React Hook Form does not assign default values, so email starts off as undefined. As expected, when watching the value, we get undefined (matching React Hook Form's internal state), but the getValues() function will return an empty string if called after the email input has been rendered, presumably b/c it actually checks the email input's .value property.

const {
  register,
  handleSubmit,
  control,
  getValues,
  formState: { errors },
} = useForm<FormSchema>({
  resolver: zodResolver(formSchema),
});

// useWatch() gets the form's values as tracked in JavaScript
const watchedEmail = useWatch({ control, name: "email" });
console.log("Watched email:", { email: watchedEmail });

// getValues() gets the form's values from the DOM
console.log("Email field value:", { email: getValues().email });

// Force re-rendering after email field has been rendered
const [, forceUpdate] = useReducer((x) => x + 1, 0);
useEffect(() => {
  console.log("Force re-rendering...");
  forceUpdate();
}, []);

This may not sound like a big deal, but can really bite you if you have any logic based on email, as it would break during the forms first render despite Typescript being positive that it's a string!

The solution is simple: provide default values (like the React Hook Form docs recommend)

} = useForm<FormSchema>({
  resolver: zodResolver(formSchema),
  defaultValues: { email: "" }, // <-- Default value
});

Note: if using reset() to clear the form, be sure to pass default values so the form is actually reset to it's initial values as you'd expect.

Fields with more complex types

Sometimes it's nice to be able to store a value in a form that isn't a simple string, number, or other primitive type.

For instance, let's say we want the user to select a color from a list of options in dropdown list, where each option has a label and a value.

type Option = { label: string, value: string }
const colorOptions: Option[] = [
  { label: 'Red', value: '#E83845' },
  { label: 'Brown', value: '#9B634C' },
  { label: 'Tan', value: 'F2DFB4' },
]

Representing this in Zod schema is trivial:

const optionSchema = z.object({
  name: z.string(),
  label: z.string(),
});
const formSchema = z.object({
  email: z.string().email(),
  color: optionSchema,
});
type FormSchema = z.infer<typeof formSchema>;

But what should our default value for this field be? I would reach for null in this case, since it clearly indicates that it's an empty value.

} = useForm<FormSchema>({
  resolver: zodResolver(formSchema),
  defaultValues: { email: "", color: null }, // <-- Default values
});

Hmm, null doesn't satisfy FormSchema:

Of course you could use a different initial value that does satisfy FormSchema, e.g. {value: '', label: ''}, but at the end of the day your initial values shouldn't be constrained by your final values. I eventually threw up my hands, took a deep breath, and made peace with defining two schemas:

formSchema represents a valid (complete) form submission
blankFormSchema represents the form's default (blank) values

// Valid form submission
export const formSchema = z.object({
  email: z.string().email(),
  color: optionSchema,
});
export type FormSchema = z.infer<typeof formSchema>;

// Blank form values
const blankFormSchema = formSchema.extend({ color: optionSchema.nullable() });
export type BlankFormSchema = z.infer<typeof blankFormSchema>;

Whew! That's a lot of boilerplate just for type safety. FWIW, I typically create a separate file with a form's schemas (e.g. my-form-schemas.ts) to avoid cluttering my React component file.

If you sweat the details as much as I do (sigh), you may have noticed that blankFormSchema still expects email to be a valid email, not an empty string. While this is true, since the purpose of this schema is just to generate a more permissive Typescript type, it doesn't matter. If it bothers you, could you construct that type explicity:

type Modify<T, R> = Omit<T, keyof R> & R;
export type BlankFormSchema = Modify<
  FormSchema,
  {
    color: FormSchema["color"] | null;
  }
>;

I typically define default values below my schemas. Verbose, I know. Evidently, it is possible to programmatically generate default values from a Zod schema using a hacky workaround like this one, but I am reluctant to adopt it until it's actually documented and supported officially by Zod.

export const defaultValues: BlankFormSchema = {
  email: "",
  color: null,
};

// or if I'm worried about downstream mutation...
export const getDefaultValues = (): BlankFormSchema => ({
  email: "",
  color: null,
});

Now that we've got two schemas, how do we apply them to the form itself?

const {
  register,
  handleSubmit,
  formState: { errors },
} = useForm<BlankFormSchema>({ // BlankFormSchema encompasses all possible field values
  resolver: zodResolver(formSchema), // formSchema is used for validation
  defaultValues,
});

const onSubmitSuccess = (data: BlankFormSchema) => {
  // We know that upon submission, data satisfies FormSchema
  console.log("Valid submission:", data as FormSchema);
};

I'm not thrilled with this solution, but it gets the job done. Within our component, Typescript uses BlankFormSchema so that we are sure to handle all possible field values, but upon submission we are certain that the resulting data matches FormSchema.

Conditional Logic

I wish there was an easy solution to this.

At its core, conditional logic means using the values in some fields to determine whether to show or hide other fields that depend on them. If we think of our typical form as a list of fields (or a tree with no branches), conditional logic introduces branches: multiple paths a user can take through our form.

             [name]
                |
             [email]
                |
(condition: is this email domain in our database?)
        / yes           \ no
      (submit)      [company name]
                           |
                   [company location]
                       (submit)

To represent this accurately in Zod & Typescript, we'd need to extend the base schema for every conditional logic branch in our form. Will this scale to large forms? Probably not. But let's at least go through the exercise of seeing what it would entail for the above example:

const formSchemaLeft = z.object({
  name: z.string().trim().min(3),
  email: z.string().email(),
});
const formSchemaRight = formSchemaLeft.extend({
  companyName: z.string().trim().min(2),
  companyLocation: z.string().trim().min(5),
});

const formSchema = z.union([formSchemaLeft, formSchemaRight]);
type FormSchema = z.infer<typeof formSchema>;

...which gives us a beautiful union type:

Let's imagine a user fills out this form on the right branch but fails to enter a company location before submitting. React Hook Form passes the following object to Zod for validation:

formSchema.parse({
  name: "Micah",
  email: "micah@my-company.com",
  companyName: "My Company",
  companyLocation: "",
}) 
// Result = {name: 'Micah', email: 'micah@my-company.com'}
// What? Company location is blank!

For union schemas, Zod will return the first schema that passes (left branch). It does not know that we only really care about the right branch in this scenario. We could try and make these schemas .strict() (not accepting additional properties), in which case the left branch would also fail validation, but in the case of both branches failing (left due to extra properties, right due to missing company name), Zod only exposes the error for the first failing schema in the union.

The solution is to conditionally pass React Hook Form the schema that matches our current branch. Here's how that might look...

const { register } = useForm<BlankFormSchema>({
  resolver: function (values: BlankFormSchema, ...args) {
    // Choose correct schema for current branch
    const currentBranchSchema = isKnownDomain(values.email)
      ? formSchemaLeft
      : formSchemaRight;
    // Validate form data against chosen schema
    return zodResolver(currentBranchSchema)(values, ...args);
  },
  defaultValues,
});

Wow, that actually works! Unfortunately, this solution has some major issues:

At a high level, do we really want to have to define a new schema for every conditional logic branch?
Our conditional logic now lives in 3 different places
- In our various schemas
- In our custom resolver function (above)
- In our Form component JSX (useWatch() to show/hide fields)

Gross!

Problem #1 (a new schema for every branch) is inherent to Zod, and will hopefully get better in a future version of Zod. Maybe z.switch(), the potential successor to z.discriminatedUnion() will make this possible (see discussion here). The TLDR is fixing it requires deep knowledge of Zod and Typescript, something I don't have.

I was able to solve Problem #2 by giving up on the "multiple Zod schemas" approach and instead creating a single Zod schema where every conditional field is .optional(). Then I defined the conditional logic in a separate JavaScript object which could be re-used in both the custom resolver (to prune hidden fields before validating) and in the Form component (to determine if a field is visible). I added some other fancy stuff like dependency tracking and published a tiny library that makes this pretty easy: rhf-conditional-logic.

Our above example becomes:

// Define a single form schema with conditional fields optional
const formSchema = z.object({
  name: z.string().trim().min(3),
  email: z.string().email(),
  companyName: z.string().trim().min(2).optional(),
  companyLocation: z.string().trim().min(5).optional(),
});
type FormSchema = z.infer<typeof formSchema>;

// Define conditional logic (and track dependencies)
const conditions = {
  companyName: getValues => !isKnownDomain(getValues('email')),
  companyLocation: getValues => !isKnownDomain(getValues('email')),
};

While I'd love if there was a way to define conditional logic within the schema, this feels like the next best thing.

Note that using a single schema with optional values results in a less precise Typescript type. The benefit of course is that you no longer need to define a new schema for every branch!

In the form component we consume this conditional logic with the useConditionalForm and useCondition hooks, which wire up the custom resolver and track dependencies for us.

export function Form() {
  const { register, getValues, control } = useConditionalForm<FormSchema>({
    conditions, // Your conditional logic definition goes here
    resolver: zodResolver(formSchema), // Required
    defaultValues, // Required
  });

  // Should the "Company Name" field be shown?
  const [showCompanyName] = useCondition(['companyName'], conditions, getValues, control)
}

You can read the full README here, but hopefully this gives you the gist.

If you have other ideas for how to improve this, don't hestitate to leave a comment or shoot me an email. Thanks!

A Configurable Hamburger Icon to Close (X) Icon Transition

Micah Engle-Eshleman — Wed, 11 Jan 2023 19:08:00 +0000

Over the past few weeks I've been gradually swapping out Emotion CSS-in-JS with CSS modules. While I do love Emotion's developer experience, I just ran into too many issues in production with jank during animations. And given that CSS modules can be SASS or SCSS, the transition was pretty easy.

In this process, I stumbled across a nifty hamburger icon to close (X) icon animation (that I spent waaay to much time perfecting last year) written in Emotion. While I will be converting it to CSS modules so I can finally lay this dependency to rest, I figured it'd be worth sharing.

Note: the blue outline is just a focus style, not part of the icon

There are MANY examples of this type of animation, but I have yet to find one that allows you to customize line thickness and spacing without skewing the animation. This implementation also ensures the resulting icon animation fits perfectly in a square.

Emotion CSS in JS

// Emotion styles
import styled from '@emotion/styled';
import { css } from '@emotion/react';

const boxSize = 18;
const xSide = boxSize / Math.SQRT2;
const spaceBetween = Math.round(boxSize / 5);
const lineHeight = 3;
const spaceAbove = (boxSize - 3 * lineHeight - 2 * spaceBetween) / 2;
const durationAndEasing = '.25s ease';

const HamburgerLineStyles = css`
  display: block;
  position: absolute;
  height: ${lineHeight}px;
  width: 100%;
  background: currentColor;
  border-radius: ${lineHeight}px;
  opacity: 1;
  left: 0;
  transform-origin: left center;
`;

export const HamburgerLine1 = styled('span')`
  ${HamburgerLineStyles}
  top: ${spaceAbove}px;
  transform: rotate(0deg);
  transition: transform ${durationAndEasing};
`;

export const HamburgerLine2 = styled('span')`
  ${HamburgerLineStyles}
  top: ${spaceAbove + lineHeight + spaceBetween}px;
  transition: width ${durationAndEasing}, opacity ${durationAndEasing};
`;

export const HamburgerLine3 = styled('span')`
  ${HamburgerLineStyles}
  top: ${spaceAbove + (lineHeight + spaceBetween) * 2}px;
  transform: rotate(0deg);
  transition: transform ${durationAndEasing};
`;

type ToggleProps = {
  isOpen: boolean;
};

export const Hamburger = styled('span')<ToggleProps>`
  position: relative;
  width: ${boxSize}px;
  height: ${boxSize}px;
  flex-shrink: 0;
  ${HamburgerLine1} {
    ${(props) =>
      props.isOpen
        ? `transform: 
          translateY(${-1 * (xSide / 2 - lineHeight - spaceBetween)}px) 
          translateX(${(boxSize - xSide) / 2}px) 
          rotate(45deg);`
        : ''};
  }
  ${HamburgerLine2} {
    ${(props) => (props.isOpen ? `width: 0%; opacity: 0;` : '')};
  }
  ${HamburgerLine3} {
    ${(props) =>
      props.isOpen
        ? `transform: 
          translateY(${xSide / 2 - lineHeight - spaceBetween}px) 
          translateX(${(boxSize - xSide) / 2}px) 
          rotate(-45deg);`
        : ''};
  }
`;

// Usage in React component
<button>
  <Hamburger isOpen={isOpen}>
    <HamburgerLine1 />
    <HamburgerLine2 />
    <HamburgerLine3 />
  </Hamburger>
</button>

Here's the version of the code migrated to CSS modules (compiled from SCSS). This should work independent of React or any other framework.

Pure SCSS

// style.scss

.hamburger {
  // Variables
  --box-size: 18px;
  --x-side: calc(var(--box-size) / 1.4142135623730951);
  --space-between: 4px;
  --line-thickness: 3px;
  --space-above: calc(
    (var(--box-size) - 3 * var(--line-thickness) - 2 * var(--space-between)) / 2
  );

  position: relative;
  width: var(--box-size);
  height: var(--box-size);
  flex-shrink: 0;

  // All 3 lines
  > * {
    display: block;
    position: absolute;
    height: var(--line-thickness);
    width: 100%;
    background: currentColor;
    border-radius: var(--line-thickness);
    opacity: 1;
    left: 0;
    transform-origin: left center;
    transition-duration: 0.25s;
    transition-timing-function: ease;
  }

  // Top line
  > :nth-child(1) {
    top: var(--space-above);
    transform: rotate(0deg);
    transition-property: transform;
  }
  &.hamburgerToX {
    > :nth-child(1) {
      transform: translateY(
          calc(-1 * (var(--x-side) / 2 - var(--line-thickness) - var(--space-between)))
        )
        translateX(calc((var(--box-size) - var(--x-side)) / 2)) rotate(45deg);
    }
  }

  // Middle line
  > :nth-child(2) {
    top: calc(var(--space-above) + var(--line-thickness) + var(--space-between));
    transition-property: width, opacity;
  }
  &.hamburgerToX {
    > :nth-child(2) {
      width: 0%;
      opacity: 0;
    }
  }

  // Bottom line
  > :nth-child(3) {
    top: calc(var(--space-above) + 2 * (var(--line-thickness) + var(--space-between)));
    transform: rotate(0deg);
    transition-property: transform;
  }
  &.hamburgerToX {
    > :nth-child(3) {
      transform: translateY(
          calc(var(--x-side) / 2 - var(--line-thickness) - var(--space-between))
        )
        translateX(calc((var(--box-size) - var(--x-side)) / 2)) rotate(-45deg);
    }
  }
}

// React or any other framework...
<span class={classNames(style.hamburger, isOpen ? style.hamburgerToX : '')}>
  <span></span>
  <span></span>
  <span></span>
</span>

Enjoy!

A Better Data Structure for Reconciling Late-Arriving User Data

Micah Engle-Eshleman — Sun, 17 Jul 2022 17:45:00 +0000

I’m a big fan of approaches to web development that take offline support seriously. Air travel and subway commutes aside, offline-friendly patterns dramatically improve UX by enabling optimistic UI updates in the much more common scenario of a flaky network or a slow API.

While you could roll your own, implementations like Replicache's "Realtime Sync" make it easy to implement this pattern across your app: update the UI right away, store all data locally, and then lazily sync changes to your API in the background.

This last bit—lazily syncing changes to your API—sounds deceptively simple. I've learned the hard way that whenever there's more than one device involved and significant delays in syncing updates (e.g. a device comes back online after a few hours of offline edits), you end up having to reconcile conflicting changes. There's a reason Replicache makes you write your own conflict resolution—it's often domain specific and tricky to get right.

I've dealt with this problem when syncing user settings to the BeFunky API and more recently when building a podcast app. In both cases, users can be logged-in even if they're offline (at least for a short while), and when they come back online their updated settings must be reconciled.

This post is an attempt at finding a common solution for merging simple out-of-sync collections of user data.

A high level view of the problem

The best mental model I've found for thinking about merging collections of data (think objects or arrays) from various devices is Git branches. Two principles emerge:

Merging branches is trivial when they don't modify the same files. Thus, if two collections' changed keys/indices don't overlap, they'll merge without conflicts.
Git keeps track of what changed (additions and removals) and when it changed (timestamps). It's not enough to simply know the final state of two branches to merge them reliably, it's helpful to know how they both changed over time.

Dealing with Arrays

Adblock Podcast has a "Listen Later" feature, stored as an array of episode IDs that get removed once the podcast has been listened to. Imagine this scenario on a subway commute:

Three episodes are in the "Listen Later" queue: [A1, B2, C3]
User listens to the second episode on their phone: [A1, C3]. They're in the subway so the data doesn't sync.
At the office, they open the web app in a browser and finish listening to the third episode [A1, B2] and queue up another one: [N0, A1, B2].
At lunch, they pull out their phone and it finally syncs up with the API.

API State: [N0, A1, B2]
Phone State: [A1, C3]
Ideal Merge: [N0, A1]

If we just know the state of the API and the phone, it's unclear what to do with items N0, B2, & C3. It's essential to know the diff from the phone:

API State: [N0, A1, B2]
Phone Diff: (Remove B2)
Easy Merge: [N0, A1]

Suddenly, the problem gets much simpler.

Keep in mind that it's important to diff the array's values, not its indices. From the API's perspective, B2 is at index 2 and from the phone's perspective, B2 is at index 1. Thus, sending an standard JS Array Diff (e.g. a JSON Patch) isn't going to cut it.

Instead, ensure each value has a unique & stable identifier. In the case of episode IDs, the ID itself will do, but in other cases you may want to use a hash or UUID.

Store a timestamp with every change

In the above example, the conflict was relatively easy to resolve b/c the phone and browser mutated different array items.

When multiple devices update the same item in a collection, a conflict occurs, much like a Git merge of two branches that modify the same files. Unlike Git, we can't press pause and sort out the conflict manually, so the resolution I usually reach for is "newest change wins".

To accurately do this it's important to know when a change was made on the client, not just when that change was synced to the server. This may seem like an edge case, but there's nothing worse than a device coming back online after a couple hours and syncing stale data, overwriting more recent changes!

Storing a timestamp for each piece of data in a collection could look something like this:

// Data structure that's easy to diff & merge
const listenLaterAddedItems = {
  N0: <timestamp>,
  A1: <timestamp>,
};
const listenLaterRemovedItems = {
  B2: <timestamp>,
  C3: <timestamp>,
};
// Derived state for other purposes (e.g. rendering)
const listenLaterArray = Object.keys(listenLaterAddedItems);

Notice we also track removed items to avoid re-adding an item from a stale update when it has been more recently removed. Ugh!

This is just pseudocode and it's already looking more complex than I'd like. All of this just to store a simple array of episode ids?

Using a Library

Fortunately, this pattern generalizes pretty well to any collection of data where:

Each piece of data has a unique and stable identifier
Preserving item order isn't essential
Old items should be replaced by newer items with the same key

If you're interested in trying it out, I wrote a tiny library (1.0KB gzipped) that implements this pattern, making it easy to store any object or array in this format and sync it between multiple clients and an API:

github.com/micahjon/timestamp-collection (name tbd)

Stores a "collection" of unique keys, each with a timestamp and an optional value

listenLaterCollection.add('A1');
// Stored as { A1: [<timestamp>] }, representing array ['A1']

podcastProgressCollection.add('A1', Date.now(), 14.2)
// Stored as { A1: [<timestamp>, 14.2] }, representing object { A1: 14.2 }

add() and remove() methods that only mutate the collection if the timestamp is newer
get() computed properties with stable references, making it trivial to derive an array (or any other data structure) and use it for rendering with minimal performance impact.

// store.js
// Will return same array reference if IDs have not changed
export const getEpisodeIds = () => listenLaterCollection.get(Object.keys); 

// listen-later.js
export const ListenLater = () => {
  const episodeIds = getEpisodeIds();

  useEffect(() => {
    // episodeIds reference needs to be stable or this will run every time
  }, [episodeIds]);
}

A built-in hash that makes it easy to compare collections (e.g. on client and API) to know if they're identical and avoid syncing data when they are.
You can import/export the entire collection or a subtree of recent changes as JSON, making it easy to sync without sending the whole thing over the wire.

I have a hard time believing that someone else hasn't already written this sort of thing, but I haven't been able to find it. If you know of a library that implements this pattern already, or have other suggestions, please let me know!

How to Track Audio/Video File Loading Progress with Workbox

Micah Engle-Eshleman — Mon, 23 May 2022 07:00:00 +0000

When working on my Adblock Podcast side project, I had to figure out how to download and cache audio files using a service worker, and show the download progress in the UI. You'll run into the same problem for video and other large files that need to be downloaded for offline viewing.

Workbox is a fantastic library for this, and has great documentation on how to write a plugin to hook into particular parts of the fetching & caching lifecycle. I'd also recommend their article on the nuances of serving cached audio & video.

Fortunately, there's an excellent Github repo (AnthumChris/fetch-progress-indicators) with a service worker example of tracking download progess using a ReadableStream, which we can adapt as a Workbox plugin.

First off, register a CacheFirst route that deals exclusively with the files you'd like to track (in my case, audio files).

// service-worker.js
import { registerRoute } from 'workbox-routing';
import { CacheFirst } from 'workbox-strategies';
import { RangeRequestsPlugin } from 'workbox-range-requests';
import { TrackFileProgressPlugin } from './lib/track-file-progress-plugin'; // to-do

registerRoute(
  ({ request }) => {
    // Match all relevant audio files here...
    return request.url.startsWith('https://your-domain.com/audio/') 
      && /\.(mp3|m4a|wav)$/.test(request.url)
  },
  new CacheFirst({
    cacheName: 'audio-files',
    plugins: [
      // We must store full files in ServiceWorker cache, but this plugin allows
      // requests with range: header (e.g. from <audio> or <video> elements) to
      // be served only a portion of the cached file instead of the whole thing
      new RangeRequestsPlugin(), 
      // Our custom plugin!
      new TrackFileProgressPlugin(),
    ],
  })
);

// Pre-cache other files *after* defining the above

Then define a TrackFileProgressPlugin that tracks download progress as it saves the file.

Note: the following file is written in TypeScript. To convert it to plain JavaScript you can just paste it in on this website: www.typescriptlang.org/play

// track-file-progress-plugin.ts
import { WorkboxPlugin } from 'workbox-core';

export class TrackFileProgressPlugin implements WorkboxPlugin {
  broadcast: BroadcastChannel;

  constructor() {
    // Setup broadcast channel for communicating download progress
    this.broadcast = new BroadcastChannel('audio-downloads');
  }

  /**
   * Called before a Response is used to update a cache
   * @return {Response|null} - Return null to avoid caching
   */
  cacheWillUpdate: WorkboxPlugin['cacheWillUpdate'] = async ({
    response,
    request,
    state,
  }) => {
    if (!shouldCacheResponse(response)) return null;

    // Helper function for sending progress to client
    // Added to state object so cacheDidUpdate method can access it
    state.reportProgress = (progressPercent: number) => {
      this.broadcast.postMessage({
        type: 'DOWNLOAD_PROGRESS',
        url: request.url,
        progress: progressPercent,
      });
    }

    // Clone response b/c stream can only be used once (either for tracking download
    // or for saving to cache, not both)
    const clonedResponse = await response.clone();
    trackDownloadProgress(clonedResponse, state.reportProgress);

    // Response is ready to cache
    if (response.status === 200 && response.headers.has('content-range')) {
      return response;
    }

    // Convert status from 206 -> 200 to make it cacheable (needed if response was
    // requested by fetch instead of by an Audio element)
    const status = 200;

    // Add content-range header if missing from 200 response (needed for iOS Safari)
    const headers = new Headers(response.headers);
    if (!response.headers.has('content-range')) {
      const contentLength = getFileSize(response);
      headers.set('content-range', `bytes 0-${contentLength - 1}/${contentLength}`);
    }

    return new Response(response.body, { status, headers });
  };

  /**
   * Called after Response is successfully saved to cache
   */
  cacheDidUpdate: WorkboxPlugin['cacheDidUpdate'] = async ({ state }) => {
    // Optional: guarantee that file is marked as fully downloaded in the event
    // that progress tracking fails
    if (state.reportProgress) state.reportProgress(1);
  };
}

// Helper function for tracking download progress
// Adapted from: https://github.com/AnthumChris/fetch-progress-indicators/blob/master/sw-basic/sw-simple.js#L41
function trackDownloadProgress(response: Response, reportProgress: Function) {
  // Start tracking
  reportProgress(0);

  let totalBytes: number;
  try {
    // Ensure that the browser supports ReadableStream and we know total file size
    if (!response.body) throw 'response.body missing';
    totalBytes = getFileSize(response);
  } catch (error) {
    console.error('Failed to track download progress', error);
    return;
  }

  let loadedBytes = 0;
  const reader = response.body.getReader();

  new ReadableStream({
    async start(controller) {
      read();

      function read() {
        reader
          .read()
          .then(({ done, value }: { done: boolean; value: Uint8Array }) => {
            if (done) {
              controller.close();
              return;
            }

            controller.enqueue(value);
            loadedBytes += value.byteLength;
            reportProgress(loaded / totalBytes);
            read();
          })
          .catch((error: any) => {
            // Error only typically occurs if network fails mid-download
            console.error('error in read()', error);
            controller.error(error);
          });
      }
    },

    // Firefox excutes this on page stop, Chrome does not
    cancel(reason) {
      console.log('cancel()', reason);
    },
  });
}

/**
 * Get total file size in bytes
 */
function getFileSize(response: Response) {
  // If content is encoded, then content-length will not be accurate
  if (response.headers.get('content-encoding')) throw 'content-encoding header';
  
  // We use content-length header to get total file size
  const contentLength = response.headers.get('content-length');
  if (contentLength === null) throw 'content-length missing';

  return parseInt(contentLength);
}

/**
 * Only cache file if the full file is provided. Don't cache
 * partial 206 responses  
 */
function shouldCacheResponse(response: Response) {
  if (response.status === 200) {
    // Ensure the file size is known (to derive content-range header)
    try {
      getFileSize(response);
    } catch (err) {
      return false;
    }
    return true;
  }

  if (response.status === 206) {
    try {
      // Did 206 response include entire file?
      const contentLength = getFileSize(response);
      return (
        `bytes 0-${contentLength - 1}/${contentLength}` ===
        response.headers.get('content-range')
      );
    } catch (err) {}
  }

  return false;
}

The only remaining item is to setup and listen to the broadcast channel on your page:

// app.js
const broadcast = new BroadcastChannel('audio-downloads');
broadcast.onmessage = (event) => {
  if (event.data && event.data.type === 'DOWNLOAD_PROGRESS') {
    const { url, progress } = event.data;
    // Handle progress...
  }
};

BroadcastChannel is a super simple API, but is only supported by Safari 15.4+, so you may want to use a different way of communicating with the client to support older browsers. There's actually quite a few options.

Also, if you're explicitly downloading files with fetch() requests instead of <audio> or <video> elements, then you can omit the RangeRequestsPlugin and my conditions for 206 responses.

And if you run into any snags or have suggestions for improvement, don't hesitate to leave a comment! I simplied some of this code for this post, so apologies in advance if there are any typos.

How to Move a CenturyLink Fiber Modem & ONT

Micah Engle-Eshleman — Thu, 10 Feb 2022 07:50:00 +0000

Last week I switched from Comcast to CenturyLink. 200 mbps upload and download for $50/month feels like a good deal, and the service has been great so far.

I've ran a good bit of ethernet and coax cable wire over the years, but fiber was new to me. Evidently, it's much more finicky:

You can't bend it as much
You're not supposed to look into the end of the cable (lasers?)
You can't get the end of the cable dirty

Moving the Fiber, Modem & ONT downstairs

Just to recap, the ONT is the box that sits between your fiber cable and your modem/router combo.

Fiber cable <-> ONT box <-> Ethernet cable <-> Modem/Router box

1. Unplug Ethernet from ONT

Before unplugging the Ethernet cable from your ONT, make note of which Ethernet port is being used.

Evidently, the ONT Ethernet port matters (it's not like a switch). CenturyLink can use a single ONT to service multiple customer modems. You'll need to plug your modem back into the same Ethernet port on the ONT after moving it, or your internet won't work.

I learned this the hard way... thought I screwed up the fiber cable only to have a CenturyLink guy fix my internet in 3 seconds by plugging the modem into the chosen Ethernet port on the ONT.

2. Unplug Fiber from ONT and protect the end of the cable

Here's a Reddit thread on unplugging the fiber cable from your ONT.

Before you unplug the cable, have a plan for how to protect the end of it while you're moving it to the new location. In my case, I did several things:

Used a sticky note and some tape to wrap a paper tube around the end of the cable, extending past the end to protect the tip. This way nothing is touching the tip if you set it down.
Later when running the wire through the wall, this paper tube got in the way, so I took it off and replaced it with a tape tube make of painter's tape that tapered to a cone at the end. This was much less sturdy, but kept the cord thin enough it went through the hole in the wall easily.
Stick the end of the cord in a ziplock bag when you're setting it down so it doesn't get dirty (especially if you're outside). Of course, you still need to protect the end of the cord first.

Unplugging the cable is actually pretty easy. Just pull on the squarish green cable terminator, not on the cable itself. Pull it straight out with a steady hand -- don't bump it against the ONT! Then protect the tiny exposed fiber end w/out touching it or looking into the end of the cable.

3. Run Fiber to new location

This part will depend a lot on the job, but in my case the steps were:

Pry silicon caulk off the hole where fiber enters house
Pull fiber outside of house (put a bag on the end to keep it from getting dirty)
Drill hole into different part of the house (e.g. my basement)
Very carefully push protected cable end into that hole, then pull it all through from the inside
Seal up holes with exterior-rated silicon caulk

4. Hook it all back up

Remember to connect the modem to the same Ethernet port on the ONT that it was connected to before.

Good luck!

Caching Fly.io Apps with Cloudflare

Micah Engle-Eshleman — Sat, 22 Jan 2022 06:30:00 +0000

Services like Fly.io and Render have generous free tiers for compute and bandwidth, but if you're trying to scale a side project for free, sticking Cloudflare in front of your API can dramatically reduce billable usage and improve performance.

I'll go over proxying your Fly.io application through Cloudflare, and then some gotchas regarding caching headers & Cloudflare settings.

Setup Node API on Fly.io

First step is simple -- just follow the Fly.io tutorial to deploy a Node application.

You'll end up with a URL that looks something like this:
https://crazy-cats-4499.fly.dev/

Setup DNS to Route Subdomain to Fly.io

I'm no expert on DNS entries or SSL certificates, but I was able to find my way by reading through the forums. This post in particular was really helpful.

Add your domain to Cloudflare. This will take a while since you'll need to wait for your domain's nameservers to propagate. Full tutorial here
Set Cloudflare SSL/TLS encryption mode to "Full". This means it will encrypt traffic between itself and the origin/backend (in this case Fly.io).
Go to the Fly.io dashboard and find your app, e.g.
https://fly.io/apps/crazy-cats-4499
Navigate to "Certificates" and click "Add Certificate".
Enter a hostname for the certificate. This is a "single-name" certificate and you can read more about it here.

In my case, I just wanted to proxy a subdomain to Fly.io, so I entered my subdomain, e.g. yesiam.crazyaboutcats.com

For more flexibility, you could use a wilcard certificate, e.g. *.crazyaboutcats.com.

Follow the Fly.io instructions to add one or more CNAME records to your DNS settings in Cloudflare.

Make sure the "Proxy status" option is set to "DNS only": grey, not orange!

The _acme-challenge CNAME is required for wildcard certificates but optional for simple subdomain ones.

Back in Fly.io dashboard, try clicking "Check Again" every few minutes until you see the certificates show up. This can take a while, especially if you're using a wildcard certificate. Be patient and go grab a cup of coffee! (could take 20 minutes)

Once these certificates show up, you should be able to visit your subdomain and see the response from Fly.io.

Once you've verified it's working, go back to the DNS entry in Cloudflare and turn on proxying (grey -> orange).

You can verify that Cloudflare is proxying your application by checking the server response header. It should be set to "cloudflare", not "Fly/...".

Cloudflare Caching

Once Cloudflare is proxying your application, you can set HTTP headers and/or Page Rules to configure caching. A couple gotchas that I've had to wrap my head around:

By default, Cloudflare only caches URLs with certain file extensions (e.g. .css) AND the Cache-Control: public, max-age={integer} header must be set. If you want your /rest/api/endpoint to be cached, you'll need to add a Page Rule with Cache Level = Cache Everything, since it's missing a file extension.
Cloudflare will cache your content for a minimum of 14400s (4 hours) by default (or longer if a larger value is set in max-age). You can reduce this to 2 hours in the free tier by setting an "Edge Cache TTL" Page Rule. But you're better off just setting a caching header that specifically targets the CDN (not the browser). Here's some example headers that tell Cloudflare to only cache your endpoint for 1 minute:
- Cache-Control: s-maxage=60
- CDN-Cache-Control: max-age=60
- Cloudflare-CDN-Cache-Control: max-age=60

Hope you found this helpful! If I missed anything, don't hesitate to let me know in the comments.

Detecting a PWA on the Microsoft Store

Micah Engle-Eshleman — Tue, 26 Oct 2021 05:40:00 +0000

A few weeks ago we submitted the BeFunky PWA on the Microsoft Store, after packaging it up with pwabuilder.com.

After launching, we realized we had no way of distinguishing it from our PWA that users could install from their Edge or Chrome browser on Windows.

It turns out that Edge Chromium sets a special document.referrer when first opening your Microsoft Store PWA's start URL:

// Microsoft Store PWA (first page load only)
document.referrer === 'app-info://platform/microsoft-store';

However, when users navigate to subsequent pages (or refresh the start URL), this special referrer will no longer be set. To accurately track additional page views, you'll need to use SessionStorage.

/**
 * If the PWA's referrer is "app-info://platform/microsoft-store", we know that it 
 * was installed from the Microsoft Store
 * Save flag in SessionStorage for tracking future page views
 * @return {boolean}
 */
function isMicrosoftStorePWA() {
  // Check referrer when PWA is first opened
  if (document.referrer === 'app-info://platform/microsoft-store') {
    sessionStorage.setItem('is_ms_store_pwa', 'yes');
    return true;
  }

  // Subsequent navigations
  return sessionStorage.getItem('is_ms_store_pwa') === 'yes';
}

A couple minor notes:

sessionStorage is used instead of localStorage so tracking is limited to the installed PWA session. localStorage is shared between all installed PWAs and website tabs in a given browser engine (in this case Edge Chromium).
For simplicity I'm using sessionStorage directly, but keep in mind that in some rare instances it is not available due to privacy settings. At BeFunky, we wrap sessionStorage and localStorage calls in helper functions with try-catch, e.g.

/**
 * Attempt to get value from SessionStorage
 * @return {string|null|undefined}
 */
function getSessionStorageValue(key) {
  try {
    return sessionStorage.getItem(key);
  } catch (err) {}
}

/**
 * Attempt to set value to SessionStorage
 * @return {boolean}
 */
function setSessionStorageValue(key, value) {
  try {
    sessionStorage.setItem(key, value);
    return true;
  } catch (err) {
    return false;
  }
}

This has worked for us thus far, but if you have any feedback or other ideas for improving this code, please leave a comment below.

Bringing Your PWA to App Stores

Micah Engle-Eshleman — Thu, 07 Oct 2021 20:45:00 +0000

I just gave a talk on the pros/cons and implementation details of packaging your PWA and distributing it through app stores at the PWA Summit conference.

View the slides here

Watch the talk here

How to Accept Payments in Your Play Store TWA

Micah Engle-Eshleman — Mon, 19 Jul 2021 05:29:00 +0000

Distributing your PWA as a TWA in the Google Play Store is relatively painless, but transitioning for a single payment provider (e.g. Stripe) to supporting Play Store billing in your TWA can be tricky. Here are a few significant differences:

Price localization: In Stripe, you choose what currencies you accept and display only those currencies to your users. In the Play Store, USD prices are automatically converted to the user's preferred currency and you need to be able to display arbitrary currencies in your UI.
New browser APIs: Instead of Stripe elements or checkout, you'll be using the Digital Goods API and the Payment Request API to show a native payment prompt to your users and track purchases.
Keeping track of users: The Play Store handles purchases by Google Play accounts and does not expose the email of the paying user to you. This results in some tricky situations we'll get into later.

High Level Overview

At a high level, here's how the Play Store payment flow works on a TWA:

On each page load:

Detect TWA session
Ask Digital Goods API what the prices are and display them in the UI
Ask Digital Goods API whether the current Google Play user is subscribed (optional)

When a user is ready to pay:

Use Payment Request API to show native browser prompt and get token
Send token to your backend API to validate & record payment
Acknowledge purchase using Digital Goods API

Detect TWA session

Detecting that the user is using your TWA and needs to pay with Play Store billing instead of Stripe is trivial.

Get Prices from Digital Goods API

First off, enter your USD prices for one-time purchases or subscriptions in the Google Play console. You can manually specify prices in other currencies, or just let the Play Store do the currency conversions automatically.

The price objects you'll be working with look something like this:

/**
 * Play Store prices are used by TWA
 * @typedef GooglePlayPrice 
 * @property {string} id - Plny Storen-us - e.g. 'es' or 'es-mx' for English
 * @property {string} currency - e.g. "USD"
 * @property {string} value - e.g. "2.99"
 */

const priceObjects = {
  monthly: {
    id: 'plus_tier_monthly_2_99',
    currency: 'USD',
    value: '2.99',
  },
  yearly: {
    id: 'plus_tier_yearly_24_99',
    value: '2',
  },
];

Notice how above I've hard-coded the default USD currency and value. I wouldn't have to, but it's good to have a fallback in case the Digital Goods API request fails (e.g. user is offline).

Next, we'll update these prices to match the user's preferred currency:

async function updatePlayStorePrices(monthlyPrice, yearlyPrice) {
  // Connect to Digital Goods service
  const service = await window.getDigitalGoodsService('https://play.google.com/billing');

  // Request latest price data from Play Store
  const [
    monthlyPriceDetails, 
    yearlyPriceDetails
  ] = await service.getDetails([monthlyPrice.id, yearlyPrice.id]))

  // Update currency & value to match user's preferred currency
  Object.assign(monthlyPrice, {
    currency: monthlyPriceDetails.price.currency,
    value: monthlyPriceDetails.price.value,
  });
  Object.assign(yearlyPrice, {
    currency: yearlyPriceDetails.price.currency,
    value: yearlyPriceDetails.price.value,
  });
}

try {
  await updatePlayStorePrices(priceObjects.monthly, priceObjects.yearly)
} catch (err) {
  // Log a warning in your error reporting software
  reportError(err, 'Failed to update Play Store prices. Falling back to USD.');
}

Rendering Arbitrary Currencies in UI

Given that a price's currency and value will change based on the user's preferred payment method, it's important that your UI is flexible enough to render an arbitrary price string. The Intl.NumberFormat browser method makes it easy to format any currency in any language:

/**
 * Format price for display in UI
 * @param {GooglePlayPrice} price 
 * @param {string} langId - e.g. 'en' or 'en-us' for English
 * @return {string}
 */
export function formatPrice(price, langId) {
    const { value, currency } = price;

    const numberFormat = new Intl.NumberFormat([langId], {
        style: 'currency',
        currency,
        currencyDisplay: 'symbol',
    });

    return numberFormat.format(value);
}

For the befunky.com/pricing page, we do some additional formatting to emphasize the major unit (dollars) of the price over the minor unit (cents). Getting this right for all language/currency combinations is nigh impossible, but we are able to do it for some common scenarios.

/**
 * Template for rendering price
 * @param {GooglePlayPrice} price 
 * @param {string} langId - e.g. 'en' or 'en-us' for English
 * @return {string}
 */
export function priceTemplate(priceObject, languageId) {
  const formattedPrice = formatPrice(priceObject, languageId);
  
  // Grab the symbol & suffix from the formatted price
  const nonNumericParts = formattedPrice.split(/\d/);
  const prefix = nonNumericParts[0];
  const suffix = nonNumericParts[nonNumericParts.length - 1];

  // Grab part of price that is numbers
  let numericPrice = formattedPrice.slice(prefix.length);
  numericPrice = numericPrice.slice(0, numericPrice.length - suffix.length);

  // If there's decimal values at the end, make them small
  const pricePartsMatch = numericPrice.match(/^(.+)([.,])(\d\d)$/);
  if (pricePartsMatch) {
      const [, majorAmount, delimiter, minorAmount] = pricePartsMatch;
      return template({ prefix, majorAmount, minorAmount, suffix, delimiter });
  }

  // Decimal value not found, just render the full price string
  return template({ prefix, majorAmount: numericPrice, suffix });

  /**
    * Template for displaying large dollars beside small cents
    * @param {string} largeAmount - dollars
    * @param {string} [smallAmount] - cents
    * @param {string} [prefix] - $
    * @param {string} [suffix] - suffix to indicate currency, e.g MXN in "$100 MXN"
    * @param {string} [delimiter] - between dollars & cents
    * @return {TemplateResult}
    */
  function template(prefix, largeAmount, smallAmount, suffix, delimiter = '.') {
    return html`
      ${prefix ? html`<span class="small-text">${prefix}</span>` : ''}
      <span class="large-amount">${largeAmount}</span>
      ${smallAmount ? html`<span class="small-amount">${delimiter}${smallAmount}</span>` : ''}
      ${suffix ? html`<span class="small-text">${suffix}</span>` : ''}
    `;
  }
}

Handling Already-Subscribed Google Play Users

When using your own payment processor (e.g. Stripe), your user database is the source of truth concerning what users have active subscriptions.

In contrast, the Play Store manages subscribing and unsubscribing, and just exposes the ID of the subscription to your backend (via a webhook or API request). You don't actually know what the Google Play user's name or email is. You just know what account they were logged-in to in your application when they subscribed.

From your perspective, a particular user account (a@a.com) is subscribed and associated with a Play Store subscription ID.
From the Play Store's perspective, all devices with a particular Google Play account (b@b.com) are subscribed to your app.

This leads to a disconnect whenever a user is on a device profile with the subscribed Google Play account (b@b.com) but not logged-in to the associated user account on your app. For instance:

They log out of the a@a.com account in the TWA and let a friend use their device. The friend logs into the app as c@c.com, and when they try to upgrade, the Play Store says "you've already subscribed".
Their a@a.com session expires. They're confused why they can no longer access premium features, since they know the have an active Play Store subscription for your app.

In both cases, you'll need to develop a flow that acknowledges that the Google Play user on this device is subscribed, but that they'll need to login to your app with the associated account (e.g. a@a.com) to use premium features.

Here's a basic outline of a solution:

export async function handlePlayStoreUser(currentAppUser) {
  // Current app user is already subscribed, no need to bother them
  if (currentAppUser.isSubscribed) return;

  // Connect to Digital Goods service
  const service = await window.getDigitalGoodsService('https://play.google.com/billing');
  
  // Get purchases made by current Google Play account
  const purchases = await service.listPurchases();
  if (purchases.length === 0) return;

  // Get app user associated with these purchases from your backend
  const subscribedUser = await requestApi('get-user-for-play-store-purchases', purchases);

  if (subscribedUser && subscribedUser.email !== currentAppUser.email) {
    // Prompt user to login with the email address of the upgraded account associated 
    // with their active Play Store subscription
    showDialog(`Welcome back! Please login to this app as ${subscribedUser.email} to use premium features.`);
  }
}

Alternatively, you could let any account on the device use premium features, but that would probably involve a much deeper refactor of your app's logic.

Handling Payments

This part is actually pretty straightforward!

Use PaymentRequest API to display a native payment prompt to your user.
Send token for the approved payment to your backend.
On the backend, validate token using Google Play API and upgrade the associated user account.
Acknowledge the payment using the Digital Goods API.

async function processPayment(priceId) {

  // Build payment request
  const paymentMethod = [
    {
      supportedMethods: 'https://play.google.com/billing',
      data: { sku: priceId },
    },
  ];

  // 1. Open native dialog to prompt user to pay
  let token;
  try {
    paymentResponse = await new PaymentRequest(paymentMethod).show();
    token = paymentResponse.details.token;

  } catch (error) {
    if (error.message.includes('was cancelled')) {
        // User dismissed native dialog
        logWarning('User chose not to subscribe:', error);
    } else {
        // Report unexpected error
        reportError(error, 'PaymentRequest.show() failed');
    }
    return;
  }

  // 2. Send token to our API to validate subscription & update database
  let validationResponse;
  try {
      // 3. Validate on backend using Google Play Developer API
      // https://developers.google.com/android-publisher/api-ref/rest/v3/purchases.subscriptions/get
      validationResponse = await requestAPI('validate-play-store-payment', { priceId, token });
      
      if (!validationResponse.isValid) throw new Error('Failed to validate token');

  } catch (error) {
    // Report unexpected error
    reportError(error, validationResponse);

    // Show payment failed UI
    await paymentResponse.complete('fail');
    return;
  }

  // 4. Acknowledge purchase with Digital Goods API. Otherwise, user will get refund in 3 days.
  try {
    const service = await getDigitalGoodsServiceOnce();
    const response = await service.acknowledge(token, 'onetime');

  } catch (error) {
    // Report unexpected error
    reportError(error, 'Failed to acknowledge purchase');

    // Show payment failed UI
    await paymentResponse.complete('fail');
    return;
  }

  // Thank user for purchasing a subscription!
  // ...
}

Feel free to leave a comment if any of this doesn't make sense or could be improved!

I'm in the process of building out this flow for BeFunky, and will update this article as our approach evolves over time.

Network-first Pre-cached PWA Start URLs with Workbox

Micah Engle-Eshleman — Sun, 18 Jul 2021 07:18:00 +0000

When deploying BeFunky as a PWA, the trickiest problem I ran into was how to pre-cache the Start URL using a Network-First Strategy. In other words, how to meet these simple requirements:

#1. If the user is online, fetch the latest Start URL page from the network

This is important for a variety of reasons:

Hashed asset URLs may be hard-coded in the page's HTML. Serving a stale page means serving an outdated release.
The Start URL page may contain dynamic server-rendered content.

#2. If the user is offline, serve a pre-cached Start URL from the cache.

Offline support makes your PWA feel like any other app on the user's device. It should at least open when offline.
An offline Start URL is now a requirement for Chrome to show the native PWA install prompt.

This is called a "Network-first" caching strategy. It ensures your users are only served stale content when necessary, instead of by default while the service worker checks for and caches new assets.

Pre-caching assets is very easy with Workbox's precacheAndRoute method, but it uses a Cache-first strategy.

To pre-cache a PWA Start URL with a Network-First strategy, we must do the following:

Setup a Network-First caching strategy
Match Navigation routes (used when navigating to new pages), not standard subresource requests (used for fetching assets).
Limit the cache to only the PWA Start URL, so each time you have a new release, the old entry is replaced

// service-worker.js

// Load Workbox
importScripts('https://storage.googleapis.com/workbox-cdn/releases/6.1.2/workbox-sw.js');
workbox.setConfig({ debug: false });

// Load Expiration plugin (optional)
workbox.loadModule('workbox-expiration');

// Setup logging & versioning (used for debugging a particular service-worker version)
const version = '3508937123';
const log = (...args) => { console.log(`SW ${version} -`, ...args); };

// Pre-cache assets
workbox.precaching.precacheAndRoute([...]);

// Setup Network-First caching strategy
workbox.routing.registerRoute(
  // Match Navigation Routes
  new workbox.routing.NavigationRoute(
    new workbox.strategies.NetworkFirst({
      cacheName: 'pwa-start-url',
      plugins: [
        new workbox.expiration.ExpirationPlugin({
          maxEntries: 1,
          maxAgeSeconds: YEAR_IN_SECONDS,
        }),
      ],
      matchOptions: {/* Optional, see note below */},
    }),
    // Limit cache to only PWA Start URL
    { allowlist: [/^\/start-url\/$/] },
  ),
);

// Cache Start URL during installation
self.addEventListener('install', (event) => event.waitUntil(() => {
  const startUrl = 'https://www.your-domain.com/start-url/';
  return caches.open('pwa-start-url')
      .then(cache => cache.add(startUrl))
      .then(() => {
        log('Pre-cached NetworkFirst Start url:', startUrl);
      })
      .catch((error) => {
        // If pre-caching fails, continue with installation & activation 
        // since it's a nice to have, not a hard requirement
        log('Failed to pre-cache NetworkFirst Start url:', startUrl, error);
      });
  }));
);

matchOptions

In some cases, the pre-cached Start URL will still fail to match the requested Start URL when the PWA user is offline, and Chrome will give you a warning about the Start URL not being available offline.

In these scenarios, CacheQueryOptions can be helpful. It's not particularly well documented, but you can pass flags to matchOptions that tell the service worker to use the pre-cached page even if...

ignoreSearch: it has different query params (e.g. ?utm_medium=pwa) than the requested page
ignoreMethod: the method (e.g. GET/POST) differs between requests (probably irrelevant for PWA Start URLs)
ignoreVary: the cached page has a Vary header

I had to use ignoreVary b/c we set Vary headers to differentiate between browsers for polyfilling purposes.

Good luck!

If this technique works for you, or if I missed something, I'd love to hear about it in the comments.

Bulletproof PWA and TWA Detection

Micah Engle-Eshleman — Sun, 18 Jul 2021 06:02:00 +0000

Detecting an Installed PWA

The simplest way to tell between a standard browser session and an installed PWA session is by checking display-mode. Assuming your PWA runs in "standalone" mode (defined in your web app manifest file, learn more about display modes here), you can simply check the display-mode property.

/**
 * Is the page currently in standalone display mode (used by PWA)?
 * @return {boolean}
 */
function isInStandaloneMode() {
  return Boolean(
    (window.matchMedia && window.matchMedia('(display-mode: standalone)').matches)
    || window.navigator.standalone, // Fallback for iOS
  );
}

There is one gotcha: users can change the display mode of your PWA by making it fullscreen. In this case, your best bet is to detect standalone mode initially, store a flag in SessionStorage, and then check that flag for every page view in your session.

// Run this code as soon as possible (before user has a chance to change display mode)
let hasBeenInStandaloneMode;
if (isInStandaloneMode()) {
  hasBeenInStandaloneMode = true;
  sessionStorage.setItem('is_standalone_mode', 'yes');
} else {
  hasBeenInStandaloneMode = sessionStorage.getItem('is_standalone_mode') === 'yes';
}

/**
 * Is this an installed PWA?
 * @return {boolean}
 */
export function isInstalledPwaSession() {
  return hasBeenInStandaloneMode;
}

Note: It's important to use SessionStorage instead of LocalStorage b/c LocalStorage is shared with the browser, and you don't want to mark tabs in the browser as part of a PWA session just because at some point in the past a user opened your PWA.

Detecting a TWA (Trusted Web Activity)

Packaging your PWA as a TWA is required for listing it in the Play Store. Essentially, you generate tiny Android app that references your PWA start URL, key asset URLs, and a version number.

Most of this data should already be in your web app manifest, so the process is relatively painless using a tool like the Bubblewrap CLI.

When a TWA is opened on Android, the TWA app ID is exposed to the start URL page via the referrer. We can use a similar technique to detect the Android TWA as we did the PWA above:

// Read more about this solution here: https://stackoverflow.com/a/54580415/1546808
let isAndroidTwaSession;
if (document.referrer.includes('android-app://<your.twa.package.name>')) {
  isAndroidTwaSession = true;
  sessionStorage.setItem('has_android_twa_referrer', 'yes');
} else {
  isAndroidTwaSession = sessionStorage.getItem('has_android_twa_referrer') === 'yes';
}

Unfortunately, this doesn't work on ChromeOS (document.referrer is empty). Hopefully it does at some point in the future, but for the time being, if you're releasing a TWA on Chromebook, you'll need to customize the start URL in the twa-manifest.json file created by Bubblewrap. In my case, I kept using the same start URL as the PWA, but added a #play-store-twa hash:

// manifest.json (for PWA)
"startUrl": "/pwa-start-url/"

// twa-manifest.json (just for TWA, used by Bubblewrap CLI)
"startUrl": "/pwa-start-url/#play-store-twa"

Then check for this hash as soon as possible on the start URL page:

// Check for #play-store-twa hash in start URL to identify TWA
let hasPlayStoreTwaHash;
if (window.location.pathname === '/pwa-start-url/' && window.location.hash === '#play-store-twa') {
    hasPlayStoreTwaHash = true;
    sessionStorage.setItem('is_play_store_twa', 'yes');
} else {
    hasPlayStoreTwaHash = sessionStorage.getItem('is_play_store_twa') === 'yes';
}

export function isTwaSession() {
  return hasPlayStoreTwaHash && isInstalledPwaSession();
}

How to Create a Repo-scoped Github Access Token in an Organization Account

Micah Engle-Eshleman — Fri, 09 Jul 2021 21:32:00 +0000

Evidently, the only way to trigger a Github Action from a third-party (e.g. headless CMS) is via a repository_dispatch event. The only tricky part is that the request must be authenticated by an Access Token.

curl \
  -X POST \
  -H "Authorization: token {ACCESS TOKEN}"
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/{org}/{repo}/dispatches \
  -d '{"event_type":"rebuild_static_site","client_payload":{"post_id":54}}}'

The problem with personal Access Tokens is that, by default, they give access to every repo you have access to. When working on private repos in organizational accounts at work, this is a non-starter!

Unfortunately, the only way to generate an Access Token scoped to a repository (public or private) in your organization is via a Github App. Hopefully this won't always be the case, but for now you can follow these steps to set one up in a few minutes:

Step 1: Create a new Github App

github.com/settings/apps/new

Give it a short unique name (max length is 34 characters).

Uncheck "Expire user authorization tokens" and leave "Callback URL" blank

Uncheck "Active" checkbox under "Webhook" and leave "Webhook URL" blank

Under "Repository Permissions" find "Contents" and enable Read & Write access.

Since this Github action is for an Organization Account, you'll need to allow it to be installed on "Any Account". That makes the app Public, but it will not compromise your Access Token, since that's generated when you install it.

Click "Create Github App".

Step 2: Jot down the App ID and download the Private Key file (PEM)

You'll need these for step #4.

The App ID should be near the top of the page.

Download the PEM (private key) file.

Step 3: Install the App on your Organization Account

The link to install it should show up in the left-hand menu

Jot down the Installation ID from the App Installation page that shows up after installing. You'll need this in the next step.

Step 4: Extract Access Token using Node library

Install github-app-installation-token and use it to extract the repo-specific Access Token from your new Github App.

You'll need the following data:

App ID (from step #2)
Installation ID (from step #3)
Private Key (PEM file in step #2)

There are detailed instructions for using this library in the README.

After running the script it should give you an Access Token, e.g.

{ token: 'ghs_jksGSQQkmtXSthnGnCsb9AY9yTWU58jhytm6' }

Step 5: You're done!

You can now use this Access Token to trigger a repository_dispatch event and fire a Github Action in your organization's private repo.

curl \
  -X POST \
  -H "Authorization: token ghs_jksGSQQkmtXSthnGnCsb9AY9yTWU58jhytm6"
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/test-inc/test-repo/dispatches \
  -d '{"event_type":"rebuild_static_site","client_payload":{"post_id":54}}'

A More Reliable FileReader

Micah Engle-Eshleman — Fri, 29 Jan 2021 04:58:00 +0000

FileReader is not a web API you bump into on most websites, but at BeFunky we use it all the time for transforming image files (aka "Blobs") from our users.

FileReader.readAsArrayBuffer() is the fastest way to know if a Blob can be read at all.
FileReader.readAsDataURL() converts a Blob to a base64 string that can be used as an <img> src.
FileReader.readAsText() is great for reading text-based files (e.g. JSON) that a user drags into your website.

The problem is that FileReader has an old event-driven API that relies on load and error events, so it's a bit icky to work with. Fortunately, it's trivial to wrap in a Promise.

const result = new Promise((resolve, reject) => {
  const fileReader = new FileReader();
  fileReader.onload = () => {
    resolve(fileReader.result);
  };
  fileReader.onerror = (error) => {
    fileReader.abort();
    reject(error);
  };
  fileReader.readAsDataURL(yourFile);
});
result.then(...)

Given that reading a file can fail unexpectedly, we've encountered situations where neither onload or onerror are called. To avoid leaving users in infinite loading states, we've introduced a timeout period (varies by task) to ensure that our app remains responsive no matter what.

setTimeout(() => {
  reject(new Error('FileReader timed out after 15s');
}), 15000);

When FileReader calls onload/onerror after our timeout period is up, we log the result and duration in Sentry we can adjust it over time.

I went ahead and wrote a tiny 1kb library for this on Github.

import reliableFileReader from 'reliable-filereader';

reliableFileReader('readAsDataURL', blob)
    .then((base64String) => {
        // Do something with base64 string
    })
    .catch((error) => {
        if (/timed out/.test(error)) {
            // FileReader has not responded within 15 seconds
        } else {
            // FileReader fired an error event.
            const errorEvent = error;
        }
    });

In addition to wrapping FileReader in a Promise and allowing you to set the timeout period...

It notifies you of the result/error if your timeout period was too short.
You can customize the timer function. One use case would be pausing the timer if the user changes tabs, which may also slow down FileReader. Full disclosure: I haven't tried this code in production yet, but I'm looking forward to sometime this spring!

If you use reliable-filereader or have other suggestions, let me know!

Simple Depth-First Search with Object.entries()

Micah Engle-Eshleman — Thu, 17 Dec 2020 05:30:00 +0000

Sometimes at work I need to find a particular type of object in a large JSON file whose hierarchy is bound to change over time.

For instance, imagine we're trying to grab all image URLs in a JSON response from our headless CMS, so we can upload them to an S3 bucket. Image objects could be top level (e.g. for SEO metadata) or deeply nested in a content hierarchy. Regardless, there are probably a couple properties that all image objects share, e.g.

function isImageObject(value) {
  return value 
    && value.url 
    && (value.dimensions || value.kind === 'image');
}

It turns out this is really easy to do using a "depth-first search" algorithm, which traverses a graph (the JSON file) and calls a function on each key-value pair using Object.entries. The wonderful thing is Object.entries will even work on arrays, where the "keys" are just indices.

/**
 * Traverse an object, calling filter on each key/value
 * pair to know whether to continue
 * @param  {object} obj
 * @param  {function} filter
 */
function traverse(obj, filter) {
    if (typeof obj !== 'object' || obj === null) return;

    Object.entries(obj).forEach(([key, value]) => {
        // Key is either an array index or object key
        if (filter(key, value)) traverse(value, filter);
    });
};

Essentially you call this function on the top level object (JSON file) and then just return a truthy value to continue traversing down the tree. Once you've found your object, you can return falsy to avoid going any deeper. Here's an example:

// Find all image URLs in the JSON file
const imageUrls = [];

traverse(jsonObject, (key, value) => {
  if (isImageObject(value)) {
    // Found an image, 
    imageUrls.push(value.url);
    // No need to go any deeper (b/c we don't have nested images)
    return false; 
  }
  // Keep looking for image
  return true; 
})

Thanks to David Housman's Algorithmic Graph Theory course at Goshen College. Without it I wouldn't have thought to try to approach the problem this way.

Working with Images

Micah Engle-Eshleman — Fri, 24 May 2019 04:26:00 +0000

A couple days ago I gave a polyglot talk at JS Admirers about working with images in the browser.

A brief outline:

Blobs (aka binary files)
What are they, how to store them, and how to render them in <img> tags using base64/object URLs.
Image performance
Images aren't in the critical path, and are intentionally loaded late using HTTP/2 prioritization.
Serving the right image
Use Webp and srcset+sizes. The <picture> tag is rarely needed.
Lazy loading
Use IntersectionObserver or passive scroll listener. To preserve aspect ratio, use padding-bottom trick or inline SVG as src.
Critical image pre-rendering
Demo of <canvas> blur technique.
Animated SVG cursors
Most browsers support SVGs as cursors. You can even update the SVG to animate the cursor.

If any of that sounds interesting, check out the slides!

CrUX - Real World Website Performance Data

Micah Engle-Eshleman — Fri, 26 Apr 2019 04:26:00 +0000

At few days ago I gave a lightning talk on the Chrome User Experience Report (CrUX), based on a longer talk by Rick Viscomi at the #PerfMatters conference.

CrUX is a publicly queryable dataset of real world performance data about your domain. I came back from the conference and was able to build a few charts for BeFunky that help us track FID and FCP and compare against our competitors over time.

For instance, here's a chart of the befunky.com domain's fast FCP (percent of users experiencing first-contentful-paint under 1 second) compared against our competitors.

Check out Rick's Gist for a bunch of great links on CrUX and building charts like the one above.

And, here's my slides from the lightning talk.

A Priority+ Navigation With Scrolling and Dropdowns

Micah Engle-Eshleman — Tue, 18 Oct 2016 05:00:00 +0000

One Regex to Rule them all: Removing wrapping
tags around images in WordPress while preserving links

Micah Engle-Eshleman — Fri, 18 Mar 2016 21:40:00 +0000

Wordpress has the unfortunate tendency of wrapping nearly everything in <p> tags. While this is often helpful for blocks of text in a post, it limits our ability to succinctly and consistently style images regardless of their containing <a> tag, <p> tag, or both. In other words, you rarely want <a> tag styles (e.g. border-bottom & :hover) or <p> styles (e.g. margin-bottom & max-width) to affect the styling of their child images.

The cleanest solution I've found is to write a Regular Expression that filters postcontent and removes wrapping <p> tags around <img>'s while preserving surrounding text and it's styling (e.g. bold, _italic). For instance, in the Wordpress editor I'll add some bold text, a right-aligned image, and then some more text:

Visual editor:

Text editor:

On the page:

You'll notice the above example is contrived: I manually added a nasty 400px max-width to the paragraph in Chrome Dev Tools. The point is that the image, even though it's right-aligned, it subject to the constraints of it's containing paragraph.

The goal now is to take the image and move it outside of the wrapping paragraph. We also need to preserve any wrapping link around the image and move it outside as well. Here's the current code:

I've iterated on this regex quite a few times, but would appreciate your feedback and improvements. In terms of performance, it's clocking in around .5ms for most posts.

Creating a Custom Unsubscribe Form in Marketo that Doesn’t Skew Email Analytics

Micah Engle-Eshleman — Thu, 15 Oct 2015 22:41:00 +0000

The easy part was creating a new form--with a hidden Unsubscribed field, among others--and embedding it in a Wordpress page. More difficult was testing the new Unsubscribe link in emails. I needed a fully functioning Unsubscribe link that met these requirements:

Link is not global! It only appears on a new "test email template" and doesn't affect existing settings.
Clicking link in email should not increment the email click count.
Unsubscribing by clicking on link and using the form on the Wordpress page should increment the email unsubscribe count.
Email should only contain the new Unsubscribe link defined in the template, not the global Unsubscribe HTML injected by Marketo below it.

#1 is easy. Just add the link to a new template and test it in a new program.

#2 involves adding the mktNoTrack class the the link. This means it will not be routed through your subdomain (e.g. go.goshen.edu) which tracks link clicks and associates them with leads and email campaigns.

#3 is tricky, and I couldn't find documentation on it. Add a mkt_tok=##MKT_TOK## url parameter to the link. Without this the form will still work, but Marketo needs the mkt_tok parameter to associate the unsubscribe action to the email.

#4 is a hack. The global Unsubscribe HTML will be added to the email unless it contains the {{system.unsubscribeLink}} token. Create an tiny invisible link with this token, e.g.


<!-- Prevent injection of Unsubscribe HTML since we're using a custom link -->
<a href="{{system.unsubscribeLink}}" style="display: none; font-size: 0; color: transparent"></a>

and be sure to comment it!

Note: #4 is not necessary if you choose to replace the global Unsubscribe HTML with your new link. It's super handy for testing the new link ahead of time though.

I'm new to Marketo development and this is my first post on the topic. I'd love to hear your feedback.

Setting up Custom Authorization in Ember-CLI using Google oAuth2 for the Initial Login

Micah Engle-Eshleman — Mon, 15 Jun 2015 13:32:00 +0000

I've found it handy in Ember-CLI apps to use Google logins for initial authentication (that way I don't have to store passwords). The question is how to transform a Google login into a session that can be used to query user-specific resources from my own server (not just Google's APIs). Here's a high level view of the process:

User logs in to their Google account, and authorizes Ember app. In this process the app receives a Google token.
App sends Google token to the node server in response for a new token. The server validates the Google token and then uses it to get user info from Google. It then creates a new custom token containing a user id, which it sends to the app.
App uses it's new custom token in all further requests to the server. Right before this token expires, it will be refreshed.

In this process, token refers to a JSON web token. We'll be using the following open-source projects to build a basic app (for the finished product, see my GitHub repo).

Ember Client:

Ember-Cli
Ember Simple Auth and Ember CLI Simple Auth Token extension
Torii Google Oauth2 Bearer provider

Node Server:

Express for creating REST API routes
jsonwebtoken and express-jwt for encoding and decoding JSON web tokens
request for talking with Google's APIs
bodyParser for extracting the body (data) from the app's requests.

Step 1: Setup Ember-CLI app structure

Create a simple folder structure:

basic-auth-demo
    ember-client
    node-server

Instead of actually making an "ember-client" folder, just run:

ember new ember-client

within the basic-auth-demo folder. Then we'll set up the following route structure:

ember g route s
ember g route s/notes

The idea is that any subroute of s (for "secure") will be protected from public access by a Google login. Ember Simple Auth has some mixins that make this a synch. Initially, I'll set up some really basic templates to keep track of our nested routes.


<h2 id="title">Welcome to Ember.js</h2>

{{#if session.isAuthenticated}}
 <a {{ action 'invalidateSession' }}>Logout</a>
{{else}}
 <a {{ action 'sessionRequiresAuthentication' }}>Login</a>
{{/if}}

{{outlet}}


<!-- s.hbs template -->

<hr>This is a secure route.<hr>

{{outlet}}


<!-- s/notes.hbs route -->

<p>A list of secure notes...</p>

{{outlet}}

Then if you run ember s and navigate to localhost:4200/s/notes you should see something like this:

Step 2: Setup node server to return tokens

In the node-server folder, create a server.js file.

// express framework for routing
var express = require('express')

// create and validate json web tokens
var createJWT = require('jsonwebtoken')
var validateJWT = require('express-jwt')

Then setup a node project with npm init (it creates a package.json to track dependencies--you can just press Enter at the prompts)

cd node-server
npm init

and install the modules we required in server.js.

npm install express --save
npm install jsonwebtoken --save
npm install express-jwt --save

Great, now you can try to run your node server!

node server

If nothing happened, that's a good thing. It means there were no errors. Now let's actually set up the node server that listens on port 4500. Add these lines to server.js

// startup and listen on port 4500
var app = express()
app.listen('4500')

// setup HTTP headers
app.use(function (req, res, next) {
  res.setHeader('Access-Control-Allow-Origin', 'http://localhost:4200')
  res.header(
    'Access-Control-Allow-Headers',
    'Origin, X-Requested-With, Content-Type, Accept, Authorization'
  )
  res.header('Access-Control-Allow-Methods', 'POST, GET, PUT, DELETE, OPTIONS')
  res.header('Content-Type', 'application/json')
  next()
})

// respond with "yolo" to a GET request to the root (localhost:4500)
app.get('/', function (req, res) {
  res.send('yolo')
})

This time when your run node server it should pause (until you hit Ctrl-C) b/c it's listening to port 4500. Try going to localhost:4500 in your browser and you should see "yolo".

We can use this same syntax to respond to Ember when it requests a json web token with a POST request. Add the following to server.js to serve up a token when a POST request is sent to localhost:4500/get-token.

// respond w/ token to POST request to /get-token
app.post('/get-token', function (req, res) {
  var token = createJWT.sign(
    // payload
    { currentUserId: 1 },
    // secret
    '09htfahpkc0qyw4ukrtag0gy20ktarpkcasht',
    // options
    { expiresInMinutes: 10 }
  )
  res.send({ token: token })
  console.log('\tsent token')
})

Now, use a browser extension like Postman for Chrome to send a POST request to localhost:4500/get-token. You'll need to restart the server (ctrl-c, then node server again) first to incorporate the new code or you'll get a "Cannot POST /get-token" error.

Just for fun: to get a good grasp of what the token actually stores, paste it into the jwt.io.

Step 3: Hook up Ember Simple Auth with node server

Install Ember Simple Auth and its token-based extension.

cd ../ember-client
ember install ember-cli-simple-auth
ember install ember-cli-simple-auth-token

Tip: I generally find it best to open up two terminal tabs open so I can have node server running while I work in Ember Then let's create an adapter so Ember knows where it's API server is:

ember g adapter application

// application.js adapter

import DS from 'ember-data'

export default DS.RESTAdapter.extend({
  host: 'http://localhost:4500',
})

And setup the authentication addons by adding the following properties to ENV in config/environment.js:

contentSecurityPolicy: {
  'connect-src': "'self' http://localhost:4500",
},

'simple-auth-token': {
  serverTokenEndpoint: 'http://localhost:4500/get-token',
},

'simple-auth': {
  authorizer: 'simple-auth-authorizer:token'
}

Great, now we'll setup s as a protected route and handle authentication in the application route.

// s (secure) route

import Ember from 'ember'
import AuthenticatedRouteMixin from 'simple-auth/mixins/authenticated-route-mixin'

export default Ember.Route.extend(AuthenticatedRouteMixin, {})

Note: I had to manually create an application.js file in app/routes

// application route

import Ember from 'ember'
import ApplicationRouteMixin from 'simple-auth/mixins/application-route-mixin'

export default Ember.Route.extend(ApplicationRouteMixin, {
  actions: {
    sessionRequiresAuthentication: function () {
      this.get('session')
        .authenticate('simple-auth-authenticator:jwt', { password: '' })
        .then(
          function () {
            console.log('custom token authentication successful!')
          },
          function (error) {
            console.log('custom token authentication failed!', error)
          }
        )
    },
  },
})

Assuming that both ember s and node server are running, you should now be able to log in and out of your Ember application and only access /s/ routes when logged in. At this point there is still no real security. Anonymous users are handed tokens willy-nilly, but it's a good start.

Step 4: Setting up Google Account logins

It may be tempting at this point to use a the Google oAuth2 Torii provider bundled into the Ember Simple Auth Torii extension, and simply authenticate with Google and right before authenticating w/ our node server. The problem is that upon authentication, Ember Simple Auth lets the user access secure routes. In other words, /s/notes can be transitioned to before the second authentication with our node server happens and an authorizer is actually set up. I imagine there is a workaround to this problem but I found it simpler to just use Torri providers directly w/out the authenticator wrapper.

ember install torii

You can setup the Google oAuth2 Bearer provider by adding a torii property to ENV in environment.js.

torii: {
      providers: {
        'google-oauth2-bearer': {
          apiKey: '10514958320583-as0utyahvaekgprntiashtoasht.apps.googleusercontent.com',
          redirectUri: 'http://localhost:4200',
        },
      }
    }

To get an apiKey, go to the Google Developer Console and create a new project. Click on Credentials under APIs & auth in the left side nav and select Create new Client ID under OAuth, choosing Web Application in the popup. (You may need to Configure the consent screen first--just enter the Product Name and click Save). Now, in the popup, enter http://localhost:4200 in both fields (authorized javascript origins and authorized redirect uri). The first requires an actual domain and the second doesn’t need to redirect to any particular Ember route since Ember will handle this on it’s own. Copy the Client ID string, and paste it into environment.js as (you guessed it) the apiKey. At this point, we just need to set up the Application route to log in with Google first and then send the Google token to the node server for custom authentication. Making a few minor edits to sessionRequiresAuthentication:

sessionRequiresAuthentication: function() {
    var session = this.get('session');
    this.get('torii')
        .open('google-oauth2-bearer')
        .then(function(googleAuth){
            var googleToken = googleAuth.authorizationToken.access_token;
            console.log('Google authentication successful.');

            session
                .authenticate('simple-auth-authenticator:jwt', { password: googleToken} )
                .then(function(){
                    console.log('custom token authentication successful!');
                }, function (error) {
                    console.log('custom token authentication failed!', error.message);
                });

        }, function (error) {
            console.error('Google auth failed: ', error.message);
        });
}

Notice how I pass googleToken to the node server as a password. Now, we just need to set up the node server to parse this token, fetch the current user from Google, and return an user-identifying token that can be used for authorizing secure resources. If everything is working, when you click on Login you should see a Request for Permission popup from Google.

Step 5: Fetching user info from Google

In server.js, instead of just returning a token, we must first validate the Google token against Google's API. In this process, Google will send back a unique user id and an email address. We can also call the Google+ People API to get additional data, such as the displayName and image. All of this information can then be used to create/update/find the current user in our backend database and return a unique token, which contains a valid user id, to Ember. The {password: googleToken} object is passed to the server as JSON in the body of the POST request. To get it, we'll need to install the body-parser middleware. We'll also install the request library to make a server-side request to Google.

cd ../node-server
npm install body-parser --save
npm install request --save

and then at the top of server.js:

// decodes json in the body of a request and stores it as req.body
var bodyParser = require('body-parser')

// simple HTTP request client
var request = require('request')

The next step is simply expanding app.post('/get-token') to validate the token.

// secret used to construct json web token
app.secret = '09htfahpkc0qyw4ukrtag0gy20ktarpkcasht'

// send token to user that contains their id
app.sendToken = function (res, userId) {
  var token = createJWT.sign(
    // payload
    { userId: userId },
    // secret
    app.secret,
    // options
    { expiresInMinutes: 10 }
  )
  res.send({ token: token })
  console.log('\tsent token')
}

// respond w/ token to POST request to /get-token
app.post('/get-token', bodyParser.json(), function (req, res) {
  // get Google token from Ember: { password: googleToken }
  var googleToken = req.body.password

  // send token to Google for validation
  request(
    'https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=' +
      googleToken,
    function (error, response, body) {
      if (!error && response.statusCode == 200) {
        console.log('\tGoogle Token Valid')
        var userId = JSON.parse(body).user_id
        var userEmail = JSON.parse(body).email
        app.sendToken(res, userId)
      } else {
        console.log('\tFailed to validate Google Token')
        res.send({})
      }
    }
  )
})

Here, I created a new app.sendToken function for token creation. The created token now contains the Google id, so the node server will always know which Google user it's getting requests from. A more typical setup would be to exchange the Google id and email for whatever the associated user id is in your backend, and then use this id in the token, but I thought I'd keep things as simple as possible.

Step 6: Refreshing the token

Our token expires in 10 minutes, so user's will have to log in w/ Google every 10 minutes. That's no good! We're going to create a refresh token endpoint at localhost:4500/refresh-token that will get used 1 minute before the token expires. Just update simple-auth-token in environment.js.

'simple-auth-token': {
  // get token
  serverTokenEndpoint: 'http://localhost:4500/get-token',
  // refresh token
  serverTokenRefreshEndpoint: 'http://localhost:4500/refresh-token',
  // timeFactor * refreshLeeway = milliseconds before token refresh
  timeFactor: 1000,
  refreshLeeway: 60, // 1 minute
},

Then we'll create that endpoint in server.js. Here, we'll parse the token, get the userId, and then send back a new token with the same userId.

// Refresh token
app.post('/refresh-token', bodyParser.json(), function (req, res) {
  // verify token and extract contents (including userId)
  var oldToken = req.body.token
  createJWT.verify(oldToken, app.secret, function (err, decodedToken) {
    if (!err) {
      // send new token
      console.log('\tRefreshing token for user ', decodedToken.userId)
      app.sendToken(res, decodedToken.userId)
    } else {
      // send error
      console.log('\tError while trying to refresh token:', err)
      res.send({})
    }
  })
})

Just for testing, modify { expiresInMinutes: 10 } in app.sendToken so tokens expire every 2 minutes. This way you should see a token refresh every minute in your node server terminal and in the Network tab of the Chrome console on localhost:4200 once you're logged in. [

Congrats! If you've gotten this far you're good to go! The next step is authorizing resources, which is pretty straightforward. Set up a model in Ember and then handle it's request like so:

// User requests list of notes
app.get('/notes', validateJWT({secret: app.secret}), function(req, res) {

    // get userId from token
    var userId = req.user.userId;

    // lookup notes for user...
    var notes = {
        'id': 1
        'content': 'A note for user '+ userId,
        'user': userId
    };

    // send notes to Ember
    res.send({ notes: notes });
});

Here, validateJWT (actually the express-jwt library), automatically parses the token and provides req.user. This is handy for quickly getting the userId in each request from Ember to serve up the appropriate resources. In the above example, I return some JSON to Ember w/ notes as the top-level element. This is currently how Ember Data likes to receive data, but is changing with version 1.13 and the introduction of JSON API.

Hey, this is my first Ember tutorial and I'd appreciate your feedback! Authentication & Authorization can be tricky to get right, and I hope this pattern helps you get a better feel for things. If you run into issues, please consult my git repo github.com/micahjon/basic-auth-demo or leave a comment. Special thanks to Martin Genev whose post on 100PercentJS got me started.

Named Anchors and Fixed Headers

Micah Engle-Eshleman — Mon, 15 Jun 2015 13:32:00 +0000

One of the issues with fixed header navigation, especially when it’s fairly tall (e.g. goshen.edu) is that when you use named anchors to scroll you to a particular page section, the top of the relevant section is covered by the header.

The solution I’ve found is simply to use an <a> tag with a tall pseudo-element whose height matches the height of the header but doesn’t actually affect the layout of the page (negative margin prevents it from taking up vertical-space and zero width makes the bottom of the previous section clickable).

For example:

.anchor-target:before {
  content: '';
  display: block;
  width: 0;
  height: $fixed-header-anchor-offset; /* fixed header height*/
  margin: -$fixed-header-anchor-offset 0 0; /* negative fixed header height */
}

If this <a class=”anchor-target”> tag is before the relevant section, the browser will jump to it, leaving the appropriate amount of buffer so that the sections begins right underneath the fixed header.

This is also an issue for Gravity Forms confirmation messages. Simply apply the same styles to .gform_anchor:before and make sure to enable the <a class=”gform_anchor”> by using the gform_confirmation_anchor filter:

// enable confirmation anchor (scrolls to the right place for the confirmation message)
add_filter( 'gform_confirmation_anchor', '__return_true' );

Loading jQuery Immediately before Gravity Forms

Micah Engle-Eshleman — Mon, 18 May 2015 22:51:00 +0000

Loading jQuery asychronously or in the footer on pages with ajax-enabled Gravity Forms is tricky because the form relies on inlined jQuery-dependent scripts that immediately follow it in the DOM:

![Inline jQuery-dependent Gravity Form scripts](/assets/images/Screen Shot 2016-10-31 at 12.55.36 AM.png)

In this case, the typical gform_init_scripts_footer solution doesn't work, because moving jQuery to the footer would break these inline scripts.

Another solution I've run across uses gform_cdata_open and gform_cdata_close to wrap these inline scripts in a function that's called once jQuery has been loaded. However, this clever technique can generate errors on payment forms (e.g. Uncaught ReferenceError: gf_global is not defined) due to the scripts being run after DOMContentLoaded instead of before as Gravity Form expects.

Ultimately, the most reliable (and performant) solution I've found is to inject a synchronous jQuery <script> immediately before the inline scripts using the gform_cdata_open hook once.

/**
 * Deregister jQuery in <head>, so it can be loaded in the footer or before the
 * first Gravity Form on the page.
 */
function gc_deregister_default_jquery()
{
	wp_deregister_script('jquery');
}
add_action( 'wp_enqueue_scripts', 'gc_deregister_default_jquery' );

/**
 * Inject synchronous jQuery dependency before the Gravity Form inline scripts
 */
function inject_jquery_above_gravity_form( $content = '' )
{
	// keep track of jquery so it's not loaded twice!
	global $jquery_already_injected;

	if ( !isset($jquery_already_injected) ) {

		$jquery_already_injected = true;

		// End inline script
		$content .= "</script>\n";

		// Inject jQuery
		$content .= "<script src='/path/to/jquery.min.js'></script>\n";

		// Start inline script again
		$content .= "<script>";
	}

	return $content;
}
add_filter( 'gform_cdata_open', 'inject_jquery_above_gravity_form' );

/**
 * Enqueue jQuery in footer unless it's already been injected above Gravity Form.
 * In this case, enqueue a fake version to trigger dependent scripts, and then remove this fake version.
 */

function enqueue_jquery()
{
	global $jquery_already_injected;

	// jQuery has not been loaded
	if ( !isset($jquery_already_injected) ) {
		wp_enqueue_script('jquery');
	}
	// jQuery has already been loaded above a Gravity Form
	else {

		// Enqueue fake script to trigger dependencies
		wp_enqueue_script( 'jquery', '//fake-jquery-script.js', [], null );

		// Remove this fake script's HTML before it's actually injected into the DOM
		function gc_remove_fake_jquery_script( $tag ) {
			$tag = ( strpos($tag, 'fake-jquery-script') !== false ) ? '' : $tag;
			return $tag;
		}
		add_filter( 'script_loader_tag', 'gc_remove_fake_jquery_script' );
	}

}
add_action('wp_footer', 'enqueue_jquery', 9);

Your comments and feedback are welcome!

How to Deselect Radio Buttons on Web Forms

Micah Engle-Eshleman — Mon, 18 Feb 2013 18:53:00 +0000

Today I filled out yet another poorly built online survey. This penultimate page asked for my academic major:

Notice the three groupings of radio buttons. One in case you're undecided, one for social science majors, and one for folks (like me) in the hard sciences.

Just for fun, let's select one of each:

Cool, but now we want to be serious and just select one major. Hmm, we can't deselect radio buttons!

At this point the frustrated user who accidentally clicked on the wrong button does one of the following:

Throws up their hands and moves on, leaving the 3 unintelligible selections in place.
Refreshes the form and fills in everything a second time. (nasty if this section was near the end).
Googles furiously for how to deselect radio buttons (and writes this article).

Here's a quick fix in Google Chrome

Using another browser? No worries, the steps are nearly identical in Firefox, Safari and Internet Explorer.

We'll be using a single line of javascript to identify the radio button and deselect it.

Right click on the radio button and select Inspect. Google Developer Tools will appear at the bottom of the screen.

Notice the <input> tag that represents this radio button has a unique id of "choice_31_8_0".

Double click to select this id, and copy it.

In the Console tab, paste the following code. Replace "choice_31_8_0" with your radio button's unique id.

document.getElementById("choice_31_8_0'").checked = false

Update: when the <input> tag is selected, you can also use the $0 shortcut to refer to it:

$0.checked = false

Press enter to run code! This should deselect your radio button.

What is this code doing?

The HTML

<input
  name="input_8"
  type="radio"
  value="No, I'm still deciding"
  id="choice_31_8_0"
  tabindex="3"
/>

The <input> tag is used for user inputs in web forms. This particular tag has type= "radio" (radio button) and id = "choice_38_8_0".

The Javascript

document.getElementById("choice_31_8_0").checked = false

This script looks within the document (webpage) for the first element with id equal to choice_31_8_0, and sets the checked property of this object to false.

Micah's Blog

Conditional Logic with Zod + React Hook Form

A super simple form

The importance of default values

Fields with more complex types

Conditional Logic

A Configurable Hamburger Icon to Close (X) Icon Transition

Emotion CSS in JS

Pure SCSS

A Better Data Structure for Reconciling Late-Arriving User Data

A high level view of the problem

Dealing with Arrays

Store a timestamp with every change

Using a Library

How to Track Audio/Video File Loading Progress with Workbox

How to Move a CenturyLink Fiber Modem & ONT

Moving the Fiber, Modem & ONT downstairs

1. Unplug Ethernet from ONT

2. Unplug Fiber from ONT and protect the end of the cable

3. Run Fiber to new location

4. Hook it all back up

Caching Fly.io Apps with Cloudflare

Setup Node API on Fly.io

Setup DNS to Route Subdomain to Fly.io

Cloudflare Caching

Detecting a PWA on the Microsoft Store

Bringing Your PWA to App Stores

How to Accept Payments in Your Play Store TWA

High Level Overview

Detect TWA session

Get Prices from Digital Goods API

Rendering Arbitrary Currencies in UI

Handling Already-Subscribed Google Play Users

Handling Payments

Network-first Pre-cached PWA Start URLs with Workbox

#1. If the user is online, fetch the latest Start URL page from the network

#2. If the user is offline, serve a pre-cached Start URL from the cache.

matchOptions

Good luck!

Bulletproof PWA and TWA Detection

Detecting an Installed PWA

Detecting a TWA (Trusted Web Activity)

How to Create a Repo-scoped Github Access Token in an Organization Account

Step 1: Create a new Github App

Step 2: Jot down the App ID and download the Private Key file (PEM)

Step 3: Install the App on your Organization Account

Step 4: Extract Access Token using Node library

Step 5: You're done!

A More Reliable FileReader

Simple Depth-First Search with Object.entries()

Working with Images

CrUX - Real World Website Performance Data

A Priority+ Navigation With Scrolling and Dropdowns

One Regex to Rule them all: Removing wrapping tags around images in WordPress while preserving links

Visual editor:

Text editor:

On the page:

Creating a Custom Unsubscribe Form in Marketo that Doesn’t Skew Email Analytics

Setting up Custom Authorization in Ember-CLI using Google oAuth2 for the Initial Login

Ember Client:

Node Server:

Step 1: Setup Ember-CLI app structure

Step 2: Setup node server to return tokens

Step 3: Hook up Ember Simple Auth with node server

Step 4: Setting up Google Account logins

Step 5: Fetching user info from Google

Step 6: Refreshing the token

Named Anchors and Fixed Headers

Loading jQuery Immediately before Gravity Forms

How to Deselect Radio Buttons on Web Forms

Here's a quick fix in Google Chrome

What is this code doing?

The HTML

The Javascript

One Regex to Rule them all: Removing wrapping
tags around images in WordPress while preserving links