Learn Shiki Rehype Pretty Code - Performance & Bundle Optimization

#Introduction

Slow highlighting is felt when the build takes long or when a client-side page flickers before the code is colored. Shiki's performance is determined by three things: the regex engine that tokenizes, where the highlighting runs, and how much of the result can be reused.

Episode 13 covers the Oniguruma WASM loading mechanism, the engine option with the @shikijs/engine-oniguruma package, the difference between SSG and client-side strategies, and grammar caching techniques. The goal is one: make highlighting fast, light, and unobtrusive to the user experience.

#Tokenization Engine and WASM

#The Role of Oniguruma in Tokenization

TextMate grammar uses special regex syntax that standard JavaScript regex doesn't fully support. The solution is Oniguruma, the regex engine from Ruby ported to WebAssembly. Shiki calls this engine through a WebAssembly binding to match grammar rules line by line.

Consequently, Shiki needs a WASM file. Since v4, you import WASM from shiki/wasm and inject it into the engine:

Engine with WASM

import { createHighlighterCore } from "shiki/core";
import { createOnigurumaEngine } from "@shikijs/engine-oniguruma";
 
const highlighter = await createHighlighterCore({
  langs: [],
  themes: [],
  engine: createOnigurumaEngine(import("shiki/wasm")),
});

The WASM file contains the entire Oniguruma regex logic. In a build-time context, this file only needs to be loaded once, and the bundler will include it as a separate asset.

#Other Engine Variations

Besides Oniguruma, Shiki provides a pure JavaScript engine: @shikijs/engine-javascript. This engine translates grammar rules into JavaScript regex, so no WASM is needed. The trade-off: some very complex grammar patterns aren't supported, and tokenization performance can be slower for large code.

Choose Oniguruma for maximum accuracy, or the JavaScript engine when you can't provide WASM at all, for example in environments that restrict executable files.

#Where Highlighting Runs

#Build Time with SSG

In static site generation, the rehype-pretty-code pipeline runs during bun run build. Shiki tokenizes all the code, and the result is static HTML containing colored spans. The browser doesn't need to load WASM or run any regex at all.

This strategy is the fastest on the client side. The first page load already shows perfectly colored code with no flicker. There's no dynamic code evaluation, which also makes applying a strict Content-Security-Policy easier.

#Client-Side Highlighting

Content rendered dynamically in the browser, such as user input or code fetched from an API, can only be highlighted on the client. You load the engine and grammars into the client bundle:

Highlight on the client

import { codeToHtml } from "shiki";
 
const html = await codeToHtml(input, {
  lang: "javascript",
  theme: "github-dark-default",
});

Client-side requires WASM, grammars, and themes sent to the browser. For short code this is still reasonable, but avoid it for entire article content because it adds network and CPU load.

#Criteria for Choosing

Use SSG for static content, like blogs and documentation. Use client-side only for truly dynamic code, and limit the languages sent to the client. A common combination: article code is highlighted at build time, user input code is highlighted at runtime.

#Caching and Lazy Loading

#Caching Tokenization Results

Tokenizing the same code repeatedly wastes time. Shiki provides a cache option on the highlighter to store tokenization results and reuse them when the same code and language are called again:

Enable cache

import { createHighlighterCore } from "shiki/core";
 
const cache = new Map();
const highlighter = await createHighlighterCore({
  langs: ["typescript"],
  themes: ["github-dark-default"],
  engine: createOnigurumaEngine(import("shiki/wasm")),
  cache,
});

The cache is a data structure following the CacheStorage contract. With a cache, re-rendering pages that use identical code blocks becomes much faster, especially in a dev server that frequently restarts the pipeline.

#Lazy Loading Languages and Themes

Grammars and themes should be loaded dynamically so they don't become one giant bundle:

Lazy loading

const highlighter = await createHighlighterCore({
  langs: [() => import("@shikijs/langs/typescript")],
  themes: [() => import("@shikijs/themes/github-dark-default")],
  engine: createOnigurumaEngine(import("shiki/wasm")),
});

Dynamic imports split each grammar into a separate chunk. At build time, unused chunks are removed by the bundler, so the space taken is only for languages that actually appear in the content.

#Measuring the Impact

Measure build time and chunk size before and after optimization. The logs of bun run build show each chunk and its size. Aim for a stable build time even as the number of posts grows, and make sure no grammar chunk leaks into the client JavaScript.

#Conclusion

Key takeaways:

• Oniguruma WASM tokenizes TextMate grammar and is loaded through the engine.
• @shikijs/engine-oniguruma handles WASM, @shikijs/engine-javascript offers a WASM-free alternative.
• SSG moves highlighting to build time so the browser doesn't load WASM.
• Client-side highlighting is only for truly dynamic code.
• The cache option stores tokenization results for reuse.
• Lazy loading splits grammars into chunks loaded as needed.

In episode 14 you'll learn security and output sanitization: preventing XSS by escaping code content, being wary of user-controlled meta strings, using rehype-sanitize, and composing a Content-Security-Policy without inline scripts.