Learn Shiki Rehype Pretty Code - Core Concepts & Main Architecture

#Introduction

Episode 1 already covered why the Shiki and rehype-pretty-code combination exists. Now it's time to open the hood and see how this combination works from the inside. Episode 2 dissects core concepts and main architecture: Shiki's tokenization flow from TextMate grammar to HTML spans, the node transformation done by rehype-pretty-code, and a complete map of the packages involved.

Understanding the architecture matters because every following episode builds on top of it. When you later configure transformers, choose an engine, or optimize the bundle, you need to know which component works at which stage. So don't rush, because this foundation determines how deep you can dig into the features.

#Shiki's Tokenization Flow Behind the Scenes

#TextMate Grammar as the Source of Rules

Everything starts with TextMate grammar, the same files VSCode uses to color code. Grammar contains regex-based patterns that produce semantic scopes, such as keyword.control, string.quoted.double, and entity.name.function. These scopes are not colors; colors are determined later by the theme.

The consequence is important: as long as the grammar is the same as VSCode, Shiki's tokenization result will be exactly the same as the editor. This is what makes the colors in your documentation feel familiar to developers.

#Oniguruma WASM and the Tokenization Engine

TextMate regex uses Oniguruma-specific syntax, not ordinary JavaScript regex. That's why Shiki runs Oniguruma through WebAssembly to tokenize text. The flow works like this:

• Code text is split line by line.
• Each line is matched against the grammar rules to produce tokens.
• Each token is given a scope that can be nested.
• Scopes are mapped to colors according to the theme.
• The results are assembled into HTML spans.

In Shiki v4, the tokenization engine can be chosen via the engine option. The default engine is Oniguruma WASM from @shikijs/engine-oniguruma, and there's a JavaScript engine alternative from @shikijs/engine-javascript for a smaller bundle at slower speed. We'll go deeper into engine selection in episode 13.

Map of the Shiki tokenization flow

const kode = "const nilai = 42;";
const highlighter = await createHighlighter({
  langs: ["javascript"],
  themes: ["github-dark-default"],
});
const html = highlighter.codeToHtml(kode, {
  lang: "javascript",
  theme: "github-dark-default",
});

Notice that createHighlighter(options) accepts a list of langs and themes loaded into memory. Once the highlighter is ready, codeToHtml(kode, options) tokenizes and directly produces HTML containing colored spans.

#Output in the Form of Colored Spans

The pure Shiki end result is a <pre> element with a <code> element inside containing many color-styled token <span> elements. This is finished HTML that can be rendered directly in the browser without JavaScript.

rehype-pretty-code doesn't replace this mechanism; it adds a structural layer around it. The roles of both are clear: Shiki does the coloring, rehype-pretty-code does the code block experience.

#rehype-pretty-code's Role in the Pipeline

#Transforming Nodes on the HAST Tree

rehype-pretty-code is a rehype plugin, meaning it works on the HAST (Hypertext Abstract Syntax Tree) tree after Markdown has been converted to HTML. Its job is simple in words: detect <pre> and <code> nodes originating from fenced code blocks, then replace them with a richer structure.

Before the plugin runs, the nodes are plain: <pre> and <code> with no metadata. After the plugin runs, the node becomes:

• A <figure> as the main wrapper with the data-rehype-pretty-code-figure attribute.
• A <pre> with the data-theme attribute containing the already-tokenized code.
• Title and caption elements if the meta string provides them.
• A <code> with Shiki's token spans.

This transformation happens on the HAST tree before the page is rendered, so there's no extra client-side step.

#Data Attributes as the Styling Language

The most important thing rehype-pretty-code adds is data attributes that become the anchors for your CSS. Some of the most frequently used:

• data-line on each token line.
• data-highlighted-line on lines requested for highlighting via meta.
• data-highlighted-chars on marked characters or words.
• data-line-numbers when line numbering is active.
• data-changed-line, data-added-line, and data-removed-line for diff blocks.

Because it uses data attributes, you can style from the outside without touching Shiki's HTML output. This keeps flexibility and lets the theme be swapped at any time.

#Key Components You Need to Know

#Core Packages from the Shiki Ecosystem

The Shiki v4 ecosystem is split into small packages that you can pick based on your needs:

• shiki as the main package that bundles the highlighter, langs, and themes.
• @shikijs/langs contains the collection of language grammars.
• @shikijs/themes contains a collection of ready-to-use themes.
• @shikijs/transformers contains built-in transformers like transformerNotationDiff.
• @shikijs/engine-oniguruma and @shikijs/engine-javascript as tokenization engines.
• @shikijs/rehype as the official rehype plugin from the Shiki team.

This package split is one of the big changes in Shiki v4. By selecting packages selectively, the build bundle size can be much smaller — a topic we go deeper into in episodes 12 and 13.

#The unified Pipeline That Connects Everything

All the components meet inside the unified pipeline. This pipeline works sequentially: Markdown text is parsed, converted to HTML, highlighted, then stringified. Plugin order heavily determines the final result.

Complete unified pipeline

unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypePrettyCode, {
    theme: "github-dark-default",
    defaultLang: "plaintext",
  })
  .use(rehypeStringify);

In this pipeline, rehypePrettyCode(options) is installed after remarkRehype. This order is required because rehype-pretty-code can only work on HAST that has already been formed. If you install it before remarkRehype, the plugin won't find the <pre> nodes it's looking for.

#Conclusion

Key takeaways:

• Shiki tokenizes code using TextMate grammar through Oniguruma WASM.
• Shiki's output is finished, colored HTML spans.
• rehype-pretty-code works on the HAST tree as a rehype plugin.
• The plugin turns <pre> and <code> nodes into metadata-rich <figure> elements.
• Data attributes like data-line and data-highlighted-line become the anchors for CSS.
• The unified pipeline manages the order of parse, transform, and stringify.

In episode 3 you'll practice hands-on: installing shiki and rehype-pretty-code, assembling a minimal pipeline, and rendering your first code block. Make sure Node.js and a package manager are installed, because from now on you'll be typing a lot of npm commands.