Learn Shiki Rehype Pretty Code - Transformers (Shiki)

#Introduction

Meta strings give you control through Markdown syntax, but not every need can be expressed through meta. When you want to manipulate tokens, mark lines with special notation, or add metadata to the output, it's transformers' turn to work.

Episode 10 covers transformers in the Shiki ecosystem: the basic transformer concept, the ready-to-use transformer catalog from @shikijs/transformers, how to use them via the transformers option on rehype-pretty-code, and how to create your own custom transformer. Transformers are the key to breaking past the limits of built-in features.

#What Is a Transformer

#The Position of Transformers in the Shiki Pipeline

Shiki processes code in several stages: tokenization, transformation, then rendering to HAST. A transformer is a function that inserts logic between those stages. Through hooks like preprocess, code, line, and span, you can modify the code, tokens, and elements before the final HTML is formed.

Each transformer is an object carrying a name property and a set of hooks. Hooks receive the data of a certain stage and may return new values to pass on to the next stage.

#Installing the Package

Transformers have been separated into their own package since Shiki v4. Install it alongside shiki:

npm i shiki @shikijs/transformers

The npm i shiki @shikijs/transformers command adds both dependencies to the project. The @shikijs/transformers package contains a collection of ready-to-use transformers written and tested by the Shiki team.

#Ready-to-Use Transformers

#Comment-Based Notation Transformers

This group of transformers reads special comments inside the code as visual instructions. The most commonly used examples:

• transformerNotationDiff recognizes // [!code ++] and // [!code --] comments to mark added and removed lines.
• transformerNotationHighlight recognizes // [!code highlight] to highlight lines.
• transformerNotationFocus recognizes // [!code focus] to focus lines.
• transformerNotationWordHighlight marks words after // [!code word:...].

This notation is popular because authors only need to write a comment on the original code line, without an extra meta string. It suits content written in the VitePress style.

#Other Utility Transformers

Besides notation, there are transformers for technical needs:

• transformerRenderWhitespace visually displays spaces and tabs.
• transformerCompactLineOptions tidies line options in the output.
• transformerMetaHighlight translates meta strings into line highlighting.
• transformerRemoveLineBreak removes unwanted empty lines.

Only use the transformers you really need, because each transformer adds work at render time.

#Installing Transformers in rehype-pretty-code

#The transformers Option in Configuration

rehype-pretty-code forwards the transformers option to Shiki. Arrange the transformer list in the plugin configuration:

Install transformers

import {
  transformerNotationDiff,
  transformerNotationHighlight,
} from "@shikijs/transformers";
 
const options = {
  theme: "github-dark-default",
  transformers: [
    transformerNotationDiff(),
    transformerNotationHighlight(),
  ],
};

With this configuration, you can write // [!code ++] comments inside a code block and that line is automatically marked as added. The order in the array determines the execution order of the hooks.

#Example Code with Notation

Here's a TypeScript block using diff notation via a transformer:

Code with diff notation

const lama = 1;
const baru = 2;
console.log(lama);
console.log(baru);

Notice that you don't need to write the diff meta string on the opening line. The notation transformer reads comments inside the code content independently. This is the advantage of using transformers: features are moved from the meta string into the code content.

#Creating a Custom Transformer

#The Basic Structure of a Custom Transformer

You can create your own transformer as a ShikiTransformer object with hooks. The following example adds a data-theme-source attribute to the code block element:

Custom transformer

import type { ShikiTransformer } from "shiki";
 
export const themeSourceTransformer: ShikiTransformer = {
  name: "theme-source-transformer",
  preprocess(code, options) {
    options.theme = options.theme ?? "github-dark-default";
    return code;
  },
  code(hast) {
    hast.properties["data-theme-source"] = "shiki";
  },
};

The preprocess hook modifies the input before tokenization, while the code hook receives the <pre> HAST node and may modify its properties. The combination of the two covers most customization needs.

#Frequently Used Hooks

Choose a hook according to the stage you want to change:

• preprocess to modify the code string before tokenization.
• tokens to change the set of already-annotated tokens.
• line to modify the HAST element of a single line.
• span to modify the HAST element of a single token.
• code to modify the HAST element of the whole code block.

Each hook receives that stage's data and can return new data. You can also use a dev version of shiki to test a custom transformer before using it in production.

#Conclusion

Key takeaways:

• Transformers insert logic between the tokenization and render stages.
• The @shikijs/transformers package provides ready-to-use transformers.
• Comment notation like // [!code ++] enriches code without meta strings.
• The transformers option in rehype-pretty-code accepts an array of transformers.
• A custom transformer is created as an object with a name property and hooks.
• The preprocess, line, span, and code hooks control every output layer.

In episode 11 you'll learn integration with content frameworks: installing the same pipeline in Next.js with MDX or Velite, plus integration patterns in Astro, SvelteKit, and VitePress, each differing only at the configuration point.