October 10, 2024
How To Use Shiki In Next.js with MDX

ShikiJS is the most popular code syntax highlighter. But how do you integrate it into your Next.JS project?

How To Use Shiki In Next.js with MDX

Introduction#

ShikiJS is a powerful code syntax highlighter based on TextMate Grammar, which is the same engine that powers VSCode. ShikiJS has been used in many popular projects such as NextJS landing page, and Vercel. In this post, we will dive into how to use ShikiJS and integrate it into NextJS and MDX ecosystem.

Why ShikiJS?#

There are some existing syntax highlighters that are well supported and powerful such as PrismJS, HighlightJS. However, these are designed to run in the browser. ShikiJS takes a different approach called highlighting ahead of time. Instead of shipping raw code content to the browser then highlighting it, ShikiJS will ship the highlighted code content as HTML to the browser. This is a more efficient approach for static site generators and server-rendering like NextJS.

How to use Shiki in Next with MDX#

Markdown is a popular format for writing content. It’s a markup language like HTML, but it provides a more lightweight, simpler set of rules for formatting text. Most static sites use Markdown to write content.

MDX is a superset of Markdown that allows you to use JSX inside Markdown. Sometimes you want to include some specific, custom components such as charts, code blocks or interactive elements which Markdown does not support.

NextJS supports MDX very well. You can use the official @next/mdx package to render MDX content from local files. With this pattern, it allows you to write pages in Markdown or MDX in /pages (Page Router) or /app (App Router) directory just like how you write pages in React.

Another way is to use a third-party package called next-mdx-remote. Unlike @next/mdx, next-mdx-remote allows you to render MDX content from remote sources such as a CMS or a database, or even from a file system by using node:fs. If you want to locate all your MDX files in a specific directory, not in NextJS directory patterns, next-mdx-remote is a good choice. For simplicity, I will use next-mdx-remote and the App Router pattern in this post.

Set up#

NextJS has a solution for creating a blog website using App Router and MDX. You can use this template to visualize how it works. First, create a new NextJS project with the following command:

Terminal
npx create-next-app --example https://github.com/vercel/examples/tree/main/solutions/blog shiki-mdx-next-app
 
...
 
Success! Created shiki-mdx-next-app at shiki-mdx-next-app
Inside that directory, you can run several commands:
 
  npm run dev
    Starts the development server.
 
  npm run build
    Builds the app for production.
 
  npm start
    Runs the built app in production mode.
 
We suggest that you begin by typing:
 
  cd shiki-mdx-next-app
  npm run dev

You can replace shiki-mdx-next-app with your project name.

This package has already included essential packages and configurations for MDX and App Router pattern.

You can run the development server with npm run dev and open the browser to http://localhost:3000 to see the result.

Install ShikiJS and its dependencies#

Shiki contains many packages to work and integrate with different frameworks.

Let’s install these packages:

Terminal
npm install -D shiki @shikijs/rehype @shikijs/transformers

You did not mistake. Shiki dependencies are suggested to be installed as dev dependencies. This is because Shiki will be used in development mode only. During runtime, Shiki will not be used therefore, it’s not necessary to include Shiki in the production build.

Create a fine-grained Bundle#

Shiki comes with a full bundle, which allows you to use all the supported themes and languages. However, this bundle can be quite large and unnecessary. However, you can create a fine-grained bundle1 that explicitly loads themes and languages you specify.

Create a lib directory in the /app directory. Then, create a new file called shiki.ts with the following content:

typescript
lib/shiki.ts
import { createHighlighterCore } from "shiki/core";
import { createOnigurumaEngine } from "shiki/engine-oniguruma.mjs";
import githubDark from "shiki/themes/github-dark.mjs";
import githubLight from "shiki/themes/github-light.mjs";
import javascriptLang from "shiki/langs/javascript.mjs";
import cssLang from "shiki/langs/css.mjs";
import graphqlLang from "shiki/langs/graphql.mjs";
import pythonLang from "shiki/langs/python.mjs";
import shellLang from "shiki/langs/shellscript.mjs";
import protobufLang from "shiki/langs/protobuf.mjs";
 
import wasm from "shiki/wasm";
 
/**
 * Create a Shiki core highlighter instance, with no languages or themes
 * bundled. Wasm and each language and theme must be loaded manually.
 */
const highlighter = createHighlighterCore({
  // Specify the themes you want to use. You can include as many as you want.
  // See https://shiki.style/themes for a list of available themes.
  themes: [githubDark, githubLight],
 
  // Specify the languages you want to use. You can include as many
  // as you want.
  langs: [
    javascriptLang,
    cssLang,
    graphqlLang,
    pythonLang,
    shellLang,
    protobufLang,
  ],
 
  // Default grammar parser. This is recommended for most use cases. You can
  // also use your own custom engine.
  // See https://shiki.style/guide/regex-engines#oniguruma-engine
  engine: createOnigurumaEngine(wasm),
});
 
export default highlighter;

Use rehype plugins#

The way Shiki works is to parse the code string directly and return the HTML content. For example:

js
JavaScript
await highlighter.codeToHtml('console.log("Hello World")', {
  lang: 'javascript',
  theme: 'github-dark',
});

However, it is tedious to manually parse the code string in every code block in every MDX file. This is where @shikijs/rehype comes in. This plugin will be used by next-mdx-remote to parse the code string in the MDX file and return the HTML content automatically.

Go to the app/components/mdx.tsx file and modify it as follow:

reactts
app/components/mdx.tsx
// Other imports ...
import { highlight } from "sugar-high"; 
import rehypeShikiFromHighlighter, { 
  type RehypeShikiCoreOptions, 
} from "@shikijs/rehype/core"; 
­  
import shikiHighlighter from "../lib/shiki"; 
 
// Other components ...
 
function Code({ children, ...props }) { 
  let codeHTML = highlight(children) 
  return <code dangerouslySetInnerHTML={{ __html: codeHTML }} {...props} /> 
} 
 
let components = {
  h1: createHeading(1),
  h2: createHeading(2),
  h3: createHeading(3),
  h4: createHeading(4),
  h5: createHeading(5),
  h6: createHeading(6),
  Image: RoundedImage,
  code: Code, 
  a: CustomLink,
  Table,
};
 
export function CustomMDX(props) { 
export async function CustomMDX(props) { 
  return (
    <MDXRemote
      {...props}
      components={{ ...components, ...(props.components || {}) }}
      options={{ 
        mdxOptions: { 
          format: "mdx", 
          rehypePlugins: [ 
            [ 
              rehypeShikiFromHighlighter, 
              await shikiHighlighter, 
              { 
                themes: { 
                  dark: "github-dark", 
                  light: "github-light", 
                }, 
              } as RehypeShikiCoreOptions, 
            ], 
          ], 
        }, 
      }}
    />
  );
}

What we did here:

  • Remove Code component and remove highlight from sugar-high package since we will use @shikijs/rehype to parse the code string.
  • Add rehypeShikiFromHighlighter from @shikijs/rehype package to rehypePlugins array in the CustomMDX component. See rehype with fine-grained bundle for more information.
  • Update CustomMDX component to be an async component and use await for shikiHighlighter.

Let’s create a simple code block in the index.mdx file to test the syntax highlighting:

mdx
how-to-use-shiki-in-next-with-mdx.mdx
{/* Other content */}
```ansi
npx create-next-app --example https://github.com/vercel/examples/tree/main/solutions/blog shiki-mdx-next-app
 
...
 
Success! Created shiki-mdx-next-app at shiki-mdx-next-app
Inside that directory, you can run several commands:
 
  npm run dev
    Starts the development server.
 
  npm run build
    Builds the app for production.
 
  npm start
    Runs the built app in production mode.
 
We suggest that you begin by typing:
 
  cd shiki-mdx-next-app
  npm run dev
```

The result will look like this:

MDX No Style

Adding dual themes#

By default, Shiki will use the light theme you specify in the rehype options. It’s not that Shiki does not include the dark theme, but it does not include the CSS style for dark theme. In order to use the dark theme, you need to add your own CSS style for that.

In NextJS projects, you can use either class or media query to switch between light and dark themes. Here is an example of how to add a dark theme with class in the app/globals.css file:

file_type_css
app/globals.css
/* For dark mode using query */
@media (prefers-color-scheme: dark) {
  .shiki,
  .shiki span {
    color: var(--shiki-dark) !important;
    background-color: var(--shiki-dark-bg) !important;
    /* Optional, if you also want font styles */
    font-style: var(--shiki-dark-font-style) !important;
    font-weight: var(--shiki-dark-font-weight) !important;
    text-decoration: var(--shiki-dark-text-decoration) !important;
  }
}

For more information about how to add dual themes, see ShikiJS’s Official Dual Theme Guide.

With that change, now our code block looks better in dark mode:

MDX With Dark Theme

Or on light mode:

MDX With Light Theme

Custom code block component#

Often, you want to customize the code block component to include certain features such as title, line numbers, or copy button. In order to do that, you need to intercept the component rendering via components prop in the MDXRemote.

Traditionally, next-mdx-remote does not parse the code block in the MDX files. It passes the raw code string to the native pre and code HTML elements.

reactts
app/components/mdx.tsx
let components = {
  // other components
  pre: (props) => {
    console.log(props);
 
    return <pre {...props} />;
  },
};
 
export async function CustomMDX(props) {
  return (
    <MDXRemote
      {...props}
      components={{ ...components, ...(props.components || {}) }}
    />
  );
}

For example, this simple code block:

js
JavaScript
await highlighter.codeToHtml('console.log("Hello World")', {
  lang: 'javascript',
  theme: 'github-dark',
});

will produce the following output:

Terminal
{
  children: {
    '$$typeof': Symbol(react.element),
    type: 'code',
    key: null,
    ref: null,
    props: {
      className: 'language-js',
      children: `await highlighter.codeToHtml('console.log("Hello World")', {\n` +
        "  lang: 'javascript',\n" +
        "  theme: 'github-dark',\n" +
        '});\n'
    },
    _owner: null,
    _store: {}
  }
}

Then you can use React components to intercept the code string and parse it with tools such as PrismJS, or HighlightJS to create your own code block component.

However, Shiki Rehype Plugin will parse the code string and modify the pre and code HTML elements to include the highlighted HTML content. Therefore, you don’t need to handle the parsing process.

reactts
app/components/mdx.tsx
export async function CustomMDX(props) {
  return (
    <MDXRemote
      {...props}
      components={{ ...components, ...(props.components || {}) }}
      options={{ 
        mdxOptions: { 
          format: "mdx", 
          rehypePlugins: [ 
            [ 
              rehypeShikiFromHighlighter, 
              await shikiHighlighter, 
              { 
                themes: { 
                  dark: "github-dark", 
                  light: "github-light", 
                }, 
              } as RehypeShikiCoreOptions, 
            ], 
          ], 
        }, 
      }}
    />
  );
}

With the same code block, the terminal output looks slightly different:

Terminal
{
  className: 'shiki shiki-themes github-light github-dark',
  style: {
    backgroundColor: '#fff',
    '--shiki-dark-bg': '#24292e',
    color: '#24292e',
    '--shiki-dark': '#e1e4e8'
  },
  tabIndex: '0',
  children: {
    '$$typeof': Symbol(react.element),
    type: 'code',
    key: null,
    ref: null,
    props: { children: [Array] },
    _owner: null,
    _store: {}
  }
}

You can inspect the children property further using console.log and combining it with functions like JSON.stringify or util.inspect to see the full content.

After inspecting the props, you can select which one you want to use in your final code block, or you can just pass the props directly to the pre with spreading operator {...props}.

For example:

reactts
app/components/mdx.tsx
let components = {
  pre: (props) => {
    console.log(props);
 
    return <CustomPre {...props} />
  },
};
reactts
app/components/custom-pre.tsx
import * as React from "react";
 
import CopyButton from "./copy-button";
 
export interface MetaMap {
  title: string;
  lang: string | undefined;
  rawCode: string;
}
 
type BlogCodeProps = React.PropsWithChildren<
  React.DetailedHTMLProps<
    React.HTMLAttributes<HTMLPreElement>,
    HTMLPreElement
  > &
    MetaMap
>;
 
const CustomPre: React.FC<BlogCodeProps> = ({
  title,
  children,
  lang,
  rawCode,
  ...rest
}) => {
  return (
    <pre {...rest}>
      <div className="flex items-center justify-between ">
        <div>title</div>
        <CopyButton content="" />
      </div>
      <div className="w-full overflow-x-auto">{children}</div>
    </pre>
  );
};
 
export default CustomPre;
reactts
app/components/copy-button.tsx
"use client";
import * as React from "react";
 
interface Props {
  content: string;
}
 
const CopyButton: React.FC<Props> = ({ content }) => {
  const [title, setTitle] = React.useState("Copy");
  const [timerId, setTimerId] = React.useState<NodeJS.Timeout | null>(null);
 
  const handleClick = async () => {
    try {
      await navigator.clipboard.writeText(content);
      setTitle("Copied!");
 
      if (timerId) {
        clearTimeout(timerId);
        setTimerId(null);
      }
 
      setTimerId(
        setTimeout(() => {
          setTitle("Copy");
        }, 2000)
      );
    } catch (error) {
      console.error("Failed to copy: ", error);
    }
  };
 
  return (
    <button
      onClick={handleClick}
      className="px-2 py-1 rounded-lg bg-slate-500/20 hover:bg-slate-500/40 hover:cursor-pointer"
    >
      {title}
    </button>
  );
};
 
export default CopyButton;

Parse metadata#

Metadata is a common feature in Markdown and MDX files, which allows you to specify some additional information about the content such as title, displaying line numbers, highlighting certain lines or allowing copying content.

Typically, metadata is written in the following format:

mdx
MDX
```js title="index.js" displayLineNumbers="true" allowCopy="true"
console.log('Hello World');
```

Now how you structure this metadata format is up to you. You can use it with or without quotes, or you can use a different delimiter such as : or =. With that said, using = is the most common way to separate the key and value, and using quotes allows you to use spaces in the value.

Using the above code block with metadata, you can access the raw metadata string during Shiki transformer hooks2. For example:

reactts
app/components/mdx.tsx
export async function CustomMDX(props) {
  return (
    <MDXRemote
      {...props}
      components={{ ...components, ...(props.components || {}) }}
      options={{
        mdxOptions: {
          format: "mdx",
          rehypePlugins: [
            [
              rehypeShikiFromHighlighter,
              await shikiHighlighter,
              {
                themes: {
                  dark: "github-dark",
                  light: "github-light",
                },
                transformers: [ 
                  { 
                    preprocess(code, options) { 
                      console.log("preprocess", code, options.meta); 
                    }, 
                  }, 
                ], 
              } as RehypeShikiCoreOptions,
            ],
          ],
        },
      }}
    />
  );
}

The output will look like this:

Terminal
preprocess await highlighter.codeToHtml('console.log("Hello World")', {
  lang: 'javascript',
  theme: 'github-dark',
}); { __raw: 'displayLineNumbers="false" allowCopy="false"' }

However, we want to access the metadata as a hash map or a key-value object so we can easily access the metadata values in other transformer hooks. Luckily, Shiki provides a helper function called parseMetaString. This function will take the raw metadata string and allows us to parse it into a key-value object.

reactts
app/components/mdx.tsx
interface MetaValue { 
  name: string; 
  regex: RegExp; 
} 
­  
export interface MetaMap { 
  title: string; 
  displayLineNumbers: boolean | undefined; 
  allowCopy: boolean | undefined; 
  lang: string | undefined; 
} 
­  
const metaValues: MetaValue[] = [ 
  { 
    name: "title", 
    regex: /title="(?<value>[^"]*)"/, 
  }, 
  { 
    name: "lang", 
    regex: /lang="(?<value>[^"]*)"/, 
  }, 
  { 
    name: "displayLineNumbers", 
    regex: /displayLineNumbers="(?<value>true|false)"/, 
  }, 
  { 
    name: "allowCopy", 
    regex: /allowCopy="(?<value>true|false)"/, 
  }, 
]; 
 
export async function CustomMDX(props) {
  return (
    <MDXRemote
      {...props}
      components={{ ...components, ...(props.components || {}) }}
      options={{
        mdxOptions: {
          format: "mdx",
          rehypePlugins: [
            [
              rehypeShikiFromHighlighter,
              await shikiHighlighter,
              {
                themes: {
                  // Other themes
                },
                parseMetaString(metaString) { 
                  const map: MetaMap = { 
                    title: "", 
                    displayLineNumbers: undefined, 
                    allowCopy: undefined, 
                    lang: "txt", 
                  }; 
­  
                  for (const value of metaValues) { 
                    const result = value.regex.exec(metaString); 
­  
                    if (result && value.name === "title") { 
                      map.title = result?.groups?.value || ""; 
                    } 
­  
                    if (result && value.name === "displayLineNumbers") { 
                      map.displayLineNumbers = result.groups?.value === "true"; 
                    } 
­  
                    if (result && value.name === "allowCopy") { 
                      map.allowCopy = result.groups?.value === "true"; 
                    } 
                  } 
­  
                  return map; 
                }, 
                transformers: [
                  {
                    preprocess(code, options) {
                      console.log("preprocess", code, options.meta);
                    },
                  },
                ],
              } as RehypeShikiCoreOptions,
            ],
          ],
        },
      }}
    />
  );
}

The parseMetaString function will append the metadata object to the options object in the transformer hook cycle. You can access the metadata object by using options.meta in the transformer hooks such as preprocess.

Terminal
{
  title: '',
  displayLineNumbers: false,
  allowCopy: false,
  lang: 'txt',
  __raw: 'displayLineNumbers="false" allowCopy="false"'
}

Then you can pass the metadata object into your CustomPre component:

reactts
app/components/custom-pre.tsx
const CustomPre: React.FC<BlogCodeProps> = ({
  title,
  children,
  lang,
  rawCode,
  ...rest
}) => {
  return (
    <pre {...rest}>
      <div className="flex items-center justify-between ">
        <div>{title}</div> // [!code ++]
        <CopyButton content="" />
      </div>
      <div className="w-full overflow-x-auto">{children}</div>
    </pre>
  );
};

Query the language#

You might have noticed that the lang metadata is not set properly. Obviously, the code block is written in JavaScript, but the lang metadata is set to txt.

This is because the lang is not considered as a metadata in the code block but it is a part of the MDX code block. In order to access the language used in code blocks, you can reference to options.lang in the transformer hooks.

reactts
app/components/mdx.tsx
transformers: [
  {
    preprocess(code, options) { 
      const optionsMeta = options.meta as MetaMap; 
      optionsMeta.lang = options.lang || "txt"; 
    }, 
    // other hooks
  },
],

Now the lang metadata will be set to javascript:

Terminal
{
  title: '',
  displayLineNumbers: false,
  allowCopy: false,
  lang: 'js',
  __raw: 'displayLineNumbers="false" allowCopy="false"'
}

The lang metadata does not effect anything about parsing. It’s merely an an additional property that you can use in your custom code block component such as adding a language icon, or displaying the language name in the code block. For example, we can add the default title to the lang metadata if the title is missing:

reactts
app/components/custom-pre.tsx
const CustomPre: React.FC<BlogCodeProps> = ({
  title,
  children,
  lang,
  rawCode,
  ...rest
}) => {
  return (
    <pre {...rest}>
      <div className="flex items-center justify-between">
        <div>{title && title.length > 0 ? title : lang}</div> // [!code ++]
        <CopyButton content="" />
      </div>
      <div className="w-full overflow-x-auto">{children}</div>
    </pre>
  );
};

Copy button#

Copy button is a common feature in writing code blocks. It allows users to copy the raw code content to the clipboard. A simple and secure way to implement this feature is to use the navigator.clipboard API as we did in the CopyButton component.

However, this API only works with strings. Since Shiki returns the highlighted HTML content, we cannot use children props directly. But thanks to transformer hook preprocess, we can access the raw code content and pass it as a metadata.

reactts
app/components/mdx.tsx
transformers: [
  {
    preprocess(code, options) {
      const optionsMeta = options.meta as MetaMap;
      optionsMeta.lang = options.lang || "txt";
      optionsMeta.rawCode = code; 
    },
    // other hooks
  },
],

Now, you can use the rawCode metadata to pass the raw code content to the CopyButton component.

reactts
app/components/custom-pre.tsx
const CustomPre: React.FC<BlogCodeProps> = ({
  title,
  children,
  lang,
  rawCode,
  ...rest
}) => {
  return (
    <pre {...rest}>
      <div className="flex items-center justify-between ">
        <div>{title && title.length > 0 ? title : lang}</div>
        <CopyButton content={rawCode} /> // [!code ++]
      </div>
      <div className="w-full overflow-x-auto">{children}</div>
    </pre>
  );
};

Transformer hooks#

We have been mentioning the term transformer hooks throughout this post but never really explain what it is. Transformer hooks are a set of functions that allow you to intercept the parsing process of the code block.

Builtin transformer hooks#

Shiki provides a set of builtin transformer hooks that you can use to customize the parsing process. ShikiJS provides the list of builtin transformer hooks here

Custom transformer hooks#

You can also create your own custom transformer hooks for your specific use case. A common case while writing MDX code blocks is to add line numbers. However, Shiki does not provide a transformer hook for such feature. But you can create it by yourself.

There are many ways to create such feature. You can intercept deeply into the HTML content and add the line numbers manually. Or you can follow the best practice that Shiki transformer hooks follow, which is to add a CSS class to the HTML content. For simplicity, we are going to use the latter approach.

reactts
app/components/mdx.tsx
export async function CustomMDX(props) {
  return (
    <MDXRemote
      {...props}
      components={{ ...components, ...(props.components || {}) }}
      options={{
        mdxOptions: {
          // Other options
          rehypePlugins: [
            // Other plugins
            [
              rehypeShikiFromHighlighter,
              await shikiHighlighter,
              {
                // code configurations
                transformers: [
                  // Other transformers
                  {
                    // Other hooks
                    line(node, line) { 
                      node.properties["class"] = `${node.properties["class"]} line-number`; 
                    }, 
                  },
                ],
              } as RehypeShikiCoreOptions,
            ],
          ],
        },
      }}
    />
  );
}

The HTML output will look like this:

html
HTML
<pre class="shiki shiki-themes github-light github-dark" style="background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8" tabindex="0">
  <div class="flex items-center justify-between ">
    <div>shiki.ts</div>
    <button class="px-2 py-1 rounded-lg bg-slate-500/20 hover:bg-slate-500/40 hover:cursor-pointer">Copy</button>
  </div>
  <div class="w-full overflow-x-auto">
    <code>
      <span class="line has-line-number">...</span>
      <span class="line has-line-number">...</span>
      <span class="line has-line-number">...</span>
      <span class="line has-line-number">...</span>
      <span class="line has-line-number">...</span>
      <!-- other lines -->
    </code>
  </div>
</pre>

Now, it’s up to you on how you want to style using this CSS class. An example of that is to use counter and content CSS properties to add line numbers.

file_type_css
app/globals.css
.prose pre {
  @apply bg-neutral-50 dark:bg-neutral-900 rounded-lg overflow-x-auto border border-neutral-200 dark:border-neutral-900 py-2 px-3 text-sm;
  @apply [counter-reset:_line];
}
 
.prose pre .line.has-line-number:before {
  @apply content-[counter(_line)] [counter-increment:_line] text-neutral-400 dark:text-neutral-600;
  @apply mr-2 w-4 inline-block text-right;
}

You can also combine the metadata with this feature to opt your code blocks in or out displaying line numbers.

Conclusion#

Shiki is a powerful syntax highlighter that can be integrated very well into the Next.JS and MDX ecosystem. With the help of @shikijs/rehype and @shikijs/transformers, you can easily parse the code string in the MDX files, customize the code block component, and add additional features to make your code blocks more interactive and more informative.

With that said, Shiki contains many drawbacks. First, it uses a lot of memory and CPU resources to parse the code string. Secondly, it requires you to create a decent strategy for caching Shiki instances on hot reloading3. Finally, if you need something to work in the browser, it’s getting tricky to use Shiki in such cases.

There are many packages that help you integrate Shiki with MDX and NextJS such as rehype-pretty-code. These packages use Shiki under the hood to save you from the complexity of configuration while still provide you better, more fine-grained control over how to customize the code blocks.

You can follow the complete repository of this post on GitHub.

References#

Footnotes#

  1. ShikiJS - Fine-grained Bundle ↩

  2. ShikiJS - Transformer Hooks ↩

  3. Caching Shiki for Faster Build Times - By Hector Sosa ↩

Tags:

mdxnextjs