Confused by all the packages

[melMass]

Hi,

I'm probably a bit dumb but after reading all the readmes of the ecosystem I'm still not understanding the architecture.

For context, I'm building a fairly simple Svelte library around markdown. I get tokens (mdast) out of md files and build svelte markup out of this. My need for a "lower" level library is to hook to the tokenizer/lexer to support custom things like <details> and build custom token types out of it

remark keeps being refered as a md parser / tokenizer and rehype as the to html part. Yet all remark examples output html. So I then looked a micromark that looks lower level but yet again all the samples output html...

TLDR What libraries should I use if I want only mdast output with a way to hook (by extensions) to the tokenizer

[Murderlon]

Hi, I highly recommend reading extending markdown on why you almost never want to start building custom things. Your example in particular of supporting <details> is already possible. You can write HTML in markdown with rehype-raw, then just add CSS/JS to it as you see fit.

Since you're using Svelte, you could also take a look at mdsvex.

[melMass]

Thanks! That was an interesting read.
I love mdsvex but my goals here are a little different, more like svelte-markdown but fully updated to svelte 5.

I will check rehype-raw but my goal is specifically to not get raw html but "metas", my POC for more context, this would extract {summary?:string,details:Token[]} as props to svelte:

const regexDetails = new RegExp(
	`<details>\s*(?:<summary>(?<summary>([^<])+)<\/summary>(?<details>[^<])+)<\/details>`,
);
const res = regextDetails.exec(block) //  res.groups has the data I need

I have a similar POC to support Github "callouts":

Which looks like this in svelte:

<script lang="ts">
  import type { Snippet } from 'svelte'
  import type { GithubCallouts } from '../preprocessors/gh_callouts'
  import 'iconify-icon'

  let { meta, raw, children, ...props }: { children: Snippet } & GithubCallouts =
    $props()
</script>

<div class="not-prose p-0 {meta.className} {meta.className}-{meta.variant}">
  <p class={meta.titleClassName}>
    <iconify-icon inline icon={meta.icon} noobserver></iconify-icon>
    {meta.title}
  </p>
  <div class="{meta.className}-content">
    {@render children()}
  </div>
</div>

[ChristianMurphy]

Check out https://github.com/rehypejs/rehype-react
Which converts HAST/HTML into components for any JSX compatible framework.
Including svelte through https://github.com/kenoxa/svelte-jsx

[wooorm]

TLDR What libraries should I use if I want only mdast output with a way to hook (by extensions) to the tokenizer

So I doubt that this is what you want. Because you actually have HTML. And want to regex it. But regexing HTML is a bad idea. ASTs are better. So you probably want to do more complex things than what you ask.

Looking at your question literally, you look for https://github.com/syntax-tree/mdast-util-from-markdown.

But indeed, I’m thinking that we’ll have 5 back and forth question about how to best solve your problems and it’ll include rehype-raw and AST transforms of the HTML in your markdown.

[melMass]

Makes sense! After reading more about mdast/micromark my approach is probably wrong. In the case of the callouts it's fairly "safe"/easy since I do it after the tokenizer (by looking for token.type == "blockquote). But <details> are "less structured" and need to be "catch" before I think.

I have plenty of docs to look into now so thanks for those answers!