Support fenced doc block alternative syntax

[ccoVeille]

Unless I'm wrong mdsf only supports Markdown fenced block with exactly 3 backticks.

But code can be formatted differently

https://www.markdownguide.org/extended-syntax/#fenced-code-blocks

~~~ can be used

Take a look at this markdown syntax

~~~go
const fooBar = 42
~~~

And how it is rendered

const fooBar = 42

It's not so rare to find some of them in Markdown files.

https://github.com/search?q=%22%7E%7E%7E%22+language%3AMarkdown+&type=code

https://github.com/zeuscsc/dify/blob/11b4e4f19941b2a55b2fea0c512e0ceecaf5476e/build.md?plain=1#L1

[hougesen]

I have actually indirectly been looking into this and #697. Lately I have been contemplating whether mdsf should switch back to using a proper markdown parser (pulldown-cmark) instead of my personal "parser" implementation.

The end goal of such a switch would be a make it easier to support more document formats (reStructuredText, mdx, Typst), and of course have better support for markdown.

The only problem is that pulldown-cmark (markdown -> rust) and it's sister library pulldown-cmark-to-cmark (rust -> markdown) does not gurantee a deterministic reversable output (Byron/pulldown-cmark-to-cmark#94). Which means some markdown element types that have multiple ways of being written (like - and * for lists) end up being converted to what pulldown-cmark deems the standard.

Do you think that would be a deal breaker?

[hougesen]

(I hope that made sense 😅 )

[ccoVeille]

I have actually indirectly been looking into this and #697. Lately I have been contemplating whether mdsf should switch back to using a proper markdown parser (pulldown-cmark) instead of my personal "parser" implementation.

The end goal of such a switch would be a make it easier to support more document formats (reStructuredText, mdx, Typst), and of course have better support for markdown.

It sounds reasonable move, if we ignore the huge red flags you rose.

The only problem is that pulldown-cmark (markdown -> rust) and it's sister library pulldown-cmark-to-cmark (rust -> markdown) does not gurantee a deterministic reversable output (Byron/pulldown-cmark-to-cmark#94).

I'm a bit worried by the "non-deterministic" here.

Does it mean that on first run

pulldown-cmark + pulldown-cmark-to-cmark could bring one result (such as using - for list) and on next run (using *) ?

If so, then it's not fun.

Which means some markdown element types that have multiple ways of being written (like - and * for lists) end up being converted to what pulldown-cmark deems the standard.

Or does it mean no matter the list type (- or *) used in source, pulldown-cmarkwill always render one of them ?

Do you think that would be a deal breaker?

Not really, because we could think about diff at different level.

I mean if you parse something. Then you apply transformation with the linter/formatter you already support.

Mdsf is mainly about the fenced doc block. The whole document formatting is optional right now.

So maybe you could either limit yourself in extracting the ~~~ or back tick ones, then apply the pulldown cmark on it.

Using pulldown-cmark formatting on document could become an option like what can be done with prettier right now on the document formatting level.

It would restrict the issues.

But then it means parsing in a strange way.

Can't you even have the exact string extracted in the rust representation?

Because then you can think about (talk is easy for me especially because I have no idea what I'm talking about), are you able to compare the rust representation of these objects?

Are you able to compare that the rust representation of list is with * or - ?

If you can then you can think about doing strange thing like

Source => Parsing in rust => convert back to markdown (this would be what I will call A)

Source => Parsing in rust => apply the changes with linters => convert back to markdown (this would be what I will call B)

We don't care about updating the source if A=B at the end.

Another way is to compare something like this

Source => Parsing in rust (this would be what I will call C)

Source => Parsing in rust => apply the changes with linters => convert back to markdown => Parsing in rust (this would be what I will call D)

We don't care about updating the source if you are about to compare C=D at the end.

[hougesen]

I'm a bit worried by the "non-deterministic" here.

Does it mean that on first run

pulldown-cmark + pulldown-cmark-to-cmark could bring one result (such as using - for list) and on next run (using *) ?

No, I simply worded it confusingly. It will always use the same token (* for list, \``` for code blocks, etc.).

You are right that the best option would be to find a way to simply ignore any non-codeblock related lines.