TASVideos Pandoc

Check-in [09a390f980]
Login

Check-in [09a390f980]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Refactor/rewrite to use a preliminary sequence of tokens.

This is to make it possible to do proper escaping of an input like

<span>http:</span><span>//example.com</span>

where the pattern "://" the escaper is looking for is broken across two nodes. Flattening the tree into a sequence before emitting any text or doing any escaping will make it possible to merge adjacent text tokens so that all the proper escaping context is available.

Unintentionally, this strategy seems to resemble the internal FlatDoc representation in Text.DocLayout:
https://github.com/jgm/doclayout/blob/0.5.0.1/src/Text/DocLayout.hs#L264

Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 09a390f980355652e2402c1cb4c821c71894278cd7d898f1662f36049a47fb0a
User & Date: Sand 2026-01-14 17:13:36.377
Original Comment: Refactor/rewrite to use a preliminary sequence of tokens.

This is to make it possible to do proper escaping of an input like http://example.com where the pattern "://" the escaper is looking for is broken across two nodes. Flattening the tree into a sequence before emitting any text or doing any escaping will make it possible to merge adjacent text tokens so that all the proper escaping context is available.

Unintentionally, this strategy seems to resemble the internal FlatDoc representation in Text.DocLayout: https://github.com/jgm/doclayout/blob/0.5.0.1/src/Text/DocLayout.hs#L264

Context
2026-01-14
17:23
Restore hr to TAGS.

Also add handling for void tags like this one. I erroneously removed hr from TAGS when refactoring to use a sequence of tokens, thinking that elements that do not have children don't have to go on the stack and don't need to be present in TAGS. But the renderer didn't know not to put hr on the stack, so it was interpreted as an element that has contents and an end tag like any other.

I missed the error because my test input file, though it contained a horizontal rule, did not contain any escapeable text after it (i.e., "inside" it, from the point of view of the renderer that left an hr tag on the stack). If I had, it would have resulted in the '"hr" missing from TAGS' error from nesting_allowed. check-in: 81063d6f1d user: Sand tags: trunk

17:13
Refactor/rewrite to use a preliminary sequence of tokens.

This is to make it possible to do proper escaping of an input like

<span>http:</span><span>//example.com</span>

where the pattern "://" the escaper is looking for is broken across two nodes. Flattening the tree into a sequence before emitting any text or doing any escaping will make it possible to merge adjacent text tokens so that all the proper escaping context is available.

Unintentionally, this strategy seems to resemble the internal FlatDoc representation in Text.DocLayout:
https://github.com/jgm/doclayout/blob/0.5.0.1/src/Text/DocLayout.hs#L264 check-in: 09a390f980 user: Sand tags: trunk

06:53
Add missing stack to call to enclose_text. check-in: 3beec84ea0 user: Sand tags: trunk
Changes
Unified Diff Ignore Whitespace Patch
Changes to tasvideos_forum.lua.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

16
17
18
19

20
21
22
23
24
25
26
-- https://tasvideos.org/ForumMarkup

-- The format recognized by RawBlock and RawInline.
local MY_FORMAT = "tasvideos_forum"

local function assert_not_nil(x, msg)
	assert(x ~= nil, msg)
	return x
end

-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L80
-- The important piece of information about each BBCode tag is whether child
-- elements are allowed in it or not. Some tags, like [b], always allow new
-- child elements; some, like [code], never allow child elements; and some,
-- like [url], allow child elements only if a parameter is set on the tag: in

-- `[url]http://example.com/[b]path[/b][/url]`, the `[b]...[/b]` is a literal
-- part of the URL path, but in
-- `[url=http://example.com/path]text [b]label[/b][/url]`, the `[b]...[/b]`
-- results in bold text.

local TAGS = {
	b      = {nesting = true},
	i      = {nesting = true},
	u      = {nesting = true},
	s      = {nesting = true},
	sub    = {nesting = true},
	sup    = {nesting = true},





<
<
<
<
<

|
|
|
|
>
|
|
|
|
>







1
2
3
4
5





6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
-- https://tasvideos.org/ForumMarkup

-- The format recognized by RawBlock and RawInline.
local MY_FORMAT = "tasvideos_forum"






-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L80
-- The important piece of information about each BBCode tag, for writing, is
-- whether child elements are allowed in it or not. Some tags, like [b], always
-- allow new child elements; some, like [code], never allow child elements; and
-- some, like [url], allow child elements only if a parameter is set on the
-- tag. For example, in:
-- 	[url]http://example.com/[b]path[/b][/url]
-- the `[b]...[/b]` is a literal part of the URL path. But with a parameter:
-- 	[url=http://example.com/path]text [b]label[/b][/url]
-- the `[b]...[/b]` results in bold text.

local TAGS = {
	b      = {nesting = true},
	i      = {nesting = true},
	u      = {nesting = true},
	s      = {nesting = true},
	sub    = {nesting = true},
	sup    = {nesting = true},
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

89
90
91

92
93
94

95
96
97


98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140


141
142



143
144
145


146
147
148
149
150
151
152

153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172

173
174







175
176
177
178

179
180
181
182
183
184
185
186
187
188
189
190
191
192
193

194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248

249

250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292

293
294

295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370

371
372
373
374
375
376
377
378

379
380

381
382
383
384
385
386





387

388
389
390
391
392
393

394
395
396

397
398
399
400

401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479

480

481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508

























































509
510

511
512
513
514






515
516














































517


518
519
520
521
522





523










524



525




526

















527








528
529

530




531

532
533


534
535
536

537
538

	table  = {nesting = true},
	tr     = {nesting = true},
	td     = {nesting = true},
	th     = {nesting = true},
}

-- We maintain a stack to keep track of what BBCode tags are open as we write
-- the output. The context of open tags affects how the BBCode is parsed, and
-- therefore affects how we must do escaping in the BBCode we output. The two
-- important considerations are:
--
-- 1. Whether the immediately enclosing element permits nested child elements.
--    If it does, then the parser will look for and interpret opening tags in
--    the input, and we must escape them to prevent such interpretation. If it
--    does not, then the parser will copy anything that looks like an opening
--    tag verbatim to the output, and we must *not* escape it.
--    https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L270
-- 2. What tags are open all the way up the stack. This affects the parser's
--    interpretation of closing tags in the input. If the parser finds
--    something that looks like a closing tag, it looks up the stack to see if
--    there is a matching opening tag. If there is, it closes that open tag and
--    all intervening tags on the stack:
--    https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L316
--    If there is not, it copies the text that looks like a closing tag
--    verbatim to the output:
--    https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L342
--
-- Whenever we output text that contains something that looks like an opening
-- or closing tag, we escape it with [noparse] if the current context allows
-- nesting. If the current context does not allow nesting, no escaping is
-- possible, but we may still output the tag-resembling text as long as there
-- is no danger of its being interpreted by the parser. An opening tag may be
-- output verbatim, as the parser will not interpret it in no-nesting contexts.
-- A closing tag is similarly safe to output verbatim, as long as it doesn't
-- match anything up the stack. The only case that is impossible to handle is a
-- closing tag in a non-nesting context that matches a tag somewhere up the
-- stack: such an input is impossible to represent and we raise an error.

-- The stack of open BBCode tags is represented as a linked list of cons cells.
-- Each element of the list has "tag" and "param" fields, where "param" may be
-- nil. For example:
--
--   {tag = "b"}
--   {tag = "url"}
--   {tag = "url", param = "http://example.com"}
local function cons(a, b)
	return {car = a, cdr = b}

end

local function nesting_allowed(stack)

	if stack == nil then
		-- Empty stack means top level, tags are allowed.
		return true

	end
	local tag = assert_not_nil(TAGS[stack.car.tag], string.format("%q missing from TAGS", stack.car.tag))
	return assert_not_nil(tag.nesting) or (stack.car.param ~= nil and tag.nesting_with_param)


end

-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L444
-- Technically the character class should also exclude \p{C} (control and
-- "other" characters).
local CLOSING_TAG_PATTERN = "(%[/([^%[%]=/]+)%])"
-- Strings to escape with [noparse]. The "[" is for BBCode tags, the "://" is
-- for URLs.
local ESCAPE_RE = re.compile([[ "["+ / "://" ]])

local function escape(str, stack)
	if nesting_allowed(stack) then
		-- Nested child elements are allowed here, so we may use
		-- noparse escaping. Enclose all "[" and "://" (for URLs) in
		-- [noparse][/noparse] to avoid their being interpreted as a
		-- BBCode opening/closing tag or a URL.
		--
		-- Escaping just the "[" of a BBCode tag, like so:
		--   [noparse][[/noparse]/center]
		-- is more robust than escaping the whole tag:
		--   [noparse][/center][/noparse]
		-- because the parser interprets closing tags even inside
		-- [noparse]: https://github.com/TASVideos/tasvideos/issues/2247.
		-- Doing it this way also means we can escape the strings
		-- "[noparse]" and "[/noparse]" themselves. Finally, escaping
		-- all "[" is easier than trying to pattern-match tags exactly.
		return re.gsub(str, ESCAPE_RE, "[noparse]%0[/noparse]")
	else
		-- Nested children are not allowed at this point, which means
		-- we cannot use noparse escaping. We do not need to escape
		-- opening tags, and closing tags are likewise safe as long as
		-- they do not match a tag up the stack. Raise an error if a
		-- closing tag matches something in the stack and therefore
		-- would be misinterpreted by the parser.
		local live = {}
		local p = stack
		while p ~= nil do
			live[p.car.tag] = true
			p = p.cdr
		end
		for text, tag in string.gmatch(str, CLOSING_TAG_PATTERN) do
			if live[tag] then
				error(string.format("cannot escape %q in %q", text, str))


			end
		end



		return str
	end
end



local Blocks = {}
local Inlines = {}

local function render_blocks(blocks, stack, opts)
	local x = {}
	for _, el in ipairs(blocks) do

		local render = assert(Blocks[el.tag], el.tag)
		x[#x + 1] = render(el, stack, opts)
	end
	-- TODO eliminate blank line after [/list] here, maybe
	return pandoc.layout.concat(x, pandoc.layout.blankline)
end

local function render_inlines(inlines, stack, opts)
	local x = {}
	for _, el in ipairs(inlines) do
		local render = assert(Inlines[el.tag], el.tag)
		x[#x + 1] = render(el, stack, opts)
	end
	return pandoc.layout.concat(x)
end

-- Check that a tag name is syntactically valid.
-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L426
local function tag_is_valid(tag)
	return string.match(tag, "^[^%[%]=/]+$")

end








local function opening_tag(tag, param)
	assert(tag_is_valid(tag), tag)
	if param == nil then
		return string.format("[%s]", tag)

	else
		-- TODO Literal '[' and ']' are allowed in param, as long as they come in balanced pairs.

		-- Check that param is syntactically valid. Technically the
		-- character class should also exclude \p{C} (control and
		-- "other" characters).
		assert(string.match(param, "^[^%[%]]+$"), param)
		-- If param begins and ends with quote characters, they will be
		-- stripped by the parser. So add another pair of quotes to
		-- protect them.
		-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L274
		if string.match(param, "^\".*\"$") then
			param = "\"" .. param .. "\""
		end
		return string.format("[%s=%s]", tag, param)

	end
end

local function closing_tag(tag)
	assert(tag_is_valid(tag), tag)
	return string.format("[/%s]", tag)
end

-- Enclose some text in a BBCode tag. This function pushes the given tag/param
-- onto the given stack and calls get_text with the modified stack. The return
-- value of get_stack is then enclosed in opening and closing tags.
--
-- param is optional, and for that matter tag is optional: if tag is nil, this
-- function just returns the return value of get_text called on the unmodified
-- stack, with no added opening and closing tags. sep is an optional separator
-- to insert between the tags and the text (defaults to the empty string).
--
-- This function uses the result of get_text verbatim. Any necessary escaping,
-- etc. needs to happen inside get_text.
local function enclose(tag, param, stack, get_text, sep)
	if tag == nil then
		assert(param == nil, param)
		return get_text(stack)
	else
		stack = cons({tag = tag, param = param}, stack)
		return pandoc.layout.concat({
			opening_tag(tag, param),
			get_text(stack),
			closing_tag(tag),
		}, sep)
	end
end

local function enclose_blocks(tag, param, blocks, stack, opts)
	return enclose(tag, param, stack, function(stack)
		return render_blocks(blocks, stack, opts)
	end)
end

local function enclose_inlines(tag, param, inlines, stack, opts)
	return enclose(tag, param, stack, function(stack)
		return render_inlines(inlines, stack, opts)
	end)
end

local function enclose_text(tag, param, text, stack)
	return enclose(tag, param, stack, function(stack)
		return escape(text, stack)
	end)
end

local function render_list(param, items, stack, opts)
	return enclose("list", param, stack, function (stack)
		local x = {}
		for _, item in ipairs(items) do

			-- TODO no blank line between list item blocks.

			x[#x + 1] = enclose_blocks("*", nil, item, stack, opts)
		end
		return pandoc.layout.concat(x, pandoc.layout.cr)
	end, pandoc.layout.cr)
	-- TODO should remove blank line after list.
end

function Blocks.BlockQuote(el, stack, opts)
	return enclose_blocks("quote", nil, el.content, stack, opts)
end

function Blocks.BulletList(el, stack, opts)
	return render_list(nil, el.content, stack, opts)
end

function Blocks.CodeBlock(el, stack, opts)
	-- TODO the language tag will be interpreted as a download filename if it includes a '.':
	-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/Node.cs#L298

	-- TODO at least ignore sourceCode numberSource numberLines

	-- The language tag, if present, is one of the members of
	-- el.attr.classes. Using the shortcut syntax,
	--   ``` haskell
	-- the language tag will be the first class. So take the first class to
	-- be the language tag, if present. This will produce the wrong result
	-- in cases where the language is absent or given in another order:
	--   ``` {.numberLines}
	--   ``` {.numberLines .haskell}
	return enclose_text("code", el.attr.classes[1], el.text, stack)
end

-- TODO Blocks.DefinitionList

function Blocks.Div(el, stack, opts)
	return render_blocks(el.content, stack, opts)
end

function Blocks.Figure(el, stack, opts)
	local caption = enclose_blocks("b", nil, el.caption.long, stack, opts)

	return pandoc.layout.concat({
		render_blocks(el.content, stack, opts),

		caption,
	}, pandoc.layout.cr)

end

function Blocks.Header(el, stack, opts)
	-- https://html.spec.whatwg.org/multipage/rendering.html#sections-and-headings
	local size = ({
		[1] = "2em",
		[2] = "1.5em",
		[3] = "1.17em",
		[4] = "1em",
		[5] = "0.83em",
	})[el.level] or "0.67em"
	local bold = pandoc.Inlines({pandoc.Strong(el.content)})
	return enclose_inlines("size", size, bold, stack, opts)
end

function Blocks.HorizontalRule(el, stack, opts)
	return opening_tag("hr")
end

-- TODO Blocks.LineBlock

function Blocks.OrderedList(el, stack, opts)
	assert(el.listAttributes.start == 1, el.listAttributes)
	-- ignore el.listAttributes.style
	-- ignore el.listAttributes.delimiter
	return render_list("1", el.content, stack, opts)
end

function Blocks.Para(el, stack, opts)
	return render_inlines(el.content, stack, opts)
end

function Blocks.Plain(el, stack, opts)
	return render_inlines(el.content, stack, opts)
end

function Blocks.RawBlock(el, stack, opts)
	if el.format == MY_FORMAT then
		return el.text
	else
		pandoc.log.warn(string.format("not rendered: %q", tostring(el)))
	end
end

local function table_cell_align_tag(cell_alignment, col_alignment)
	local alignment
	if cell_alignment ~= "AlignDefault" then
		alignment = cell_alignment
	else
		alignment = col_alignment
	end
	if alignment == "AlignDefault" then
		return nil
	else
		return assert(({
			AlignLeft = "left",
			AlignRight = "right",
			AlignCenter = "center",
		})[alignment], alignment)
	end
end

local function render_table_cell(cell, cell_tag, col_alignment, stack, opts)
	assert(cell.col_span == 1, cell.col_span)
	assert(cell.row_span == 1, cell.row_span)
	local align_tag = table_cell_align_tag(cell.alignment, col_alignment)
	return enclose(cell_tag, nil, stack, function (stack)
		return enclose_blocks(align_tag, nil, cell.contents, stack, opts)
	end)
end

local function render_table_row(row, cell_tag, col_alignments, stack, opts)
	return enclose("tr", nil, stack, function (stack)
		local x = {}
		local i = 1
		for _, cell in ipairs(row.cells) do

			x[#x + 1] = render_table_cell(cell, cell_tag, col_alignments[i], stack, opts)
			i = i + cell.col_span
		end
		return pandoc.layout.concat(x, pandoc.layout.cr)
	end, pandoc.layout.cr)
end

function Blocks.Table(el, stack, opts)

	local caption = enclose_blocks("b", nil, el.caption.long, stack, opts)


	local col_alignments = {}
	for i, colspec in ipairs(el.colspecs) do
		col_alignments[i] = colspec[1]
	end
	local table = enclose("table", nil, stack, function (stack)
		local x = {}





		for _, row in ipairs(el.head.rows) do

			x[#x + 1] = render_table_row(row, "th", col_alignments, stack, opts)
		end
		for _, body in ipairs(el.bodies) do
			-- TODO row_head_columns
			assert(body.row_head_columns == 0, body.row_head_columns)
			for _, row in ipairs(body.head) do

				x[#x + 1] = render_table_row(row, "th", col_alignments, stack, opts)
			end
			for _, row in ipairs(body.body) do

				x[#x + 1] = render_table_row(row, "td", col_alignments, stack, opts)
			end
		end
		for _, row in ipairs(el.foot) do

			x[#x + 1] = render_table_row(row, "td", col_alignments, stack, opts)
		end
		return pandoc.layout.concat(x, pandoc.layout.cr)
	end, pandoc.layout.cr)

	return pandoc.layout.concat({
		caption,
		table,
	}, pandoc.layout.cr)
end

-- TODO Inlines.Cite

function Inlines.Code(el, stack, opts)
	return enclose_text("tt", nil, el.text, stack)
end

function Inlines.Emph(el, stack, opts)
	return enclose_inlines("i", nil, el.content, stack, opts)
end

function Inlines.Image(el, stack, opts)
	-- ignore el.caption
	-- ignore el.title
	-- Include a size param only if w and h are provided and have integer
	-- values. (Or if just w is provided.)
	local w = el.attr.attributes.width  and tonumber(el.attr.attributes.width, 10)
	local h = el.attr.attributes.height and tonumber(el.attr.attributes.height, 10)
	local size
	if w and h then
		size = string.format("%dx%d", w, h)
	elseif w then
		size = string.format("%d", w)
	end
	return enclose_text("img", size, el.src, stack)
end

function Inlines.LineBreak(el, stack, opts)
	-- A literal newline character, not the collapsing pandoc.layout.cr.
	-- Multiple blank lines in the input are retained as multiple blank
	-- lines by the parser.
	return "\n"
end

function Inlines.Link(el, stack, opts)
	-- If the element has the class "uri", this is a bare URL link. Prefer
	-- to output it as `[url]http://example.com[/url]`. But do this only if
	-- el.target and el.content are equal: they may differ due to percent
	-- escaping, for example.
	if el.attr.classes:includes("uri") and el.target == pandoc.utils.stringify(el.content) then
		return enclose_inlines("url", nil, el.content, stack, opts)
	else
		-- We expect Pandoc to have hex-escaped link.target, so it
		-- meets the param syntax check in enclose_inlines. Otherwise
		-- an error will be raised.
		return enclose_inlines("url", el.target, el.content, stack, opts)
	end
end

function Inlines.Math(el, stack, opts)
	-- TODO maybe do something more fancy with math.
	if el.mathtype == "InlineMath" then
		return render_inlines({pandoc.Code(el.text, {class = "latex"})})
	elseif el.mathtype == "DisplayMath" then
		return render_blocks({pandoc.CodeBlock(el.text, {class = "latex"})})
	else
		error(el.mathtype)
	end
end

-- TODO Inlines.Note
-- http://chulsky.com/pandoc/
-- https://github.com/tarleb/djot/blob/d682d018f8c8fd26e7bd40aef64b1bfdca2b8441/djot-writer.lua#L392-L406

function Inlines.Quoted(el, stack, opts)
	local q = assert(({
		SingleQuote = {open = [[']], close = [[']]},
		DoubleQuote = {open = [["]], close = [["]]},
	})[el.quotetype], el.quotetype)

	return pandoc.layout.concat({q.open, render_inlines(el.content, stack, opts), q.close})

end

function Inlines.RawInline(el, stack, opts)
	if el.format == MY_FORMAT then
		return el.text
	else
		pandoc.log.warn(string.format("not rendered: %q", tostring(el)))
	end
end

function Inlines.SmallCaps(el, stack, opts)
	return escape(el.text, stack)
end

function Inlines.SoftBreak(el, stack, opts)
	return " "
end

function Inlines.Space(el, stack, opts)
	-- A literal space character, not the reflowable pandoc.layout.space.
	-- Line breaks in the input are always interpreted as line breaks by
	-- the parser, so retain long lines.
	return " "
end

function Inlines.Span(el, stack, opts)
	return render_inlines(el.content, stack, opts)
end


























































function Inlines.Str(el, stack, opts)

	return escape(el.text, stack)
end

function Inlines.Strikeout(el, stack, opts)






	return enclose_inlines("s", nil, el.content, stack, opts)
end

















































function Inlines.Strong(el, stack, opts)
	return enclose_inlines("b", nil, el.content, stack, opts)
end

function Inlines.Subscript(el, stack, opts)





	return enclose_inlines("sub", nil, el.content, stack, opts)










end








function Inlines.Superscript(el, stack, opts)

















	return enclose_inlines("sup", nil, el.content, stack, opts)








end


function Inlines.Underline(el, stack, opts)




	return enclose_inlines("u", nil, el.content, stack, opts)

end



Writer = pandoc.scaffolding.Writer

function Writer.Pandoc(doc, opts)

	return render_blocks(doc.blocks, nil, opts)
end







|
|
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<

<
<
<
<
<
<
<
|
<
>


|
>
|
|
|
>
|
<
|
>
>


<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
|
|
<
<
>
>
|
|
>
>
>
|
<
<
>
>




|
<
|
>
|
|

<
<


|
<

|
|

<


<
<
|
<
>


>
>
>
>
>
>
>
|
|
|
<
>

<
|
<
<
<
<
<
<
|
<
<
<
|
<
>
|
|
<
<
<
<

<
<
<
<
<
<
<
|
<
<
|
<
|
<
<
|
<
<
<
<
<
<
<
|


|
|
|



|
|
|



|
<
<
<
<
|
|
|
<
|
>

>
|

<
|



|
|


|
|


|













|




|
|


|
<
|
<
|
>
|
<
>


|









|


|
|




|



|


|
|


|
|


|

|






|
|
<
<













|



|
|



|
|
<
|
|
>
|
|

<
|


|
>
|
|
>




|
|
>
>
>
>
>
|
>
|




|
>
|

|
>
|


|
>
|

<
<
|
<
<
<
<




|
|


|
|


|












|


|
|


|


|





|




|



|


|

|









|




>
|
>


|

|





|
|


|
|


|



|


|
|

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

|
>
|


|
>
>
>
>
>
>
|
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

>
>
|
|


|
>
>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
|
>
>
>

>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
>
>
>
>
|

>
|
>
>
>
>
|
>


>
>



>
|

38
39
40
41
42
43
44
45
46







47










48










49







50

51
52
53
54
55
56
57
58
59
60

61
62
63
64
65








66























67




68
69


70
71
72
73
74
75
76
77


78
79
80
81
82
83
84

85
86
87
88
89


90
91
92

93
94
95
96

97
98


99

100
101
102
103
104
105
106
107
108
109
110
111
112

113
114

115






116



117

118
119
120




121







122


123

124


125







126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141




142
143
144

145
146
147
148
149
150

151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186

187

188
189
190

191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238


239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262

263
264
265
266
267
268

269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307


308




309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607

	table  = {nesting = true},
	tr     = {nesting = true},
	td     = {nesting = true},
	th     = {nesting = true},
}

-- We first process the Pandoc AST into a linear sequence of tokens, where a
-- token is one of the yield_* types below: a start tag, an end tag, text to be







-- escaped, raw text to be output without escaping, a carriage return, or a










-- blank line.


















local function yield_start_tag(tag, param)

	coroutine.yield({type = "start_tag", tag = tag, param = param})
end

local function yield_end_tag(tag)
	coroutine.yield({type = "end_tag", tag = tag})
end

local function yield_text(text)
	coroutine.yield({type = "text", text = text})
end


local function yield_raw_text(text)
	coroutine.yield({type = "raw_text", text = text})
end









local function yield_cr()























	coroutine.yield({type = "cr"})




end



local function yield_blankline()
	coroutine.yield({type = "blankline"})
end

-- Tables of tokenization functions for Pandoc node types. This is like the
-- pandoc.scaffolding.Writer machinery, though we do not actually use
-- pandoc.scaffolding.Writer beyond the very top level. tokenize_doc is called
-- on the top-level pandoc.Pandoc, and in turn recursively calls


-- tokenize_blocks and tokenize_inlines, which consult the Blocks and Inlines
-- tables of per-node tokenization functions.

local Blocks = {}
local Inlines = {}

local function tokenize_blocks(blocks, opts)

	for i, el in ipairs(blocks) do
		if i > 1 then yield_blankline() end
		local tokenize = assert(Blocks[el.tag], string.format("missing Blocks[%q]", el.tag))
		tokenize(el, opts)
	end


end

local function tokenize_inlines(inlines, opts)

	for _, el in ipairs(inlines) do
		local tokenize = assert(Inlines[el.tag], string.format("missing Inlines[%q]", el.tag))
		tokenize(el, opts)
	end

end



local function tokenize_doc(doc, opts)

	tokenize_blocks(doc.blocks, opts)
end

-- Put start and end BBCode tags around some other tokens. This function yields
-- a start tag/param, then calls fn with no arguments, then yields an end tag.
--
-- param is optional, and for that matter tag is also optional: if tag is nil,
-- this function just calls fn, without adding any start or end tags. sep is an
-- optional function that returns tokens to insert between the tags and the
-- result of calling fn.
local function enclose(tag, param, fn, sep)
	if tag == nil then
		assert(param == nil, param)

		fn()
	else

		yield_start_tag(tag, param)






		if sep then sep() end



		fn()

		if sep then sep() end
		yield_end_tag(tag)
	end




end










local function enclose_blocks(tag, param, blocks, opts)

	return enclose(tag, param, function()


		return tokenize_blocks(blocks, opts)







	end)
end

local function enclose_inlines(tag, param, inlines, opts)
	return enclose(tag, param, function()
		return tokenize_inlines(inlines, opts)
	end)
end

local function enclose_text(tag, param, text)
	return enclose(tag, param, function()
		return yield_text(text)
	end)
end

-- Helper function for pandoc.BulletList and pandoc.OrderedList. param should




-- be either nil (for a bullet list) or "1" (for an ordered list).
local function tokenize_list(param, items, opts)
	return enclose("list", param, function ()

		for i, item in ipairs(items) do
			if i > 1 then yield_cr() end
			-- TODO no blank line between list item blocks.
			-- TODO maybe omit end tag [/*].
			enclose_blocks("*", nil, item, opts)
		end

	end, yield_cr)
	-- TODO should remove blank line after list.
end

function Blocks.BlockQuote(el, opts)
	enclose_blocks("quote", nil, el.content, opts)
end

function Blocks.BulletList(el, opts)
	tokenize_list(nil, el.content, opts)
end

function Blocks.CodeBlock(el, opts)
	-- TODO the language tag will be interpreted as a download filename if it includes a '.':
	-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/Node.cs#L298

	-- TODO at least ignore sourceCode numberSource numberLines

	-- The language tag, if present, is one of the members of
	-- el.attr.classes. Using the shortcut syntax,
	--   ``` haskell
	-- the language tag will be the first class. So take the first class to
	-- be the language tag, if present. This will produce the wrong result
	-- in cases where the language is absent or given in another order:
	--   ``` {.numberLines}
	--   ``` {.numberLines .haskell}
	enclose_text("code", el.attr.classes[1], el.text)
end

-- TODO Blocks.DefinitionList

function Blocks.Div(el, opts)
	tokenize_blocks(el.content, opts)
end

function Blocks.Figure(el, opts)

	-- Figure

	tokenize_blocks(el.content, opts)
	yield_cr()
	-- Caption

	enclose_blocks("b", nil, el.caption.long, opts)
end

function Blocks.Header(el, opts)
	-- https://html.spec.whatwg.org/multipage/rendering.html#sections-and-headings
	local size = ({
		[1] = "2em",
		[2] = "1.5em",
		[3] = "1.17em",
		[4] = "1em",
		[5] = "0.83em",
	})[el.level] or "0.67em"
	local bold = pandoc.Inlines({pandoc.Strong(el.content)})
	enclose_inlines("size", size, bold, opts)
end

function Blocks.HorizontalRule(el, opts)
	yield_start_tag("hr")
end

-- TODO Blocks.LineBlock

function Blocks.OrderedList(el, opts)
	assert(el.listAttributes.start == 1, el.listAttributes)
	-- ignore el.listAttributes.style
	-- ignore el.listAttributes.delimiter
	tokenize_list("1", el.content, opts)
end

function Blocks.Para(el, opts)
	tokenize_inlines(el.content, opts)
end

function Blocks.Plain(el, opts)
	tokenize_inlines(el.content, opts)
end

function Blocks.RawBlock(el, opts)
	if el.format == MY_FORMAT then
		yield_raw_text(el.text)
	else
		pandoc.log.warn(string.format("not rendered: %q", tostring(el)))
	end
end

local function table_cell_align_tag(cell_alignment, col_alignment)
	local alignment = cell_alignment
	if alignment == "AlignDefault" then


		alignment = col_alignment
	end
	if alignment == "AlignDefault" then
		return nil
	else
		return assert(({
			AlignLeft = "left",
			AlignRight = "right",
			AlignCenter = "center",
		})[alignment], alignment)
	end
end

local function tokenize_table_cell(cell, cell_tag, col_alignment, opts)
	assert(cell.col_span == 1, cell.col_span)
	assert(cell.row_span == 1, cell.row_span)
	local align_tag = table_cell_align_tag(cell.alignment, col_alignment)
	enclose(cell_tag, nil, function ()
		enclose_blocks(align_tag, nil, cell.contents, opts)
	end)
end

local function tokenize_table_row(row, cell_tag, col_alignments, opts)
	enclose("tr", nil, function ()

		local col = 1
		for i, cell in ipairs(row.cells) do
			if i > 1 then yield_cr() end
			tokenize_table_cell(cell, cell_tag, col_alignments[col], opts)
			col = col + cell.col_span
		end

	end, yield_cr)
end

function Blocks.Table(el, opts)
	-- Caption
	enclose_blocks("b", nil, el.caption.long, opts)
	yield_cr()
	-- Table
	local col_alignments = {}
	for i, colspec in ipairs(el.colspecs) do
		col_alignments[i] = colspec[1]
	end
	-- Helper function to yield a cr before every row but the first.
	local first = true
	local function yield_sep()
		if not first then yield_cr() end
		first = false
	end
	enclose("table", nil, function ()
		for i, row in ipairs(el.head.rows) do
			yield_sep()
			tokenize_table_row(row, "th", col_alignments, opts)
		end
		for _, body in ipairs(el.bodies) do
			-- TODO row_head_columns
			assert(body.row_head_columns == 0, body.row_head_columns)
			for i, row in ipairs(body.head) do
				yield_sep()
				tokenize_table_row(row, "th", col_alignments, opts)
			end
			for i, row in ipairs(body.body) do
				yield_sep()
				tokenize_table_row(row, "td", col_alignments, opts)
			end
		end
		for i, row in ipairs(el.foot) do
			yield_sep()
			tokenize_table_row(row, "td", col_alignments, opts)
		end


	end, yield_cr)




end

-- TODO Inlines.Cite

function Inlines.Code(el, opts)
	enclose_text("tt", nil, el.text, opts)
end

function Inlines.Emph(el, opts)
	enclose_inlines("i", nil, el.content, opts)
end

function Inlines.Image(el, opts)
	-- ignore el.caption
	-- ignore el.title
	-- Include a size param only if w and h are provided and have integer
	-- values. (Or if just w is provided.)
	local w = el.attr.attributes.width  and tonumber(el.attr.attributes.width, 10)
	local h = el.attr.attributes.height and tonumber(el.attr.attributes.height, 10)
	local size
	if w and h then
		size = string.format("%dx%d", w, h)
	elseif w then
		size = string.format("%d", w)
	end
	enclose_text("img", size, el.src, opts)
end

function Inlines.LineBreak(el, opts)
	-- Always a literal newline character, not a collapsing "cr" token.
	-- Multiple blank lines in the input are retained as multiple blank
	-- lines by the parser.
	yield_text("\n")
end

function Inlines.Link(el, opts)
	-- If the element has the class "uri", this is a bare URL link. Prefer
	-- to output it as `[url]http://example.com[/url]`. But do this only if
	-- el.target and el.content are equal: they may differ due to percent
	-- escaping, for example.
	if el.attr.classes:includes("uri") and el.target == pandoc.utils.stringify(el.content) then
		enclose_inlines("url", nil, el.content, opts)
	else
		-- We expect Pandoc to have hex-escaped link.target, so it
		-- meets the param syntax check in enclose_inlines. Otherwise
		-- an error will be raised.
		enclose_inlines("url", el.target, el.content, opts)
	end
end

function Inlines.Math(el, opts)
	-- TODO maybe do something more fancy with math.
	if el.mathtype == "InlineMath" then
		tokenize_inlines({pandoc.Code(el.text, {class = "latex"})})
	elseif el.mathtype == "DisplayMath" then
		tokenize_blocks({pandoc.CodeBlock(el.text, {class = "latex"})})
	else
		error(el.mathtype)
	end
end

-- TODO Inlines.Note
-- http://chulsky.com/pandoc/
-- https://github.com/tarleb/djot/blob/d682d018f8c8fd26e7bd40aef64b1bfdca2b8441/djot-writer.lua#L392-L406

function Inlines.Quoted(el, opts)
	local q = assert(({
		SingleQuote = {open = [[']], close = [[']]},
		DoubleQuote = {open = [["]], close = [["]]},
	})[el.quotetype], el.quotetype)
	yield_text(q.open)
	tokenize_inlines(el.content, opts)
	yield_text(q.close)
end

function Inlines.RawInline(el, opts)
	if el.format == MY_FORMAT then
		yield_raw_text(el.text)
	else
		pandoc.log.warn(string.format("not rendered: %q", tostring(el)))
	end
end

function Inlines.SmallCaps(el, opts)
	yield_text(el.text)
end

function Inlines.SoftBreak(el, opts)
	yield_text(" ")
end

function Inlines.Space(el, opts)
	-- A literal space character, not the reflowable pandoc.layout.space.
	-- Line breaks in the input are always interpreted as line breaks by
	-- the parser, so retain long lines.
	yield_text(" ")
end

function Inlines.Span(el, opts)
	tokenize_inlines(el.content, opts)
end

function Inlines.Str(el, opts)
	yield_text(el.text)
end

function Inlines.Strikeout(el, opts)
	enclose_inlines("s", nil, el.content, opts)
end

function Inlines.Strong(el, opts)
	enclose_inlines("b", nil, el.content, opts)
end

function Inlines.Subscript(el, opts)
	enclose_inlines("sub", nil, el.content, opts)
end

function Inlines.Superscript(el, opts)
	enclose_inlines("sup", nil, el.content, opts)
end

function Inlines.Underline(el, opts)
	enclose_inlines("u", nil, el.content, opts)
end

-- We maintain a stack to keep track of what BBCode tags are open as we write
-- the output. The context of open tags affects how the BBCode is parsed, and
-- therefore affects how we must do escaping in what we output. The two
-- important considerations are:
--
-- 1. Whether the immediately enclosing element permits nested child elements.
--    If nested child elements are permitted, then the parser will look for and
--    interpret start tags and URLs in the input, and we must escape them to
--    prevent such interpretation. If nested child elements are not permitted,
--    then the parser will copy anything that looks like a start tag or URL
--    verbatim to the output, and we must *not* escape it.
--    https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L270
-- 2. What tags are open all the way up the stack. This affects the parser's
--    interpretation of end tags in the input. If the parser finds something
--    that looks like an end tag, it looks up the stack to see if there is a
--    matching start tag. If there is, it closes that open tag and all
--    intervening tags on the stack:
--    https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L316
--    If there is not, it copies the text that looks like an end tag verbatim
--    to the output:
--    https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L342
--
-- Whenever we output text that contains something that looks like a start tag,
-- end tag, or URL, we escape it with [noparse] if the current context allows
-- nesting. If the current context does not allow nesting, no escaping is
-- possible, but we may still output the text as long as there is no danger of
-- its being interpreted by the parser. A start tag or URL may be output
-- verbatim, as the parser will not interpret it in no-nesting contexts. An end
-- tag is similarly safe to output verbatim, as long as it doesn't match
-- anything up the stack. The only case that is impossible to handle is an end
-- tag in a non-nesting context that matches a tag somewhere up the stack: such
-- an input is impossible to represent and we raise an error.

local function assert_not_nil(x, msg)
	assert(x ~= nil, msg)
	return x
end

local function nesting_allowed(stack)
	local head = stack[#stack]
	if head then
		local tag = assert_not_nil(TAGS[head.tag], string.format("%q missing from TAGS", head.tag))
		return assert_not_nil(tag.nesting) or (head.param ~= nil and tag.nesting_with_param)
	else
		-- Empty stack means top level, tags are allowed.
		return true
	end
end

-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L444
-- Technically the character class should also exclude \p{C} (control and
-- "other" characters).
local CLOSING_TAG_PATTERN = "%[/([^%[%]=/]+)%]"
-- Strings to escape with [noparse]. The "[" is for BBCode tags, the "://" is
-- for URLs.
local ESCAPE_RE = re.compile([[ "["+ / "://" ]])

local function escape(text, stack)
	if nesting_allowed(stack) then
		-- Nested child elements are allowed here, so we may use
		-- noparse escaping. Enclose all "[" and "://" (for URLs) in
		-- [noparse][/noparse] to avoid their being interpreted as a
		-- BBCode start/end tag or a URL.
		--
		-- Escaping just the "[" of a BBCode tag, like so:
		--   [noparse][[/noparse]/center]
		-- is more robust than escaping the whole tag:
		--   [noparse][/center][/noparse]
		-- because the parser may interpret end tags even inside
		-- [noparse]: https://github.com/TASVideos/tasvideos/issues/2247.
		-- Doing it this way also means we can escape the strings
		-- "[noparse]" and "[/noparse]" themselves. Finally, escaping
		-- all "[" is easier than trying to pattern-match tags exactly.
		return re.gsub(text, ESCAPE_RE, "[noparse]%0[/noparse]")
	else
		-- Nested children are not allowed at this point, which means
		-- we cannot use noparse escaping. We do not need to escape
		-- start tags, and end tags are likewise safe as long as they
		-- do not match a tag up the stack. Raise an error if an end
		-- tag matches something in the stack and therefore would be
		-- misinterpreted by the parser.
		local live = {}
		for _, elem in ipairs(stack) do
			live[elem.tag] = true
		end
		for tag in string.gmatch(text, CLOSING_TAG_PATTERN) do
			if live[tag] then
				error(string.format("cannot escape [/%s] in %q", tag, text))
			end
		end
		return text
	end
end

-- Check that a tag name is syntactically valid.
-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L426
local function tag_is_valid(tag)
	return string.match(tag, "^[^%[%]=/]+$")
end

local function start_tag(tag, param)
	assert(tag_is_valid(tag), tag)
	if param == nil then
		return "[" .. tag .. "]"
	else
		-- TODO Literal '[' and ']' are allowed in param, as long as they come in balanced pairs.

		-- Check that param is syntactically valid. Technically the
		-- character class should also exclude \p{C} (control and
		-- "other" characters).
		assert(string.match(param, "^[^%[%]]+$"), param)
		-- If param begins and ends with quote characters, they will
		-- be stripped by the parser. So add another pair of quotes to
		-- protect them.
		-- https://github.com/TASVideos/tasvideos/blob/de99c4233e3457c3abb526b5355eb5cea85c90e5/TASVideos.ForumEngine/BbParser.cs#L274
		if string.match(param, "^\".*\"$") then
			param = "\"" .. param .. "\""
		end
		return "[" .. tag .. "=" .. param .. "]"
	end
end

local function end_tag(tag)
	assert(tag_is_valid(tag), tag)
	return "[/" .. tag .. "]"
end

local function render_token(token, stack)
	if token.type == "start_tag" then
		return start_tag(token.tag, token.param)
	elseif token.type == "end_tag" then
		return end_tag(token.tag)
	elseif token.type == "text" then
		return escape(token.text, stack)
	elseif token.type == "raw_text" then
		return token.text
	elseif token.type == "cr" then
		return pandoc.layout.cr
	elseif token.type == "blankline" then
		return pandoc.layout.blankline
	else
		error(token.type)
	end
end

local function render_tokens(tokens)
	local parts = {}
	local stack = {}

	for token in tokens do
		if token.type == "end_tag" then
			local tag = assert(table.remove(stack), string.format("empty stack for %q", token.tag))
			assert(token.tag == tag.tag, string.format("popping %q, found %q", token.tag, tag.tag))
		end

		table.insert(parts, render_token(token, stack))

		if token.type == "start_tag" then
			table.insert(stack, {tag = token.tag, param = token.param})
		end
	end

	return pandoc.layout.concat(parts)
end

-- We use pandoc.scaffolding.Writer for its template and metadata services.
-- Otherwise we override Writer.Pandoc to do all our own processing of the AST.
Writer = pandoc.scaffolding.Writer

function Writer.Pandoc(doc, opts)
	local tokens = coroutine.wrap(function () tokenize_doc(doc, opts) end)
	return render_tokens(tokens)
end