Skip to content

Use HtmlKit tokenizer for HTML5 parsing #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jhaygood86 wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Use HtmlKit tokenizer for HTML5 parsing #243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e73aa47
Use HtmlKit tokenizer for HTML5 parsing
jhaygood86 Jul 14, 2026
8fd853c
Polish HTML5 parsing script sample
jhaygood86 Jul 14, 2026
2445dc6
Remove HtmlParserVerify internals exposure
jhaygood86 Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Source/Demo/Common/HtmlRenderer.Demo.Common.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
<EmbeddedResource Include="Samples\12.HtmlToolTip.htm" />
<EmbeddedResource Include="Samples\13.HtmlRender.htm" />
<EmbeddedResource Include="Samples\14.HtmlContainer.htm" />
<EmbeddedResource Include="Samples\15.HTML5 Parsing.htm" />
<EmbeddedResource Include="Samples\20.About.htm" />
<EmbeddedResource Include="PerfSamples\1.Big table.htm" />
<EmbeddedResource Include="PerfSamples\2.Lots blocks in inline.htm" />
Expand Down
128 changes: 128 additions & 0 deletions Source/Demo/Common/Samples/15.HTML5 Parsing.htm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
<html>
<head>
<title>HTML5 Parsing</title>
<link rel="Stylesheet" href="StyleSheet" />
</head>
<body>
<h1>HTML5 Parsing
</h1>
<blockquote>
<p>
The html is parsed using a real <b>HTML5 tokenizer</b> (from the
<a href="https://github.com/jstedfast/HtmlKit">HtmlKit</a> library) instead of a hand-rolled
scanner, so the pages below - all of which are <i>intentionally malformed</i> - render
correctly instead of breaking the layout. Every example shows the raw (broken) html
source followed by the actual <b>live rendering</b> of that exact same markup.
</p>
<hr />
<h2>Optional / omitted end tags
</h2>
<p>
Real world html is full of tags whose closing tag was never written. The parser now
follows the <a href="https://html.spec.whatwg.org/dev/dom.html#concept-element-tag-omission">
HTML5 end-tag omission rules</a> and automatically closes them at the right
place, instead of nesting everything into one deep, broken tree.
</p>
<div class="comment">Source:</div>
<pre>&lt;p&gt;First paragraph, never closed
&lt;p&gt;Second paragraph, never closed
&lt;p&gt;Third paragraph, never closed</pre>
<div class="comment">Live result:</div>
<div class="example">
<p>First paragraph, never closed
<p>Second paragraph, never closed
<p>Third paragraph, never closed
</div>
<hr />
<h2>Tables with missing &lt;tr&gt;/&lt;td&gt; end tags
</h2>
<p>
The same omission rules apply to table rows and cells, so a table written without a
single closing <code>&lt;/tr&gt;</code> or <code>&lt;/td&gt;</code> still comes out as
a proper grid.
</p>
<div class="comment">Source:</div>
<pre>&lt;table border="1" cellpadding="4"&gt;
&lt;tr&gt;&lt;td&gt;A&lt;td&gt;B
&lt;tr&gt;&lt;td&gt;C&lt;td&gt;D
&lt;/table&gt;</pre>
<div class="comment">Live result:</div>
<div class="example">
<table border="1" cellpadding="4">
<tr><td>A<td>B
<tr><td>C<td>D
</table>
</div>
<hr />
<h2>Anonymous table boxes
</h2>
<p>
Per the <a href="https://www.w3.org/TR/CSS2/tables.html#anonymous-boxes">CSS 2.1 anonymous
table object rules</a>, a stray <code>&lt;td&gt;</code> with no <code>&lt;tr&gt;</code>
(or even no <code>&lt;table&gt;</code>) around it is automatically wrapped in the missing
row/table boxes, rather than producing a broken or invisible layout.
</p>
<div class="comment">Source:</div>
<pre>&lt;table border="1"&gt;&lt;td&gt;Anonymous row generated around me&lt;/td&gt;&lt;/table&gt;</pre>
<div class="comment">Live result:</div>
<div class="example">
<table border="1"><td>Anonymous row generated around me</td></table>
</div>
<hr />
<h2>Comments containing "&gt;"
</h2>
<p>
A real tokenizer knows where a comment actually ends, so a stray <code>&gt;</code> character
inside a comment no longer truncates it early or corrupts the rest of the page.
</p>
<div class="comment">Source:</div>
<pre>&lt;!-- a comment that contains a &gt; character right here --&gt;
&lt;p&gt;This paragraph renders normally right after it.&lt;/p&gt;</pre>
<div class="comment">Live result:</div>
<div class="example">
<!-- a comment that contains a > character right here -->
<p>This paragraph renders normally right after it.</p>
</div>
<hr />
<h2>&lt;script&gt; content is not mistaken for markup
</h2>
<p>
<code>&lt;script&gt;</code> content is tokenized as raw text (this renderer does not execute
script, so the code itself is not shown), which means <code>&lt;</code> and <code>&gt;</code>
characters used for comparisons inside the script can no longer be misread as the start
of a nested tag and swallow the rest of the document.
</p>
<div class="comment">Source:</div>
<pre>&lt;script&gt;
if (1 &lt; 2 &amp;&amp; 3 &gt; 2) {
runDemo('&lt;b&gt;looks like a tag&lt;/b&gt;, but is just a string');
}
&lt;/script&gt;
&lt;p&gt;This paragraph still renders correctly right after the script block.&lt;/p&gt;</pre>
<div class="comment">Live result:</div>
<div class="example">
<script>
if (1 < 2 && 3 > 2) {
runDemo('<b>looks like a tag</b>, but is just a string');
}
</script>
<p>This paragraph still renders correctly right after the script block.</p>
</div>
<hr />
<h2>&lt;noscript&gt; content is parsed as real markup
</h2>
<p>
Per the HTML5 tokenizer spec, <code>&lt;noscript&gt;</code> content is normally tokenized
as raw text too. Since this renderer never runs script, its <code>&lt;noscript&gt;</code>
content is specifically re-parsed as nested html, so tags placed inside it render as real
elements instead of showing up as literal text.
</p>
<div class="comment">Source:</div>
<pre>&lt;noscript&gt;&lt;p&gt;This is &lt;b&gt;real, nested markup&lt;/b&gt; inside noscript.&lt;/p&gt;&lt;/noscript&gt;</pre>
<div class="comment">Live result:</div>
<div class="example">
<noscript><p>This is <b>real, nested markup</b> inside noscript.</p></noscript>
</div>
</blockquote>
</body>
</html>
26 changes: 17 additions & 9 deletions Source/HtmlRenderer/Core/Dom/CssBox.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ internal class CssBox : CssBoxProperties, IDisposable
/// <summary>
/// the inner text of the box
/// </summary>
private SubString _text;
private string _text;

/// <summary>
/// Do not use or alter this flag
Expand Down Expand Up @@ -146,6 +146,14 @@ public bool IsBrElement
}
}

/// <summary>
/// Is the box "Display" is one of the table-row-group/table-header-group/table-footer-group values.
/// </summary>
public bool IsTableRowGroupBox
{
get { return Display == CssConstants.TableRowGroup || Display == CssConstants.TableHeaderGroup || Display == CssConstants.TableFooterGroup; }
}

/// <summary>
/// is the box "Display" is "Inline", is this is an inline box and not block.
/// </summary>
Expand Down Expand Up @@ -278,7 +286,7 @@ public bool IsSpaceOrEmpty
/// <summary>
/// Gets or sets the inner text of the box
/// </summary>
public SubString Text
public string Text
{
get { return _text; }
set
Expand Down Expand Up @@ -541,7 +549,7 @@ public void ParseToWords()

int startIdx = 0;
bool preserveSpaces = WhiteSpace == CssConstants.Pre || WhiteSpace == CssConstants.PreWrap;
bool respoctNewline = preserveSpaces || WhiteSpace == CssConstants.PreLine;
bool respoctNewline = preserveSpaces || WhiteSpace == CssConstants.PreLine || IsBrElement;
while (startIdx < _text.Length)
{
while (startIdx < _text.Length && _text[startIdx] == '\r')
Expand Down Expand Up @@ -795,27 +803,27 @@ private void CreateListItemBox(RGraphics g)

if (ListStyleType.Equals(CssConstants.Disc, StringComparison.InvariantCultureIgnoreCase))
{
_listItemBox.Text = new SubString("•");
_listItemBox.Text = "•";
}
else if (ListStyleType.Equals(CssConstants.Circle, StringComparison.InvariantCultureIgnoreCase))
{
_listItemBox.Text = new SubString("o");
_listItemBox.Text = "o";
}
else if (ListStyleType.Equals(CssConstants.Square, StringComparison.InvariantCultureIgnoreCase))
{
_listItemBox.Text = new SubString("♠");
_listItemBox.Text = "♠";
}
else if (ListStyleType.Equals(CssConstants.Decimal, StringComparison.InvariantCultureIgnoreCase))
{
_listItemBox.Text = new SubString(GetIndexForList().ToString(CultureInfo.InvariantCulture) + ".");
_listItemBox.Text = GetIndexForList().ToString(CultureInfo.InvariantCulture) + ".";
}
else if (ListStyleType.Equals(CssConstants.DecimalLeadingZero, StringComparison.InvariantCultureIgnoreCase))
{
_listItemBox.Text = new SubString(GetIndexForList().ToString("00", CultureInfo.InvariantCulture) + ".");
_listItemBox.Text = GetIndexForList().ToString("00", CultureInfo.InvariantCulture) + ".";
}
else
{
_listItemBox.Text = new SubString(CommonUtils.ConvertToAlphaNumber(GetIndexForList(), ListStyleType) + ".");
_listItemBox.Text = CommonUtils.ConvertToAlphaNumber(GetIndexForList(), ListStyleType) + ".";
}

_listItemBox.ParseToWords();
Expand Down
2 changes: 1 addition & 1 deletion Source/HtmlRenderer/Core/Dom/CssLayoutEngine.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ private static void FlowBox(RGraphics g, CssBox blockbox, CssBox box, double lim
}

// handle box that is only a whitespace
if (box.Text != null && box.Text.IsWhitespace() && !box.IsImage && box.IsInline && box.Boxes.Count == 0 && box.Words.Count == 0)
if (box.Text != null && CommonUtils.IsNonEmptyWhitespace(box.Text) && !box.IsImage && box.IsInline && box.Boxes.Count == 0 && box.Words.Count == 0)
{
curx += box.ActualWordSpacing;
}
Expand Down
Loading