SgmlReader

SgmlReader is an XmlReader API over any SGML document (including built in support for HTML).  A command line utility is also provided which outputs the well formed XML result.

Download the zip file including the standalone executable and the full source code: SgmlReader.zip

See online demo at demo.aspx.
See also online source.

Command Line Usage

The command line executable version has the following options:

 sgmlreader <options> [InputUri] [OutputFile]
-e "file" Specifies a file to write error output to.  The default is to generate no errors.  The special name "$stderr" redirects errors to stderr output stream.
-proxy "server" Specifies the proxy server to use to fetch DTD's through the fire wall.
-html Specifies that the input is HTML.
-dtd "uri" Specifies some other SGML DTD.
-base

Add an HTML base tag to the output.

-pretty Pretty print the output.
-encoding name Specify an encoding for the output file (default UTF-8)
-noxml Stops generation of XML declaration in output.
-doctype Copy <!DOCTYPE tag to the output.
InputUri The input file name or URL. Default is stdin.  If this is a local file name then it also supports wildcards.
OutputFile The optional output file name. Default is stdout.  If the InputUri contains wildcards then this just specifies the output file extension, the default being ".xml".

Examples

sgmlreader -html *.htm *.xml
Converts all .htm files to corresponding .xml files using the built in HTML DTD.
sgmlreader -html http://www.msn.com -proxy myproxy:80 msn.xml
Converts all the MSN home page to XML storing the result in the local file "msn.xml".
sgmlreader -dtd ofx160.dtd test.ofx ofx.xml
Converts the given OFX file to XML using the SGML DTD "ofx160.dtd" specified in the test.ofx file.
 

SgmlReader Usage

The SgmlReader is an implementation of the XmlReader API so the only thing you really need to know is how to construct it. SgmlReader has a default constructor, then you need to set some of the following properties. To load a DTD you must specify DocType="HTML" or you must provide a SystemLiteral. To specify the SGML document you must provide either the InputStream or Href. Everything else is optional.

SgmlDtd Dtd
Specify the SgmlDtd object directly. This allows you to cache the Dtd and share it across multipl SgmlReaders. To load a DTD from a URL use the SystemLiteral property.
string DocType
The name of root element specified in the DOCTYPE tag. If you specify "HTML" then the SgmlReader will use the built-in HTML DTD. In this case you do not need to specify the SystemLiteral property.
string PublicIdentifier
The PUBLIC identifier in the DOCTYPE tag. This is optional.
string SystemLiteral
The SYSTEM literal in the DOCTYPE tag identifying the location of the DTD.
string InternalSubset
The DTD internal subset in the DOCTYPE tag. This is optional.
TextReader InputStream
The input stream containing SGML data to parse. You must specify this property or the Href property before calling Read().
string Href
Specify the location of the input SGML document as a URL.
string WebProxy
Sometimes you need to specify a proxy server in order to load data via HTTP from outside the firewall. For example: "itgproxy:80".
string BaseUri
The base Uri is used to resolve relative Uri's like the SystemLiteral and Href properties.
TextWriter ErrorLog
DTD validation errors are written to this stream.
string ErrorLogFile
DTD validation errors are written to this log file.

Then you can read from this reader like any other XmlReader class.

 

Features

SGML CDATA to XML <![CDATA[...]]> conversion

SGML DTD's describe a special DTD element type named "CDATA".  This is used in HTML for <SCRIPT> for example and the contents of the script block can be any text terminated by </SCRIPT> including script code containing "<" symbol and so forth, but this would not be well formed in an XML document so the contents of the script block are automatically converted to an XML CDATA block.

Support

Please email bugs, feedback and/or feature requests to Chris Lovett.

Change History

Version Description
1.7 Fix lots of reported bugs:
  1. Fix bug reported by chriswang - MoveToAttribute didn't save state properly.
  2. Fix bug reported by starascendent - build on Visual Studio 2003 was broken.
  3. Fix bug reported by sanchen - ExpandCharEntity was messed up on hex entities.
  4. Fix bug reported by kojiishi - off by one bug in SniffName()
  5. Fix bug reported by kojiishi - bug in loading XmlDocument from SgmlReader - this was caused by the HTML documernt containing an embedded <?xml version='1.0'?> declaration, so the SgmlReader now strips these.
  6. Added special stripping of punctuation characters between attributes like ",".
1.6 Improve wrapping of HTML content with auto-generated <html></html> container tags.
1.5

Fix detection of ContentType=text/html and switch to HTML mode.
Fix problems parsing DOCTYPE tag when case folding is on. 
Fix reading of XHTML DTD.
Fix parsing of content of type CDATA that resulted in the error message 'Cannot have ']]>' inside an XML CDATA block'.
Fix parsing of http://www.virtuelvis.com/download/162/evilml.html.
Fix parsing of attributes missing the equals sign: height"4"  (thanks to Ulrich Schwanitz for his fix).
Fix 'SniffWhitespace' thanks to "Windy Winter".
Added TestSuite project.

1.4 Added UserAgent string "Mozilla/4.0 (compatible;);" so that SgmlReader gets the right content from webservers.  Fixed handling of HTML that does not start with root <html> element tag. Fixed handling of built in HTML entities.
1.3

Changed ToUpper to CaseFolding enum and added support for "auto-folding" based on input.
Added support for <![CDATA[...]]> blocks.
Added proper encoding support, including support for HTML <META http-equiv="content-type".  This means output now has the correct XML declaration (unless you specify the new -noxml option) and any existing xml declarations in the input are stipped out so you don't end up with two.
Added support for ASP <%...%> blocks (thanks to Dan Whalin).
Now strips out DOCTYPE by default since HTML DocTypes can cause problems for XmlDocument when it tries to load the HTML DTD.  but added "-doctype" switch for those who really need it to come through.
Fix handling of Office 2000 <?xml:namespace .../> declarations.
Remove bogus attributes that have no name, in cases like <class= "test">.

1.2 Converted back to Visual Studio 7.0 since this is the lowest common denominator.
Added ToUpper switch for upper case folding, instead of the default lower case.
Fix handling of UNC paths.
Added OFX test suite.
Fixed bug in parsing CDATA type elements (like <script><!-- --></script>)
1.1

Upgraded project to Visual Studio 7.1.
Fixed bug in accessing https authenticated sites.
Fixed bug in handling of content that contains nulls.
Improved handling of <!DOCTYPE with PUBLIC and no SYSTEM literal.
Fixed bug in losing attributes when auto-closing tags.
Fixed pretty printing output by adding WhitespaceHandling flag to SgmlReader.

1.0.4 Added -encoding option so you can change the encoding of the output file.
1.0.3.26932 Implemented ReadOuterXml and ReadInnerXml and fix some bugs in dealing with xmlns attributes and dealing with non-HTML tags.
1.0.3 Fixed some CLS compliance problems with using SgmlReader from VB and a null reference exception bug when loading SgmlReader from XmlDocument
1.0.2.21225 Fixed bug in handling of encodings. Now uses the correct encoding returned from the HTTP server
1.0.2.21105 Fixed bug in handling of input that contains blank lines at the top.
1.0.2 Added fix for the way IE & Netscape deal with characters in the range 0x80 through 0x9F in HTML.
1.0.1 Fixed bug in handling of empty elements, like <INPUT>
1.0 Add wildcard support for command line utility.
0.5 Initial