CVS log for validator/httpd/cgi-bin/check

Request diff between arbitrary revisions

configuring the default doctype for HTML and XHTML modes. Should be in config file?

* if the root element has a namespace, and even neither mime type,
  doctype nor xml declaration have helped detect the proper parse mode,
  switch to XML checking mode.

* If the document has no doctype but we are in XML mode and a root namespace
  was found, filter out the "no doctype" errors.

subroutine namespace disambiguation, by setting all the main routines
under the W3C::Validator::MarkupValidator namespace.

 ----------------------------------------------------------------------

Trying to solve the headache of the default fallback character encoding,
when different specs suggest different encodings, and we try to promote utf-8
as the best practice. This patch tried to go around the issue by trying, in sequence,
a fallback of utf8, win-1252, then iso-8859-1 - in sequence.

* Renaming the W3C::Validator::SAXHandler (misleading name) module
as what it is, some hybrid event handler.

* Giving W3C::Validator::EventHandler a proper namespace and packacge
structure.

* subclassing W3C::Validator::EventHandler with extra handling of char/data
for the creation of the document outline => W3C::Validator::EventHandler::Outliner

We could move the various packages out of the check script and into their own cpan'd
modules, I guess...

adding accept-charset option

additional fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=4848
- we were checking whether the SI matched a recognized FPI
- we are now also checking, in case the FPI is not recognized, whether it matches a known SI.

fixed bug for display of FPI in W26

Rewording of the warning title and text for doctype FPI/SI mismatch, as suggested by Terje.

check that in a DOCTYPE declaration for a known document type, FPI and SI match

Make sure (looks like a regression) that we do not perform override of a doctype upon itself
See e.g:
http://lists.w3.org/Archives/Public/www-validator/2007Nov/0038.html

UTF-8-encode "authorization failed" template output (Chuck Houpt)

Include some warning info from LWP in fatal HTTP error messages.

IIS workaround, thanks to David Tibbe and Allan Smith.

patch by Brett Bieber for Bug 5154
http://lists.w3.org/Archives/Public/www-validator/2007Oct/0027.html

Fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=3663

Checking if doctype-less documents display patterns specific to SVG
(svg root element, with or without version/baseProfile attributes)
and, upon detection, applying "fake" doctype override to help validation
of doctype-less SVG 1.0, 1.1 (including Tiny and Basic).

adding experimental parameters to set the Accept and Accept-Language headers

all our templates are utf-8

giving spopt options twice, working around the usual bug with sgml::parser::opensp 0.99

in sgml mode, reporting shorttag usage as a warning. TODO: add explanations for the warning messages

* Cleanup of the template variables used to determine whether upload or direct input has been used.
This complicates the template structure a little, since HTML::Template does not have <TMPL_ELSIF>, but in the long run it is cleaner than using is_upload
for direct input AND upload
* using a filtering routine to work around a HTML::Template bug with utf-8 encoded template files, see
http://lists.w3.org/Archives/Public/public-qa-dev/2007Sep/0006.html

Tighten matching of content types that we treat as special in parse_content_type()

Making doctype detection work with document types using dashes or dots
in root element (allowed in XML, at least).
This fixes Bug 5031: http://www.w3.org/Bugs/Public/show_bug.cgi?id=5031

Now that we have a decent XML processor, we can accept documents sent with a
media type which is unknown, if the media type follows the convention
"application/*+xml" (as a subset of the convention defined in RFC 3023 sec 7)
we'll treat it as incoming XML.

This should also fix http://www.w3.org/Bugs/Public/show_bug.cgi?id=4926

for resources served as application/atom+xml, passing to feed validator

sending cache-control header in requests, telling in-the-middle
caches we want as fresh a version as possible:
This fixes http://www.w3.org/Bugs/Public/show_bug.cgi?id=4998

applying karl's idea that the doctype declaration
we sneak in should not be at the beginning of the doc,
but rather, just before the root element.

This should fix http://www.w3.org/Bugs/Public/show_bug.cgi?id=857

for the sake of a lighter memory footprint, making sure that the whole source of the document
is only passed to the template object if necessary (that is, at this point, if the show
source option is on, or if the file was uploaded or direct input was used).

patch by Brett Bieber for Bug/Enhancement 4342
http://www.w3.org/Bugs/Public/show_bug.cgi?id=4342

including the content for revalidation even when using upload or direct input.

There are two warnings for "unknown parse mode". Obsoleting W08
and merging with W06, as per discussion around:
http://lists.w3.org/Archives/Public/www-validator/2007Jul/0168.html

Comment out the "OpenSP reports errors beyond EOL" workaround altogether for now.

* fixing a bug that would run Tidy on content even when not requested (that's a DUH award for me...)
* silencing some debug output

silencing HTTP / XML decl charset mismatch warning in case of direct input mode

fix for borken language-negotiation cases:
http://www.w3.org/Bugs/Public/show_bug.cgi?id=4899

not clogging our logs with a warning about a libxml bug we can't control

* muting the perl -w warnings
"check: \\1 better written as $1 at /usr/local/validator/httpd/cgi-bin/check line 642"

* undef-ing content copy after use

More robust parsing of the error output from XML::LibXML,
taking into account the three lines structure of the error

:12:  parser error : Error Message
   ... here markup quoted ...
           ^

(first the error message,
then a second line with quoted content which we ignore,
then the pointer for the column number)

This should fix http://www.w3.org/Bugs/Public/show_bug.cgi?id=4892

Back down to unversioned XML::LibXML, load_catalog no longer used.

workaround for borked /check/referer with mod_perl2

change of strategy for the use of XML::LibXML
apparently, even loading a local catalog does not stop it from
fetching tons of schema/entities stuff (even in non-validating mode)

Using load_ext_dtd(0) solves the issue, but the parser would complain about unknown entities
=> filtering that at the post-parsing level.

XML::LibXML >= 1.53 is needed for load_catalog().

Use top-level "use" for pulling in non-optional modules (XML::LibXML,
HTTP::Negotiate) instead of on-demand "require" so they need to be compiled
only once with mod_perl.  While at it, sort list of imported modules (except
HTTP::Headers::Auth which still needs to be imported after other HTTP::*).

workaround for people upgrading without updating their config files => we give a default for the language variable

It looks like the flattenecd DTDs for e.g SVG 1.1 are not going to be fixed just yet
http://lists.w3.org/Archives/Public/www-html-editor/2007AprJun/0010.html
=> filtering out the warning about re-declaration of xml:space

See also:
http://www.w3.org/Bugs/Public/show_bug.cgi?id=301

two warnings were sharing the same identifier - fixing

re-populating the list of charsets (from older revisions in both branch 0.7 and 0.8)
to be used thus:
* no fatal error if the charset is supported by encode
* a warning with the suggestion for a better alias if we know one
* a warning that the encoding may be "odd" if not in the list but encode says it's OK

The XML parser will check the value of encoding attribute in XML declaration
so we have to amend it to reflect transcoding: using a regexp to match the xml declaration
and replace the part with encoding="foo" with encoding="utf-8".

Note: this may mess up column/offset numbering if there are errors on that line.

See also Bug 4867:
http://www.w3.org/Bugs/Public/show_bug.cgi?id=4867

removing the list of supported character encodings,
as we have a better and more reliable technical way to do this.

Replacing with a mechanism to:
* suggest a better alias if a "bad" encoding alias is used
* refuse to transcode if encoding used is forbidden by policy
(I don't know any, but the mechanism is here...)

More details:
http://lists.w3.org/Archives/Public/public-qa-dev/2007Jul/0009.html
and surrounding thread.

for the sake of readability, at least until the xmlwf errors have explanations,
we push the errors from the XML parser at the END of the error list.

fixing a bug with error location highlighting when the beginning of the line
is replaced with the ellipsis. The wrong location (2 chars off) was pointed to.

Bug spotted and patched by Brett Bieber.

fixing xml whitespace detection to use two-digits hex characters, so that perl properly recognises them

requiring extra perl libs for CJK character encodings, and documenting the new requirements

making the     X-W3C-Validator-Status header more consistently report:
* Valid if validation passed
* Invalid if validation proceeded, failed
* Abort if validation could not be performed

Using local images for display, when available.

As noted by Ville, in regexp character classes | is a literal character.
=> fixing the regex for matching encoding value, in xml declaration detection code.

* we were passing some fake debug info to the warning for parse mode fallback
   (should not have made it to CVS, my mistake)
* fixing info about doctype passed to parse mode fallback warning

better matching of xml declarations with line breaks

regexp for xml declaration encoding attribute was bogus...

adding debug info to soap output. Useful for automation of test suite.

Revisiting the parse mode detection algorithm.
In order of priority:
* Media type (if Content-Type HTTP info present)
* Doctype Declaration
* XML Declaration (Which we now detect - wasn't the case before)

We now also run the parse mode detection algorithm even for doctype-less documents.

Another addition: Parse Mode Factor parameter passed to template
showing (in debug mode) which was the deciding factor for the parse mode.

not filtering the 'parsing without validation' error message for doctypeless sgml documents (that aren't html).

adding a case for non-html, non-xml document without a doctype.

giving XML::LibXML the path to our XML catalog, for entities.
See discussion at http://www.w3.org/Bugs/Public/show_bug.cgi?id=4623

make feedback link absolute (especially for soap output)
Patch by Brett Bieber
http://www.w3.org/Bugs/Public/show_bug.cgi?id=4612

re-wording for documents checked with xml-wf only

bringing back line count for fatal_byte_error, output of raw decoding error message

Simplify xmlwf error list population.

The fix for bug 4474 actually broke the transcoding routine for some versions of Encode lib.
http://lists.w3.org/Archives/Public/www-validator-cvs/2007Apr/0159.html

This patch:
* fixes the transcoding routine back to normal
  (hence removing mistaken "non-sgml character" for unicode content)
* makes sure that content we output is properly encoded from perl's internal to utf-8
* adds proper decoding/encoding for tidy processing

small hack to fix the soap fault output.
We should however sooner or later look into a real clean fix,
see: http://www.w3.org/Bugs/Public/show_bug.cgi?id=2131

 we are not sorting errors by line, as it would break the position
 of auxiliary messages such as "start tag was here". We'll have to live with
 the fact that XML well-formedness errors are listed first, then validation errors

for documents served as XML, and for which the root element is not <html>,
the fallback to XHTML is not a great choice. Instead, we can perform a basic xml-wf checking,
and report a positive result for these.

better display of information in case of fatal error (show preparse warning,
e.g http://www.w3.org/Bugs/Public/show_bug.cgi?id=2572 ) and revalidation table.

Better Error Msg for missing xmlns

Escape SPO's croaking on some rare error message formats.
As reported in
http://lists.w3.org/Archives/Public/www-validator/2007Apr/0118.html
and filed in
http://www.w3.org/Bugs/Public/show_bug.cgi?id=4516

Applying patch from http://www.w3.org/Bugs/Public/show_bug.cgi?id=800
based on ideas and code proposed by Shane McCarron.

fixing SOAP12 output - thanks to Struan Donald for the Bug Report

Workaround and diagnostics for http://bugzilla.gnome.org/show_bug.cgi?id=434196

Better diagnostics about OpenSP reporting error columns > line length in case
it's still a problem.

Don't treat any '^' as a column locator when parsing libxml errors.

Eliminate global templates to make mod_perl happier, improve naming.

More warning cleanups.

2007

Warning cleanups.

The error template needs to be constructed before invoking prepCGI().

Cleanups, NFC.

Looks like we were decoding bytes, but forgetting to re-encode in utf8...

See http://www.w3.org/Bugs/Public/show_bug.cgi?id=4474

fixing http://www.w3.org/Bugs/Public/show_bug.cgi?id=4472
(outline for non xml-based languages)
by making heading-selection regexp case insensitive.

Since we have a number of different sources for errors (xml-wf, parsing, ...)
better make sure that they are sorted before output
(+test case)

detection and listing of non-root namespaces had disappeared from 0.8 branch.
re-adding via sax/opensp api parsing

making prefilling of document structure on the server-side rather than via scripting.
See: http://lists.w3.org/Archives/Public/www-validator/2007Mar/0078.html
+thread
and: http://lists.w3.org/Archives/Public/www-validator/2007Apr/0049.html
+thread

Fixing display of fallback doctype when missing

Went through all the RECs of the document types we support, and found which ones require
the xmlns attribute on the root element, and which don't. Making the check script whine
about its absence only if required.

Whitespace/indentation, NFC

Load HTTP::Negotiate and XML::LibXML only when needed.

silent recovery if HTML::Tidy not present or needed

* checking presence and value of xmlns for any root element of XML docs, not just html (e.g smil, etc)
* bug fix for location of error when xmlns value is incorrect

(Ironically, see Bug 4420) making sure our source snippet doesn't go beyond 80 chars.

 working around an apparent odd limitation of libxml
 which only gives context for lines up to 80 chars
 http://www.w3.org/Bugs/Public/show_bug.cgi?id=4420
 http://bugzilla.gnome.org/show_bug.cgi?id=424017

in grouped error mode, give the specific error message for each instance

routine to check presence and value of xmlns attribute, generally necessary for conformance, but not well expressed in DTD

Small selfish edit, the m(^\d+$) construct systematically confused my editor.
Replacing with equivalent (I think) matching construct, which should work
as well and make my life easier.

bringing back the document outline, with SAX.
Using a <pre> and whitespace to show the level of outline is unfortunate,
but is the only sane way of doing it in case of broken markup with unclosed <hx>

during pre-parsing to figure out the doctype, only use preparser in xml mode
if content-type is non-ambiguous. For text/html, assume sgml mode until
and XHTML doctype has been found.
see also http://www.w3.org/Bugs/Public/show_bug.cgi?id=978

small hack for grouping of xml-wf errors  -we really need to extract a catalog of all errors generated by the xml parser

Adding XML well-formedness parsing for XML docs
with ideas from Jacques Distler http://golem.ph.utexas.edu/~distler/blog/archives/001054.html

We could be using only the XML parser and not parse twice, but
at the moment I am tempted to keep using opensp, for as long
as we have the best library of error message and explanations for it.

Code is not really pretty, and should be amended at some point to
use a real SAX ErrorHandler. That said, it very much tolls the bell for the
"Validator XML support has some limitations." message.

rewriting parse mode choice routine, adding comments to code
See http://www.w3.org/Bugs/Public/show_bug.cgi?id=22 for explanations and testing

more info in debug. fixing parse mode fallback for unknown document types

extra output in debug mode

tentative fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=3992
(parser tripping on DOS newlines)

For default auto-detection of charset and doctype, changing the
form so that the values be empty, hence creating validation results URIs like
...check?uri=http%3A%2F%2Fwww.example.org&doctype=&charset=
rather than
...check?uri=http%3A%2F%2Fwww.example.org%2F&charset=%28detect+automatically%29&doctype=Inline

showing blue and "gold" icons upon successful validation

first pass at implementing language negotiation/selection. Can certainly be improved, but it works

Workaround for CGI.pm's _name_and_path_from_env query string issues, #4365

Make /check/path/info behave the same way as /check also when there's no URI available.

Whitespace, appease Perl::Critic a bit more (NFC)

Workaround for CGI.pm's _name_and_path_from_env query string issues, #4365

Adding a check for a mime type / doctype conflict.
Warning wording in progress, including suggestions by Philip "nikitathespider"
and Frank Ellermann.

Should not hardcode tidy conf path, make it configurable in main conf file

Adding a bit of complexity to the code for the sake of localization.
Instead of having messages like "no error" or "1 error" or "2 error"
directly created in the check script, the script will instead pass
parameters to the templates stating whether there is 0, 1, 2 or more
error(s) (and warnings alike).

This should keep translation work away from check - we want it contained
in documentation and templates.

removing diff code pasted by mistake

Adding an option for the grouping of error by error id, keeping the ordering
of messages according to parsing sequence.

* fixing (silly) version requirement issue with S:P:O (for perl, v 0.100 < 0.99)
* adding proof of concept HTM::Tidy output

* workaround for bug 798 does not need s:p:o 0.100 any more, we can put req back to .99
* port from 0.7.4 to HEAD - bringing back outline option (still need filter) and commenting out parse tree etc.

Plugging the (old, unused? so far) option to enable/disable the soap output
to the actual API. Changing default to enabled in config file, maintainers
wishing to disable can now use the directive in configuration file, and
validator would then return 503 service not available to output=soap12 requests.

More default paths, slightly improved message in case configuration files are missing.
courtesy of Ask Bjørn Hanse
See http://www.w3.org/Bugs/Public/show_bug.cgi?id=3334

bringing back the outline, by popular demand...

* fix for latest s:p:o named 0.100, confuses perl thinking it's <= 0.99, which we require...

* fixing the passing of options (including XML mode, see bug #798) to s:p:o, hat tip to Bjoern Hoehrmann

workaround for bug #798 (apparently in s:p:o 0.99)

updating credits and pointer to list of contributors

change in opensp 1.5.2 broke our parsing of the message-id - fixed.

avoid perl subroutine prototype warning

* moving all outputs to templates instead of inline code
* adding ucn and soap12 outputs as experimental additions
  ( will allow us to plug unicorn to main validator, and also let people
  actually develop libraries for these outputs)
* deprecating homegrown xml format (in documentation, needs to be in the changelog and announce)

These involve some heavy porting of code from CVS HEAD to 0.7 branch, needs some testing.

fixing template order and logic to show warning about absence of character encoding
before fatal error when transcoding to utf-8 failed
http://www.w3.org/Bugs/Public/show_bug.cgi?id=3642

(porting to HEAD code)

fixing template order and logic to show warning about absence of character encoding
before fatal error when transcoding to utf-8 failed
http://www.w3.org/Bugs/Public/show_bug.cgi?id=3642

Include more environment variables in debug output.