CVS log for validator/httpd/cgi-bin/Attic/checklink.pl

Request diff between arbitrary revisions

The link checker has moved to /perl/modules/W3C/LinkChecker/ and is no longer included in the validator distribution.

The link checker has moved to /perl/modules/W3C/LinkChecker/ and is no longer included with the validator distribution.

2004.

Markup improvements: "fussy"-clean, towards HTML 4.01 Strict and ISO-HTML.

Fix HTML "escaping" of URIs.  Some were escaped twice, some not at all.

Check recursion scope also for redirected documents.
http://lists.w3.org/Archives/Public/www-validator/2003Nov/0074.html

Improve "To do" message wrapping in command line mode.

Fix URI->abs() usage in absolutizing URIs.

Make ip_allowed() not choke on URIs that don't implement host().
http://lists.w3.org/Archives/Public/www-validator/2003Oct/0056.html

Unbreak --masquerade, enhance documentation and error messages.  W3C Bugzilla
#340, Debian bug #187791, thanks to Dan Jacobson and Frederic Schutz for the
report.

Make the configuration file optional.  Thanks to Hugo Haas for the heads up.
Also, load Config::General only if needed.

Fixing the following:
[Thu Sep 11 12:44:53 2003] checklink.pl: Argument "help" isn't numeric in exit at /home/hugo/bin/checklink.pl line 410.

Don't conditionalize the Term::ReadKey requirement, but defer its loading
until actually needed.  system('stty') doesn't work under -T without laundering
$ENV{PATH}, and I don't feel like going there :P

Also, don't load CGI and CGI::Carp in command line mode; defer by using
require() instead of use().

Implement option to disallow checking non-public IP addresses;
disallow by default in CGI mode and allow in command line mode.

New requirement: Net::IP.

Use Text::Wrap for beautifying command line output, some other output tweaks.

Fix redirect loop detection, and add a descriptive error message.

Previously, only loops like L1 -> L2 -> L3 -> L1 were detected; ones like
L1 -> L2 -> L3 -> L2 resulted in infinite recursion.

Fixes <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=188372>, thanks to
Dan Jacobson and Frederic Schutz for reporting this.

Plug myself into the man page's AUTHOR section, per Frederic's request :)

Include POD documentation, courtesy of Frederic Schutz.

Document env vars affecting FTP configuration, and make passive mode the default (#226).

Trust and launder the NNTPSERVER and NEWSHOST environment variables in order to
support news: URIs without explicit hostnames under -T.

Call get_timestamp always without arguments.

-? == --help, be a bit more verbose with the usage of --location and --recursive (Debian bug report #189642).

Merging from validator-0_6_0-branch, at tag validator-0_6_2-beta1, to HEAD.

- Abort check in case of non-understood command line options
  (#203, Debian bug #187792).
- Allow bundling of command line options.
- Bump 2002->2003 in copyright.

Skip 'base' tags with empty hrefs (#138).
Put HTML validity a bit more prominently into beginning of results page.

- Implemented external configuration file stuff (#127).
  We now require Config::General >= 2.06.
- Fixed basic authentication (#122).  If the trusted domain is not specificed,
  we only send the authentication to the same host which requested it.
  Also added a notice about this to the error page/command line message if
  no suitable authentication info was given.
- Changed text/javascript to application/x-javascript, thanks to
  Nicolas Roeser for the report.
  (see <http://lists.w3.org/Archives/Public/www-validator/2003Jan/0158.html>)
- Fixed missing results from the summary when a file:// URI was present
  (also reduces number of error log warnings emitted).
- Command line --help clarifications, reformatting.
- Code cleanups.

Fix markup in "Authorization Required" error document.

- Fix multiple brainos in cookie code.

- Rewrote recursion scope checking, was failing with URIs with no slash after
  the hostname; eg. <http://validator.w3.org> caused checklink to wander to
  other sites (#115).
- Report CVS revision as version number.

- Send Accept: application/xhtml+xml, text/html, */*;q=0.5 (#112)
- Sanitize Accept-Language headers; either send the ones from a browser or *.
  In command line mode, really default to * and send the ones given with
  the command line option. (#110)
- Add messages for codes 303 (#111), 307 and 406 (#55).
- Add "for" attribute for all <label>'s and the corresponding id's so that
  MSIE users can enjoy clicking the labels.
- Some internal cleanup.

- Options can be saved in a cookie (#56).
- Added a JavaScript check against submitting empty URIs.
- Style tweaks, ala Validator.
- Point to new docs location (relative).

- Tweak "form page" layout.
- Don't say "Valid anchors!" if no anchors found.
- Fix HTML output from command line.

Output and style nits.

Removed the debug logic and reworked the guess whether the script is being run
as a CGI or from the command line; the problems this caused outweigh its
usefulness [1].  Oh, and this fixes #26.

[1] <http://nagoya.apache.org/bugzilla/show_bug.cgi?id=13914>

Unbreak the source download link (introduced in my previous commit).
Found by checking the output of checklink with itself.  Yay :)

- Load Getopt::Long only in command line mode.
- Tighter lexical scoping for variables.
- Output version/revision number consistently everywhere.

Use keep-alive if the installed LWP is >= 5.6.

- Made W3C::UserAgent more OO avoiding use of W3C::CheckLink's lexicals.
- Tuned HTTP header sending to make mod_perl 1.99_05 happy.
- Some style tweaks; reorganized use()'s, added missing ones.

Swap $REVISION and $VERSION in order to be more consistent with Perl stuff in
general, make sure that we have at least two digits in the version number.

Whitespace only (reindentation).

Portability update, now actually works on Win2K.

Term::ReadKey is now used for reading the password from the command line if
it is available, falling back to stty if it isn't.

Fix all function prototypes, and remove them from methods.

Security fix: cross-scripting vulnerability found and fixed by
ITO Tsuyoshi <tsuyoshi@is.s.u-tokyo.ac.jp>.

Security fix: cross-scripting vulnerability found and fixed by
ITO Tsuyoshi <tsuyoshi@is.s.u-tokyo.ac.jp>.

Applying checklink-options.patch from Ville SKytta. Thanks Ville!
Use GetOpt::Long for command-line option handling. Add &version/-V.

Applying checklink-cleanup.patch from Ville SKytta. Thanks Ville!
(A bunch of cleanups; police whitespace, namespace, HTML, etc.)

Merging to HEAD from validator-0_6_0 branch.
This will be validator-0_6_0b1c1 (Beta 1, Candidate 1).

Apply "Local Validator" patch from Ville Skytta. Thanks Ville!

s/netloc/host/ to reflect API change.
made it easy to rerun HTTP sessions in the debugger.
covering for that slacker Hugo.

Updated copyright info

Added support for application/xhtml+xml documents

Applied patch from Henk P. Penning <henkp@cs.uu.nl>:
Using Time::HiRes instead of sys/syscall.ph (should be more portable)

See: http://lists.w3.org/Archives/Public/www-validator/2001OctDec/0224.html

Tiny change: escaping less characters in URIs (0-9 and . are no longer escaped)

Fixed a URI escaping problem

Added support for *_proxy environment variables

Specifying the charset (Latin-1)

Apllied patch from Terje Bless <link@tss.no>: http://lists.w3.org/Archives/Public/www-validator/2001AprJun/0496.html

typo

Ooops, left a closing tag...

Fixed an HTML validity problem in the summary only mode:

http://lists.w3.org/Archives/Public/www-validator/2001AprJun/0001.html

Added directions for inaccessible javascript: URIs

Got rid of $_contact_address
Changed the mailto: URIs and instructions to include 'checklink' in the
subject when sending a mail to www-validator

Made the "Summary only" function do the same as
John <rowe@excc.ex.ac.uk>'s quiet function from:
http://lists.w3.org/Archives/Public/www-validator/2001JanMar/0261.html
This is a tiny change.

Added a --depth option based on the patch provided by
John <rowe@excc.ex.ac.uk>:
http://lists.w3.org/Archives/Public/www-validator/2001JanMar/0261.html

Had to tweak it to make the combination of -r and -D work.

Maybe the base element problem is going to go away completely now.
I have added a change by John <rowe@excc.ex.ac.uk> which basically does not
trust HTTP::Response's base function, and it seems to be the way to go.

Fixed typo

Made it clearer that www-validator is a mailing list to avoid confusion

Now suggests to send bug reports and comments to www-validator@w3.org
Stopped suggesting people to send me email when the max number of documents
  checked is reached in recursive mode.

Changed the text before the input box too

Changed title to "W3C Link Checker" only for the form instead of "W3CŪ Link Checker: Revision: 2.68 (c) 1999-2001 W3C" which was too verbose

Committing changes that I made a long time ago...

- Added 410 HTTP error code in the list of errors.
- Simplified a few regular expressions (got rid of some /'s).
- Added support for //... URI's.
- Commented a bit more the code because I was getting lost.
- Simplified already_processed().
- Added a --hide-same-realm option: hide 401's that are in the same
    realm as the document checked.
- Added a --masquerade option to check documents on the local system
    as if they were on the Web (based on a patch by David Carlisle
    <davidc@nag.co.uk>). Example:
    checklink -q -b --masquerade /afs/w3.org/pub/WWW/Math/Group/draft/cr/ http://www.w3.org/TR/2000/CR-MathML2-20001113/ appendixe.html
- 500 errors are now split in two cases: DNS problems (LWP reports
    that as a 500...) and real internal server errors.
- Fixed a footer problem in the online problem.
- Fixed a duplicate id problem in the error code references.

Undid change done in 2.62 about base tag handling.
I'm not really sure why I did this change in the first place, I might have
to fix a bug later...

Replaced require by use and specified that HTML::Parser version 3 at least
is required.

Reverting line list code because it needs tweaking

Got rid of unnecessary verbosity in the line list.
Added a valid links message in the text version.

Now checks <blockquote cite="...">

Fixed a confusion in the use of URI::abs()

Do not consider base's href as a link anymore (was wrong and was causing a loop in recursive mode)

Discovered the existence of HTTP::Headers::authorization_basic

Fixed typo

Added links to the documentation

Added a test in the case when the encoding would be identity

Now checks the Content-Encoding too

Fixed a bug due to the migration to the version 3 API

Clarified the footer

Increased the limit for the online version to 150 documents.

Fixed a bad wording

Fixed a display problem

Clean up. Fixed a tiny bug.

Cosmetic changes

Speeded up parsing.
Fixed display problem in the recursive version.
Made things simpler in check_uri().

Next time, I'll think twice before committing:
- Removed superfluous test
- Using the same strategy for text version

Report fragments as really broken is the HTTP end code is 200

Removed unnecessary test. Added a header for the link results.

Improved presentation for 500's

Improved presentation for redirected broken URI's.
Added note about directory redirects.

Added anchor to the results for better accessibility.
Removed a lot of colors: it's more readable.
Added a heading for anchors report.

Now uses the version 3 of the API of HTML::Parser.
Got rid of the chunk stuff, since the new parser doesn't care about the
  size of the data parsed (good).
Added attributes checked. Still not perfect, but better.
Fixed a display bug in the text version.
Now parses <?xml:stylesheet ...?>. I'm not happy about the way it's done
  though.

Changed language from 'en' to '*'.
Fixed a bug in the online version when the document cannot be processed.
Changed the presentation to make clearer what needs to be done.
Changed the directions to fix the links.

Only use XML extensions when required in the parser

Fixed a bug when accessing a file which does not exist.
Now counts the lines in <? ... ?> parts.
Now reports when an anchor is defined twice on the same line.

Changed the format of the HTML output to make it accessible.
Broken links sorted by priority.
Fixed a display issue for 200 replies.
Removed useless statistics and cleaned up a few things.
Changed the email address to report configuration problems.
Added links to HTML Validator and CSS Validator.
Added optional Accept-Language headers.

Better handling of doctypes.

I knew I would break something changing the data structures: I love it when a plan comes together!
Fixed a table display problem.

Fixed authentication problem.
Added a limit for the number of documents checked with the online version.
Fixed the -h command line option

Major changes:
- can check documents recursively
- fixed a bug in the parsing of the URI in the CGI version
- now checks URI's for DTD's
- now checks id attributes
- doesn't check <a name="..."> for XHTML Basic 1.0

Changed the way results are stored internally
Added an option to hide directory redirects: http://www.w3.org/TR -> http://www.w3.org/TR/
Fixed some stats errors
Fixed some display problems
Separated directory redirects in the report

Fixed nasty bug: when providing authentication information, 401's were not reported.

Fixed markup problem (thanks to Patrick D. F. Ion <ion@ams.org>)

Changed the default for HTML version to "Show redirects"

"use strict;" which had somehow disappeared is back.
It helped me fixing a typo.

Added comments
Now correctly reports broken fragments with redirects

Now handling empty anchors.
Worked on the command line version.
Added link to source code.

Added links to online versions

Added W3C logo

W3C is registered

Encoding HTML correctly

Fixed HTTP answer

Some more explanation

Fixed problem in the summary table

Backward compatibility with Renaud's script

Fixed escaped URI's problem

Fixed line count

Fixed copyright

Added summary and advices.

Now validates

Fixed form