Cover image for Perl and LWP : [fetching Web pages, parsing HTML, writing spiders & more]
Title:
Perl and LWP : [fetching Web pages, parsing HTML, writing spiders & more]
Author:
Burke, Sean M.
Personal Author:
Edition:
First edition.
Publication Information:
Beijing ; Cambridge [Mass.] : O'Reilly, [2002]

©2002
Physical Description:
xv, 242 pages : illustrations ; 24 cm
General Note:
Subtitle from cover.

Includes index.
Language:
English
ISBN:
9780596001780
Format :
Book

Available:*

Library
Call Number
Material Type
Home Location
Status
Central Library QA76.73.P22 B87 2002 Adult Non-Fiction Central Closed Stacks
Searching...

On Order

Summary

Summary

Perl soared to popularity as a language for creating and managing web content, but with LWP (Library for WWW in Perl), Perl is equally adept at consuming information on the Web. LWP is a suite of modules for fetching and processing web pages.The Web is a vast data source that contains everything from stock prices to movie credits, and with LWP all that data is just a few lines of code away. Anything you do on the Web, whether it's buying or selling, reading or writing, uploading or downloading, news to e-commerce, can be controlled with Perl and LWP. You can automate Web-based purchase orders as easily as you can set up a program to download MP3 files from a web site. Perl & LWP covers:

Understanding LWP and its design Fetching and analyzing URLs Extracting information from HTML using regular expressions and tokens Working with the structure of HTML documents using trees Setting and inspecting HTTP headers and response codes Managing cookies Accessing information that requires authentication Extracting links Cooperating with proxy caches Writing web spiders (also known as robots) in a safe fashion Perl & LWP includes many step-by-step examples that show how to apply the various techniques. Programs to extract information from the web sites of BBC News, Altavista, ABEBooks.com, and the Weather Underground, to name just a few, are explained in detail, so that you understand how and why they work.Perl programmers who want to automate and mine the web can pick up this book and be immediately productive. Written by a contributor to LWP, and with a foreword by one of LWP's creators, Perl & LWP is the authoritative guide to this powerful and popular toolkit.


Author Notes

Sean M. Burke is an active member of the Perl community and one of CPAN's most prolific module authors. Since 1998 he has been a contributor to LWP and a columnist for The Perl Journal


Table of Contents

Forewordp. ix
Prefacep. xi
1. Introduction to Web Automationp. 1
The Web as Data Sourcep. 1
History of LWPp. 3
Installing LWPp. 4
Words of Cautionp. 9
LWP in Actionp. 10
2. Web Basicsp. 15
URLsp. 15
An HTTP Transactionp. 17
LWP::Simplep. 19
Fetching Documents Without LWP::Simplep. 24
Example: AltaVistap. 25
Http Postp. 27
Example: Babelfishp. 28
3. The LWP Class Modelp. 31
The Basic Classesp. 31
Programming with LWP Classesp. 32
Inside the do_GET and do_POST Functionsp. 33
User Agentsp. 34
HTTP::Response Objectsp. 42
LWP Classes: Behind the Scenesp. 47
4. URLsp. 48
Parsing URLsp. 48
Relative URLsp. 54
Converting Absolute URLs to Relativep. 55
Converting Relative URLs to Absolutep. 57
5. Formsp. 58
Elements of an HTML Formp. 59
LWP and GET Requestsp. 59
Automating Form Analysisp. 62
Idiosyncrasies of HTML Formsp. 64
POST Example: License Platesp. 70
POST Example: ABEBooks.comp. 74
File Uploadsp. 81
Limits on Formsp. 84
6. Simple HTML Processing with Regular Expressionsp. 85
Automating Data Extractionp. 85
Regular Expression Techniquesp. 87
Troubleshootingp. 91
When Regular Expressions Aren't Enoughp. 93
Example: Extracting Links from a Bookmark Filep. 93
Example: Extracting Links from Arbitrary HTMLp. 96
Example: Extracting Temperatures from Weather Undergroundp. 98
7. HTML Processing with Tokensp. 100
HTML as Tokensp. 100
Basic HTML::TokeParser Usep. 101
Individual Tokensp. 105
Token Sequencesp. 107
More HTML::TokeParser Methodsp. 112
Using Extracted Textp. 117
8. Tokenizing Walkthroughp. 119
The Problemp. 119
Getting the Datap. 120
Inspecting the HTMLp. 121
First Codep. 122
Narrowing Inp. 123
Rewrite for Featuresp. 125
Alternativesp. 131
9. HTML Processing with Treesp. 132
Introduction to Treesp. 132
HTML::TreeBuilderp. 133
Processingp. 137
Example: BBC Newsp. 142
Example: Fresh Airp. 145
10. Modifying HTML with Treesp. 148
Changing Attributesp. 148
Deleting Imagesp. 152
Detaching and Reattachingp. 153
Attaching in Another Treep. 156
Creating New Elementsp. 161
11. Cookies, Authentication, and Advanced Requestsp. 165
Cookiesp. 165
Adding Extra Request Header Linesp. 169
Authenticationp. 172
An HTTP Authentication Example: The Unicode Mailing Archivep. 175
12. Spidersp. 178
Types of Web-Querying Programsp. 178
A User Agent for Robotsp. 180
Example: A Link-Checking Spiderp. 181
Ideas for Further Expansionp. 197
A. LWP Modulesp. 199
B. HTTP Status Codesp. 203
C. Common MIME Typesp. 205
D. Language Tagsp. 207
E. Common Content Encodingsp. 209
F. ASCII Tablep. 211
G. User's View of Object-Oriented Modulesp. 224
Indexp. 235

Google Preview