MoonPoint Support Logo

 

Shop Amazon Warehouse Deals - Deep Discounts on Open-box and Used ProductsAmazon Warehouse Deals



Advanced Search
May
Sun Mon Tue Wed Thu Fri Sat
   
27    
2012
Months
May


Sun, May 27, 2012 10:51 pm

Downloading Webpages with Perl Using Get

I had written a Perl script to dowload specific webpages from a site and peform some processing on the downloaded copies of the webpages. When I moved the script to another system, I received the error message below when I ran it.
Can't locate LWP/Simple.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at ./get_webpage.pl line 5.
BEGIN failed--compilation aborted at ./get_webpage.pl line 5.

The script uses the get command from the LWP::Simple Perl module, which allows me to download a webpage similar to how I would use the GNU wget command from a shell prompt to download webpages. I wanted to use the Perl script, instead of wget, though, since I wanted to examine the webpages for specific links and then download other pages based on specific URLs present in the downloaded web pages.

When I checked for the presence of the LWP::Simple module, which provides support for the get, command on the system on which the script worked, I saw the following:

$ perldoc -l LWP::Simple
/usr/lib/perl5/vendor_perl/5.8.8/LWP/Simple.pm
$ cd /home/joe/www/support/languages/perl

When I checked on the system on which the script failed, I saw the following:

$ perldoc -l LWP::Simple
You need to install the perl-doc package to use this program.

I decided just to proceed with the installation of the LWP::Simple module by running the command perl -MCPAN -e shell. Since it was the first time I had run it, I was prompted to answer a number of questions. I hit Enter to accept the default answer for all of the questions. When the configuration process was completed, I entered install LWP::Simple at the cpan prompt. When the module was installed, I entered exit.

$ sudo bash
[sudo] password for joe: 
root@Saturn:~/Documents/bin# perl -MCPAN -e shell

CPAN.pm requires configuration, but most of it can be done automatically.
If you answer 'no' below, you will enter an interactive dialog for each
configuration option instead.

Would you like to configure as much as possible automatically? [yes] 

 <install_help>

Warning: You do not have write permission for Perl library directories.

To install modules, you need to configure a local Perl library directory or
escalate your privileges.  CPAN can help you by bootstrapping the local::lib
module or by configuring itself to use 'sudo' (if available).  You may also
resolve this problem manually if you need to customize your setup.

What approach do you want?  (Choose 'local::lib', 'sudo' or 'manual')
 [local::lib] 

Autoconfigured everything but 'urllist'.

Now you need to choose your CPAN mirror sites.  You can let me
pick mirrors for you, you can select them from a list or you
can enter them by hand.

Would you like me to automatically choose some CPAN mirror
sites for you? (This means connecting to the Internet) [yes] 
Trying to fetch a mirror list from the Internet
Fetching with HTTP::Tiny:
http://www.perl.org/CPAN/MIRRORED.BY

Looking for CPAN mirrors near you (please be patient)
.............................. done!

New urllist
  http://httpupdate3.cpanel.net/CPAN/
  http://httpupdate23.cpanel.net/CPAN/
  http://mirrors.rit.edu/CPAN/

Autoconfiguration complete.

Attempting to bootstrap local::lib...

Writing /home/joe/.cpan/CPAN/MyConfig.pm for bootstrap...
commit: wrote '/home/joe/.cpan/CPAN/MyConfig.pm'
Fetching with HTTP::Tiny:
http://httpupdate3.cpanel.net/CPAN/authors/01mailrc.txt.gz
Going to read '/home/joe/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with HTTP::Tiny:
http://httpupdate3.cpanel.net/CPAN/modules/02packages.details.txt.gz
Going to read '/home/joe/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Mon, 28 May 2012 01:35:04 GMT
  HTTP::Date not available
..............
  New CPAN.pm version (v1.9800) available.
  [Currently running version is v1.960001]
  You might want to try
    install CPAN
    reload cpan
  to both upgrade CPAN.pm and run the new version without leaving
  the current session.


..............................................................DONE
Fetching with HTTP::Tiny:
http://httpupdate3.cpanel.net/CPAN/modules/03modlist.data.gz
Going to read '/home/joe/.cpan/sources/modules/03modlist.data.gz'
............................................................................DONE
Going to write /home/joe/.cpan/Metadata
Running make for A/AP/APEIRON/local-lib-1.008004.tar.gz
Fetching with HTTP::Tiny:
http://httpupdate3.cpanel.net/CPAN/authors/id/A/AP/APEIRON/local-lib-1.008004.tar.gz
Fetching with HTTP::Tiny:
http://httpupdate3.cpanel.net/CPAN/authors/id/A/AP/APEIRON/CHECKSUMS
Checksum for /home/joe/.cpan/sources/authors/id/A/AP/APEIRON/local-lib-1.008004.tar.gz ok

  CPAN.pm: Going to build A/AP/APEIRON/local-lib-1.008004.tar.gz

Attempting to create directory /home/joe/perl5

*** Module::AutoInstall version 1.03
*** Checking for Perl dependencies...
*** Since we're running under CPAN, I'll just let it take care
    of the dependency's installation later.
[Core Features]
- ExtUtils::MakeMaker ...loaded. (6.57_05 >= 6.31)
- ExtUtils::Install   ...loaded. (1.56 >= 1.43)
- Module::Build       ...loaded. (0.38 >= 0.36)
- CPAN                ...loaded. (1.960001 >= 1.82)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Writing Makefile for local::lib
Writing MYMETA.yml
cp lib/POD2/PT_BR/local/lib.pod blib/lib/POD2/PT_BR/local/lib.pod
cp lib/lib/core/only.pm blib/lib/lib/core/only.pm
cp lib/local/lib.pm blib/lib/local/lib.pm
cp lib/POD2/DE/local/lib.pod blib/lib/POD2/DE/local/lib.pod
Manifying blib/man3/POD2::PT_BR::local::lib.3pm
Manifying blib/man3/lib::core::only.3pm
Manifying blib/man3/local::lib.3pm
Manifying blib/man3/POD2::DE::local::lib.3pm
  APEIRON/local-lib-1.008004.tar.gz
  /usr/bin/make -- OK
'YAML' not installed, will not store persistent state
Running make test
PERL_DL_NONLAZY=1 /usr/bin/perl -I/home/joe/perl5/lib/perl5/i686-linux-gnu-threa
d-multi-64int -I/home/joe/perl5/lib/perl5 "-MExtUtils::Command::MM" "-e" "test_h
arness(0, 'inc', 'blib/lib', 'blib/arch')" t/classmethod.t t/coderefs_in_inc.t t
/de-dup.t t/install.t t/lib-core-only.t t/pipeline.t t/stackable.t
t/classmethod.t ...... Name "File::Spec::rel2abs" used only once: possible typo 
at t/classmethod.t line 20.
t/classmethod.t ...... 1/? Attempting to create directory t/var/splat
t/classmethod.t ...... ok   
t/coderefs_in_inc.t .. ok   
t/de-dup.t ........... ok   
t/install.t .......... skipped: Install Capture::Tiny to test installation
t/lib-core-only.t .... ok   
t/pipeline.t ......... ok   
t/stackable.t ........ ok     
All tests successful.
Files=7, Tests=29, 41 wallclock secs ( 0.17 usr  0.04 sys +  1.32 cusr  0.14 csys =  1.67 CPU)
Result: PASS
  APEIRON/local-lib-1.008004.tar.gz
  /usr/bin/make test -- OK
Running make install
Installing /home/joe/perl5/lib/perl5/POD2/PT_BR/local/lib.pod
Installing /home/joe/perl5/lib/perl5/POD2/DE/local/lib.pod
Installing /home/joe/perl5/lib/perl5/lib/core/only.pm
Installing /home/joe/perl5/lib/perl5/local/lib.pm
Installing /home/joe/perl5/man/man3/local::lib.3pm
Installing /home/joe/perl5/man/man3/lib::core::only.3pm
Installing /home/joe/perl5/man/man3/POD2::PT_BR::local::lib.3pm
Installing /home/joe/perl5/man/man3/POD2::DE::local::lib.3pm
Appending installation info to /home/joe/perl5/lib/perl5/i686-linux-gnu-thread-multi-64int/perllocal.pod
  APEIRON/local-lib-1.008004.tar.gz
  /usr/bin/make install  -- OK
Tried to deactivate inactive local::lib '/home/joe/perl5'

local::lib is installed. You must now add the following environment variables
to your shell configuration files (or registry, if you are on Windows) and
then restart your command line shell and CPAN before installing modules:

Use of uninitialized value $deactivating in numeric eq (==) at /home/joe/perl5/l
ib/perl5/local/lib.pm line 354.
Use of uninitialized value $deactivating in numeric eq (==) at /home/joe/perl5/l
ib/perl5/local/lib.pm line 356.
Use of uninitialized value $interpolate in numeric eq (==) at /home/joe/perl5/li
b/perl5/local/lib.pm line 366.
export PERL_LOCAL_LIB_ROOT="/home/joe/perl5";
export PERL_MB_OPT="--install_base /home/joe/perl5";
export PERL_MM_OPT="INSTALL_BASE=/home/joe/perl5";
export PERL5LIB="/home/joe/perl5/lib/perl5/i686-linux-gnu-thread-multi-64int:/home/joe/perl5/lib/perl5";
export PATH="/home/joe/perl5/bin:$PATH";

Would you like me to append that to /home/joe/.bashrc now? [yes] 


commit: wrote '/home/joe/.cpan/CPAN/MyConfig.pm'

You can re-run configuration any time with 'o conf init' in the CPAN shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.960001)
Enter 'h' for help.

cpan[1]> install LWP::Simple
<text snipped> cpan[2]> exit

I was then able to successfully run the Perl script.

[/languages/perl] permanent link

Valid HTML 4.01 Transitional

Privacy Policy   Contact

Blosxom logo