Running Sitescooper
The easiest way to get started with sitescooper is to simply run it:
(UNIX users can leave out the perl command at the start of the line,
but will have to provide the correct path to the sitescooper.pl
script. Linux users installing using the RPM get sitescooper in their path,
so they can just type sitescooper to run it.)
The first time you run sitescooper, it will pop up a list of sites in
your editor, and ask you to pick which sites you wish to scoop. This creates
a file in your temporary directory called site_choices.txt with
these choices. Your temporary directory is the .sitescooper subdirectory
of your home directory on UNIX, or C:\Windows\TEMP for Windows users;
this can be changed by editing the built-in configuration in the script.
Once you've chosen some sites, it'll run through them, retrieve the
pages, and convert them into iSilo format. See Changing Output
Format if you wish to change this.
The Sites Directory
Versions of sitescooper before 2.0 used a different mechanism to choose
sites; instead of picking them from a list, you had to copy them manually
from the site_samples directory into a sites directory, and
sitescooper would use the site files in that directory. This is still supported
in 2.0; if there are any site files in the sites directory, they'll
be read and those sites will be downloaded when you run sitescooper. If
you're a pre-2.0 user and don't want to keep doing things that way, just
delete those files.
Selecting Sites On The Command Line
If you want to scoop from one specific site, you can use the -site
argument to do this. Provide the path to the site file and sitescooper
will only read that one site.
perl sitescooper.pl -site site_samples/web/alertbox.site
Multiple sites can be chosen by providing multiple -site arguments,
one before each site name.
Scooping a URL
Let's say you want to scoop a URL which doesn't have a site file written
for it. Run sitescooper with the URL on the command line and it will scoop
that URL, tidy up the HTML as best it can without knowledge of the site's
layout, and convert it into the chosen format.
perl sitescooper.pl http://jmason.org/
You can even provide a limited form of multi-level scooping with this,
using the -levels and -storyurl arguments:
perl sitescooper.pl -levels 2 -storyurl 'http://jmason.org/.*\.html'
http://jmason.org/
Personally, I think this is only handy when prototyping a site file, but
it's possible nonetheless.
Stopping Sitescooper From Being Too Clever
The next time you run sitescooper, it will avoid reloading the stories
it's visited before. Sometimes, however, you will want it to do so; if
this is the case, use the -refresh argument. This will cause sitescooper
to ignore any historical accesses of the site and will scoop the lot.
perl sitescooper.pl -site site_samples/web/alertbox.site
perl sitescooper.pl -refresh -site site_samples/web/alertbox.site
This is very handy when you're writing a site file.
What's This "File Size Limit Exceeded" Message?
Sitescooper imposes a limit of 200 kilobytes on the HTML or text scooped
from any one site; otherwise it's quite easy to produce site files which
can generate a 800K PRC file in one sitting!
By the way, note that the resulting PRC files may be well under 200Kb
in size; sitescooper imposes the limit on the raw HTML or text as it goes
along, and the conversion tools used may do a great job of compressing
the data.
Also it should be noted that often, when you hit the limit on a site,
the missed stories will simply be scooped next time you run the script.
This depends on the site file though.
If you want to increase the limit, use the -limit argument:
perl sitescooper.pl -limit 500
will scoop your chosen sites, with a limit of 500Kb.
Changing Output Format
Currently, these are your options. The command-line switch is provided
in bold text after the description.
-
plain text, with all the articles listed one after the other. (-text)
-
DOC format, for reading with AportisDoc or another DOC reader on a Palm
handheld. This is essentially plain-text format converted using MakeDoc.
(-doc)
-
HTML format, with all the indexes and articles listed one after the other
in one file, with hyperlinks between them. (-html)
-
M-HTML (multiple-page HTML) format, which is the same as HTML except it
separates the stories and indexes into separate files with hyperlinks between
them. (-mhtml)
-
iSilo format, for reading with iSilo on a Palm handheld. This is HTML converted
using the iSilo conversion tool. (-isilo, this is the default)
-
M-iSilo format, iSilo format files with multiple pages. This is an M-HTML
article converted using the iSilo tool. (-misilo)
-
RichReader format, for reading with RichReader on a Palm. This is HTML
converted using the RichReader conversion tool. (-richreader)
It's also possible to run any command you like to convert the resulting
output; see the documentation for the -pipe switch
if you're interested
in this.
If you want to convert to multiple output formats, you need to run sitescooper
once for each output format, and use a shared cache between the separate
invocations. Ask on the mailing list for more information on this.
What's Going Wrong?
If sitescooper is acting up and not doing what it's supposed to do, try
the -debug switch. This will turn on lots of optional debugging
output to help you work out what's wrong.
perl sitescooper.pl -debug
This is very very handy when writing a site file.
There's also a -nowrite argument which will stop sitescooper
from writing cache files, already_seen entries, and output files.
If the worst comes to the worst, you can get sitescooper to copy the
HTML of every page accessed to a journal file using the -admin journal
switch. This HTML is logged, first, in its initial form straight from the
website, secondly, after the StoryStart and StoryEnd has been stripped
from the page, and finally, as text. This is handy for debugging a site
file, but is definitely not recommended during normal use, as a big site
will produce a lot of journal output.
If you have all the files in your cache, use the -fromcache switch
and network accesses will be avoided entirely. This is handy for debugging
your site offline, or for producing output in multiple formats from the
same files, if you have a shared cache set up.
Getting The Output
Normally, the output from sitescooper is written to the installation directory
of your Pilot Desktop software, where possible.
-
On UNIX, it'll be copied into the ~/pilot/install directory if it
exists, or ~/pilot otherwise. Users of gnome-pilot, PilotManager, or
JPilot should enter gnome-pilot, PilotManager, or JPilot
in the PilotInstallApp field of the configuration, as sitescooper
includes built-in support for those tools. (If you use KPilot, mail me and
tell me where it should go!)
-
Windows users have it easy, as sitescooper will automatically install
PRC files into the Pilot Desktop Install directory. If you use
multiple Palm devices however, you will still need to edit the configuration
to name the correct directory.
-
Mac users get the worst of all worlds, as the output is simply left in
the txt sub-folder of the temporary folder. They need to copy it
over themselves manually (sorry).
If you want the output directly, use the -dump switch. This will
cause readable formats (text, HTML) to be dumped directly to standard output,
i.e. to the screen, or to a file if you've redirected stdout.
-dumpprc does the same thing for the binary formats, such as
DOC, iSilo, M-iSilo or RichReader. Note that multi-file formats such as
M-HTML don't get dumped either way; the path to the first page of
that output is printed instead.
Some versions of Windows perl have difficulty redirecting stdout, so
the -stdout-to argument allows the same thing to be done from within
the script itself.
Output Renaming
By default, the output files are generated with the current date in the
filename and in the title used in the PRC file (when a PRC file is generated).
Use the -prctitle and -filename arguments to change this
behaviour. More information on this can be found in the command-line reference
documentation.
[
README ]|[
Installing ]|[
on UNIX ]|[
on Windows ]|[
on a Mac ]|[
Running ]|[
Command-line Arguments Reference ]|[
Writing a Site File ]|[
Contributing ]|[
GPL ]|[
Home Page ]