Running Sitescooper

The easiest way to get started with sitescooper is to simply run it: (UNIX users can leave out the perl command at the start of the line, but will have to provide the correct path to the sitescooper.pl script. Linux users installing using the RPM get sitescooper in their path, so they can just type sitescooper to run it.)

The first time you run sitescooper, it will pop up a list of sites in your editor, and ask you to pick which sites you wish to scoop. This creates a file in your temporary directory called site_choices.txt with these choices. Your temporary directory is the .sitescooper subdirectory of your home directory on UNIX, or C:\Windows\TEMP for Windows users; this can be changed by editing the built-in configuration in the script.

Once you've chosen some sites, it'll run through them, retrieve the pages, and convert them into iSilo format. See Changing Output Format if you wish to change this.

The Sites Directory

Versions of sitescooper before 2.0 used a different mechanism to choose sites; instead of picking them from a list, you had to copy them manually from the site_samples directory into a sites directory, and sitescooper would use the site files in that directory. This is still supported in 2.0; if there are any site files in the sites directory, they'll be read and those sites will be downloaded when you run sitescooper. If you're a pre-2.0 user and don't want to keep doing things that way, just delete those files.

Selecting Sites On The Command Line

If you want to scoop from one specific site, you can use the -site argument to do this. Provide the path to the site file and sitescooper will only read that one site. Multiple sites can be chosen by providing multiple -site arguments, one before each site name.

Scooping a URL

Let's say you want to scoop a URL which doesn't have a site file written for it. Run sitescooper with the URL on the command line and it will scoop that URL, tidy up the HTML as best it can without knowledge of the site's layout, and convert it into the chosen format. You can even provide a limited form of multi-level scooping with this, using the -levels and -storyurl arguments: Personally, I think this is only handy when prototyping a site file, but it's possible nonetheless.

Stopping Sitescooper From Being Too Clever

The next time you run sitescooper, it will avoid reloading the stories it's visited before. Sometimes, however, you will want it to do so; if this is the case, use the -refresh argument. This will cause sitescooper to ignore any historical accesses of the site and will scoop the lot. This is very handy when you're writing a site file.

What's This "File Size Limit Exceeded" Message?

Sitescooper imposes a limit of 200 kilobytes on the HTML or text scooped from any one site; otherwise it's quite easy to produce site files which can generate a 800K PRC file in one sitting!

By the way, note that the resulting PRC files may be well under 200Kb in size; sitescooper imposes the limit on the raw HTML or text as it goes along, and the conversion tools used may do a great job of compressing the data.

Also it should be noted that often, when you hit the limit on a site, the missed stories will simply be scooped next time you run the script. This depends on the site file though.

If you want to increase the limit, use the -limit argument:

will scoop your chosen sites, with a limit of 500Kb.

Changing Output Format

Currently, these are your options. The command-line switch is provided in bold text after the description.
  1. plain text, with all the articles listed one after the other. (-text)
  2. DOC format, for reading with AportisDoc or another DOC reader on a Palm handheld. This is essentially plain-text format converted using MakeDoc. (-doc)
  3. HTML format, with all the indexes and articles listed one after the other in one file, with hyperlinks between them. (-html)
  4. M-HTML (multiple-page HTML) format, which is the same as HTML except it separates the stories and indexes into separate files with hyperlinks between them. (-mhtml)
  5. iSilo format, for reading with iSilo on a Palm handheld. This is HTML converted using the iSilo conversion tool. (-isilo, this is the default)
  6. M-iSilo format, iSilo format files with multiple pages. This is an M-HTML article converted using the iSilo tool. (-misilo)
  7. RichReader format, for reading with RichReader on a Palm. This is HTML converted using the RichReader conversion tool. (-richreader)
It's also possible to run any command you like to convert the resulting output; see the documentation for the -pipe switch if you're interested in this.

If you want to convert to multiple output formats, you need to run sitescooper once for each output format, and use a shared cache between the separate invocations. Ask on the mailing list for more information on this.

What's Going Wrong?

If sitescooper is acting up and not doing what it's supposed to do, try the -debug switch. This will turn on lots of optional debugging output to help you work out what's wrong. This is very very handy when writing a site file.

There's also a -nowrite argument which will stop sitescooper from writing cache files, already_seen entries, and output files.

If the worst comes to the worst, you can get sitescooper to copy the HTML of every page accessed to a journal file using the -admin journal switch. This HTML is logged, first, in its initial form straight from the website, secondly, after the StoryStart and StoryEnd has been stripped from the page, and finally, as text. This is handy for debugging a site file, but is definitely not recommended during normal use, as a big site will produce a lot of journal output.

If you have all the files in your cache, use the -fromcache switch and network accesses will be avoided entirely. This is handy for debugging your site offline, or for producing output in multiple formats from the same files, if you have a shared cache set up.

Getting The Output

Normally, the output from sitescooper is written to the installation directory of your Pilot Desktop software, where possible. If you want the output directly, use the -dump switch. This will cause readable formats (text, HTML) to be dumped directly to standard output, i.e. to the screen, or to a file if you've redirected stdout.

-dumpprc does the same thing for the binary formats, such as DOC, iSilo, M-iSilo or RichReader. Note that multi-file formats such as M-HTML don't get dumped either way; the path to the first page of that output is printed instead.

Some versions of Windows perl have difficulty redirecting stdout, so the -stdout-to argument allows the same thing to be done from within the script itself.

Output Renaming

By default, the output files are generated with the current date in the filename and in the title used in the PRC file (when a PRC file is generated). Use the -prctitle and -filename arguments to change this behaviour. More information on this can be found in the command-line reference documentation.

[ README ]|[ Installing ]|[ on UNIX ]|[ on Windows ]|[ on a Mac ]|[ Running ]|[ Command-line Arguments Reference ]|[ Writing a Site File ]|[ Contributing ]|[ GPL ]|[ Home Page ]