
readme_ghostitems.txt


README for WebSphere Product Center, version 5.3.1, bad blob and ghost items detect and cleanup scripts.
Version of January 17, 2007
Copyright 2007 IBM Corporation

NOTE: This readme is for people interested in the ghost items scripts (and perhaps also bad blob detection).  
If you're just interested in bad blob detection, you should close this file and open readme.txt instead.

----------------------------------------------------------------------------------------------------
INTRODUCTION
----------------------------------------------------------------------------------------------------

Two shell scripts doreportblob.sh and docleanblob.sh are delivered.  The first reports the items, if any, in a given WPC Catalog
that have corrupted serialized forms.  The second purges (the latest version) of such items from that catalog, 
and also reverts (i.e. undoes) all checkouts of such items to collaboration areas associated with this source catalog.
Thus work in progress in collaboration areas will be deleted by docleanblob.sh .

Also included are 5.3.1 versions of older analysis scripts called the "ghost items scripts".  See below for more information on these.

----------------------------------------------------------------------------------------------------
REQUIRED PRELIMINARY SETUP STEPS in the database (prior to running any of the shell scripts)
----------------------------------------------------------------------------------------------------

1. Give the Oracle user either the "CREATE SNAPSHOT" or the "CREATE MATERIALIZED VIEW" privilege.

2. Using sqlplus and logging into your WPC database, run @drop_indexes, @create_indexes, @drop_backup_tables, @create_backup_tables.


NOTE:   Be sure to carry out these steps each time you use a new version of these scripts, even if you have previously done these steps
when using a prior version of these scripts.  Also, repeat these steps when switching between the "badblob" scripts and the "ghost items" scripts.

----------------------------------------------------------------------------------------------------
OTHER REQUIRED PRELIMINARY STEPS
----------------------------------------------------------------------------------------------------

1. Unpack the zipfile containing the scripts into any directory.  The usual WPC environment variables need to be defined ($TOP, $JAVA_RT).

2. The shell scripts require the database connection that is to be used, to be the one that is currently 
entered in $TOP/etc/default/common.properties.  Thus be sure you have the right values entered there. 
Both the bad blob and the ghost items scripts (see below) also require the (same) database connection 
information to be explicitly supplied in their command line arguments.

3. Although not strictly required, it is a good idea to shutdown WPC when running these shell scripts.

4.  Create (by hand) the directory '$TOP/logs/reportLog', to store the "longer reports" generated by running the shell scripts 

The shell invocations of the two shell scripts doreportblob.sh and docleanblob.sh produce brief sufficient output on the console about what they did.  
The above directory is where longer reports from each of their runs will be stored.  

The filenames of the long reports define the nature and time of the run, for exmample:

  56 -rw-r--r--    1 darnon   unixuser    50048 Jan 15 15:31 console_reportblob_Jan_15_2007_163028.txt
   8 -rw-r--r--    1 darnon   unixuser     5606 Jan 15 15:31 reportblob_Jan_15_2007_163028.txt

  24 -rw-r--r--    1 darnon   unixuser    21949 Jan 15 15:31 cleanblob_Jan_15_2007_163028.txt
  44 -rw-r--r--    1 darnon   unixuser    40047 Jan 15 15:31 console_cleanblob_Jan_15_2007_163028.txt

The first two are what you get from a run of doreportblob.sh. 
When you run docleanblob.sh, you would get all four files above.  This is because docleanblob.sh itself invokes doreportblob.sh.

The "long reports" are the files  reportblob_Jan_15_2007_163028.txt and cleanblob_Jan_15_2007_163028.txt.  The filenames beginning
"console..." are copies of the console output produced from the actual invocations of the shell scripts.

The longer reports are not intended to be either economical of output (rather, they are verbose), 
or understandable by the customer (rather, they are only understandable by IBM).   
The longer reports are generated only as a backup audit trail.  If they are needed, they should be sent to IBM for analysis.

=========================================================================================================================
"BAD BLOB" SHELL SCRIPT USAGE INSTRUCTIONS
=========================================================================================================================

The basic usage cycle for bad blob detection is: run doreportblob.sh and see if it finds any corrupted items.  If so, then run docleanblob.sh

You can get usage instructions for each script by just typing its name without arguments to a command line.
For example, type to a shell prompt:
    > doreportblob.sh
This will return the following six or so lines of output:
	Usage 
	doreportblob.sh --db=databasename --dbuser=dbusername --dbpassword=dbuserpassword --cmpName=cmpName --ctgId=ctgId
	    where databasename, dbusername, and dbuserpassword are viewable in the WPC System Administrator > Properties menu
	    and ctgId is the (numerical) id of a 'source' catalog in company 'cmpName'.
	    ctgId is obtainable via the WPC DBAdmin screen, with the query 'select ctg_id from ctg where ctg_name = CATALOGNAME'.
	
	Sample command line: doreportblob.sh --db=wpcdb --dbuser=wpc1 --dbpassword=wpc1 --cmpName=wildwest --ctgId=99

The command line for docleanblob.sh is similar.

Thus note that doreportblob.sh and docleanblob.sh operate on a single source catalog.

=========================================================================================================================
RELATIONSHIP OF THE "BAD BLOB" DETECTION SCRIPTS AND THE GHOST ITEMS SCRIPTS
=========================================================================================================================

There is a general relationship between the ghost items scripts and blob check scripts:  they both use the same Oracle views and the same backup tables.  
Plus, the blob scripts call (invoke) the ghost items scripts.  If you're going to run both, run the ghost items scripts first.

It doesn't hurt to run the ghost items scripts first if you want to, just to confirm that there are no ghost items.  
After doing so, drop and recreate the backup tables (as noted above).

As noted above, the bad blob detection and cleanup scripts require the database connection that is to be used, 
to be the one that is currently entered in $TOP/etc/default/common.properties.
Thus be sure you have the right values entered there. 

Both the bad blob and the ghost items scripts also require the (same) database connection 
information to be explicitly supplied in their command line arguments.

=========================================================================================================================
ADDITIONAL INFORMATION ON GHOST ITEMS SCRIPTS
=========================================================================================================================

The main ghost items shell scripts are: doreportall.sh, docleanall.sh, and dorestoreall.sh.  
You can get usage instructions for each by just typing its name without arguments to a command line.
    For example:  doreport.sh --db=wpcdb --dbuser=wpc1 --dbpassword=wpc1 --collabid=6

-----------------------------------------
Cautions and notes about the ghost items scripts
-----------------------------------------
- Ideally you should shut down WPC when running these scripts.  Short of that, run stop_workflowengine.sh before the scripts, and then 
    start_workflowengine.sh afterwards.   And if you must have human users on the system when the scripts are running, have them not
    do any workflow-related tasks.

- Be careful not to do a second run of the clean script, before you have reviewed the results of the preceding clean script run 
    and are happy with the results.   The reason is that each run of the clean script wipes out the backup table data for the previous run, 
    that is needed if you're going to run restore.  

- Please verify that your application never has items checked out to more than one collab area simultaneously, before
    using these scripts.  If your application can ever have items checked out to more than one collab area at a time, 
    please contact us for further special assistance, and please do NOT run doclean.sh or docleanall.sh before doing so.

-------------------------------------------------------------------------
Further details about the ghost items scripts
-------------------------------------------------------------------------

a) The six core scripts (that handle a single collaboration area) are:
    doclean.sh , doreport.sh, and dorestore.sh  (for item-workflow collaboration areas)
    doclean_category.sh , doreport_category.sh, and dorestore_category.sh  (for category-workflow collaboration areas)

You may not need to use these scripts directly if you always want to do all collaboration areas.
In that case, use doreportall.sh, docleanall.sh, dorestoreall.sh

Type any script's name on a command line by itself, i.e. without any arguments, to get usage instructions.


b) The output files now have meaningful names. To be specific, their names now have this format:

 9-{report_item_ca_Apr_6_2006_130522-.txt

or

 7-{report_category_ca_Apr_6_2006_130536-.txt

where 9 and 7 are collab area ids, 'report'signifies it is a from "doreport" command, and the rest is time stamp.


c)  Output files are written into standard subdirectories under $TOP/logs,
of to standard subdirectories under current $CWD if $TOP is null, where "$CWD" stands for "current working directory".
The output subdirectories are:
    $TOP/logs/reportLog, $TOP/logs/cleanLog , and $TOP/logs/restoreLog.


d) There is a standard convention for each shell script to indicate that it has completed: 
it has exit status 0 if sucessful, 1 if not.


e) There is a standard convention to record the "last time ran", for each doall shell script. 
Namely, there are three standard files in the $TOP/logs subdirectories:

    $TOP/logs/lastreportall.txt, lastcleanall.txt, and lastrestoreall.txt, 

i.e. one per script, and the contents of each of them is the "last time ran" timestamp.
Let's take docleanall.sh for example.  
It does its work (i.e. cleans all the collab areas), then removes the existing version 
of its standard file ($TOP/logs/ghostitems/lastcleanall.txt), if any, 
and finally recreates a new lastcleanall.txt file and writes the current system time into it.   
That way once you start docleanall.sh running, you can poll to test for the
updating of the lastcleanall.txt file, to know that it's done.  

The same idea applies to the other two scripts.

If you want, you could edit the shell scipt to delete the standard file first, each time doall starts.

Note that the very first time you run each of the doall scripts, you'll get an error trying to remove its standard file, e.g.
    rm: cannot remove `/home/darnon/perforce/dev/release/5_2_1/main/logs/cleanLog/lastcleanall.txt': No such file or directory
This is because it tries to remove the file first, before creating a new version of it, so the first time you get the above error message. 
The second and subsequent times running the doall script, since it has already created the file, 
it will be able to remove it successfully, i.e. no error message.


----------------------------------------------------------------------------------------------------
New features in the APRIL 11, 2006 version of the ghost items scripts
----------------------------------------------------------------------------------------------------

The commands doreportall.sh, docleanall.sh, and dorestoreall.sh have been enhanced to handle category-workflow collaboration areas,
in addition to the item-workflow collaboration areas already handled.  There is no change to the command line format for those three commands.
Each of the three "doall" commands will first cycle through and process each of the collaboration areas whose associated workflow is an 
item workflows, then cycle through and process each of the collaboration areas whose associated workflow is a category workflow.


For category-workflow collaboration areas, the new "ghost categories" check looks for the following:
a) categories which appear to be checked out in the source category tree, but which are not properly present
in the (database tables for the) collaboration area.

b) "orphan" category id's which are present in the workflow database tables for this collaboration area, but
which do not point at any actual categories in the (category) database tables.


If you wish to do a report, clean, or restore for a single collaboration area, it is necessary to know whether that collaboration area's
associated workflow is an item workflow or a category workflow.   
For a collaboration area with an item workflow, use the procedures doreport.sh, doclean.sh, and dorestore.sh, just as
before.  For a collaboration area with a category worklow, use the new procedures 
doreport_category.sh, doclean_category.sh, and dorestore_category.sh.


The names of the output files now identify what type of collaboration area it was.  For example, the output filename:

 9-{report_item_ca_Apr_6_2006_130522-.txt

identifies the collaboration area id=9 as having an item workflow, and the output filename

 7-{report_category_ca_Apr_6_2006_130536-.txt

identifies collaboration area id=7 as having a category workflow.

-------------------------------------------------------------------------
END OF README_GHOSTITEMS.TXT
-------------------------------------------------------------------------



