
readme_fullIntegrityVerify.txt


README for WebSphere Product Center, version 5.3.2, fullIntegrityVerify.sh 
Version of September 20, 2007
Copyright 2007 IBM Corporation


----------------------------------------------------------------------------------------------------
INTRODUCTION
----------------------------------------------------------------------------------------------------
Data Integrity Scripts detect data corruption such as duplicate rows,ghost items or bad serialized forms 
in DB2/Oracle databases. These data corruptions can be detected by a combination of SQL scripts, shell 
scripts and java programs.
There are 3 types of integrity checks to detect the corruption in your database

Basic Integrity Check:
       Checks for the duplicate rows in the tables of target company
   
Workflow Integrity Check:
	Checks for ghost items in the collaboration areas of the target company

Serialization Integrity Check:
	Checks for bad serialized forms in all the containers in the target company

---------------------------------------------------------------------------------------------------
Running Data Integrity Scripts for Oracle Database
---------------------------------------------------------------------------------------------------
The shell script fullIntegrityVerify.sh can be run to detect the above three types of corruptions.
This script runs three previously existing reports in succession.  

1) basic_integrity_verify.sh

2) workflow_integrity_verify.sh - ghost item checks of all collaboration areas.

3) serialization_integrity_verify.sh - bad blob checking on all source catalogs of the target company, and their associated collaboration areas, if any.
  (please communicate with IBM Level-2 support re: what "target company" means here).

----------------------------------------------------------------------------------------------------
REQUIRED PRELIMINARY SETUP STEPS in the database (prior to running any of the shell scripts)
----------------------------------------------------------------------------------------------------

1. Give the Oracle user either the "CREATE SNAPSHOT" or the "CREATE MATERIALIZED VIEW" privilege.
   TO do this login as database administrator, then run the following sqlplus commands to set 
   your database privileges to either CREATE SNAPSHOT or CREATE MATERIALIZED VIEW
   syntax:
   sqlplus db_admin_userid/db_admin_password@dbname
   grant create snapshot to db_username;
   quit

2. Using sqlplus and logging into your WPC database, run @drop_indexes, @create_indexes, @drop_backup_tables, @create_backup_tables.


NOTE:   Be sure to carry out these steps each time you use a new version of these scripts, even if you have previously done these steps
when using a prior version of these scripts.  Also, repeat these steps when switching between the "badblob" scripts and the "ghost items" scripts.


----------------------------------------------------------------------------------------------------
OTHER REQUIRED PRELIMINARY STEPS
----------------------------------------------------------------------------------------------------

1. The shell scripts require the database connection that is to be used, to be the one that is currently 
entered in $TOP/etc/default/common.properties.  Thus be sure you have the right values entered there. 
Both the serialization and the workflow  scripts also require the (same) database connection 
information to be explicitly supplied in their command line arguments.

2. Although not strictly required, it is a good idea to shutdown WPC when running these shell scripts.

------------------------------------------------------------------------------------------------------
After completing the above preliminary steps, run one of the following version of fullIntegrityVerify.sh



1)To check for the above three types of corruptions in all the catalogs and collaboration areas in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmp directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --basic<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'
       
	For example if the users want to check for all the corruptions, the script can be run as
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=y --serialization=y
        To check for basic and serialization corruptions,
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=n --serialization=y


2)To check for the above three types of corruptions in a given collaboration area in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCA directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --coaname=Collabareaname --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 coaname refers to Collaboration Area Name
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

3)To check for the above three types of corruptions in a given Catalog in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCat directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --ctg=CatalogName --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 ctg refers to name of the catalog
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

The shell script fullIntegrityVerify.sh runs the basic_integrity_verify.sh, workflow_integrity_verify.sh 
and Serialization_integrity_verify.sh shell scripts in succession based on the user options as specified above.


Here are the associated cleanup procedures, which need to be run 
individually (they are in subdirectories of the root directory):
1)  basic_integriy_verify.sh/basic_integrity_restore.sh
2)  workflow_integrity_verify.sh/oracle/docleanall.sh
3)  serialization_integrity_verify.sh/oracle/docleanbloball.sh

The best way is to "cd" to the respective subdirectory and run the shell script there.

Reports by 2) and 3) are produced in two subdirectories of $TOP/logs:
badBlobsCleanings
ghostItemsCleanings

Some notes Re: 1)
basic_integrity_verify.sh is mainly concerned to detect duplicate rows in specific WPC tables. 
Its associated cleanup script (basic_integriy_restore.sh) will never delete entire items.
To run  basic_integrity_restore.sh:
From the $TOP/src/integrity/basic/oracle directory run:
  basic_integrity_restore.sh --code=CmpCode
  where CmpCode is the code of the company

Re: 2)
Workflow_integrity_verify.sh detects ghost items.
The associated ghost items cleanup script reverts (i.e. undoes) all checkouts of such items from their source catalog.,
and purges (deletes) them from their collaboration area.
To run clean up scripts for ghost items:
From $TOP/src/integrity/oneCmp/workflow/oracle directory run :
  docleanall.sh 

Re: 3)
Bad blob checking reports the items, if any, in each source catalog of the target company, and all its associated collaboration areas,
that have corrupted serialized forms.  
The associated bad blob cleanup script purges (the latest version) of such items from that catalog, 
and also reverts (i.e. undoes) all checkouts of such items to collaboration areas associated with this source catalog.
Thus "work in progress" in such items in collaboration areas,  will be deleted by bad blob cleanup
To run the cleanup script for badblobs:
From the directory $TOP/src/integrity/oneCmp/serialization/oracle run :
   docleanbloball.sh --cmpname=CmpName
   cmpname refers to the name of your company.

(Note: The scripts could run for a very long time depending upon the size of the database)

Reports by 2) and 3) are produced in two subdirectories of $TOP/logs:
badBlobsReports
ghostItemsReports

Thus note: the user running fullIntegrityVerify.sh needs to have write and execute permission in $TOP/logs.  And of course, the user needs to have
write and execute permissions in all subdirectories of the "home" or "root" directory in which the delivery zipfile/tarfile is unpacked.

Report names are descriptive (containing catalog or collab area ids, and timestamps). 
They alway come in pairs of two files"  one called "console_report_..." 
which is a replica of console output,
and the other called just "report_...", which is a longer detailed report and should be sent to IBM for analysis if and when errors are found.
Archive the reportLogs and the command line output and send to L2 for review


Note:Before running the scripts again, backup the reports directory. Otherwise, new script runs will be writing to the same reports directory and they can get very confusing as to which logs are from which run.

Note: this version is only for item (and not category) workflow/collab areas.


--------------------------------------------------------------------------------------------------------
Running Integrity Scripts for DB2
--------------------------------------------------------------------------------------------------------
DB2 version of the scripts can be run similar to the oracle version. 

1)To check for the three types of corruptions in all the catalogs and collaboration areas in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmp directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

       For example if the users want to check for all the corruptions, the script can be run as
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=y --serialization=y
        To check for basic and serialization corruptions,
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=n --serialization=y

2)To check for the three types of corruptions in a given collaboration area in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCA directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --coaname=Collabareaname --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 coaname refers to Collaboration Area Name
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

3)To check for the three types of corruptions in a given Catalog in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCat directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --ctg=CatalogName --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 ctg refers to name of the catalog
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

The shell script fullIntegrityVerify.sh runs the basic_integrity_verify.sh, workflow_integrity_verify.sh 
and Serialization_integrity_verify.sh shell scripts in succession based on the user options.

The logs for workflow integrity scripts are under $TOP/logs/ghost_items_reports.
 
The logs for serialization scripts are under $TOP/logs/badBlobReports. Also, the entries with badblobs are written to the table cet_badblob. 
In the case of oneCmp, the serialization script verifies for bad serialized forms in Catalogs, Category Trees, CollaborationAreas, Lookup Tables. The consolidated report for all these entries in the company is stored under $TOP/logs/badBlobReports. The name of this report file starts with “Container_report_ALL_time stamp”. Along with this consolidated report, the console report files for each catalog can also be found in this directory.

Contact IBM Software support if you must run the clean up script.


Currently for DB2 database, there are clean up scripts for only 2 types of corruptions. They can be
invoked as follows:

1)Clean the duplicate rows in the tables of yur DB2 database:
Run the following from $TOP/src/integrity/basic/db2 directory:
 basic_integrity_restore.sh --code=companycode
  --code is the company code

2)Clean ghost items:
 Run the following from $TOP/src/integrity/oneCmp/workflow/db2 directory to revert all your source catalog 
 item checkouts and then delete them from your collaboration areas:
   clean_ghost_items_db2.sh

 Two summary report files are stored in $TOP/logs/ghostitemsreports directory.
   Console_report_catalog_OR_collab_area_id_timestamp - is a replica of the console output where
	 catalog_OR_collab_area_id is the Catalog or Collaboration area id and 
	timestamp is the time at which the file is created.

   report_catalog_OR_collab_area_id_timestamp contains the detailed report.

   The detailed report should be sent to IBM software support for analysis if errors are found.

The cleanup script for serialization corruptions is not yet available for DB2 database.This will be
provided in future.

-------------------------------------------------------------------------
END OF README
-------------------------------------------------------------------------



