
readme_fullIntegrityVerify.txt


README for WebSphere Product Center, version 5.3.2, fullIntegrityVerify.sh 
Version of April 8, 2009
Copyright 2007 IBM Corporation


----------------------------------------------------------------------------------------------------
 I  	INTRODUCTION
----------------------------------------------------------------------------------------------------
Data Integrity Scripts detect data corruption such as duplicate rows,ghost items or bad serialized forms 
in DB2/Oracle databases. These data corruptions can be detected by a combination of SQL scripts, shell 
scripts and java programs.
There are 3 types of integrity checks to detect the corruption in your database

Basic Integrity Check:
       Checks for the duplicate rows in the tables of target company
   
Workflow Integrity Check:
	Checks for ghost items in the collaboration areas of the target company

Serialization Integrity Check:
	Checks for bad serialized forms in all the containers in the target company

---------------------------------------------------------------------------------------------------
II    	Running Data Integrity Scripts for Oracle Database
---------------------------------------------------------------------------------------------------
The shell script fullIntegrityVerify.sh can be run to detect the above three types of corruptions.
This script runs three previously existing reports in succession.  

1) basic_integrity_verify.sh

2) workflow_integrity_verify.sh - ghost item checks of all collaboration areas.

3) serialization_integrity_verify.sh - bad blob checking on all source catalogs of the target company, and their associated collaboration areas, if any.
  "target company" here refers to the company for which these scripts are run. 

----------------------------------------------------------------------------------------------------
	REQUIRED PRELIMINARY SETUP STEPS in the database (prior to running any of the shell scripts)
----------------------------------------------------------------------------------------------------

1. Give the Oracle user either the "CREATE SNAPSHOT" or the "CREATE MATERIALIZED VIEW" privilege.
   TO do this login as database administrator, then run the following sqlplus commands to set 
   your database privileges to either CREATE SNAPSHOT or CREATE MATERIALIZED VIEW
   syntax:
   sqlplus db_admin_userid/db_admin_password@dbname
   grant create snapshot to db_username;
   quit

2. Go to $TOP/src/integrity/SQL/oracle. Using sqlplus and logging into your WPC database, run  @drop_backup_tables, @create_backup_tables.


NOTE:   Be sure to carry out these steps each time you use a new version of these scripts, even if you have previously done these steps
when using a prior version of these scripts.  Also, repeat these steps when switching between the "badblob" scripts and the "ghost items" scripts.


----------------------------------------------------------------------------------------------------
		OTHER REQUIRED PRELIMINARY STEPS
----------------------------------------------------------------------------------------------------

1. The shell scripts require the database connection that is to be used, to be the one that is currently 
entered in $TOP/etc/default/common.properties.  Thus be sure you have the right values entered there. 
Both the serialization and the workflow  scripts also require the (same) database connection 
information to be explicitly supplied in their command line arguments.

2. Although not strictly required, it is a good idea to shutdown WPC when running these shell scripts.

------------------------------------------------------------------------------------------------------
After completing the above preliminary steps, run one of the following version of fullIntegrityVerify.sh



1)To check for the above three types of corruptions in all the catalogs and collaboration areas in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmp directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --basic<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'
       
	For example if the users want to check for all the corruptions, the script can be run as
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=y --serialization=y
        To check for basic and serialization corruptions,
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=n --serialization=y


2)To check for the above three types of corruptions in a given collaboration area in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCA directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --coaname=Collabareaname --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 coaname refers to Collaboration Area Name
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

3)To check for the above three types of corruptions in a given Catalog in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCat directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --ctg=CatalogName --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 ctg refers to name of the catalog
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

The shell script fullIntegrityVerify.sh runs the basic_integrity_verify.sh, workflow_integrity_verify.sh 
and serialization_integrity_verify.sh shell scripts in succession based on the user options as specified above.

Some notes Re: 1)
basic_integrity_verify.sh is mainly concerned to detect duplicate rows in specific WPC tables. 
Its associated cleanup script (basic_integriy_restore.sh) will never delete entire items.
To run  basic_integrity_restore.sh:
From the $TOP/src/integrity/basic/oracle directory run:
  basic_integrity_restore.sh --code=CmpCode
  where CmpCode is the code of the company

Re: 2)
Workflow_integrity_verify.sh detects ghost items.
The associated ghost items cleanup script reverts (i.e. undoes) all checkouts of such items from their source catalog.,
and purges (deletes) them from their collaboration area.


Re: 3)
Bad blob checking reports the items, if any, in each source catalog of the target company, and all its associated collaboration areas,
that have corrupted serialized forms.  
The associated bad blob cleanup script purges (the latest version) of such items from that catalog, 
and also reverts (i.e. undoes) all checkouts of such items to collaboration areas associated with this source catalog.
Thus "work in progress" in such items in collaboration areas,  will be deleted by bad blob cleanup

(Note: The scripts could run for a very long time depending upon the size of the database)

Reports by 2) and 3) are produced in two subdirectories of $TOP/logs:
badBlobsReports
ghostItemsReports

Thus note: the user running fullIntegrityVerify.sh needs to have write and execute permission in $TOP/logs.  And of course, the user needs to have
write and execute permissions in all subdirectories of the $TOP/src/integrity. 

Report names are descriptive (containing catalog or collab area ids, and timestamps). 

Note:Before running the scripts again, backup the reports directory. Otherwise, new script runs will be writing to the same reports directory and they can get very confusing as to which logs are from which run.
Note: this version is only for item (and not category) workflow/collab areas.

Here are the associated cleanup procedures:
1)  basic_integriy_verify.sh/basic_integrity_restore.sh
2)  workflow_integrity_verify.sh/workflow_integrity_restore.sh
3)  serialization_integrity_verify.sh/serialization_integrity_restore.sh

To clean the corruptions and restore the database the following scripts can be run.

1)To clean  the above three types of corruptions in all the catalogs and collaboration areas in one company :
	Run fullIntegrityRestore.sh script from $TOP/src/integrity/oneCmp directory.
	The syntax for running this script:
	./fullIntegrityRestore.sh --code=cmpcode --basic<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 basic refers to the option to run basic_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_restore.sh. The options for this parameter are 'y' or 'n'
       
	For example if the users want to clean all the corruptions, the script can be run as
         ./fullIntegrityRestore.sh --code=trigo --basic=y --wf=y --serialization=y
        To clean  basic and serialization corruptions,
         ./fullIntegrityRestore.sh --code=trigo --basic=y --wf=n --serialization=y
		 
2)To clean the above three types of corruptions in a given collaboration area in one company :
	Run fullIntegrityRestore.sh script from $TOP/src/integrity/oneCmpOneCA directory.
	The syntax for running this script:
	./fullIntegrityRestore.sh --code=cmpcode --coaname=Collabareaname --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 coaname refers to Collaboration Area Name
	 basic refers to the option to run basic_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 
	 
3)To clean the above three types of corruptions in a given Catalog in one company :
	Run fullIntegrityRestore.sh script from $TOP/src/integrity/oneCmpOneCat directory.
	The syntax for running this script:
	./fullIntegrityRestore.sh --code=cmpcode --ctg=CatalogName --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 ctg refers to name of the catalog
	 basic refers to the option to run basic_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_restore.sh. The options for this parameter are 'y' or 'n'

Note : If there are any bad blobs in collaboration area, the items with bad blobs are purged. This can lead to ghost items. In that case, after running the fullIntegrityRestore.sh 
	 with serialization option "y", run the script again with wf option="y".
Reports by workflow_integrity_restore and serialization_integrity_restore are produced in two subdirectories of $TOP/logs:
badBlobsCleanings
ghostItemsCleanings



--------------------------------------------------------------------------------------------------------
III		Running Integrity Scripts for DB2
--------------------------------------------------------------------------------------------------------
DB2 version of the scripts can be run similar to the oracle version. 

1)To check for the three types of corruptions in all the catalogs and collaboration areas in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmp directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

       For example if the users want to check for all the corruptions, the script can be run as
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=y --serialization=y
        To check for basic and serialization corruptions,
         ./fullIntegrityVerify.sh --code=trigo --basic=y --wf=n --serialization=y

2)To check for the three types of corruptions in a given collaboration area in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCA directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --coaname=Collabareaname --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 coaname refers to Collaboration Area Name
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

3)To check for the three types of corruptions in a given Catalog in one company :
	Run fullIntegrityVerify.sh script from $TOP/src/integrity/oneCmpOneCat directory.
	The syntax for running this script:
	./fullIntegrityVerify.sh --code=cmpcode --ctg=CatalogName --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 ctg refers to name of the catalog
	 basic refers to the option to run basic_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_verify.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_verify.sh. The options for this parameter are 'y' or 'n'

The shell script fullIntegrityVerify.sh runs the basic_integrity_verify.sh, workflow_integrity_verify.sh 
and Serialization_integrity_verify.sh shell scripts in succession based on the user options.

The logs for workflow integrity scripts are under $TOP/logs/ghost_items_reports.
 
The logs for serialization scripts are under $TOP/logs/badBlobReports. Also, the entries with badblobs are written to the table cet_badblob. 
In the case of oneCmp, the serialization script verifies for bad serialized forms in Catalogs, Category Trees, CollaborationAreas, Lookup Tables. The consolidated report for all these entries in the company is stored under $TOP/logs/badBlobReports. The name of this report file starts with “Container_report_ALL_time stamp”. Along with this consolidated report, the console report files for each catalog can also be found in this directory.
For more details about the output from these scripts and interpreting them see : "Output - Details And Interpretation" below.

If you have questions about interpreting the output from these logs, contact IBM Support

contact IBM Support if you must run the clean up script.


To clean the corruptions and restore the database the following scripts can be run.

1)To clean  the above three types of corruptions in all the catalogs and collaboration areas in one company :
	Run fullIntegrityRestore.sh script from $TOP/src/integrity/oneCmp directory.
	The syntax for running this script:
	./fullIntegrityRestore.sh --code=cmpcode --basic<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 basic refers to the option to run basic_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_restore.sh. The options for this parameter are 'y' or 'n'
       
	For example if the users want to clean all the corruptions, the script can be run as
         ./fullIntegrityRestore.sh --code=trigo --basic=y --wf=y --serialization=y
        To clean  basic and serialization corruptions,
         ./fullIntegrityRestore.sh --code=trigo --basic=y --wf=n --serialization=y
		 
2)To clean the above three types of corruptions in a given collaboration area in one company :
	Run fullIntegrityRestore.sh script from $TOP/src/integrity/oneCmpOneCA directory.
	The syntax for running this script:
	./fullIntegrityRestore.sh --code=cmpcode --coaname=Collabareaname --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 coaname refers to Collaboration Area Name
	 basic refers to the option to run basic_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 
	 
3)To clean the above three types of corruptions in a given Catalog in one company :
	Run fullIntegrityRestore.sh script from $TOP/src/integrity/oneCmpOneCat directory.
	The syntax for running this script:
	./fullIntegrityRestore.sh --code=cmpcode --ctg=CatalogName --basic=<y/n> --wf=<y/n> --serialization=<y/n>
	 cmpcode refers to the company code
	 ctg refers to name of the catalog
	 basic refers to the option to run basic_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 wf refers to the option to run workflow_integrity_restore.sh. The options for this parameter are 'y' or 'n'
	 serialization to the option to run serialization_integrity_restore.sh. The options for this parameter are 'y' or 'n'


 Output files for workflow corruption cleanup are stored in $TOP/logs/ghost_items_cleanings directory.
   There are 3 output files for each collaboration area :
   1.report_<timestamp>_<collaborationareaId>_before_cleanup.txt
   2.report_<timestamp>_<collaborationareaId>_cleanup.txt
   3.report_<timestamp>_<collaborationareaId>_after_cleanup.txt
   
  Output files for serialization corruption cleanup are stored in $TOP/logs/badBlobsCleanings directory.
  These reports come in pairs - one console report and another detailed report.

   The reports should be sent to IBM Support for analysis if errors are found.


Please exercise utmost caution and discretion before using the information in these reports to modify your database.If you have questions about that, contact IBM Support.

-------------------------------------------------------------------------
END OF README
-------------------------------------------------------------------------



