WPC Implementation Analyzer
---------------------------

I. Introduction

This tool was developed to have a good sense about the size and complexity of the Data Model within WPC. It analyzes the WPC db schema and provides detailed information about the data model, like number of specs vs attributes per specs vs. specs per items vs. rules per spec etc. The tool automatically collects the following pertinent parameters or metadata definitions and computes totals and distributed averages for the given set of groups.

     1  Attributes per PrimarySpec
     2  Rules per PrimarySpec
     3  Strings per PrimarySpec
     4  Flags per PrimarySpec
     5  Numerals per PrimarySpec
     6  Attributes per SecondarySpec
     7  Rules per SecondarySpec
     8  Strings per SecondarySpec
     9  Flags per SecondarySpec
    10  Numerals per SecondarySpec
    11  Size per Script
    12  Categories per Hierarchy
    13  Depth per Hierarchy
    14  Categories per Depth
    15  SecondarySpecs per Category
    16  PopulatedAttributes per Category
    17  StandAloneSpecs per Category
    18  Attributes per Category
    19  DataSize per Category
    20  CategoryTrees per Catalog
    21  Views per Catalog
    22  Specs per Items
    23  Attributes per Item
    24  Items per Catalog
    25  Items per Category
    26  Categorizations per Item
    27  LocationAttributes per Item
    28  DataSize per Item
    29  Entries per LookupTable
    30  Attributes per AttributeCollection

II. Distribution Averages

Instead of just reporting the average number of Xs per Y, this tool reports the distributions of Xs in groups of Ys. This could be achieved by counting the number of Xs in Ys, sort by counts, divide the results in n groups (n=1 for a simple average, n=4 for average per quartiles, n=10 for average per deciles etc) and averaging up the counts of Xs per Y in each group.

For example consider the following information on Items per catalog:
                        Total ITEMS per catalog
Catalog id = 622           211229
Catalog id = 630           68642 
Catalog id = 620           48829 
Catalog id = 627           38229 
Catalog id = 594           4265  
Catalog id = 596           2111  
Catalog id = 629           887   
Catalog id = 624           772   
Catalog id = 588           759   
Catalog id = 601           562   

From the above data the average items per catalog (376285/10) = 37,625

Now consider splitting the average in 3 groups as below - then the averages are Average of Items per Catalog in 3 groups = 109567,14868,745
Number of catalogs in each of the 3 groups = 10/3 = 3.33 ~ 3  

211229  \
68642    average of 3 largest catalogs = 109,567 
48829   /
--------
38229   \               
4265     average of next 3 largest catalogs = 14,868
2111    /    
--------           
887     \               
772      average of last group of catalogs (smallest) = 745
759      /               
562     /
--------
Similarly the Average of Items per Catalog in 5 groups = 139936,43429,3188,830,661
Here the number of catalogs in a group (group size) = 10/5 = 2

211229  \
68642   / average of 2 largest catalogs = 139,936
-------- 
48829  \
38229  /  average of next 2 largest catalogs = 43,429
-------
4265   \
2111   /  average of next 2 largest catalogs = 3,188
-------
887    \
772    /  average of next 2 largest catalogs = 830
-------
759    \
562    /  average of last group of catalogs (smallest) = 661
-------

Tool provides the above distribution averages for all the definitions listed in the introduction section.

III. Tool Design and usage options

Tool has 2 shell scripts:
1. setup_wpcAnalyzer.sh - this script is for the initial set up of analyzer script
2. wpcAnalyzer_db2.sh (wpcAnalyzer_oracle.sh for Oracle)

wpcAnalyzer script has 2 parts
1. Statistics Collection: Collect all the statistics using sqls and insert into a table - METADATA_STATISTICS table. This is the MOST time and resource intersive operation.
2. Average Calculation: Compute distribution averages for the number of groups provided. For readability purposes, the number of groups is restricted between 1 and 20.

Tool is designed for collect statistics once and slice data for averages multiple times. Users can provide the different group number for a statistics collected on a specific rundate.

wpcAnalyzer script accepts company code, number of groups and either a definition file OR an existing rundate. Tool uses the db username, password and connection information from $TOP/etc/default/common.properties file. If definition file is provided, statistics is always collected then distribution averages are calculated for given number of groups. If an existing rundate is provided, it is verified and ONLY distribution averages are computed, statistics collection is skipped.

Tool writes out all the collected information into a csv file. CSV report with groups greater than 1 is considered the complex report. The group number indicates the complexity level of the report.

setup_wpcAnalyzer.sh scripts does the initial setup and retrieves useful information. It has following usages

Usage 1: setup_wpcAnalyzer.sh --option=recreate

This option creates the METADATA_STATISTICS table. Optionally --data_ts and --index_ts can be provided for place the table in custom tablespaces. By default the table is created in USERS and INDX tablespaces.

Usage 2: setup_wpcAnalyzer.sh --option=info

This option retrieves list of existing rundates from METADATA_STATISTICS table that is to be used with wpcAnalyzer.sh script's rundate option for slicing data into different groups.

Usage 3: setup_wpcAnalyzer.sh --option=delete --date=YYYY-MM-DD

This option deletes the rows in METADATA_STATISTICS table based on the provided date. This is for maintenance purposes.

III. Steps for running and maintenance 

1. Run setup_wpcAnalyzer.sh with option recreate (FIRST TIME ONLY)
2. Run wpcAnalyzer_db2 or wpcAnalyzer_oracle with valid company code, groups and definition file

For the next runs, for retrieving info for different group number for existing data:
1. Retrieve rundate from setup_wpcAnalyzer.sh with option info
2. Run wpcAnalyzer with rundate option for the rundate retrieved

IV. PRECAUTIONS

1. During statistics collection phase, tool places a heavy load on the db server. WPC should be shutdown and no wpc jobs should be running. Unless there is significant data addition, DON'T use definition file option for second time.

2. The table holding the statistics becomes very large and considering deleting it periodically. It's advised not to have more than 1 or 2 runs worth of statistics data for space considerations on the db.  

3. The following metadata definitions, collects statistics for ITEM again and again for different definitionis and as such they are time consuming and resource intensive. If you have more than 2 Million items, please plan and allocate time for running the script or comment out the following definitions in definition file (defn option).
 AttributesPerItem                 
 CategorizationsPerItem           
 DataSizePerItem                 
 LocationAttributesPerItem     
 SpecsPerItems                 

V. LIMITATIONS

1. In some cases, the average computation of defintions - Attributes per Item and Location Attributes Per Item may not be exact.  
