Platform Analytics 7 Dataflow

WI_JOBRESUSAGEBYGROUP table

This describes the wi_jobresusagebygroup table and how each column of data arises from row table.
The wi_jobresusagebygroup get its data from lsb_events table.
For each record we get, we will filter them into different grouping. There are seven grouping as Cluster, User, UserGroup, Project, Queue, JobName and JobGroup. For each grouping, there are five resources subgroup as "Memory", "LICUSAGE", "Slot", "Swap" and "Slot Usage". We then aggregate the records across the hour and inserted the result into this table.
For the "By Host", we get the individual host from the "ExecHostList" field. The field has a format of "1*hostname1 2*hostname2 ...". The first number is the slot number for the hostname. The total slot is calculated by adding all the available slot for each host together. For each host, a proportion is calculated based on the slot number it has and divide that by the TotSlot. This faction is then used to multiple for "Swap", "Memory", "Slot" and "CpuDelta" for the host usage. The LicenseUsage is calculated by getting each license from the "LicenseUsageStr" which has a formation of "License1:3 License2:2..." and then aggregated them into this table. If there is no license being used which is indicated by having "LicenseUsageStr" = "NO_LICENSE_USAGE".
For "By Queue", the slot limit comes from the "QueueJobLimit" field and the slot usage comes from the "Slot" field.
This is the column description of each data column of WI_JOBRESUSAGEBYGROUP and how each column is filled with data.
Column Name
Description
Key
CLUSTER_CODE
This comes from the "ClusterName" field in the data file. We split the "Project" field into subfields and one of them is the cluster_name. Once we get the cluster_name, we then look it up in the wi_clustercode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_clustercode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.
Primary key
TIME_STAMP
This is in GMT and is always end in the hour such as 02:00:00. This marks the time that this record is aggregated. For example, 02:00:00 means that all the records between 02:00:00 and 02:59:00 are aggregated into this record.
Primary key
GROUP_TYPE
This indicates what kind of grouping this record belongs to. The different group type are: "By Host", "By Jobname", "By JobGroup", "By Project", "By Queue", "By User", "By UserGroup" and "By Cluster".
Primary key
GROUP_CODE
This could comes from different source and it is depends on the group_type. If the group_type is "By Host", then it is one of the host name that comes from a subfield in the "Exechostlist" from the data file. The subfield contains the hostname and is being used here. If it is "By JobName", then it comes from the "JobName" field. If it is "By Project", then it comes from the "ProjectName". If it is "By Queue", then it comes from the "QueueName". If it is "By User", then it comes from the "UserName". If it is "By UserGroup", then it comes from the "UserGroupName". If it is "By Cluster", then it comes from the "ClusterName". Once we get the name, we then look it up in the wi_dimensioncode table to see if we have already has a record of it in there by matching the dimension_name. If we do, then we'll get the code back, otherwise, we will insert it into the wi_dimensioncode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.
Primary key
JOB_TYPE
This comes from the "JobType" field in the data file.
For the slot aggregation for all job type, this field will be set with "ALL JOB TYPE".
Primary key
RESOURCE_CODE
This depends on the sub grouping. It could be one of the following subgroups: "Memory", "Cputime", the actual license name that the job checked out, "Slot" and "Swap". Once we get the name, we then look it up in the wi_resourcecode table to see if we have already has a record of it in there by matching the resource_name. If we do, then we'll get the code back, otherwise, we will insert it into the wi_resourcecode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.
Primary key
TOTAL_RES_USAGE
This field is calculated in two phases. First, the data of the same sampling period are aggregated together based on "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType" and "GroupCode" as group by columns and we then sum up the value field based on the resource. For example, if the resource is "Memory", then the value field is the sum of the memory usage of the job with the same sampling interval. The second aggregation will then based on the grouping of "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType", "GroupCode" across the hour. A second record, without the jobtype in the grouping for the slot usage hourly aggregation, is produced for all job type. The total is calculated as the sum of all the data from the phase one calculation.

MIN_RES_USAGE
This field is calculated in two phases. First, the data of the same sampling period are aggregated together based on "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType" and "GroupCode" as group by columns and we then sum up the value field based on the resource. For example, if the resource is "Memory", then the value field is the sum of the memory usage of the job with the same sampling interval. The second aggregation will then based on the grouping of "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType", "GroupCode" across the hour. A second record, without the jobtype in the grouping for the slot usage hourly aggregation, is produced for all job type. The minimum of this value is calculated as the minimum of this value from the results of phase one.

MAX_RES_USAGE
This field is calculated in two phases. First, the data of the same sampling period are aggregated together based on "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType" and "GroupCode" as group by columns and we then sum up the value field based on the resource. For example, if the resource is "Memory", then the value field is the sum of the memory usage of the job with the same sampling interval. The second aggregation will then based on the grouping of "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType", "GroupCode" across the hour. A second record, without the jobtype in the grouping for the slot usage hourly aggregation, is produced for all job type. The maximum of this value is calculated as the maximum of this value from the results of phase one.

COUNTER
This field is calculated in two phases. First, the data of the same sampling period are aggregated together based on "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType" and "GroupCode" as group by columns and we then treat this as 1 record. For example, if the resource is "Memory", then the result of the first aggregation, we'll treat that as 1 record. The second aggregation will then based on the grouping of "ClusterCode", "GroupBy", "TimeDiff", "ResourceCode", "JobType", "GroupCode" across the hour. A second record, without the jobtype in the grouping for the slot usage hourly aggregation, is produced for all job type. The count will be the number of records produced in phase one.

LOCAL_SERVERTIME
This comes from the "Time_stamp" in this table. This is transformed into local server time.

INSERT_SEQ
This is a system generated sequence number. For each new record inserted, an unique sequence number is being assigned to this column.