Platform Analytics 7 Dataflow

WI_JOBSLOTUSAGEBYGROUP table

This describes the wi_jobslotusagebygroup table and how each column of data arises from LSB_EVENTS table.
The wi_jobslotusagebygroup get its data from Data Collection Table LSB_EVENTS.
For each record we get, we will filter them into different grouping. There are seven grouping as Cluster, User, UserGroup, Project, Queue, JobName and JobGroup. For each grouping, there are five resources subgroup as "Memory", "LICUSAGE", "Slot", "Swap" and "Slot Usage". For the "Slot Usage", we will calculate the usage and if the Group_By is of "By Host" or "By Queue", then we will also calculate utilization for the hour and put it into this table.
For the "By Host", we get the individual host from the "ExecHostList" field. The field has a format of "1*hostname1 2*hostname2 ...". The first number is the slot number for the hostname. The total slot is calculated by adding all the available slot for each host together. For each host, a proportion is calculated based on the slot number it has and divide that by the TotSlot. This faction is then used to multiple the "Slot" field to calculate the slot usage for this host.
The slot utilization is calculated by first getting the slot limit which is extracted from the "HostJobLimit" field. The field has a format of "hostname1=1 hostname2=2 ...". For each hostname, we do a match and extract the slot limit for each one of them. Then, we sum the Slot usage, the Slot limit and the Slot number for the host records from the same interval with the same cluster, "GroupBy", "Timediff" and "Host". We also do a count to see how many are summed. Then, we calculate the utilization as the "Sum of Usage" divided by ("Sum of slot limit"/ "No of record summed"). If the result is bigger then 1, then we return 1. If the "Sum of slot limit" is > 9999999, or "Sum of slot limit" is equal to zero or the result is equal to zero and the "Sum of slot usage" not equal to zero, then return -1 which indicates to the cube that the slot utilization is not calculated.
For "By Queue", the slot limit comes from the "QueueJobLimit" field and the slot usage comes from the "Slot" field.
This is the column description of each data column of WI_JOBSLOTUSAGEBYGROUP and how each column is filled with data.
Column Name
Description
Key
CLUSTER_CODE
This comes from the CLUSTER_NAME field in the raw table. Once we get the cluster_name, we then look it up in the wi_clustercode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_clustercode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the sequence number +1.
Primary key
TIME_STAMP
This is in GMT and is always end in the hour such as 02:00:00. This marks the time that this record is aggregated. For example, 02:00:00 means that all the records between 02:00:00 and 02:59:00 are aggregated into this record.
Primary key
GROUP_TYPE
This indicates what kind of grouping this record belongs to. The different group type are: "By Host", "By Jobname", "By JobGroup", "By Project", "By Queue", "By User", "By UserGroup" and "By Cluster".
Primary key
GROUP_CODE
This could comes from different source and it is depends on the group_type. If the group_type is "By Host", then it is one of the host name that comes from a subfield in the "Exechostlist" from the data file. The subfield contains the hostname and is being used here. If it is "By JobName", then it comes from the "JobName" field. If it is "By Project", then it comes from the "ProjectName". If it is "By Queue", then it comes from the "QueueName". If it is "By User", then it comes from the "UserName". If it is "By UserGroup", then it comes from the "UserGroupName". If it is "By Cluster", then it comes from the "ClusterName". Once we get the name, we then look it up in the wi_dimensioncode table to see if we have already has a record of it in there by matching the dimension_name. If we do, then we'll get the code back, otherwise, we will insert it into the wi_dimensioncode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.
Primary key
MIN_SLOT_USAGE
If the "GroupBy" is "By Host", then for each host, we calculate the slot usage as describe in the beginning of this page. Then, we aggregate them together based on the same "Cluster", "GroupBy", "TimeDiff" and "HostCode". For the others, we just group by "Cluster", "GroupBy", "TimeDiff" and "GroupCode" and the usage comes from the "Slot" field in the data file. We find the minimum of the slot usage for the hour and put it into this field.

MAX_SLOT_USAGE
If the "GroupBy" is "By Host", then for each host, we calculate the slot usage as describe in the beginning of this page. Then, we aggregate them together based on the same "Cluster", "GroupBy", "TimeDiff" and "HostCode". For the others, we just group by "Cluster", "GroupBy", "TimeDiff" and "GroupCode" and the usage comes from the "Slot" field in the data file. We find the maximum of the slot usage for the hour and put it into this field.

TOTAL_SLOT_USAGE
If the "GroupBy" is "By Host", then for each host, we calculate the slot usage as describe in the beginning of this page. Then, we aggregate them together based on the same "Cluster", "GroupBy", "TimeDiff" and "HostCode".
For the others, we just group by "Cluster", "GroupBy", "TimeDiff" and "GroupCode" and the usage comes from the "Slot" field in the data file. We find the sum of the slot usage for the hour and put it into this field.

COUNTER
This is the number of records being aggregated for the hour for the slot usage calculation. The records itself is first aggregated across the same interval before aggregated again for the hour.

MIN_SLOT_UTIL
If the "GroupBy" is "By Host", then for each host, we calculate the slot utilization as describe in the beginning of this page. Then, we aggregate them together based on the same "Cluster", "GroupBy", "TimeDiff" and "HostCode". For the "By Queue", we just group by "Cluster", "GroupBy", "TimeDiff" and "GroupCode" and the usage comes from the "Slot" field in the data file. We find the minimum of the slot utilization for the hour and put it into this field. For the other "GroupBy", we will not calculate utilization and will not insert any value into this field.

MAX_SLOT_UTIL
If the "GroupBy" is "By Host", then for each host, we calculate the slot utilization as describe in the beginning of this page. Then, we aggregate them together based on the same "Cluster", "GroupBy", "TimeDiff" and "HostCode". For the "By Queue", we just group by "Cluster", "GroupBy", "TimeDiff" and "GroupCode" and the usage comes from the "Slot" field in the data file. We find the maximum of the slot utilization for the hour and put it into this field. For the other "GroupBy", we will not calculate utilization and will not insert any value into this field.

AVG_SLOT_UTIL
If the "GroupBy" is "By Host", then for each host, we calculate the slot utilization as describe in the beginning of this page. Then, we aggregate them together based on the same "Cluster", "GroupBy", "TimeDiff" and "HostCode". For the "By Queue", we just group by "Cluster", "GroupBy", "TimeDiff" and "GroupCode" and the usage comes from the "Slot" field in the data file. We calculate the average of the slot utilization for each interval and then average them across the hour. For the other "GroupBy", we will not calculate utilization and will not insert any value into this field.

COUNTER_UTIL
This is the number of records being aggregated for the hour for the slot usage calculation. The records itself is first aggregated across the same interval before aggregated again for the hour. If the Min_slot_util is equal to -1 (meaning no slot utilization), then this will be zero. For the other "GroupBy", we will not calculate utilization and will not insert any value into this field.

LOCAL_SERVERTIME
This comes from the "Time_stamp" in this table. This is transformed into local server time.

INSERT_SEQ
This is a system generated sequence number. For each new record inserted, an unique sequence number is being assigned to this column.