Platform Analytics 7 Dataflow

Previous Topic Next Topic Index

WI_JOBMARTDAILY table

This describes the wi_jobmartdaily table. The data in wi_jobmartdaily table is derived from the daily aggregation of wi_jobmart data. The range of aggregation is between the last date in this table and the midnight of the most recent inserted record in the wi_jobmart for each cluster.

The data is extracted for each distinct cluster in wi_jobmart and aggregated for the localized day grouped by "Cluster", "ExecHost", "SubHost", "ExecHostModel", "ExecHostType", "User", "Project", "Queue", "NumExecProcs", "JobExitStatus", "JobExitCode", "QueueTime", "StartTime", "Timediff_hour", "CoreHour", "Pending Rank", "Job_type", and "UserGroup". We sum up the "CpuTime", "Mem_Usage", "Swap_Usage", "Run_time", "Pending_Time" and also counted the number of records being aggregated.

This is the column description of each data column of WI_JOBMARTDAILY and how each column is filled with data.

Columns

Description

Key

CLUSTER_CODE

This comes from the "ClusterCode" field in the wi_jobmart.

Primary key

QUEUE_TIME

This comes from the "Queue_time" field in the wi_jobmart. The "Queue_time" is transformed to local time and then truncated to the date (ie, set the hour part to midnight) and then transformed back to GMT time.

Primary key

START_TIME

This comes from the "Start_time" field in the wi_jobmart. The "Start_time" is transformed to local time and then truncated to the date (ie, set the hour part to midnight) and then transformed back to GMT time.

Primary key

FINISH_TIME

This comes from the "Finish_time" field in the wi_jobmart. The "Finish_time" is transformed to local time and then truncated to the date (ie, set the hour part to midnight) and then transformed back to GMT time.

Primary key

USER_CODE

This comes from the "User_Code" field in the wi_jobmart table.

Primary key

EXECHOST_CODE

This comes from the "ExecHost_Code" field in the wi_jobmart table. Then it go through a transformation to lookup the host_name in the wi_hostcode table. After that, it will go through a mapping process which based on user definition of the host_name. After that, it goes through the name to code process again and insert a new record if the code cannot be looked up in the wi_exechostcode table.

Primary key

SUBHOST_CODE

This comes from the "SubHost_Code" field in the wi_jobmart table. Then it go through a transformation to lookup the host_name in the wi_hostcode table. After that, it will go through a mapping process which based on user definition of the host_name. After that, it goes through the name to code process again and insert a new record if the code cannot be looked up in the wi_subhostcode table.

Primary key

 

QUEUE_CODE

This comes from the "Queue_Code" field in the wi_jobmart table.

Primary key

 

PROJECT_CODE

This comes from the "Project_Code" field in the wi_jobmart table.

Primary key

 

JOB_TYPE

This comes from the "Job_Type" field in the wi_jobmart table.

Primary key

NUM_EXEC_PROCS

This comes from the "Num_exec_procs" field in the wi_jobmart table.

Primary key

JOB_EXIT_STATUS

This comes from the "Job_exit_status" field in the wi_jobmart table.

Primary key

JOB_EXIT_CODE

This comes from the "Job_exit_code" field in the wi_jobmart table.

Primary key

RUN_TIME

This comes from the aggregation as a sum of the "Run_Time"

 

PENDING_TIME

This comes from the aggregation as a sum of the "Pending_Time"

 

CPU_TIME

This comes from the aggregation as a sum of the "Cpu_Time"

 

MEM_USAGE

This comes from the aggregation as a sum of the "Mem_usage"

 

SWAP_USAGE

This comes from the aggregation as a sum of the "Swap_usage"

 

TIMEDIFF_HOUR

This comes from the "Timediff_hour" field in the wi_jobmart table.

Primary key

EXECHOSTMODEL_CODE

This is the result of the lookup of the LSFHOST_MODEL in the wi_HOSTPROPERTIES table. The lookup is based on the hostname lookup in the wi_hostcode table based on the EXECHOST_CODE column in the wi_jobmart table. Then the looked up LSFHOST_MODEL will be checked whether it is null or not. If it is null, then set it as "UNKNOWN". Once we get the LSFHOST_MODEL, we then look it up in the wi_hostmodelcode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_hostmodelcode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.

Primary key

EXECHOSTTYPE_CODE

This is the result of the lookup of the LSFHOST_TYPE in the wi_HOSTPROPERTIES table. The lookup is based on the hostname lookup in the wi_hostcode table based on the EXECHOST_CODE column in the wi_jobmart table. If it is null, then set it as "UNKNOWN". Once we get the LSFHOST_TYPE, we then look it up in the wi_hosttypecode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_hosttypecode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.

Primary key

USER_GROUP_CODE

This comes from the "user_group_code" in the wi_jobmart table.

Primary key

PENDINGRANK_CODE

This comes from the sum of the pending_time of the wi_jobmart table after the aggregation.

Then it rank the pending time according to the following logic:

if (pendingTime == null) {

return "Not available";

} else if (pendingTime < 5/60) {

return "0 sec to < 5 sec";

} else if (pendingTime < 30/60) {

return "5 sec to < 30 sec";

} else if (pendingTime < 5) {

return "30 sec to < 5 min";

} else if (pendingTime < 60) {

return "5 min to < 60 min";

} else if (pendingTime < 1440) {

return "60 min to < 1 day";

} else {

return "One day or more";

}

Once we get the pending_rank, we then look it up in the wi_pendingrankcode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_pendingrankcode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.

Primary key

RUNTIMERANK_CODE

This is the code of job runtime rank. This is the result of the lookup of the RUNTIMERANK_NAME in the WI_RUNTIMERANKCODE table.

Primary key

MEMUSAGERANK_CODE

This is the code of job mem usage rank. This is the result of the lookup of the MEMUSAGERANK_NAME in the WI_RUNTIMERANKCODE table.

Primary key

CPUTIMERANK_CODE

This is the code of CPU time rank. This is the result of the lookup of the CPUTIMERANK_NAME in the WI_CPUTIMERANKCODE table.

Primary key

PENDINGRANK_IDX

This is the index of job pending rank orderred by rank values. The bigger index has higher rank.

Primary key

RUNTIMERANK_IDX

This is the index of job runtime rank orderred by rank values. The bigger index has higher rank.

Primary key

MEMUSAGERANK_IDX

This is the index of job mem usage rank orderred by rank values. The bigger index has higher rank.

Primary key

CPUTIMERANK_IDX

This is the index of CPU time rank orderred by rank values. The bigger index has higher rank.

 

NUM_OF_JOB

This is the number of jobs being aggregated into this record.

 

CORE_HOUR

The core hour is resulted from a mapping of the original time_stamp in local time. If the time_stamp is within the range from Monday to Friday, 9am to 5pm, then the core_hour will be 1, otherwise it will be 0.

Primary key

JOB_GROUP_CODE

A code representing the job group. The source is the job_group field in the wi_jobgroupcode table. If the job_group code already exists, the existing code is used; otherwise, the new group code is auto-incremented and inserted into the wi_jobgroupcode table.

Primary key

NPROCS

This is the number of physical processors per host.

 

NCORES

This is the number of cores per physical processor.

 

NTHREADS

This is the number of threads per core

 

APPLICATION_CODE

This is the code of job application tag. This is the result of the lookup of the APPLICATION_TAG in the WI_APPLICATIONCODE table.

 

INSERT_SEQ

This is a system generated sequence number. For each new record inserted, an unique sequence number is being assigned to this column.

 

Previous Topic Next Topic Index