Platform Analytics 7 Dataflow

Previous Topic Next Topic Index

WI_JOBMART_HPC_DAILY table

This describes the wi_jobmartdaily_hpc table. The data in wi_jobmartdaily_hpc table is derived from the daily aggregation of wi_jobmart_hpc data. The range of aggregation is between the last date in this table and the midnight of the most recent inserted record in the wi_jobmart_hpc_daily for each cluster.

The data is extracted for each distinct cluster in wi_jobmart_hpc and aggregated for the localized day grouped by "Cluster", "ExecHost", "SubHost", "ExecHostModel", "ExecHostType", "User", "Project", "Queue", "NumExecProcs", "JobExitStatus", "JobExitCode", "QueueTime", "StartTime", "Timediff_hour", "CoreHour", "Pending Rank", "Job_type", and "UserGroup". We sum up the "CpuTime", "Mem_Usage", "Swap_Usage", "Run_time", "Pending_Time" and also counted the number of records being aggregated.

This is the column description of each data column of WI_JOBMART_HPC_DAILY and how each column is filled with data.

Columns

Description

Key

CLUSTER_CODE

This comes from the "ClusterCode" field in the wi_jobmart.

Primary key

QUEUE_TIME

This comes from the "Queue_time" field in the wi_jobmart. The "Queue_time" is transformed to local time and then truncated to the date (ie, set the hour part to midnight) and then transformed back to GMT time.

Primary key

START_TIME

This comes from the "Start_time" field in the wi_jobmart. The "Start_time" is transformed to local time and then truncated to the date (ie, set the hour part to midnight) and then transformed back to GMT time.

Primary key

FINISH_TIME

This comes from the "Finish_time" field in the wi_jobmart. The "Finish_time" is transformed to local time and then truncated to the date (ie, set the hour part to midnight) and then transformed back to GMT time.

Primary key

USER_CODE

This comes from the "User_Code" field in the wi_jobmart table.

Primary key

EXECHOST_CODE

This comes from the "ExecHost_Code" field in the wi_jobmart table. Then it go through a transformation to lookup the host_name in the wi_hostcode table. After that, it will go through a mapping process which based on user definition of the host_name. After that, it goes through the name to code process again and insert a new record if the code cannot be looked up in the wi_exechostcode table.

Primary key

SUBHOST_CODE

This comes from the "SubHost_Code" field in the wi_jobmart table. Then it go through a transformation to lookup the host_name in the wi_hostcode table. After that, it will go through a mapping process which based on user definition of the host_name. After that, it goes through the name to code process again and insert a new record if the code cannot be looked up in the wi_subhostcode table.

 

QUEUE_CODE

This comes from the "Queue_Code" field in the wi_jobmart table.

 

PROJECT_CODE

This comes from the "Project_Code" field in the wi_jobmart table.

 

JOB_TYPE

This comes from the "Job_Type" field in the wi_jobmart table.

Primary key

NUM_EXEC_PROCS

This comes from the "Num_exec_procs" field in the wi_jobmart table.

Primary key

JOB_EXIT_STATUS

This comes from the "Job_exit_status" field in the wi_jobmart table.

Primary key

JOB_EXIT_CODE

This comes from the "Job_exit_code" field in the wi_jobmart table.

Primary key

RUN_TIME

This comes from the aggregation as a sum of the "Run_Time"

 

PENDING_TIME

This comes from the aggregation as a sum of the "Pending_Time"

 

CPU_TIME

This comes from the aggregation as a sum of the "Cpu_Time"

 

MEM_USAGE

This comes from the aggregation as a sum of the "Mem_usage"

 

SWAP_USAGE

This comes from the aggregation as a sum of the "Swap_usage"

 

TIMEDIFF_HOUR

This comes from the "Timediff_hour" field in the wi_jobmart table.

Primary key

EXECHOSTMODEL_CODE

This is the result of the lookup of the LSFHOST_MODEL in the wi_HOSTPROPERTIES table. The lookup is based on the hostname lookup in the wi_hostcode table based on the EXECHOST_CODE column in the wi_jobmart table. Then the looked up LSFHOST_MODEL will be checked whether it is null or not. If it is null, then set it as "UNKNOWN". Once we get the LSFHOST_MODEL, we then look it up in the wi_hostmodelcode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_hostmodelcode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.

Primary key

EXECHOSTTYPE_CODE

This is the result of the lookup of the LSFHOST_TYPE in the wi_HOSTPROPERTIES table. The lookup is based on the hostname lookup in the wi_hostcode table based on the EXECHOST_CODE column in the wi_jobmart table. If it is null, then set it as "UNKNOWN". Once we get the LSFHOST_TYPE, we then look it up in the wi_hosttypecode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_hosttypecode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.

Primary key

USER_GROUP_CODE

This comes from the "user_group_code" in the wi_jobmart table.

Primary key

PENDINGRANK_CODE

This comes from the sum of the pending_time of the wi_jobmart table after the aggregation.

Then it rank the pending time according to the following logic:

if (pendingTime == null) {

return "Not available";

} else if (pendingTime < 5/60) {

return "0 sec to < 5 sec";

} else if (pendingTime < 30/60) {

return "5 sec to < 30 sec";

} else if (pendingTime < 5) {

return "30 sec to < 5 min";

} else if (pendingTime < 60) {

return "5 min to < 60 min";

} else if (pendingTime < 1440) {

return "60 min to < 1 day";

} else {

return "One day or more";

}

Once we get the pending_rank, we then look it up in the wi_pendingrankcode table to see if we have already has a record of it in there. If we do, then we'll get the code back, otherwise, we will insert it into the wi_pendingrankcode table and generate the code. The code itself is a positive integer and each new code is equal to the maximum of the existing code+1.

Primary key

RUNTIMERANK_CODE

 

Primary key

MEMUSAGERANK_CODE

 

Primary key

NUM_OF_JOB

This is the number of jobs being aggregated into this record.

 

CORE_HOUR

The core hour is resulted from a mapping of the original time_stamp in local time. If the time_stamp is within the range from Monday to Friday, 9am to 5pm, then the core_hour will be 1, otherwise it will be 0.

Primary key

NPROCS

This is the number of physical processors per host.

 

NCORES

This is the number of cores per physical processor.

 

NTHREADS

This is the number of threads per core.

 

APPLICATION_CODE

This is the code of job application tag. This is the result of the lookup of the APPLICATION_TAG in the WI_APPLICATIONCODE table.

 

INSERT_SEQ

This is a system generated sequence number. For each new record inserted, an unique sequence number is being assigned to this column.

 

Previous Topic Next Topic Index