Jobsub ID 14065.81@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.81@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | UK_RAL-Tier1 | |
| Entry | LIGO_UK_RAL_arc_ce05 | |
| Last heartbeat | 2025-08-02 23:09:26 | |
| From worker node | Hostname | dune001-7093504.0-lcg2625.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:39:23 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 29m (1747s) | |
| CPU time | 10m (602s = 34%) | |
| Max RSS bytes | 1929027584 (1839 MiB) | |
| Outputting started | 2025-08-02 23:08:32 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223935Z_3_ana_tree_hd.root | |
| Finished | 2025-08-02 23:09:21 | |
| Saved logs | justin-logs:14065.81-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
oHitListU_graph.data -> graph_output_2025-08-02T_223935Z_3_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223935Z_3_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223935Z_3_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223935Z_3_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223935Z_3_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223935Z_3_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223935Z_3_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223935Z_3_ana_tree_hd.root === Start last 100 lines of lar log file === \_ MCPDG 2212, Energy 0.981048, Dist. 1.76174, nMCHits 7 (4, 3, 0) \_ MCPDG -211, Energy 0.73145, Dist. 36.6862, nMCHits 96 (56, 24, 16) \_ MCPDG 1000010020, Energy 1.93357, Dist. 1.76144, nMCHits 8 (3, 3, 2) \_ MCPDG 211, Energy 0.225962, Dist. 20.3386, nMCHits 62 (26, 11, 25) \_ MCPDG 1000180360, Energy 33.496, Dist. 0.000279698, nMCHits 6 (3, 1, 2) --Primary 3, MCPDG -211, Energy 1.79128, Dist. 23.0772, nMCHits 156 (83, 46, 27) MCPDG -211, Energy 1.79128, Dist. 23.0772, nMCHits 44 (30, 8, 6) \_ MCPDG -211, Energy 0.614218, Dist. 41.5481, nMCHits 112 (53, 38, 21) --Primary 4, MCPDG 211, Energy 2.94059, Dist. 44.8755, nMCHits 78 (59, 6, 13) MCPDG 211, Energy 2.94059, Dist. 44.8755, nMCHits 78 (59, 6, 13) --Primary 5, MCPDG -211, Energy 1.63093, Dist. 43.3066, nMCHits 60 (29, 19, 12) MCPDG -211, Energy 1.63093, Dist. 43.3066, nMCHits 60 (29, 19, 12) --Primary 6, MCPDG 22, Energy 0.964932, Dist. 26.1476, nMCHits 46 (17, 19, 10) MCPDG 22, Energy 0.964932, Dist. 26.1476, nMCHits 46 (17, 19, 10) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 97 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1113744 bytes. The eid is 97 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1454120 bytes. The eid is 97 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1382784 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1113420 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1453812 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1358524 bytes. Boundary wire vector sizes: 1100, 1320, 845 minwire 0: 700 minwire 1: 1009 minwire 2: 104 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0324425, Output 1: 0.0348561, 0.0479417, 0.0897093, 0.827493, Output 2: 0.194081, 0.244083, 0.536302, 0.0255338, Output 3: 0.184304, 0.142679, 0.30635, 0.366667, Output 4: 0.0682905, 0.224847, 0.288186, 0.418676, Output 5: 0.809489, 0.182595, 0.00617706, 0.00173844, Output 6: 0.129626, 0.372497, 0.204722, 0.293156, Running Ophitfinder with InputDigiType = 'recob' Found hits: 336! 02-Aug-2025 23:03:14 UTC Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2_graph_2025-08-02T_223935Z.root" 02-Aug-2025 23:03:14 UTC Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8a/9c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.638662 4.81797 18.3168 4.73862 2.4889 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0291687 0.0560904 0.0914663 0.0566969 0.0187092 100 reco:gaushit:GausHitFinder 0.102908 0.190431 1.31993 0.139687 0.159553 100 reco:spsolve:SpacePointSolver 0.000157864 0.115798 4.86593 0.00819674 0.517876 100 reco:hitfd:DisambigFromSpacePoints 0.000292827 0.0203717 0.669675 0.00230616 0.0758983 100 reco:rns:RandomNumberSaver 2.3371e-05 5.12819e-05 0.000378478 4.17605e-05 3.91243e-05 100 reco:pandora:StandardPandora 0.0966112 1.8157 10.4038 1.57526 1.19952 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000133223 0.000311874 0.00478138 0.000247075 0.000458706 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000164954 0.000371295 0.00644937 0.000299672 0.000615305 100 reco:pandoracalo:Calorimetry 0.000111443 0.000243759 0.0030082 0.000211345 0.000283353 100 reco:pandorapid:Chi2ParticleID 3.7881e-05 8.77446e-05 0.00165847 6.9162e-05 0.000158817 100 reco:cvnmap:CVNMapper 3.0261e-05 0.0246874 0.0723167 0.026281 0.0167987 100 reco:cvneva:CVNEvaluator 2.3301e-05 1.35473 6.51132 1.58752 0.93895 100 reco:energyrecnumu:EnergyReco 0.00208013 0.00622733 0.0146682 0.00553453 0.00292011 100 reco:energyrecnue:EnergyReco 0.00185894 0.00409154 0.0109813 0.00372387 0.00156607 100 reco:energyrecnc:EnergyReco 0.00180951 0.00399218 0.0115863 0.00354078 0.00165012 100 reco:energyrecnumurange:EnergyReco 0.00177061 0.00406382 0.0149883 0.00358258 0.00189671 100 reco:energyrecnumumcs:EnergyReco 0.0017657 0.00393076 0.0130804 0.00350397 0.00171878 100 reco:opdec:Deconvolution 0.0892451 0.38169 0.660832 0.385122 0.114316 100 reco:ophitspe:OpHitFinderDeco 0.248856 0.261429 0.288141 0.260673 0.00682454 100 reco:opflash:OpFlashFinder 0.000172274 0.000681027 0.00403308 0.000555968 0.000474926 100 reco:opslicer:OpSlicer 4.8541e-05 0.0228191 0.145287 0.00905672 0.0308229 100 [art]:TriggerResults:TriggerResultInserter 1.2571e-05 3.1572e-05 0.000100922 2.91055e-05 1.3687e-05 100 end_path:out1:RootOutput 3.449e-06 7.37037e-06 3.601e-05 5.56e-06 4.92054e-06 100 end_path:out1:RootOutput(write) 0.00702302 0.548533 1.10612 0.664882 0.293142 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 37972.3 MB Peak resident set size usage (VmHWM): 1929.03 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.81_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2_graph_2025-08-02T_223935Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2_graph_2025-08-02T_223935Z.root debugprod.log graph_output_2025-08-02T_223935Z_3_ana_tree_hd.root graph_output_2025-08-02T_223935Z_3_analysiseid.root graph_output_2025-08-02T_223935Z_3_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223935Z_3_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223935Z_3_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223935Z_3_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223935Z_3_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223935Z_3_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2_graph_2025-08-02T_223935Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_478_20231207T050122Z_gen_g4_detsim_hitreco__20240510T044005Z_reco2_graph_2025-08-02T_223935Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv