Jobsub ID 14147.46@dunegpschedd02.fnal.gov
| Jobsub ID | 14147.46@dunegpschedd02.fnal.gov | |
| Workflow ID | 270 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-03 14:52:40 | |
| Site | CZ_FZU | |
| Entry | Nova_CZ_FZU_osgce1 | |
| Last heartbeat | 2025-08-03 15:05:35 | |
| From worker node | Hostname | milan01.farm.particle.cz |
| cpuinfo | AMD EPYC 7543 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-03 14:57:40 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 6m (417s) | |
| CPU time | 6m (389s = 93%) | |
| Max RSS bytes | 1964036096 (1873 MiB) | |
| Outputting started | 2025-08-03 15:04:38 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_ana_tree_hd.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/graph_output_2025-08-03T_145746Z_10_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/training2_CaloHitListU.csv https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/training2_CaloHitListV.csv https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/training1_CaloHitListV.csv https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/training1_CaloHitListW.csv https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/training2_CaloHitListW.csv https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00270/1/001/training1_CaloHitListU.csv | |
| Finished | 2025-08-03 15:05:35 | |
| Saved logs | justin-logs:14147.46-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
===================================================================== Full event 0.0109844 0.0681948 0.876463 0.039584 0.101881 100 ------------------------------------------------------------------------------------------------------------------------ source:RootInput(read) 0.000583676 0.00144305 0.0110037 0.00119593 0.00119303 100 end_path:analysistree:AnalysisTree 0.00988355 0.0665727 0.87492 0.0376752 0.101829 100 ======================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 19162.8 MB Peak resident set size usage (VmHWM): 966.779 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-03T_145746Z_10_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-03T_145746Z_10_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-03T_145746Z_10_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-03T_145746Z_10_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-03T_145746Z_10_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-03T_145746Z_10_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-03T_145746Z_10_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-03T_145746Z_10_ana_tree_hd.root === Start last 100 lines of lar log file === 81 533.501 82 532.023 83 530.62 84 529.289 85 528.026 86 526.828 87 525.691 88 524.613 89 523.591 90 522.622 91 521.703 92 520.832 93 520.006 94 519.223 95 518.481 96 517.777 97 517.11 98 516.478 99 515.879 Now with regularization... Begin: -710537 0 -714392 1 -719981 2 -724695 3 -727820 4 -730171 5 -732062 6 -732788 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG 2212, Energy 1.02227, Dist. 5.82936, nMCHits 38 (14, 11, 13) MCPDG 2212, Energy 1.02227, Dist. 5.82936, nMCHits 38 (14, 11, 13) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 97 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 898480 bytes. The eid is 97 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1219608 bytes. The eid is 97 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1284752 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 898480 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1216900 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1281956 bytes. Running Ophitfinder with InputDigiType = 'recob' Found hits: 136! 03-Aug-2025 17:03:47 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2_graph_2025-08-03T_145746Z.root" 03-Aug-2025 17:03:47 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2f/c8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.073073 2.8541 13.4228 3.12159 1.80023 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00131099 0.00286409 0.0194721 0.00230644 0.00237974 100 reco:gaushit:GausHitFinder 0.0124341 0.0634219 0.601711 0.0314909 0.0909568 100 reco:spsolve:SpacePointSolver 0.000107129 0.0495808 0.904047 0.00279269 0.148997 100 reco:hitfd:DisambigFromSpacePoints 0.000226868 0.014032 0.538824 0.00092076 0.059297 100 reco:rns:RandomNumberSaver 2.2e-05 4.5421e-05 0.000405538 3.8895e-05 3.89057e-05 100 reco:pandora:StandardPandora 0.0102642 1.28419 8.38643 1.13097 0.861425 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000122879 0.000332929 0.00680486 0.000243813 0.000660019 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000171669 0.000354212 0.00380965 0.000312313 0.000359351 100 reco:pandoracalo:Calorimetry 9.6389e-05 0.000209752 0.0019294 0.000190409 0.00018354 100 reco:pandorapid:Chi2ParticleID 4.049e-05 7.74761e-05 0.000899615 6.73845e-05 8.50775e-05 100 reco:cvnmap:CVNMapper 2.716e-05 0.0223405 0.11141 0.0250479 0.0214373 100 reco:cvneva:CVNEvaluator 1.991e-05 0.809151 3.53862 1.09426 0.672878 100 reco:energyrecnumu:EnergyReco 0.00167133 0.00573907 0.0258744 0.00450741 0.00415104 100 reco:energyrecnue:EnergyReco 0.00152141 0.00341174 0.0239937 0.00281543 0.00280239 100 reco:energyrecnc:EnergyReco 0.00146645 0.00326502 0.0187103 0.00264358 0.00240794 100 reco:energyrecnumurange:EnergyReco 0.00149068 0.00337444 0.0140193 0.00267021 0.00227749 100 reco:energyrecnumumcs:EnergyReco 0.00145253 0.00317149 0.011614 0.00270338 0.00171593 100 reco:opdec:Deconvolution 0.00754196 0.143844 0.352052 0.137905 0.0676071 100 reco:ophitspe:OpHitFinderDeco 0.0036014 0.0124769 0.0394086 0.0108481 0.0063371 100 reco:opflash:OpFlashFinder 0.000144749 0.000573585 0.00725739 0.000458997 0.00070376 100 reco:opslicer:OpSlicer 3.643e-05 0.0130142 0.0747513 0.00648702 0.0161849 100 [art]:TriggerResults:TriggerResultInserter 1.0629e-05 2.42472e-05 0.000156449 2.2145e-05 1.52936e-05 100 end_path:out1:RootOutput 3.64e-06 1.04525e-05 0.000308508 6.545e-06 3.01239e-05 100 end_path:out1:RootOutput(write) 0.00732806 0.417164 1.02256 0.552927 0.298892 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 20717.2 MB Peak resident set size usage (VmHWM): 1964.04 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14147.46_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2_graph_2025-08-03T_145746Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2_graph_2025-08-03T_145746Z.root debugprod.log graph_output_2025-08-03T_145746Z_10_ana_tree_hd.root graph_output_2025-08-03T_145746Z_10_analysiseid.root graph_output_2025-08-03T_145746Z_10_training1_CaloHitListU_graph.data graph_output_2025-08-03T_145746Z_10_training1_CaloHitListV_graph.data graph_output_2025-08-03T_145746Z_10_training1_CaloHitListW_graph.data graph_output_2025-08-03T_145746Z_10_training2_CaloHitListU_graph.data graph_output_2025-08-03T_145746Z_10_training2_CaloHitListV_graph.data graph_output_2025-08-03T_145746Z_10_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2_graph_2025-08-03T_145746Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_211_20231207T071816Z_gen_g4_detsim_hitreco__20240510T062815Z_reco2_graph_2025-08-03T_145746Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv