Jobsub ID 14065.54@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.54@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce01 | |
| Last heartbeat | 2025-08-02 23:00:44 | |
| From worker node | Hostname | wn2210250.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:39:00 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 20m (1257s) | |
| CPU time | 10m (657s = 52%) | |
| Max RSS bytes | 1974702080 (1883 MiB) | |
| Outputting started | 2025-08-02 22:59:57 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_ana_tree_hd.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223906Z_1_training2_CaloHitListW_graph.data | |
| Finished | 2025-08-02 23:00:44 | |
| Saved logs | justin-logs:14065.54-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
================= MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 19163.2 MB Peak resident set size usage (VmHWM): 982.02 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-02T_223906Z_1_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223906Z_1_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223906Z_1_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223906Z_1_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223906Z_1_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223906Z_1_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223906Z_1_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223906Z_1_ana_tree_hd.root === Start last 100 lines of lar log file === 4 574773 5 567079 6 564741 7 563828 8 563383 Now with regularization... Begin: -1.66289e+06 0 -1.66582e+06 1 -1.66643e+06 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG 2212, Energy 1.1885, Dist. 23.5044, nMCHits 97 (54, 18, 25) MCPDG 2212, Energy 1.1885, Dist. 23.5044, nMCHits 78 (44, 15, 19) \_ MCPDG 2212, Energy 1.01227, Dist. 4.61453, nMCHits 18 (10, 2, 6) \_ MCPDG 2212, Energy 0.951038, Dist. 0.20114, nMCHits 1 (0, 1, 0) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 99 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1505724 bytes. The eid is 99 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1995656 bytes. The eid is 99 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 2095232 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1505724 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1993224 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 2092460 bytes. Boundary wire vector sizes: 67, 109, 62 minwire 0: 268 minwire 1: 2153 minwire 2: 302 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0817593, Output 1: 0.00151797, 0.000757161, 0.00197006, 0.995755, Output 2: 0.732012, 0.118263, 0.000282891, 0.149442, Output 3: 0.312165, 0.674516, 0.0129496, 0.000369162, Output 4: 0.994822, 0.00467222, 0.000375409, 0.000129991, Output 5: 0.999831, 0.000137947, 1.10459e-05, 1.97014e-05, Output 6: 0.0171902, 0.277139, 0.290738, 0.414933, Running Ophitfinder with InputDigiType = 'recob' Found hits: 176! 02-Aug-2025 23:56:48 BST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2_graph_2025-08-02T_223906Z.root" 02-Aug-2025 23:56:48 BST Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/f1/d7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 1.55938 4.78737 21.8765 4.38708 3.3216 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0113295 0.0208865 0.0396404 0.0219847 0.00660702 100 reco:gaushit:GausHitFinder 0.0526089 0.154515 0.896939 0.0869859 0.167991 100 reco:spsolve:SpacePointSolver 0.000162405 0.2813 9.67656 0.00872137 1.26086 100 reco:hitfd:DisambigFromSpacePoints 0.000289423 0.0319345 0.712535 0.00282663 0.0983655 100 reco:rns:RandomNumberSaver 2.3965e-05 6.91417e-05 0.000492234 5.56845e-05 5.38781e-05 100 reco:pandora:StandardPandora 1.12814 2.10937 11.8605 1.72112 1.58309 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000161623 0.000402007 0.00390855 0.000348499 0.000376557 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000205665 0.000404059 0.00288066 0.000357926 0.000275855 100 reco:pandoracalo:Calorimetry 0.000140323 0.000279059 0.00142814 0.000250495 0.000145756 100 reco:pandorapid:Chi2ParticleID 4.1558e-05 8.56366e-05 0.000643577 7.28065e-05 6.28916e-05 100 reco:cvnmap:CVNMapper 5.5254e-05 0.0426097 0.12108 0.0443282 0.0306366 100 reco:cvneva:CVNEvaluator 2.2171e-05 1.16214 5.37813 1.3837 0.80887 100 reco:energyrecnumu:EnergyReco 0.00227257 0.00734651 0.0238621 0.00603421 0.00425478 100 reco:energyrecnue:EnergyReco 0.00191187 0.00451597 0.01311 0.00398107 0.00189601 100 reco:energyrecnc:EnergyReco 0.00198028 0.00445734 0.0124685 0.00391519 0.00191235 100 reco:energyrecnumurange:EnergyReco 0.00197198 0.00456638 0.020171 0.00379918 0.0024268 100 reco:energyrecnumumcs:EnergyReco 0.00181252 0.00440654 0.0124891 0.00374854 0.00191275 100 reco:opdec:Deconvolution 0.117426 0.272007 0.497353 0.276713 0.0830266 100 reco:ophitspe:OpHitFinderDeco 0.0954633 0.110055 0.158756 0.108295 0.00892505 100 reco:opflash:OpFlashFinder 0.000269776 0.000857084 0.00378938 0.000732394 0.000455199 100 reco:opslicer:OpSlicer 6.1656e-05 0.022727 0.121548 0.0156123 0.0263605 100 [art]:TriggerResults:TriggerResultInserter 1.4427e-05 4.15239e-05 0.000174808 3.0272e-05 3.13684e-05 100 end_path:out1:RootOutput 3.527e-06 9.39043e-06 5.4863e-05 6.8125e-06 8.87286e-06 100 end_path:out1:RootOutput(write) 0.0429932 0.550218 1.23295 0.646564 0.309513 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 20736.6 MB Peak resident set size usage (VmHWM): 1974.7 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.54_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2_graph_2025-08-02T_223906Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2_graph_2025-08-02T_223906Z.root debugprod.log graph_output_2025-08-02T_223906Z_1_ana_tree_hd.root graph_output_2025-08-02T_223906Z_1_analysiseid.root graph_output_2025-08-02T_223906Z_1_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223906Z_1_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223906Z_1_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223906Z_1_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223906Z_1_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223906Z_1_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2_graph_2025-08-02T_223906Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_736_20231208T094018Z_gen_g4_detsim_hitreco__20240510T064132Z_reco2_graph_2025-08-02T_223906Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv