Jobsub ID 20539.0@dunegpschedd01.fnal.gov
| Jobsub ID | 20539.0@dunegpschedd01.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 23:17:48 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-08-02 23:35:51 | |
| From worker node | Hostname | wn-da-05.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 23:19:20 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 15m (950s) | |
| CPU time | 6m (375s = 39%) | |
| Max RSS bytes | 2136166400 (2037 MiB) | |
| Outputting started | 2025-08-02 23:35:11 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231929Z_7_ana_tree_hd.root | |
| Finished | 2025-08-02 23:35:51 | |
| Saved logs | justin-logs:20539.0-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
============================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 10025.2 MB Peak resident set size usage (VmHWM): 982.307 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-02T_231929Z_7_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_231929Z_7_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_231929Z_7_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_231929Z_7_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_231929Z_7_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_231929Z_7_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_231929Z_7_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_231929Z_7_ana_tree_hd.root === Start last 100 lines of lar log file === 2 6.90279e+07 3 6.87516e+07 4 6.86258e+07 5 6.85498e+07 6 6.85049e+07 Now with regularization... Begin: 5.52696e+07 0 5.51067e+07 1 5.50234e+07 2 5.496e+07 3 5.49144e+07 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG -11, Energy 1.182, Dist. 10.0675, nMCHits 1164 (558, 282, 324) MCPDG -11, Energy 1.182, Dist. 10.0675, nMCHits 1164 (558, 282, 324) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 97 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1131968 bytes. The eid is 97 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1639880 bytes. The eid is 97 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1532316 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1131968 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1638796 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1530676 bytes. Boundary wire vector sizes: 573, 384, 377 minwire 0: 1069 minwire 1: 2013 minwire 2: 612 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0148694, Output 1: 0.00598096, 0.89256, 0.0148625, 0.0865963, Output 2: 0.334051, 0.373462, 0.0570548, 0.235432, Output 3: 0.559913, 0.287881, 0.0989558, 0.0532496, Output 4: 0.91258, 0.0871569, 0.000242763, 2.03223e-05, Output 5: 0.519646, 0.474762, 0.00447202, 0.0011203, Output 6: 0.11137, 0.239984, 0.136394, 0.512251, Running Ophitfinder with InputDigiType = 'recob' Found hits: 369! 03-Aug-2025 01:31:26 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_231929Z.root" 03-Aug-2025 01:31:26 CEST Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/5a/47/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.672891 3.12357 13.8008 3.02293 1.69732 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0440134 0.0788921 0.439378 0.0726027 0.0402544 100 reco:gaushit:GausHitFinder 0.104746 0.1926 0.818278 0.143545 0.121321 100 reco:spsolve:SpacePointSolver 9.2265e-05 0.044247 1.04367 0.00289133 0.140063 100 reco:hitfd:DisambigFromSpacePoints 0.000182077 0.0175376 0.377163 0.000783376 0.0547749 100 reco:rns:RandomNumberSaver 1.7983e-05 2.44686e-05 0.000259575 2.0539e-05 2.40414e-05 100 reco:pandora:StandardPandora 0.117298 1.25504 9.11778 0.986918 1.0014 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000117663 0.000158054 0.00217119 0.000127884 0.000203846 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000157019 0.000203415 0.00326878 0.000170836 0.000308257 100 reco:pandoracalo:Calorimetry 9.6904e-05 0.000123659 0.00136335 0.000105206 0.000125426 100 reco:pandorapid:Chi2ParticleID 3.3964e-05 4.44077e-05 0.000689873 3.67795e-05 6.49953e-05 100 reco:cvnmap:CVNMapper 1.9017e-05 0.0136412 0.042759 0.0169822 0.00859494 100 reco:cvneva:CVNEvaluator 1.5329e-05 0.68751 2.80491 0.892932 0.449346 100 reco:energyrecnumu:EnergyReco 0.00201548 0.00405162 0.0102304 0.00315049 0.00201044 100 reco:energyrecnue:EnergyReco 0.00176428 0.00276512 0.00998665 0.00229681 0.00129479 100 reco:energyrecnc:EnergyReco 0.00178633 0.00270346 0.00997379 0.0022225 0.00129961 100 reco:energyrecnumurange:EnergyReco 0.00172384 0.00270668 0.0100137 0.00221661 0.00132196 100 reco:energyrecnumumcs:EnergyReco 0.00185634 0.00269668 0.0097767 0.00221152 0.00128011 100 reco:opdec:Deconvolution 0.110778 0.305602 0.495793 0.303926 0.0808008 100 reco:ophitspe:OpHitFinderDeco 0.128421 0.19166 0.530172 0.184303 0.0527362 100 reco:opflash:OpFlashFinder 0.000109088 0.00032105 0.00301727 0.000270053 0.000293758 100 reco:opslicer:OpSlicer 3.1741e-05 0.0101215 0.0455605 0.00521717 0.0120525 100 [art]:TriggerResults:TriggerResultInserter 8.647e-06 1.15732e-05 4.9825e-05 1.1051e-05 3.99364e-06 100 end_path:out1:RootOutput 3.266e-06 4.0583e-06 1.6972e-05 3.798e-06 1.50601e-06 100 end_path:out1:RootOutput(write) 0.00324136 0.31008 0.636294 0.374265 0.168481 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11731.4 MB Peak resident set size usage (VmHWM): 2136.17 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 20539.0_dunegpschedd01.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_231929Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_231929Z.root debugprod.log graph_output_2025-08-02T_231929Z_7_ana_tree_hd.root graph_output_2025-08-02T_231929Z_7_analysiseid.root graph_output_2025-08-02T_231929Z_7_training1_CaloHitListU_graph.data graph_output_2025-08-02T_231929Z_7_training1_CaloHitListV_graph.data graph_output_2025-08-02T_231929Z_7_training1_CaloHitListW_graph.data graph_output_2025-08-02T_231929Z_7_training2_CaloHitListU_graph.data graph_output_2025-08-02T_231929Z_7_training2_CaloHitListV_graph.data graph_output_2025-08-02T_231929Z_7_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_231929Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_231929Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv