Jobsub ID 14065.110@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.110@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-08-02 22:47:54 | |
| From worker node | Hostname | wn-la-13.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:38:50 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 8m (494s) | |
| CPU time | 7m (446s = 90%) | |
| Max RSS bytes | 2140585984 (2041 MiB) | |
| Outputting started | 2025-08-02 22:47:05 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223900Z_2_ana_tree_hd.root | |
| Finished | 2025-08-02 22:47:54 | |
| Saved logs | justin-logs:14065.110-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
ual memory usage (VmPeak) : 18076.3 MB Peak resident set size usage (VmHWM): 980.472 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-02T_223900Z_2_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223900Z_2_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223900Z_2_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223900Z_2_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223900Z_2_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223900Z_2_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223900Z_2_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223900Z_2_ana_tree_hd.root === Start last 100 lines of lar log file === 2 3.85878e+06 3 3.82714e+06 4 3.8145e+06 5 3.80806e+06 6 3.80431e+06 Now with regularization... Begin: 3.06204e+06 0 3.05887e+06 1 3.05727e+06 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG 13, Energy 0.160419, Dist. 11.99, nMCHits 38 (3, 23, 12) MCPDG 13, Energy 0.160419, Dist. 11.99, nMCHits 37 (3, 22, 12) \_ MCPDG 2212, Energy 0.947107, Dist. 0.108381, nMCHits 1 (0, 1, 0) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 98 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1126576 bytes. The eid is 98 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1367728 bytes. The eid is 98 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1574900 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1125892 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1367276 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1573772 bytes. Boundary wire vector sizes: 49, 29, 34 minwire 0: 982 minwire 1: 1467 minwire 2: 1137 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 2363, 2862 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0178373, Output 1: 0.0228316, 0.000547108, 0.00813362, 0.968488, Output 2: 0.530077, 0.204338, 0.00460648, 0.260979, Output 3: 0.902242, 0.0913643, 0.00422822, 0.00216515, Output 4: 0.207157, 0.789897, 0.00292729, 1.87493e-05, Output 5: 0.997941, 0.00202477, 2.06029e-05, 1.36902e-05, Output 6: 0.448735, 0.472399, 0.0533209, 0.0255461, Running Ophitfinder with InputDigiType = 'recob' Found hits: 170! 03-Aug-2025 00:45:59 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2_graph_2025-08-02T_223900Z.root" 03-Aug-2025 00:45:59 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/8b/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.0831228 3.07791 19.3001 3.02662 2.11071 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00199667 0.00341065 0.00637819 0.00333367 0.000839434 100 reco:gaushit:GausHitFinder 0.0161178 0.0666935 0.699529 0.028226 0.102659 100 reco:spsolve:SpacePointSolver 0.000127811 0.0704168 3.94312 0.00465868 0.398767 100 reco:hitfd:DisambigFromSpacePoints 0.000226519 0.0141335 0.520657 0.00113097 0.05787 100 reco:rns:RandomNumberSaver 2.2474e-05 3.37131e-05 0.000452226 2.7386e-05 4.25296e-05 100 reco:pandora:StandardPandora 0.0126437 1.46156 11.642 1.19474 1.222 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000151306 0.000234502 0.0038448 0.000181692 0.000369193 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000195803 0.000261759 0.00405287 0.0002171 0.000381762 100 reco:pandoracalo:Calorimetry 0.000124967 0.000174572 0.00230111 0.00014725 0.000214864 100 reco:pandorapid:Chi2ParticleID 4.1132e-05 8.21335e-05 0.00261955 4.55275e-05 0.000268143 100 reco:cvnmap:CVNMapper 2.5889e-05 0.011204 0.0440971 0.0124376 0.00802753 100 reco:cvneva:CVNEvaluator 2.0611e-05 0.830182 3.45784 1.08608 0.545435 100 reco:energyrecnumu:EnergyReco 0.00210103 0.00522121 0.0204647 0.00557239 0.00283511 100 reco:energyrecnue:EnergyReco 0.00203887 0.00289374 0.0123903 0.00244976 0.00134879 100 reco:energyrecnc:EnergyReco 0.00214468 0.00280617 0.0124816 0.00236254 0.00135165 100 reco:energyrecnumurange:EnergyReco 0.00215836 0.0028054 0.0123065 0.0023475 0.00134291 100 reco:energyrecnumumcs:EnergyReco 0.00214758 0.00281314 0.0122993 0.00233184 0.00135326 100 reco:opdec:Deconvolution 0.0147092 0.179373 0.310244 0.182922 0.0712214 100 reco:ophitspe:OpHitFinderDeco 0.0090963 0.0164111 0.0288614 0.0160219 0.00336927 100 reco:opflash:OpFlashFinder 0.000132138 0.000410781 0.00389135 0.000353132 0.000375044 100 reco:opslicer:OpSlicer 3.8397e-05 0.0156627 0.0681964 0.00718718 0.017898 100 [art]:TriggerResults:TriggerResultInserter 1.2369e-05 1.56888e-05 7.0795e-05 1.4957e-05 5.77805e-06 100 end_path:out1:RootOutput 3.946e-06 5.89615e-06 3.1438e-05 5.2185e-06 3.2107e-06 100 end_path:out1:RootOutput(write) 0.00962889 0.390056 0.9016 0.47365 0.208454 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 19851.1 MB Peak resident set size usage (VmHWM): 2140.59 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.110_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2_graph_2025-08-02T_223900Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2_graph_2025-08-02T_223900Z.root debugprod.log graph_output_2025-08-02T_223900Z_2_ana_tree_hd.root graph_output_2025-08-02T_223900Z_2_analysiseid.root graph_output_2025-08-02T_223900Z_2_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223900Z_2_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223900Z_2_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223900Z_2_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223900Z_2_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223900Z_2_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2_graph_2025-08-02T_223900Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1012_20231203T064857Z_gen_g4_detsim_hitreco__20240508T041927Z_reco2_graph_2025-08-02T_223900Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv