Jobsub ID 257504.1@dunegpschedd02.fnal.gov
| Jobsub ID | 257504.1@dunegpschedd02.fnal.gov | |
| Workflow ID | 10938 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-12-04 08:35:18 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc01 | |
| Last heartbeat | 2025-12-04 09:09:37 | |
| From worker node | Hostname | wn-db-11.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-12-04 08:36:29 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 32m (1942s) | |
| CPU time | 12m (775s = 39%) | |
| Max RSS bytes | 2159190016 (2059 MiB) | |
| Outputting started | 2025-12-04 09:08:52 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/10938/1/008/graph_output_2025-12-04T_083640Z_9_ana_tree_hd.root | |
| Finished | 2025-12-04 09:09:37 | |
| Saved logs | justin-logs:257504.1-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
================ MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 10294.9 MB Peak resident set size usage (VmHWM): 1258.8 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-12-04T_083640Z_9_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-12-04T_083640Z_9_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-12-04T_083640Z_9_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-12-04T_083640Z_9_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-12-04T_083640Z_9_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-12-04T_083640Z_9_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-12-04T_083640Z_9_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-12-04T_083640Z_9_ana_tree_hd.root === Start last 100 lines of lar log file === Iterating with no regularization... Begin: 2.28495e+07 0 2.0356e+07 1 2.01878e+07 2 2.01689e+07 Now with regularization... Begin: 1.41914e+07 0 1.41862e+07 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG 13, Energy 0.910111, Dist. 355.437, nMCHits 1052 (323, 563, 166) MCPDG 13, Energy 0.910111, Dist. 355.437, nMCHits 997 (303, 541, 153) \_ MCPDG 11, Energy 0.039855, Dist. 6.31238, nMCHits 55 (20, 22, 13) --Primary 1, MCPDG 2212, Energy 1.14526, Dist. 27.4358, nMCHits 19 (5, 7, 7) MCPDG 2212, Energy 1.14526, Dist. 27.4358, nMCHits 19 (5, 7, 7) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 198 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 3306508 bytes. The eid is 198 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 4491444 bytes. The eid is 198 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 4856748 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 3301500 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 4489724 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 4854932 bytes. Boundary wire vector sizes: 333, 573, 174 minwire 0: 994 minwire 1: 1380 minwire 2: 997 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0399606, Output 1: 0.896071, 0.0011224, 0.0251973, 0.0776092, Output 2: 0.677261, 0.0261127, 0.00199368, 0.294632, Output 3: 0.495024, 0.493871, 0.0108186, 0.000286354, Output 4: 0.899312, 0.100568, 0.000114359, 5.8625e-06, Output 5: 0.991267, 0.00838141, 0.000107076, 0.000244967, Output 6: 0.33322, 0.599656, 0.0518028, 0.015321, Running Ophitfinder with InputDigiType = 'recob' Found hits: 247! 04-Dec-2025 10:03:26 CET Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_083640Z.root" 04-Dec-2025 10:03:26 CET Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/90/c2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.34962 3.24287 24.31 2.91107 2.40782 200 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.015126 0.0240795 0.0439997 0.0221834 0.00799565 200 reco:gaushit:GausHitFinder 0.0579641 0.146715 0.917851 0.0896671 0.143856 200 reco:spsolve:SpacePointSolver 0.000107335 0.0959717 3.16149 0.00455014 0.305005 200 reco:hitfd:DisambigFromSpacePoints 0.000185394 0.0269266 0.804102 0.00118955 0.0866528 200 reco:rns:RandomNumberSaver 1.7833e-05 2.51336e-05 0.000331534 2.1466e-05 2.24834e-05 200 reco:pandora:StandardPandora 0.0523328 1.4454 16.9206 0.995131 1.58212 200 reco:pandoraTrack:LArPandoraTrackCreation 0.000119819 0.000177588 0.00499825 0.000142257 0.0003434 200 reco:pandoraShower:LArPandoraModularShowerCreation 0.000163132 0.000197242 0.00321742 0.000174839 0.000215474 200 reco:pandoracalo:Calorimetry 9.9629e-05 0.000124627 0.00131156 0.000111187 8.58396e-05 200 reco:pandorapid:Chi2ParticleID 3.4396e-05 4.24282e-05 0.000664109 3.74365e-05 4.47069e-05 200 reco:cvnmap:CVNMapper 2.1011e-05 0.0199657 0.0610675 0.0254521 0.0131188 200 reco:cvneva:CVNEvaluator 1.6531e-05 0.714691 2.47139 0.95016 0.43708 200 reco:energyrecnumu:EnergyReco 0.00212485 0.00473371 0.0155198 0.00413561 0.00253523 200 reco:energyrecnue:EnergyReco 0.00193543 0.0032837 0.0115103 0.00257611 0.00176639 200 reco:energyrecnc:EnergyReco 0.00189961 0.00321464 0.0117135 0.00252844 0.00176773 200 reco:energyrecnumurange:EnergyReco 0.00198644 0.00320432 0.0114827 0.00251142 0.00175526 200 reco:energyrecnumumcs:EnergyReco 0.00186774 0.00318138 0.0119102 0.00243777 0.00175656 200 reco:opdec:Deconvolution 0.0634681 0.252578 0.445264 0.255831 0.0846281 200 reco:ophitspe:OpHitFinderDeco 0.132093 0.13952 0.175746 0.138458 0.00562954 200 reco:opflash:OpFlashFinder 0.00011504 0.000334072 0.00285753 0.000286628 0.000216363 200 reco:opslicer:OpSlicer 3.5738e-05 0.0151203 0.0726241 0.0074591 0.016861 200 [art]:TriggerResults:TriggerResultInserter 1.0149e-05 1.35987e-05 0.000137633 1.20585e-05 1.00407e-05 200 end_path:out1:RootOutput 3.027e-06 4.25168e-06 2.7343e-05 3.8425e-06 2.41839e-06 200 end_path:out1:RootOutput(write) 0.0116541 0.342445 0.80064 0.405667 0.188209 200 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11750.8 MB Peak resident set size usage (VmHWM): 2159.19 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 257504.1_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_083640Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_083640Z.root debugprod.log graph_output_2025-12-04T_083640Z_9_ana_tree_hd.root graph_output_2025-12-04T_083640Z_9_analysiseid.root graph_output_2025-12-04T_083640Z_9_training1_CaloHitListU_graph.data graph_output_2025-12-04T_083640Z_9_training1_CaloHitListV_graph.data graph_output_2025-12-04T_083640Z_9_training1_CaloHitListW_graph.data graph_output_2025-12-04T_083640Z_9_training2_CaloHitListU_graph.data graph_output_2025-12-04T_083640Z_9_training2_CaloHitListV_graph.data graph_output_2025-12-04T_083640Z_9_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_083640Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_083640Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv