Jobsub ID 14065.97@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.97@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc01 | |
| Last heartbeat | 2025-08-02 22:46:49 | |
| From worker node | Hostname | wn-db-07.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:38:48 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 7m (425s) | |
| CPU time | 6m (386s = 90%) | |
| Max RSS bytes | 2003083264 (1910 MiB) | |
| Outputting started | 2025-08-02 22:45:55 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223856Z_10_ana_tree_hd.root | |
| Finished | 2025-08-02 22:46:49 | |
| Saved logs | justin-logs:14065.97-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
ata -> graph_output_2025-08-02T_223856Z_10_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-08-02T_223856Z_10_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223856Z_10_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 1459352 bytes.
The eid is 95
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 1406720 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 1021032 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 1457620 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 1404716 bytes.
Boundary wire vector sizes: 61, 45, 47
minwire 0: 1541
minwire 1: 1337
minwire 2: 1553
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.0417716,
Output 1: 0.00955552, 0.0036243, 0.0267494, 0.960071,
Output 2: 0.327991, 0.226912, 0.00265622, 0.442441,
Output 3: 0.138107, 0.846379, 0.0154781, 3.58971e-05,
Output 4: 0.0562165, 0.938886, 0.00488954, 8.3174e-06,
Output 5: 0.996892, 0.00283636, 0.00014882, 0.000122642,
Output 6: 0.111468, 0.819938, 0.0645196, 0.00407439,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 142!
Begin processing the 100th record. run: 66004350 subRun: 1 event: 71700 at 03-Aug-2025 00:44:44 CEST
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
in function: PrepareTrainingSample
in file: /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_FOUND
Operating in inference mode.
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
in function: PrepareTrainingSample
in file: /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_FOUND
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 0!
03-Aug-2025 00:44:46 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2_graph_2025-08-02T_223856Z.root"
03-Aug-2025 00:44:46 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/bf/a8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.0655945 2.57698 15.2737 2.547 2.11099 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00132538 0.00276726 0.00444825 0.00264736 0.000695816 100
reco:gaushit:GausHitFinder 0.0169937 0.0640883 0.685782 0.0247574 0.118016 100
reco:spsolve:SpacePointSolver 8.942e-05 0.0769105 2.92373 0.00237232 0.359013 100
reco:hitfd:DisambigFromSpacePoints 0.000177107 0.0183151 0.548054 0.000736729 0.0780103 100
reco:rns:RandomNumberSaver 1.6843e-05 2.51164e-05 0.000266157 2.1065e-05 2.4668e-05 100
reco:pandora:StandardPandora 0.0107619 1.21287 9.88406 0.932212 1.31528 100
reco:pandoraTrack:LArPandoraTrackCreation 0.000117694 0.000157538 0.00203863 0.000130368 0.000190237 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.00016246 0.000209709 0.00305841 0.000179727 0.000286483 100
reco:pandoracalo:Calorimetry 9.8458e-05 0.00012871 0.00128356 0.000112595 0.000116853 100
reco:pandorapid:Chi2ParticleID 3.5758e-05 4.62303e-05 0.000671521 3.8287e-05 6.30589e-05 100
reco:cvnmap:CVNMapper 1.8976e-05 0.0214147 0.0567139 0.027484 0.0154591 100
reco:cvneva:CVNEvaluator 1.5379e-05 0.67764 2.51977 0.948744 0.480164 100
reco:energyrecnumu:EnergyReco 0.00193332 0.00502185 0.0190809 0.00421722 0.00335759 100
reco:energyrecnue:EnergyReco 0.00199482 0.00289157 0.00999383 0.00247395 0.00138062 100
reco:energyrecnc:EnergyReco 0.00199039 0.00282443 0.0101352 0.00239366 0.00136547 100
reco:energyrecnumurange:EnergyReco 0.00202376 0.00282007 0.010354 0.00237052 0.00137573 100
reco:energyrecnumumcs:EnergyReco 0.00197939 0.00291683 0.0156358 0.00233716 0.00186 100
reco:opdec:Deconvolution 0.0071187 0.15393 0.324823 0.164617 0.0730848 100
reco:ophitspe:OpHitFinderDeco 0.0102454 0.0161224 0.0316758 0.0158294 0.00289779 100
reco:opflash:OpFlashFinder 9.8297e-05 0.000302369 0.00276797 0.000257791 0.00028024 100
reco:opslicer:OpSlicer 3.1761e-05 0.00924259 0.0572638 0.00496665 0.0125829 100
[art]:TriggerResults:TriggerResultInserter 9.458e-06 1.19124e-05 4.215e-05 1.1357e-05 3.46765e-06 100
end_path:out1:RootOutput 3.166e-06 4.30051e-06 2.0018e-05 3.8625e-06 2.24927e-06 100
end_path:out1:RootOutput(write) 0.00344787 0.305472 0.716076 0.396127 0.189862 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 11597.5 MB
Peak resident set size usage (VmHWM): 2003.08 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14065.97_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2_graph_2025-08-02T_223856Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2_graph_2025-08-02T_223856Z.root
debugprod.log
graph_output_2025-08-02T_223856Z_10_ana_tree_hd.root
graph_output_2025-08-02T_223856Z_10_analysiseid.root
graph_output_2025-08-02T_223856Z_10_training1_CaloHitListU_graph.data
graph_output_2025-08-02T_223856Z_10_training1_CaloHitListV_graph.data
graph_output_2025-08-02T_223856Z_10_training1_CaloHitListW_graph.data
graph_output_2025-08-02T_223856Z_10_training2_CaloHitListU_graph.data
graph_output_2025-08-02T_223856Z_10_training2_CaloHitListV_graph.data
graph_output_2025-08-02T_223856Z_10_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2_graph_2025-08-02T_223856Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_716_20231202T160948Z_gen_g4_detsim_hitreco__20240508T044308Z_reco2_graph_2025-08-02T_223856Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv