Jobsub ID 20545.0@dunegpschedd01.fnal.gov
| Jobsub ID | 20545.0@dunegpschedd01.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 23:49:51 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc01 | |
| Last heartbeat | 2025-08-03 00:16:10 | |
| From worker node | Hostname | wn-lb-17.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 23:50:50 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 24m (1470s) | |
| CPU time | 14m (882s = 59%) | |
| Max RSS bytes | 2300092416 (2193 MiB) | |
| Outputting started | 2025-08-03 00:15:20 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_235100Z_4_ana_tree_hd.root | |
| Finished | 2025-08-03 00:16:10 | |
| Saved logs | justin-logs:20545.0-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
Z_4_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_235100Z_4_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_235100Z_4_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_235100Z_4_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-08-02T_235100Z_4_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-08-02T_235100Z_4_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 32, 40, 32
minwire 0: 2385
minwire 1: 390
minwire 2: 2647
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.0221207,
Output 1: 0.00174935, 0.000794411, 0.000990917, 0.996465,
Output 2: 0.183017, 0.655281, 0.00147552, 0.160226,
Output 3: 0.0245916, 0.484385, 0.475985, 0.0150389,
Output 4: 0.995887, 0.00269876, 0.000869628, 0.00054453,
Output 5: 0.99732, 0.00228168, 0.000311523, 8.65824e-05,
Output 6: 0.303294, 0.558227, 0.120712, 0.0177664,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 106!
Begin processing the 100th record. run: 50601160 subRun: 1 event: 37400 at 03-Aug-2025 02:12:12 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:17 13 XUs and 20 XVs -> 6 XUVs
6 XUVs total
4 collection wire objects
6 potential space points
Neighbour search...
18 tests to find 12 neighbours
Iterating with no regularization...
Begin: 46960.4
0 32133.2
1 32133.2
Now with regularization...
Begin: 14291.1
0 14290.1
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
in function: PrepareTrainingSample
in file: /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_FOUND
Operating in inference mode.
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
in function: PrepareTrainingSample
in file: /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_FOUND
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 30!
03-Aug-2025 02:12:15 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_235100Z.root"
03-Aug-2025 02:12:15 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/97/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.348194 7.54678 375.279 3.3277 37.2674 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.015815 0.0243439 0.0331038 0.0228856 0.00754678 100
reco:gaushit:GausHitFinder 0.0589813 0.137072 1.29182 0.0801703 0.168985 100
reco:spsolve:SpacePointSolver 0.00012672 3.89906 370.817 0.00296023 36.8883 100
reco:hitfd:DisambigFromSpacePoints 0.000227009 0.0316501 0.858666 0.000970378 0.122957 100
reco:rns:RandomNumberSaver 2.1943e-05 3.30821e-05 0.000476511 2.62645e-05 4.50589e-05 100
reco:pandora:StandardPandora 0.0535916 1.89187 28.5514 1.2026 3.32213 100
reco:pandoraTrack:LArPandoraTrackCreation 0.000150024 0.00022543 0.0044136 0.000172928 0.000422014 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.000195312 0.000248899 0.0034993 0.000210114 0.000327073 100
reco:pandoracalo:Calorimetry 0.000126648 0.000164703 0.00155023 0.00014401 0.000140772 100
reco:pandorapid:Chi2ParticleID 4.1823e-05 5.37735e-05 0.00073005 4.4442e-05 6.83403e-05 100
reco:cvnmap:CVNMapper 2.4026e-05 0.0116584 0.0651738 0.0123174 0.0119812 100
reco:cvneva:CVNEvaluator 2.0622e-05 0.741231 3.4682 1.0809 0.58214 100
reco:energyrecnumu:EnergyReco 0.00239847 0.00523071 0.0178339 0.00494176 0.00275498 100
reco:energyrecnue:EnergyReco 0.00239306 0.00340835 0.0140995 0.00283148 0.00184737 100
reco:energyrecnc:EnergyReco 0.00231735 0.00333771 0.0140413 0.00268168 0.00184143 100
reco:energyrecnumurange:EnergyReco 0.00233634 0.00334923 0.0140844 0.00270076 0.00184866 100
reco:energyrecnumumcs:EnergyReco 0.00231843 0.00334401 0.0140568 0.00269572 0.00185084 100
reco:opdec:Deconvolution 0.0491392 0.259137 1.10322 0.250858 0.125665 100
reco:ophitspe:OpHitFinderDeco 0.134061 0.14306 0.172286 0.142191 0.00546668 100
reco:opflash:OpFlashFinder 0.000129963 0.000409635 0.00344168 0.000339171 0.000346859 100
reco:opslicer:OpSlicer 4.2724e-05 0.0174681 0.0742997 0.00753971 0.0205635 100
[art]:TriggerResults:TriggerResultInserter 1.2097e-05 1.57395e-05 7.3882e-05 1.4196e-05 6.67302e-06 100
end_path:out1:RootOutput 4.637e-06 5.68116e-06 2.3937e-05 5.2635e-06 2.56899e-06 100
end_path:out1:RootOutput(write) 0.00446979 0.369358 0.994848 0.468869 0.242162 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20010.5 MB
Peak resident set size usage (VmHWM): 2300.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20545.0_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_235100Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_235100Z.root
debugprod.log
graph_output_2025-08-02T_235100Z_4_ana_tree_hd.root
graph_output_2025-08-02T_235100Z_4_analysiseid.root
graph_output_2025-08-02T_235100Z_4_training1_CaloHitListU_graph.data
graph_output_2025-08-02T_235100Z_4_training1_CaloHitListV_graph.data
graph_output_2025-08-02T_235100Z_4_training1_CaloHitListW_graph.data
graph_output_2025-08-02T_235100Z_4_training2_CaloHitListU_graph.data
graph_output_2025-08-02T_235100Z_4_training2_CaloHitListV_graph.data
graph_output_2025-08-02T_235100Z_4_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_235100Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_235100Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv