justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14065.40@dunegpschedd02.fnal.gov

Jobsub ID14065.40@dunegpschedd02.fnal.gov
Workflow ID253
Stage ID1
User nameichong@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-02 22:37:47
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-08-02 23:25:53
From worker nodeHostnametds509.pic.es
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-08-02 22:39:16
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2.root
Outputting started2025-08-02 23:25:53
Output files
Finished2025-08-02 23:48:50
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3923Z_9_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223923Z_9_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223923Z_9_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223923Z_9_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-08-02T_223923Z_9_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223923Z_9_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 32, 40, 32
minwire 0: 2385
minwire 1: 390
minwire 2: 2647
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.022121, 
Output 1: 0.00174935, 0.00079439, 0.000990922, 0.996465, 
Output 2: 0.183007, 0.655291, 0.00147551, 0.160227, 
Output 3: 0.0245917, 0.484382, 0.475987, 0.0150391, 
Output 4: 0.995887, 0.00269882, 0.000869609, 0.000544487, 
Output 5: 0.99732, 0.0022818, 0.000311538, 8.65821e-05, 
Output 6: 0.303293, 0.558228, 0.120713, 0.0177663, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 106!
Begin processing the 100th record. run: 50601160 subRun: 1 event: 37400 at 03-Aug-2025 01:18:39 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:17 13 XUs and 20 XVs -> 6 XUVs
6 XUVs total
4 collection wire objects
6 potential space points
Neighbour search...
18 tests to find 12 neighbours
Iterating with no regularization...
Begin: 46960.4
0 32133.2
1 32133.2
Now with regularization...
Begin: 14291.1
0 14290.1
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
    in function: PrepareTrainingSample
    in file:     /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_FOUND
Operating in inference mode.
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
    in function: PrepareTrainingSample
    in file:     /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_FOUND

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 30!
03-Aug-2025 01:18:44 CEST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_223923Z.root"
03-Aug-2025 01:18:44 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/97/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.930908       8.45353       345.724       4.41951       34.4008        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0470515     0.081969      0.142038      0.0724506     0.025775        100    
reco:gaushit:GausHitFinder                             0.154785      0.255812       1.87134       0.17261      0.245655        100    
reco:spsolve:SpacePointSolver                         0.000130585     3.64145       340.229     0.00315653      33.8488        100    
reco:hitfd:DisambigFromSpacePoints                    0.000243987     0.03635       1.01651     0.00115207     0.143411        100    
reco:rns:RandomNumberSaver                            2.0749e-05    3.63685e-05   0.000445115   2.9715e-05    4.22177e-05      100    
reco:pandora:StandardPandora                           0.145724       2.18216       35.2368       1.39733       3.99061        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000117901   0.000245783   0.00460488    0.00019458    0.000440441      100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000151824   0.000289798   0.00328358    0.000241829   0.000326898      100    
reco:pandoracalo:Calorimetry                          9.5619e-05    0.000173476   0.00159083    0.000158793   0.00014516       100    
reco:pandorapid:Chi2ParticleID                        3.4855e-05    6.02257e-05   0.000703008   5.23985e-05   6.53855e-05      100    
reco:cvnmap:CVNMapper                                 3.5347e-05     0.0460106     0.151402      0.0471644     0.0397275       100    
reco:cvneva:CVNEvaluator                              1.7262e-05     0.881759       5.18866       1.24215      0.743901        100    
reco:energyrecnumu:EnergyReco                         0.00158494    0.00535541     0.0178176    0.00371563     0.0031945       100    
reco:energyrecnue:EnergyReco                          0.00155856    0.00390926     0.0160794    0.00331711    0.00210799       100    
reco:energyrecnc:EnergyReco                           0.00156479    0.00388518     0.0170251    0.00323062    0.00218787       100    
reco:energyrecnumurange:EnergyReco                    0.00157333    0.00391108     0.0174909    0.00320282    0.00224681       100    
reco:energyrecnumumcs:EnergyReco                      0.00159172    0.00390244     0.0173008     0.0032332    0.00226281       100    
reco:opdec:Deconvolution                               0.140891      0.378479      0.635394      0.389778      0.112522        100    
reco:ophitspe:OpHitFinderDeco                          0.411389      0.420533      0.468917      0.419589     0.00706607       100    
reco:opflash:OpFlashFinder                            0.000153308   0.000480428   0.00452057    0.000381366   0.000449618      100    
reco:opslicer:OpSlicer                                4.3682e-05     0.020058      0.0870034    0.00891607     0.0239895       100    
[art]:TriggerResults:TriggerResultInserter            1.2774e-05    2.02945e-05   0.000114946   1.8164e-05    1.10107e-05      100    
end_path:out1:RootOutput                               3.186e-06    4.26633e-06   2.5377e-05    3.9925e-06    2.19994e-06      100    
end_path:out1:RootOutput(write)                       0.00631631     0.485428       1.61285      0.615319      0.337859        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20852 MB
  Peak resident set size usage (VmHWM): 2031.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14065.40_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_223923Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_223923Z.root
debugprod.log
graph_output_2025-08-02T_223923Z_9_ana_tree_hd.root
graph_output_2025-08-02T_223923Z_9_analysiseid.root
graph_output_2025-08-02T_223923Z_9_training1_CaloHitListU_graph.data
graph_output_2025-08-02T_223923Z_9_training1_CaloHitListV_graph.data
graph_output_2025-08-02T_223923Z_9_training1_CaloHitListW_graph.data
graph_output_2025-08-02T_223923Z_9_training2_CaloHitListU_graph.data
graph_output_2025-08-02T_223923Z_9_training2_CaloHitListV_graph.data
graph_output_2025-08-02T_223923Z_9_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_223923Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2_graph_2025-08-02T_223923Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv
justIN time: 2025-08-04 14:23:03 UTC       justIN version: 01.04.00