justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 11089.71@dunegpschedd02.fnal.gov

Jobsub ID11089.71@dunegpschedd02.fnal.gov
Workflow ID59
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-25 00:20:41
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2025-07-25 16:58:32
From worker nodeHostnamewk01.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-07-25 00:21:58
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco.root
JobscriptExit code0
Real time4h (16242s)
CPU time3h (14168s = 87%)
Max RSS bytes1370341376 (1306 MiB)
Outputting started2025-07-25 04:52:41
Output files
Finished2025-07-25 16:58:32
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

hits
Genrating OpFlash 1 with 6 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 1163 OpFlashes
Begin processing the 49th record. run: 5100 subRun: 0 event: 223249 at 25-Jul-2025 05:43:03 BST
Failure in algorithm Alg0002, LArDLSNSignal, unknown exception
Failure in algorithm Alg0003, LArDLSNSignal, unknown exception
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:44:38 BST run: 5100 subRun: 0 event: 223249
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:44:38 BST run: 5100 subRun: 0 event: 223249
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:44:38 BST run: 5100 subRun: 0 event: 223249
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:44:38 BST run: 5100 subRun: 0 event: 223249
Comparing two wires in the same plane: return failure
%MSG
32339
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 39013!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 7 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 1063 OpFlashes
Begin processing the 50th record. run: 5100 subRun: 0 event: 223250 at 25-Jul-2025 05:48:00 BST
Failure in algorithm Alg0002, LArDLSNSignal, unknown exception
Failure in algorithm Alg0003, LArDLSNSignal, unknown exception
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:49:41 BST run: 5100 subRun: 0 event: 223250
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:49:41 BST run: 5100 subRun: 0 event: 223250
Comparing two wires in the same plane: return failure
%MSG
32737
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 39976!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 9 hits
Genrating OpFlash 9 with 5 hits
...
Generated 1156 OpFlashes
25-Jul-2025 05:52:38 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco_20250725T002210Z_reco2.root"
25-Jul-2025 05:52:38 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/2e/00/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              244.652       311.036       1199.81       297.305       129.609        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0897991     0.118597      0.213111      0.0992898     0.0390557       50     
reco:linecluster:LineCluster                           0.813931      0.849543       1.14229      0.842689      0.0463963       50     
reco:trajcluster:TrajCluster                           0.270755      0.282729      0.581674      0.274617      0.0429648       50     
reco:pandora:StandardPandora                            77.5159       109.503       980.113       90.6168       124.912        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000159427   0.000375814   0.00762449    0.000226963   0.00103608       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000178808   0.000296775   0.00278837    0.00024149    0.000358612      50     
reco:pandoracalo:Calorimetry                          0.00012923    0.000185699    0.0012548    0.000156748   0.000155522      50     
reco:pandorapid:Chi2ParticleID                        3.6412e-05    6.12203e-05   0.000648936   4.71025e-05   8.44853e-05      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00230111     0.0329319     0.209394      0.0125012     0.0494732       50     
reco:pmtracktccalo:Calorimetry                        8.8783e-05    0.000446756    0.0031793    0.000318042   0.000555986      50     
reco:pmtracktcpid:Chi2ParticleID                      3.1848e-05    0.000100595   0.000900667   8.0525e-05    0.000116544      50     
reco:emtrkmichelid:EmTrackMichelId                      4.76459       7.15894       11.4287       6.55787       1.49942        50     
reco:opdec:Deconvolution                                16.0899       18.216        22.4903       17.5628       1.85206        50     
reco:ophitspe:OpHitFinderDeco                           137.116       168.362       209.34        169.226       19.854         50     
reco:opflash:OpFlashFinder                              5.40441       6.17851       7.59511       6.00317      0.483403        50     
reco:solarflash:SolarOpFlash                           0.140288      0.185805      0.371487      0.154221      0.0572798       50     
reco:rns:RandomNumberSaver                            4.4177e-05    5.92251e-05   0.00026407    5.3195e-05    2.99997e-05      50     
[art]:TriggerResults:TriggerResultInserter            1.5186e-05    2.26547e-05   4.3503e-05    2.0505e-05    6.33004e-06      50     
end_path:out1:RootOutput                               7.249e-06    8.7926e-06     1.964e-05     8.011e-06    2.08102e-06      50     
end_path:out1:RootOutput(write)                        0.0976167     0.145514       1.4716       0.104441      0.191158        50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2858.9 MB
  Peak resident set size usage (VmHWM): 1370.34 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco_20250725T002210Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco_20250725T002210Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco_20250725T002210Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco_20250725T002210Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194808Z_gen_004465_supernova_g4_halfActiveVol_detsim_reco_20250725T002210Z_reco2.root.temp.json
reco2_hist.root
justIN time: 2025-08-04 18:35:51 UTC       justIN version: 01.04.00