justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 16330.68@dunegpschedd02.fnal.gov

Jobsub ID16330.68@dunegpschedd02.fnal.gov
Workflow ID704
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-10 13:34:58
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2025-08-11 04:45:36
From worker nodeHostnameheplnc091.pp.rl.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-10 13:36:41
Input filesfardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco.root
JobscriptExit code0
Real time15h (54249s)
CPU time13h (50347s = 92%)
Max RSS bytes1742159872 (1661 MiB)
Outputting started2025-08-11 04:40:51
Output filesfardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root
Finished2025-08-11 04:45:36
Saved logsjustin-logs:16330.68-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Printing W view candidate length: 0
Printing 2D view candidate length: 4
Printing background CaloHitList length: 727
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 29
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  11-Aug-2025 05:10:32 BST run: 249 subRun: 0 event: 183549
Comparing two wires in the same plane: return failure
%MSG
55096
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71705!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 5 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2234 OpFlashes
Begin processing the 50th record. run: 249 subRun: 0 event: 183550 at 11-Aug-2025 05:23:30 BST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 8
Printing V view candidate length: 3
Printing W view candidate length: 0
Printing 2D view candidate length: 11
Printing background CaloHitList length: 686
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
*Photon identified*
*Electron identified*
*Photon identified*
*Photon identified*
Printing U view candidate length: 4
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 4
Printing background CaloHitList length: 27
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  11-Aug-2025 05:28:44 BST run: 249 subRun: 0 event: 183550
Comparing two wires in the same plane: return failure
%MSG
55201
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71891!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 4 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 5 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 2230 OpFlashes
11-Aug-2025 05:40:38 BST  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root"
11-Aug-2025 05:40:38 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/22/29/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1005.94       1053.57        1148         1052.54       27.9761        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.106884      0.143627      0.272645      0.121267      0.0457675       50     
reco:linecluster:LineCluster                            1.05112       1.16831       1.42412       1.16282      0.0725728       50     
reco:trajcluster:TrajCluster                           0.332302      0.353885      0.713014       0.33689      0.0589414       50     
reco:pandora:StandardPandora                            292.75        315.164       362.344       315.912       10.7449        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000474338    0.0010192     0.0147939    0.000736935   0.00197075       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000480285   0.000963038   0.00711772    0.000833127   0.000885534      50     
reco:pandoracalo:Calorimetry                          0.000377257   0.00061015    0.00329022    0.000544699   0.00039349       50     
reco:pandorapid:Chi2ParticleID                        0.000122027   0.000231178   0.00185443    0.000192517   0.000233604      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00655765     0.0214877     0.0483958     0.0205738    0.00659885       50     
reco:pmtracktccalo:Calorimetry                        0.000299737   0.000988896    0.0078263    0.000849187   0.00105102       50     
reco:pmtracktcpid:Chi2ParticleID                      0.000134268   0.000299322   0.00217678    0.00027271    0.000279732      50     
reco:emtrkmichelid:EmTrackMichelId                      21.4965       29.8082       33.3892       29.716        1.85955        50     
reco:opdec:Deconvolution                                74.0256       78.1005       84.7671       78.2424       1.85308        50     
reco:ophitspe:OpHitFinderDeco                           550.839       596.196       682.442       594.276       23.7178        50     
reco:opflash:OpFlashFinder                              27.8306        30.07        31.6419       30.0983       0.89568        50     
reco:solarflash:SolarOpFlash                            1.41358       1.91798       2.1196        1.94971      0.146678        50     
reco:rns:RandomNumberSaver                            0.000104171   0.000149588   0.000869161   0.000124379   0.00010635       50     
[art]:TriggerResults:TriggerResultInserter             4.212e-05    7.7051e-05    0.000146588   7.40375e-05   1.85248e-05      50     
end_path:out1:RootOutput                               1.56e-05     2.59112e-05   6.6974e-05    2.41455e-05   9.93279e-06      50     
end_path:out1:RootOutput(write)                        0.481488      0.622688       0.70829      0.621749      0.0296944       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3219.96 MB
  Peak resident set size usage (VmHWM): 1742.16 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root.ext.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082354Z_gen_003671_supernova_g4_detsim_reco_20250810T133653Z_reco2.root.temp.json
reco2_hist.root
justIN time: 2025-08-14 03:23:32 UTC       justIN version: 01.04.00