justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 16330.57@dunegpschedd02.fnal.gov

Jobsub ID16330.57@dunegpschedd02.fnal.gov
Workflow ID704
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-10 13:34:58
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce01
Last heartbeat2025-08-11 05:52:54
From worker nodeHostnamewn59
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-10 13:40:14
Input filesfardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco.root
JobscriptExit code0
Real time15h (56603s)
CPU time13h (47597s = 84%)
Max RSS bytes1719107584 (1639 MiB)
Outputting started2025-08-11 05:23:44
Output filesfardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root
Finished2025-08-11 05:51:26
Saved logsjustin-logs:16330.57-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 5 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 6 hits
...
Generated 2118 OpFlashes
Begin processing the 49th record. run: 249 subRun: 0 event: 27449 at 11-Aug-2025 02:02:31 -03
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 2
Printing W view candidate length: 0
Printing 2D view candidate length: 2
Printing background CaloHitList length: 632
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 33
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
54888
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71688!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 10 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2188 OpFlashes
Begin processing the 50th record. run: 249 subRun: 0 event: 27450 at 11-Aug-2025 02:12:02 -03
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 8
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 8
Printing background CaloHitList length: 653
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
Printing U view candidate length: 4
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 4
Printing background CaloHitList length: 21
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
54703
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71088!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 5 hits
Genrating OpFlash 4 with 5 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2213 OpFlashes
11-Aug-2025 02:22:09 -03  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root"
11-Aug-2025 02:22:09 -03  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/d2/b7/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              491.547       993.102       8374.13       684.984       1300.55        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.168221      0.255998       0.73925      0.204945      0.111734        50     
reco:linecluster:LineCluster                            1.5665        2.22488       7.33445       1.9605       0.874241        50     
reco:trajcluster:TrajCluster                           0.512162      0.646857       1.10093      0.597221      0.141184        50     
reco:pandora:StandardPandora                            184.891       539.61        7912.21       293.305       1182.55        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.00283288     0.0259253     0.976315     0.00587515     0.135878        50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00314618    0.00831718     0.0576655     0.0061363    0.00947839       50     
reco:pandoracalo:Calorimetry                          0.00276258    0.00750092     0.0410704    0.00576857    0.00726493       50     
reco:pandorapid:Chi2ParticleID                        0.000565059   0.00186627     0.0178804    0.00116316    0.00281558       50     
reco:pmtracktc:PMAlgTrackMaker                         0.0101484     0.0663913     0.984458      0.0358893     0.140589        50     
reco:pmtracktccalo:Calorimetry                        0.00078661     0.011218      0.078965     0.00577453     0.0160078       50     
reco:pmtracktcpid:Chi2ParticleID                      0.00019627     0.0025351     0.016997     0.00172951    0.00286995       50     
reco:emtrkmichelid:EmTrackMichelId                      17.4682       44.4116       1005.24       22.9315       137.565        50     
reco:opdec:Deconvolution                                41.423         56.91        86.3489       56.6499       9.21477        50     
reco:ophitspe:OpHitFinderDeco                           214.985       279.981       391.117       273.685       35.9806        50     
reco:opflash:OpFlashFinder                              14.8748       67.147        2220.82       21.0412       307.734        50     
reco:solarflash:SolarOpFlash                            0.40402      0.974994       8.47968      0.758301       1.13035        50     
reco:rns:RandomNumberSaver                            0.000807647    0.0026441     0.0543882    0.00127583    0.00746859       50     
[art]:TriggerResults:TriggerResultInserter            5.1839e-05    0.000327167   0.00389174    0.00013734    0.000650148      50     
end_path:out1:RootOutput                              0.000194738   0.000400662   0.00495614    0.000280848   0.000659773      50     
end_path:out1:RootOutput(write)                        0.313341      0.787704       5.00081      0.633365      0.674894        50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3194.34 MB
  Peak resident set size usage (VmHWM): 1719.11 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root.ext.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T055050Z_gen_000549_supernova_g4_detsim_reco_20250810T134141Z_reco2.root.temp.json
reco2_hist.root
justIN time: 2025-08-14 03:23:31 UTC       justIN version: 01.04.00