There will be a short downtime on Monday morning UTC 8th September for a version upgrade.
Jobsub ID 22831.94@dunegpschedd01.fnal.gov
| Jobsub ID | 22831.94@dunegpschedd01.fnal.gov | |
| Workflow ID | 728 | |
| Stage ID | 1 | |
| User name | higuera@fnal.gov | |
| HTCondor Group | group_dune.prod.mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-10 14:24:25 | |
| Site | US_FNAL-FermiGrid | |
| Entry | FNAL_GPGrid_ce04_mcore_op_duneonly | |
| Last heartbeat | 2025-08-10 18:16:44 | |
| From worker node | Hostname | dunegli-6561904-0-fnpc19112.fnal.gov |
| cpuinfo | AMD EPYC 7502 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-10 16:06:27 | |
| Input files | fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco.root | |
| Jobscript | Exit code | 0 |
| Real time | 2h (7775s) | |
| CPU time | 2h (7699s = 99%) | |
| Max RSS bytes | 1668386816 (1591 MiB) | |
| Outputting started | 2025-08-10 18:16:03 | |
| Output files | fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root | |
| Finished | 2025-08-10 18:16:44 | |
| Saved logs | justin-logs:22831.94-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
ating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 11 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 6 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 4 hits
...
Generated 1168 OpFlashes
Begin processing the 49th record. run: 251 subRun: 0 event: 494199 at 10-Aug-2025 18:11:37 UTC
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 2
Printing 2D view candidate length: 2
Printing background CaloHitList length: 524
Printing New CaloHitList Names:
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
*Electron identified*
*Electron identified*
Printing U view candidate length: 2
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 2
Printing background CaloHitList length: 13
Printing New CaloHitList Names:
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
in function: Infer
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
34496
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 42596!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 10 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 5 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 6 hits
...
Generated 1249 OpFlashes
Begin processing the 50th record. run: 251 subRun: 0 event: 494200 at 10-Aug-2025 18:13:49 UTC
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 1
Printing 2D view candidate length: 1
Printing background CaloHitList length: 509
Printing New CaloHitList Names:
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 21
Printing New CaloHitList Names:
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
in function: Infer
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
33541
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40789!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 6 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 8 hits
Genrating OpFlash 7 with 7 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 5 hits
...
Generated 1165 OpFlashes
10-Aug-2025 18:16:00 UTC Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root"
10-Aug-2025 18:16:00 UTC Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/0b/a1/prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 127.072 153.685 229.537 149.276 22.9238 50
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00154489 0.00669581 0.063988 0.00242347 0.0104458 50
reco:linecluster:LineCluster 0.0136688 0.0416011 0.127549 0.0268825 0.0348593 50
reco:trajcluster:TrajCluster 0.0027882 0.00500308 0.0192048 0.00424964 0.0029154 50
reco:pandora:StandardPandora 51.1918 69.1227 103.321 68.9915 12.8713 50
reco:pandoraTrack:LArPandoraTrackCreation 0.000112251 0.000235758 0.00250929 0.000190696 0.000326857 50
reco:pandoraShower:LArPandoraModularShowerCreation 0.000149211 0.000212476 0.0014953 0.000180097 0.000184894 50
reco:pandoracalo:Calorimetry 9.9161e-05 0.000142016 0.000736797 0.000126621 8.69193e-05 50
reco:pandorapid:Chi2ParticleID 2.7481e-05 4.13666e-05 0.000385443 3.22605e-05 4.94692e-05 50
reco:pmtracktc:PMAlgTrackMaker 0.00141301 0.00388044 0.00796747 0.003669 0.00153839 50
reco:pmtracktccalo:Calorimetry 6.1211e-05 0.000176585 0.00144954 0.000152121 0.000195421 50
reco:pmtracktcpid:Chi2ParticleID 2.2841e-05 5.62301e-05 0.000405564 4.88805e-05 5.25276e-05 50
reco:emtrkmichelid:EmTrackMichelId 2.91246 3.82666 5.74779 3.63659 0.685759 50
reco:opdec:Deconvolution 8.92137 9.9371 12.4663 9.61924 0.946697 50
reco:ophitspe:OpHitFinderDeco 57.5257 66.4833 107.133 61.1975 13.0104 50
reco:opflash:OpFlashFinder 3.3509 4.03311 4.88025 3.99017 0.36834 50
reco:solarflash:SolarOpFlash 0.118764 0.142263 0.214143 0.132848 0.0244013 50
reco:rns:RandomNumberSaver 3.596e-05 5.05656e-05 0.000330503 4.43205e-05 4.03832e-05 50
[art]:TriggerResults:TriggerResultInserter 1.298e-05 1.74716e-05 5.037e-05 1.607e-05 5.64235e-06 50
end_path:out1:RootOutput 5e-06 8.05104e-06 2.568e-05 7.59e-06 2.94447e-06 50
end_path:out1:RootOutput(write) 0.0709076 0.0809486 0.105508 0.0784304 0.00867045 50
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 3140.3 MB
Peak resident set size usage (VmHWM): 1668.39 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor
============OUTPUT FILE: prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root.ext.json
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root.json
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root.temp.json
reco2_hist.root