Jobsub ID 14673.82@dunegpschedd02.fnal.gov

Jobsub ID	14673.82@dunegpschedd02.fnal.gov
Workflow ID	480
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-05 23:35:52
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2025-08-06 07:13:32
From worker node	Hostname	lupin22.farm.particle.cz
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 01:20:54
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (21044s)
	CPU time	5h (20221s = 96%)
	Max RSS bytes	2492841984 (2377 MiB)
Outputting started	2025-08-06 07:11:39
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco_20250806T012059Z_reco2.root
Finished	2025-08-06 07:13:32
Saved logs	justin-logs:14673.82-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:41 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 09:08:42 CEST run: 82 subRun: 0 event: 64040
Comparing two wires in the same plane: return failure
%MSG
06-Aug-2025 09:11:35 CEST  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco_20250806T012059Z_reco2.root"
06-Aug-2025 09:11:35 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/37/93/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1908.53       2073.59       2194.73       2098.15       84.154         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00334823     0.0320703      0.13319     0.00853236     0.0480177       10     
reco:pandora:StandardPandora                            1724.54       1881.78       2016.34       1919.67       86.8823        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000369341   0.00820517     0.0772985    0.000492896    0.0230315       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000329141   0.000990494   0.00577208    0.000378886   0.00160298       10     
reco:pandoracalo:Calorimetry                          0.00026498    0.00212126    0.00686492    0.000314585   0.00247016       10     
reco:pandorapid:Chi2ParticleID                         6.168e-05    0.000202402   0.00132207    7.81455e-05   0.000373389      10     
reco:linecluster:LineCluster                            1.07541       1.14058       1.6022        1.09116      0.154078        10     
reco:trajcluster:TrajCluster                           0.395328      0.440237      0.492839      0.434853      0.0291073       10     
reco:pmtracktc:PMAlgTrackMaker                          6.36078       7.0916        7.82865       7.12946       0.50247        10     
reco:emtrkmichelid:EmTrackMichelId                      147.819       181.328       212.839       178.755       18.2087        10     
reco:solarflash:SolarOpFlash                            1.40073       1.45934       1.67662       1.43038      0.0789711       10     
[art]:TriggerResults:TriggerResultInserter             3.234e-05    4.9264e-05     8.999e-05     4.072e-05    1.80084e-05      10     
end_path:out1:RootOutput                               9.01e-06     1.3206e-05     3.213e-05     1.031e-05    6.76041e-06      10     
end_path:out1:RootOutput(write)                        0.286391      0.312692      0.336567      0.313673      0.017633        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3966.55 MB
  Peak resident set size usage (VmHWM): 2492.84 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco_20250806T012059Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco_20250806T012059Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco_20250806T012059Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T210540Z_gen_006404_supernova_g4stage1_g4stage2_detsim_reco_20250806T012059Z_reco2.root.json
reco2_hist.root