Jobsub ID 21366.30@dunegpschedd01.fnal.gov

Jobsub ID	21366.30@dunegpschedd01.fnal.gov
Workflow ID	545
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-06 16:02:39
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc03
Last heartbeat	2025-08-06 21:30:16
From worker node	Hostname	wn-da-12.gina.surf.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 16:03:33
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (19420s)
	CPU time	5h (19115s = 98%)
	Max RSS bytes	2503831552 (2387 MiB)
Outputting started	2025-08-06 21:27:14
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco_20250806T160339Z_reco2.root
Finished	2025-08-06 21:30:16
Saved logs	justin-logs:21366.30-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:48 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:24:49 CEST run: 79 subRun: 0 event: 91900
Comparing two wires in the same plane: return failure
%MSG
06-Aug-2025 23:27:07 CEST  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco_20250806T160339Z_reco2.root"
06-Aug-2025 23:27:07 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5e/36/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1749.6        1913.82       2088.56       1923.2        106.08         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00403771     0.0287085     0.114539     0.00840251     0.0423093       10     
reco:pandora:StandardPandora                            1603.61       1757.08       1911.08       1767.94       98.3912        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000179142   0.00462111     0.0440768    0.000232529    0.013152        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000185783   0.000410447   0.00222384    0.000212776   0.000604587      10     
reco:pandoracalo:Calorimetry                          0.000162931   0.00200345    0.00459273    0.00245537    0.00147724       10     
reco:pandorapid:Chi2ParticleID                        3.3925e-05    8.48929e-05   0.000476158   4.18355e-05   0.00013046       10     
reco:linecluster:LineCluster                            0.9355       0.994585       1.36243      0.949085      0.123814        10     
reco:trajcluster:TrajCluster                           0.272715      0.297463      0.335254      0.292679      0.0159859       10     
reco:pmtracktc:PMAlgTrackMaker                          5.09546       5.33645       5.60645       5.36584      0.178837        10     
reco:emtrkmichelid:EmTrackMichelId                      134.474       148.634       169.363       147.122       8.84415        10     
reco:solarflash:SolarOpFlash                            1.18241       1.22399       1.36622       1.2016       0.0516176       10     
[art]:TriggerResults:TriggerResultInserter             1.58e-05     2.20402e-05   4.6229e-05    1.96275e-05   8.23843e-06      10     
end_path:out1:RootOutput                               4.028e-06    5.9785e-06    2.0329e-05    4.5635e-06    4.79029e-06      10     
end_path:out1:RootOutput(write)                        0.211303      0.221428      0.241548       0.21829     0.00872317       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3969.58 MB
  Peak resident set size usage (VmHWM): 2503.83 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco_20250806T160339Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco_20250806T160339Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco_20250806T160339Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T010244Z_gen_009190_supernova_g4stage1_g4stage2_detsim_reco_20250806T160339Z_reco2.root.json
reco2_hist.root