Jobsub ID 14673.98@dunegpschedd02.fnal.gov

Jobsub ID	14673.98@dunegpschedd02.fnal.gov
Workflow ID	480
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-05 23:35:53
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc03
Last heartbeat	2025-08-06 08:19:02
From worker node	Hostname	wn-lb-07.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 01:26:37
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (24472s)
	CPU time	6h (24072s = 98%)
	Max RSS bytes	2486738944 (2371 MiB)
Outputting started	2025-08-06 08:14:29
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco_20250806T012647Z_reco2.root
Finished	2025-08-06 08:19:02
Saved logs	justin-logs:14673.98-dunegpschedd02.fnal.gov.logs.tgz
List job events (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

e: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 10:08:23 CEST run: 82 subRun: 0 event: 29600
Comparing two wires in the same plane: return failure
%MSG
06-Aug-2025 10:14:18 CEST  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco_20250806T012647Z_reco2.root"
06-Aug-2025 10:14:18 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/aa/d8/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1850.72       2417.1        2967.96       2383.41       318.182        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00665712     0.0427785     0.129481       0.02787      0.0416804       10     
reco:pandora:StandardPandora                            1600.22       2140.78       2584.83       2134.38       275.018        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00040769     0.0185935     0.181313     0.000504503    0.0542398       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000335301   0.000807715   0.00450836    0.000397553   0.00123385       10     
reco:pandoracalo:Calorimetry                          0.000331476    0.0130153     0.021824      0.0165658    0.00822513       10     
reco:pandorapid:Chi2ParticleID                        6.4106e-05    0.00015465    0.000780597    8.374e-05    0.000209241      10     
reco:linecluster:LineCluster                           0.956297       1.0183        1.40753      0.970708      0.131224        10     
reco:trajcluster:TrajCluster                            0.30772      0.344891      0.398533       0.34226      0.0244052       10     
reco:pmtracktc:PMAlgTrackMaker                          5.04077       5.72962       6.20276       5.80224      0.389503        10     
reco:emtrkmichelid:EmTrackMichelId                      212.472       267.764       374.409       239.26        54.6169        10     
reco:solarflash:SolarOpFlash                            1.11123       1.15504       1.30853       1.13074      0.0596045       10     
[art]:TriggerResults:TriggerResultInserter            3.3619e-05    5.09973e-05   8.4566e-05    4.5153e-05    1.63979e-05      10     
end_path:out1:RootOutput                               8.723e-06    1.45298e-05   3.4612e-05    1.0901e-05    7.60485e-06      10     
end_path:out1:RootOutput(write)                        0.212744      0.229186       0.26853       0.23028      0.0149508       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3953.37 MB
  Peak resident set size usage (VmHWM): 2486.74 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco_20250806T012647Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco_20250806T012647Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco_20250806T012647Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T174427Z_gen_002960_supernova_g4stage1_g4stage2_detsim_reco_20250806T012647Z_reco2.root.json
reco2_hist.root