Jobsub ID 19842.4@dunegpschedd01.fnal.gov

Jobsub ID	19842.4@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:56:57
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc03
Last heartbeat	2025-07-31 19:37:41
From worker node	Hostname	wn-la-04.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 14:58:58
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13063s)
	CPU time	3h (12687s = 97%)
	Max RSS bytes	2566524928 (2447 MiB)
Outputting started	2025-07-31 18:36:41
Output files
Finished	2025-07-31 19:37:41
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

res in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:18 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:34:19 CEST run: 5055 subRun: 0 event: 9140
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 20:36:34 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco_20250731T145908Z_reco2.root"
31-Jul-2025 20:36:35 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ed/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1195.2        1275.95       1315.6        1288.8        30.7266        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00352749     0.0323736     0.126824      0.0121034     0.0446692       10     
reco:pandora:StandardPandora                            1039.23       1125.62       1166.24       1134.09       32.093         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000239478    0.007077      0.0526584    0.000917735    0.0153361       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000240919   0.00170085     0.0073248    0.000902644   0.00213603       10     
reco:pandoracalo:Calorimetry                          0.000252967   0.00167125    0.00447891    0.000350828   0.00171745       10     
reco:pandorapid:Chi2ParticleID                        4.1921e-05    0.000177725   0.00112933    7.5357e-05    0.000317486      10     
reco:linecluster:LineCluster                           0.901391       1.0181        1.47306      0.986251      0.156703        10     
reco:trajcluster:TrajCluster                           0.292781      0.365004      0.454128      0.365564      0.0414661       10     
reco:pmtracktc:PMAlgTrackMaker                          5.11159       7.22835       8.79547       7.41803       1.01257        10     
reco:emtrkmichelid:EmTrackMichelId                      132.612       140.187       145.549       140.459       4.14509        10     
reco:solarflash:SolarOpFlash                            1.11107       1.23386       1.45166       1.22727      0.0858597       10     
[art]:TriggerResults:TriggerResultInserter            2.2134e-05    2.74983e-05   5.7165e-05    2.4411e-05    9.94048e-06      10     
end_path:out1:RootOutput                               4.586e-06    6.7328e-06    2.1732e-05     5.092e-06    5.0033e-06       10     
end_path:out1:RootOutput(write)                        0.203349      0.252088      0.271132      0.258891      0.019545        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4036.89 MB
  Peak resident set size usage (VmHWM): 2566.52 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco_20250731T145908Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco_20250731T145908Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco_20250731T145908Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T235107Z_gen_000914_supernova_g4stage1_g4stage2_detsim_reco_20250731T145908Z_reco2.root.json
reco2_hist.root