Jobsub ID 19842.37@dunegpschedd01.fnal.gov

Jobsub ID	19842.37@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:56:57
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2025-07-31 20:15:26
From worker node	Hostname	milan11.farm.particle.cz
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 14:59:09
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (16236s)
	CPU time	4h (15436s = 95%)
	Max RSS bytes	2547748864 (2429 MiB)
Outputting started	2025-07-31 19:29:46
Output files
Finished	2025-07-31 20:15:26
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:17 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:18 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:19 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:26:19 CEST run: 5025 subRun: 0 event: 41840
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 21:29:40 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco_20250731T145914Z_reco2.root"
31-Jul-2025 21:29:40 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/93/fd/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1298.67       1586.02       1967.34       1544.08       212.544        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00262307     0.0483537     0.239197      0.0189502     0.0751826       10     
reco:pandora:StandardPandora                            1152.79       1395.03       1703.32       1381.03       174.141        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000215439    0.0065256     0.0401625    0.00249761     0.011554        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000243388    0.0018432    0.00715307    0.00152721    0.00188628       10     
reco:pandoracalo:Calorimetry                          0.000121699   0.00173432      0.01047     0.000538338   0.00296632       10     
reco:pandorapid:Chi2ParticleID                         3.765e-05    0.000167963   0.000508048    9.409e-05    0.00016978       10     
reco:linecluster:LineCluster                            1.01846       1.17737       1.73869       1.07239      0.237653        10     
reco:trajcluster:TrajCluster                           0.308174      0.355093      0.472521      0.339791      0.0450867       10     
reco:pmtracktc:PMAlgTrackMaker                          5.77124       8.58009       10.3459       8.69361       1.28174        10     
reco:emtrkmichelid:EmTrackMichelId                      134.668       179.169       258.624       150.938       43.8947        10     
reco:solarflash:SolarOpFlash                            1.27629       1.34587       1.38523       1.34863      0.0284746       10     
[art]:TriggerResults:TriggerResultInserter             2.262e-05    7.08278e-05   0.000358778   4.2335e-05    9.64675e-05      10     
end_path:out1:RootOutput                               6.36e-06      9.033e-06     1.876e-05     7.72e-06     3.48967e-06      10     
end_path:out1:RootOutput(write)                        0.273281      0.304631      0.380196      0.289364      0.0330153       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4038.09 MB
  Peak resident set size usage (VmHWM): 2547.75 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco_20250731T145914Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco_20250731T145914Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco_20250731T145914Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T211925Z_gen_004184_supernova_g4stage1_g4stage2_detsim_reco_20250731T145914Z_reco2.root.json
reco2_hist.root