Jobsub ID 13233.176@dunegpschedd02.fnal.gov

Jobsub ID	13233.176@dunegpschedd02.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 08:22:37
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2025-07-31 20:16:07
From worker node	Hostname	tds409.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 08:54:53
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	10h (37802s)
	CPU time	6h (23563s = 62%)
	Max RSS bytes	2511867904 (2395 MiB)
Outputting started	2025-07-31 19:24:57
Output files
Finished	2025-07-31 20:16:02
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:51 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:51 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:51 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:20:52 CEST run: 5031 subRun: 0 event: 26090
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 21:24:47 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco_20250731T085501Z_reco2.root"
31-Jul-2025 21:24:48 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b4/b4/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2175.58       2474.94       2641.6        2562.49       169.563        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00599155     0.042632      0.162085      0.0140867     0.058136        10     
reco:pandora:StandardPandora                            1972.91       2238.22       2390.7        2320.99       159.961        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000265245    0.0130504     0.108155      0.0013242     0.0317989       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000311426   0.00315179     0.0210713    0.00151922    0.00601895       10     
reco:pandoracalo:Calorimetry                          0.000201364   0.00188628    0.00623725    0.000592036   0.00217894       10     
reco:pandorapid:Chi2ParticleID                         5.144e-05    0.000278918   0.00172597    0.000140268   0.000483728      10     
reco:linecluster:LineCluster                            2.42922       5.80142       15.2764       4.57348       3.65149        10     
reco:trajcluster:TrajCluster                           0.352555      0.459648      0.514162      0.457836      0.044271        10     
reco:pmtracktc:PMAlgTrackMaker                          8.86542       10.6602       12.5317       10.641        1.10473        10     
reco:emtrkmichelid:EmTrackMichelId                      160.391       212.294       227.917       219.701       18.7145        10     
reco:solarflash:SolarOpFlash                            2.40464       7.09076       27.5772       4.45355       7.21193        10     
[art]:TriggerResults:TriggerResultInserter             2.867e-05    4.27098e-05   7.3271e-05    3.9891e-05    1.10721e-05      10     
end_path:out1:RootOutput                               6.04e-06     1.01271e-05    2.77e-05      8.155e-06    5.95555e-06      10     
end_path:out1:RootOutput(write)                        0.253013      0.359051      0.400161       0.36664      0.0382222       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4037.93 MB
  Peak resident set size usage (VmHWM): 2511.87 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco_20250731T085501Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco_20250731T085501Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco_20250731T085501Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T065315Z_gen_002609_supernova_g4stage1_g4stage2_detsim_reco_20250731T085501Z_reco2.root.json
reco2_hist.root