Jobsub ID 13385.4@dunegpschedd02.fnal.gov

Jobsub ID	13385.4@dunegpschedd02.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 16:59:03
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2025-07-31 19:33:19
From worker node	Hostname	gpu10.pic.es
	cpuinfo	AMD EPYC 9554 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 17:00:00
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (7497s)
	CPU time	1h (7129s = 95%)
	Max RSS bytes	2548322304 (2430 MiB)
Outputting started	2025-07-31 19:04:58
Output files
Finished	2025-07-31 19:33:19
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:38 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:38 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:38 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:03:39 CEST run: 5027 subRun: 0 event: 38690
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 21:04:55 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco_20250731T170004Z_reco2.root"
31-Jul-2025 21:04:55 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/25/fe/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              698.978       723.164       747.962       725.075       14.8433        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00329974     0.0379258     0.154799     0.00877505     0.0568286       10     
reco:pandora:StandardPandora                            618.693       638.369       663.589       637.586       13.1151        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000207138   0.00535574     0.0441053    0.00101264     0.012947        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000197138   0.00111962    0.00657804    0.000648843   0.00183951       10     
reco:pandoracalo:Calorimetry                          0.000172868   0.000819677   0.00209682    0.000330336   0.00072282       10     
reco:pandorapid:Chi2ParticleID                        4.1159e-05    0.000125595   0.000723122   6.5499e-05    0.00019979       10     
reco:linecluster:LineCluster                            1.19956       1.26734       1.77086       1.21117      0.167972        10     
reco:trajcluster:TrajCluster                           0.173976      0.186769      0.201557      0.186391     0.00738277       10     
reco:pmtracktc:PMAlgTrackMaker                          3.31064       3.80477       4.20955       3.77583      0.242987        10     
reco:emtrkmichelid:EmTrackMichelId                      73.8062       78.0651       82.5916       77.0888       3.00447        10     
reco:solarflash:SolarOpFlash                            1.2347        1.27902       1.51648       1.25445      0.0801608       10     
[art]:TriggerResults:TriggerResultInserter             2.025e-05    2.55096e-05   4.7479e-05    2.34045e-05   7.62604e-06      10     
end_path:out1:RootOutput                               4.03e-06      9.067e-06     2.608e-05     7.545e-06    5.7742e-06       10     
end_path:out1:RootOutput(write)                        0.134141      0.145883      0.177488      0.142103      0.0124188       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4052.82 MB
  Peak resident set size usage (VmHWM): 2548.32 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco_20250731T170004Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco_20250731T170004Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco_20250731T170004Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021741Z_gen_003869_supernova_g4stage1_g4stage2_detsim_reco_20250731T170004Z_reco2.root.json
reco2_hist.root