Jobsub ID 13336.139@dunegpschedd02.fnal.gov

Jobsub ID	13336.139@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:58:57
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2025-07-31 19:41:51
From worker node	Hostname	milan09.farm.particle.cz
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 15:07:12
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13324s)
	CPU time	3h (12762s = 95%)
	Max RSS bytes	2541305856 (2423 MiB)
Outputting started	2025-07-31 18:49:17
Output files
Finished	2025-07-31 19:41:51
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

o wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:46:50 CEST run: 4924 subRun: 0 event: 41230
Flip, endpoint closer to vStart.
%MSG
31-Jul-2025 20:49:12 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco_20250731T150717Z_reco2.root"
31-Jul-2025 20:49:12 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/e6/ea/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1206.25       1297.29       1395.48       1277.94       59.0706        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00270297     0.0291932     0.123255     0.00644632     0.0465097       10     
reco:pandora:StandardPandora                            1068.58       1152.59       1248.25       1128.32       57.306         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00026154    0.00433106     0.0312954    0.00179817    0.00902745       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000262791   0.00137059    0.00644766    0.00098551     0.0017304       10     
reco:pandoracalo:Calorimetry                          0.000287791    0.0030369     0.0217547    0.00105129    0.00627131       10     
reco:pandorapid:Chi2ParticleID                         5.556e-05    0.000173965   0.00067906    0.000108975   0.000175897      10     
reco:linecluster:LineCluster                            1.03382       1.09545       1.45065       1.05887      0.118741        10     
reco:trajcluster:TrajCluster                           0.224332      0.308654      0.338339      0.316193      0.0328419       10     
reco:pmtracktc:PMAlgTrackMaker                          6.72025       7.69894       9.06427       7.4813       0.712981        10     
reco:emtrkmichelid:EmTrackMichelId                      123.139       134.001       145.173       134.722       5.71512        10     
reco:solarflash:SolarOpFlash                            1.09837       1.2895        1.50198       1.29362      0.102001        10     
[art]:TriggerResults:TriggerResultInserter            2.4351e-05    3.84271e-05    6.454e-05     3.61e-05     1.01086e-05      10     
end_path:out1:RootOutput                               5.72e-06      9.439e-06     2.742e-05     6.75e-06     6.22295e-06      10     
end_path:out1:RootOutput(write)                        0.206199      0.262801      0.296846      0.270468      0.0285188       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4039.02 MB
  Peak resident set size usage (VmHWM): 2541.31 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco_20250731T150717Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco_20250731T150717Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco_20250731T150717Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T003535Z_gen_004123_supernova_g4stage1_g4stage2_detsim_reco_20250731T150717Z_reco2.root.json
reco2_hist.root