Jobsub ID 13239.164@dunegpschedd02.fnal.gov

Jobsub ID	13239.164@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 08:30:39
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce510
Last heartbeat	2025-07-31 16:36:14
From worker node	Hostname	b9p13p8753.cern.ch
	cpuinfo	Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-07-31 10:13:43
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 16:06:10
Output files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco_20250731T101349Z_reco2.root
Finished	2025-07-31 16:36:14
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:58 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:01:59 CEST run: 5017 subRun: 0 event: 46530
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:02:00 CEST run: 5017 subRun: 0 event: 46530
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:02:00 CEST run: 5017 subRun: 0 event: 46530
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:02:00 CEST run: 5017 subRun: 0 event: 46530
TuneFullTree failed.
%MSG
31-Jul-2025 18:06:01 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco_20250731T101349Z_reco2.root"
31-Jul-2025 18:06:01 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d7/f3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1904.31       2067.57       2264.52       2039.16       109.696        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00309746     0.030288      0.127692     0.00912558     0.045691        10     
reco:pandora:StandardPandora                            1654.78       1801.75       1977.33       1767.17       97.8467        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000311955   0.00816748     0.0492439    0.00427936     0.013815        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000339346   0.00211747     0.0105494    0.00134247    0.00283766       10     
reco:pandoracalo:Calorimetry                          0.000227414   0.00116306    0.00539678    0.000424868    0.0016325       10     
reco:pandorapid:Chi2ParticleID                        6.1395e-05    0.000201916   0.00113446    9.9383e-05    0.000311552      10     
reco:linecluster:LineCluster                            1.06567       1.1509        1.51978        1.117       0.126006        10     
reco:trajcluster:TrajCluster                           0.339445      0.480962      0.612333      0.481989      0.0620082       10     
reco:pmtracktc:PMAlgTrackMaker                          8.94911       10.5247       13.8868       10.467        1.44994        10     
reco:emtrkmichelid:EmTrackMichelId                      225.088       251.733       274.645       254.329       14.7996        10     
reco:solarflash:SolarOpFlash                            1.3113        1.52325       1.71133       1.53855      0.105295        10     
[art]:TriggerResults:TriggerResultInserter            2.6305e-05    9.63298e-05   0.000625169   3.05625e-05   0.000177623      10     
end_path:out1:RootOutput                               6.212e-06    1.39507e-05   6.6715e-05    7.2275e-06    1.77542e-05      10     
end_path:out1:RootOutput(write)                        0.245327      0.367358      0.445121      0.391967      0.0558039       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4037.54 MB
  Peak resident set size usage (VmHWM): 2557.37 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco_20250731T101349Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco_20250731T101349Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco_20250731T101349Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T034608Z_gen_004653_supernova_g4stage1_g4stage2_detsim_reco_20250731T101349Z_reco2.root.json
reco2_hist.root