Jobsub ID 19733.33@dunegpschedd01.fnal.gov

Jobsub ID	19733.33@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 08:38:39
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce513
Last heartbeat	2025-07-31 13:40:58
From worker node	Hostname	b9p09p7269.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-07-31 09:33:15
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 13:21:48
Output files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco_20250731T093323Z_reco2.root
Finished	2025-07-31 13:40:58
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:30 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:19:31 CEST run: 4985 subRun: 0 event: 13740
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 15:21:44 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco_20250731T093323Z_reco2.root"
31-Jul-2025 15:21:44 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/0d/75/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1151.16       1341.11       1477.34        1363         108.664        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00356254     0.0297469     0.113804     0.00991691     0.0418595       10     
reco:pandora:StandardPandora                            976.166       1169.71       1297.97       1186.83       101.675        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000324162   0.00691951     0.0420065    0.00333524     0.0118781       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000764154   0.00197674    0.00621021    0.00138597    0.00156942       10     
reco:pandoracalo:Calorimetry                          0.000206071   0.000567948   0.00179293    0.000409252   0.000453119      10     
reco:pandorapid:Chi2ParticleID                        5.9761e-05    0.000173286   0.000521573   0.000105895   0.000153529      10     
reco:linecluster:LineCluster                           0.976518       1.06623       1.40518       1.02114      0.127113        10     
reco:trajcluster:TrajCluster                           0.279363      0.330206       0.37105      0.326225      0.0296154       10     
reco:pmtracktc:PMAlgTrackMaker                          6.63476       7.98545       10.0216       8.13767      0.887815        10     
reco:emtrkmichelid:EmTrackMichelId                      130.273       160.416       178.213       163.113       13.6112        10     
reco:solarflash:SolarOpFlash                            1.07988       1.30047       1.49179       1.3037       0.0948487       10     
[art]:TriggerResults:TriggerResultInserter             3.839e-05    5.55432e-05   0.000122651    4.728e-05    2.40246e-05      10     
end_path:out1:RootOutput                               6.57e-06     1.07571e-05   3.4871e-05     8.245e-06    8.09272e-06      10     
end_path:out1:RootOutput(write)                        0.198132      0.253779      0.298803      0.255273      0.0319443       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4034.24 MB
  Peak resident set size usage (VmHWM): 2519.09 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco_20250731T093323Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco_20250731T093323Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco_20250731T093323Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T191910Z_gen_001374_supernova_g4stage1_g4stage2_detsim_reco_20250731T093323Z_reco2.root.json
reco2_hist.root