Jobsub ID 18834.68@dunegpschedd01.fnal.gov

Jobsub ID	18834.68@dunegpschedd01.fnal.gov
Workflow ID	102
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-29 08:59:08
Site	US_PuertoRico
Entry	CMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat	2025-07-29 19:34:43
From worker node	Hostname	compute-1-1.local
	cpuinfo	Intel(R) Xeon(R) CPU W3550 @ 3.07GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-07-29 09:10:07
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-29 19:01:04
Output files
Finished	2025-07-29 19:34:43
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:04 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 14:54:05 AST run: 5025 subRun: 0 event: 47800
Comparing two wires in the same plane: return failure
%MSG
29-Jul-2025 15:01:00 AST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco_20250729T091010Z_reco2.root"
29-Jul-2025 15:01:00 AST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/db/31/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2909.88       3512.77       4854.66       3001.14       815.835        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00236191     0.0243934     0.0966961    0.00840466     0.0347725       10     
reco:pandora:StandardPandora                            2638.79       3168.81       4357.99       2701.13       746.346        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000350015   0.00853673     0.0595804    0.00266911     0.0171817       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000417015   0.00223872     0.011854     0.00112646    0.00323082       10     
reco:pandoracalo:Calorimetry                          0.00028016    0.00122486    0.00580708    0.000400558   0.00175905       10     
reco:pandorapid:Chi2ParticleID                        6.3155e-05    0.000217299   0.00133777    8.82395e-05   0.000374131      10     
reco:linecluster:LineCluster                           0.843729       22.3187       76.2773      0.958652       32.8202        10     
reco:trajcluster:TrajCluster                           0.340832      0.412388      0.622213      0.394708      0.074308        10     
reco:pmtracktc:PMAlgTrackMaker                          7.03327       8.13967       11.8791       7.47618       1.41195        10     
reco:emtrkmichelid:EmTrackMichelId                      256.19        311.503       416.98        274.301       59.4783        10     
reco:solarflash:SolarOpFlash                            1.21527       1.27065       1.3669        1.26715      0.0438071       10     
[art]:TriggerResults:TriggerResultInserter            1.7029e-05    2.36767e-05   4.8131e-05    2.2234e-05    8.58265e-06      10     
end_path:out1:RootOutput                               3.147e-06     5.118e-06    1.5899e-05    4.0595e-06    3.63484e-06      10     
end_path:out1:RootOutput(write)                        0.262587      0.276477      0.304596      0.273341      0.0130939       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4054.72 MB
  Peak resident set size usage (VmHWM): 2353.46 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco_20250729T091010Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco_20250729T091010Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco_20250729T091010Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T220019Z_gen_004780_supernova_g4stage1_g4stage2_detsim_reco_20250729T091010Z_reco2.root.json
reco2_hist.root