Jobsub ID 18887.100@dunegpschedd01.fnal.gov

Jobsub ID	18887.100@dunegpschedd01.fnal.gov
Workflow ID	102
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-29 11:30:15
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-07-29 22:22:09
From worker node	Hostname	dunegli-44471-0-cmswn4002.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-29 11:48:05
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	9h (35121s)
	CPU time	9h (34805s = 99%)
	Max RSS bytes	2538594304 (2420 MiB)
Outputting started	2025-07-29 21:33:27
Output files
Finished	2025-07-29 22:22:09
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 21:31:47 UTC run: 5017 subRun: 0 event: 26200
Comparing two wires in the same plane: return failure
%MSG
29-Jul-2025 21:33:25 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco_20250729T114807Z_reco2.root"
29-Jul-2025 21:33:25 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/19/53/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2733.5        3492.21       3710.72       3546.38       259.437        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0122806     0.0212341     0.0447997     0.0187195    0.00891216       10     
reco:pandora:StandardPandora                            2632.73       3200.17       3419.43       3214.9        202.569        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000219016    0.0167083      0.12012     0.00522543     0.0347131       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000236479   0.00230241     0.0142855    0.00118111    0.00401673       10     
reco:pandoracalo:Calorimetry                          0.000182908   0.00156042    0.00805097    0.000458051   0.00243875       10     
reco:pandorapid:Chi2ParticleID                        4.1008e-05    0.000292098   0.00188701    0.000138819   0.000532739      10     
reco:linecluster:LineCluster                           0.124633      0.158356      0.238687       0.14246      0.0350734       10     
reco:trajcluster:TrajCluster                           0.186996      0.217369      0.285637      0.216403      0.0266241       10     
reco:pmtracktc:PMAlgTrackMaker                          2.8745        3.70956       4.20836       3.74122      0.342271        10     
reco:emtrkmichelid:EmTrackMichelId                      96.3345       287.437       338.217       316.668       71.1477        10     
reco:solarflash:SolarOpFlash                           0.292855      0.314641      0.334261      0.312666      0.0100884       10     
[art]:TriggerResults:TriggerResultInserter            1.8766e-05    4.37363e-05   0.00010006    3.79775e-05   2.00874e-05      10     
end_path:out1:RootOutput                               5.02e-06     1.5674e-05    4.7531e-05    1.28745e-05   1.0935e-05       10     
end_path:out1:RootOutput(write)                        0.137721      0.160581      0.172583      0.160153     0.00918788       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4040.37 MB
  Peak resident set size usage (VmHWM): 2538.59 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco_20250729T114807Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco_20250729T114807Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco_20250729T114807Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250201T234722Z_gen_002620_supernova_g4stage1_g4stage2_detsim_reco_20250729T114807Z_reco2.root.json
reco2_hist.root