Jobsub ID 19649.1@dunegpschedd01.fnal.gov

Jobsub ID	19649.1@dunegpschedd01.fnal.gov
Workflow ID	106
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-30 23:49:09
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-07-31 02:05:37
From worker node	Hostname	dunegli-6240620-0-fnpc23024.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-30 23:50:29
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (8063s)
	CPU time	2h (7966s = 98%)
	Max RSS bytes	2562330624 (2443 MiB)
Outputting started	2025-07-31 02:04:52
Output files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco_20250730T235041Z_reco2.root
Finished	2025-07-31 02:05:37
Saved logs	justin-logs:19649.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 02:03:15 UTC run: 5027 subRun: 0 event: 33020
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 02:04:49 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco_20250730T235041Z_reco2.root"
31-Jul-2025 02:04:49 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/91/9d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              723.36        797.452       908.387       768.997       70.3248        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00364844    0.00728467     0.010729     0.00741318    0.00181175       10     
reco:pandora:StandardPandora                            633.163       690.546       796.153       670.768       56.8249        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000203529   0.00784786     0.074889     0.000249596    0.0223501       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000185223   0.00124416     0.0094966    0.00020165    0.00276093       10     
reco:pandoracalo:Calorimetry                          0.000153663   0.00178082    0.00513207    0.00173967    0.00157385       10     
reco:pandorapid:Chi2ParticleID                         3.663e-05    0.000167633   0.00123265    4.39535e-05   0.000355217      10     
reco:linecluster:LineCluster                           0.108444      0.152058      0.363818      0.128525      0.0715184       10     
reco:trajcluster:TrajCluster                           0.173272      0.196659      0.228116      0.189277      0.016573        10     
reco:pmtracktc:PMAlgTrackMaker                          3.18167       3.86184       4.14338       3.94683      0.264285        10     
reco:emtrkmichelid:EmTrackMichelId                      85.2993       102.22        133.785       93.8065       15.5216        10     
reco:solarflash:SolarOpFlash                           0.293062      0.308178      0.321937      0.306897     0.00948072       10     
[art]:TriggerResults:TriggerResultInserter            1.5089e-05    2.6723e-05    6.4073e-05    2.4331e-05    1.30346e-05      10     
end_path:out1:RootOutput                               6.742e-06    1.08075e-05   3.4786e-05     8.041e-06    8.04653e-06      10     
end_path:out1:RootOutput(write)                        0.139593      0.147782      0.162108      0.145451     0.00730447       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4030.43 MB
  Peak resident set size usage (VmHWM): 2562.33 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco_20250730T235041Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco_20250730T235041Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco_20250730T235041Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T015237Z_gen_003302_supernova_g4stage1_g4stage2_detsim_reco_20250730T235041Z_reco2.root.json
reco2_hist.root