Jobsub ID 14823.10@dunegpschedd02.fnal.gov

Jobsub ID	14823.10@dunegpschedd02.fnal.gov
Workflow ID	480
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-06 13:26:30
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-08-06 23:06:06
From worker node	Hostname	dunegli-6345683-0-fnpc17110.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 17:33:35
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (19850s)
	CPU time	5h (19599s = 98%)
	Max RSS bytes	2478034944 (2363 MiB)
Outputting started	2025-08-06 23:04:26
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco_20250806T173338Z_reco2.root
Finished	2025-08-06 23:06:06
Saved logs	justin-logs:14823.10-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

omparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:52 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 23:00:53 UTC run: 68 subRun: 0 event: 44390
Comparing two wires in the same plane: return failure
%MSG
06-Aug-2025 23:04:23 UTC  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco_20250806T173338Z_reco2.root"
06-Aug-2025 23:04:23 UTC  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-vd/bb/37/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1720.81       1961.03       2424.51       1940.29       205.68         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00359512     0.0141089     0.0491782    0.00759315     0.0146516       10     
reco:pandora:StandardPandora                            1495.47       1725.74       2210.6        1670.92       208.287        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000250174   0.00883472     0.0853801    0.000321676    0.0255152       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000252008   0.000731746   0.00455485    0.000305181   0.00127483       10     
reco:pandoracalo:Calorimetry                          0.000246528   0.00305275    0.00528283    0.00418837    0.00198722       10     
reco:pandorapid:Chi2ParticleID                        5.0242e-05    0.000166395    0.0010766    6.71535e-05   0.000303495      10     
reco:linecluster:LineCluster                           0.380245      0.438952      0.678293      0.403745      0.0863643       10     
reco:trajcluster:TrajCluster                           0.283934      0.374316      0.536575       0.3535       0.0809714       10     
reco:pmtracktc:PMAlgTrackMaker                          4.57477       6.23012       11.0998       5.78397       1.80934        10     
reco:emtrkmichelid:EmTrackMichelId                      206.436       227.098       262.721       221.032       19.6659        10     
reco:solarflash:SolarOpFlash                           0.666787      0.810851       1.16345      0.706067      0.170284        10     
[art]:TriggerResults:TriggerResultInserter            1.6931e-05    3.10465e-05   5.9579e-05    2.73515e-05   1.15833e-05      10     
end_path:out1:RootOutput                               7.061e-06    9.4029e-06    2.2672e-05     7.714e-06    4.4713e-06       10     
end_path:out1:RootOutput(write)                        0.232558      0.312389      0.529987      0.260728      0.100118        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3944.6 MB
  Peak resident set size usage (VmHWM): 2478.03 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco_20250806T173338Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco_20250806T173338Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco_20250806T173338Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250725T215401Z_gen_004439_supernova_g4stage1_g4stage2_detsim_reco_20250806T173338Z_reco2.root.json
reco2_hist.root