Jobsub ID 14823.11@dunegpschedd02.fnal.gov

Jobsub ID	14823.11@dunegpschedd02.fnal.gov
Workflow ID	480
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-06 13:26:30
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-08-06 23:03:24
From worker node	Hostname	dunegli-6345683-0-fnpc17110.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 17:33:13
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (19699s)
	CPU time	5h (19487s = 98%)
	Max RSS bytes	2504421376 (2388 MiB)
Outputting started	2025-08-06 23:01:33
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco_20250806T173315Z_reco2.root
Finished	2025-08-06 23:03:24
Saved logs	justin-logs:14823.11-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:51 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:57:52 UTC run: 125 subRun: 0 event: 23060
Comparing two wires in the same plane: return failure
%MSG
06-Aug-2025 23:01:28 UTC  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco_20250806T173315Z_reco2.root"
06-Aug-2025 23:01:28 UTC  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-vd/84/80/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1728.61       1952.83       2448.46       1891.46       207.384        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00348582     0.0155292     0.0556911    0.00817621     0.0168308       10     
reco:pandora:StandardPandora                            1504.22       1709.66       2201.06       1650.87       200.713        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000252173   0.00723238     0.0679478    0.000427388    0.0202406       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000281768   0.000823321   0.00456624    0.000430885   0.00125265       10     
reco:pandoracalo:Calorimetry                          0.000252771   0.00349524    0.00904548    0.00324844    0.00318595       10     
reco:pandorapid:Chi2ParticleID                        5.3414e-05    0.00149925     0.0133303    8.46185e-05   0.00395278       10     
reco:linecluster:LineCluster                           0.392017      0.470087       0.67499      0.428109      0.0891321       10     
reco:trajcluster:TrajCluster                           0.343522      0.411004      0.576129      0.359289      0.0837968       10     
reco:pmtracktc:PMAlgTrackMaker                          5.2545        7.09919       11.5031       6.37074       1.86644        10     
reco:emtrkmichelid:EmTrackMichelId                      208.44        234.091       267.594       232.769       19.6885        10     
reco:solarflash:SolarOpFlash                           0.661804       0.75117       1.0829       0.706152      0.117583        10     
[art]:TriggerResults:TriggerResultInserter            2.3203e-05    3.78004e-05   5.4427e-05    3.5431e-05    8.81892e-06      10     
end_path:out1:RootOutput                               8.61e-06     1.35251e-05    3.881e-05    1.08065e-05   8.57095e-06      10     
end_path:out1:RootOutput(write)                        0.239588      0.319894      0.723989      0.253835      0.145597        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3971.08 MB
  Peak resident set size usage (VmHWM): 2504.42 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco_20250806T173315Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco_20250806T173315Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco_20250806T173315Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250729T193456Z_gen_002306_supernova_g4stage1_g4stage2_detsim_reco_20250806T173315Z_reco2.root.json
reco2_hist.root