Jobsub ID 21366.22@dunegpschedd01.fnal.gov

Jobsub ID	21366.22@dunegpschedd01.fnal.gov
Workflow ID	545
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-06 16:02:39
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc01
Last heartbeat	2025-08-06 20:15:37
From worker node	Hostname	wn-la-18.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 16:04:03
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (14885s)
	CPU time	4h (14641s = 98%)
	Max RSS bytes	2504634368 (2388 MiB)
Outputting started	2025-08-06 20:12:09
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco_20250806T160413Z_reco2.root
Finished	2025-08-06 20:15:37
Saved logs	justin-logs:21366.22-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:37 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  06-Aug-2025 22:09:38 CEST run: 226 subRun: 0 event: 79820
Comparing two wires in the same plane: return failure
%MSG
06-Aug-2025 22:12:01 CEST  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco_20250806T160413Z_reco2.root"
06-Aug-2025 22:12:01 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b1/5b/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1176.89       1458.81       1865.43       1370.91       207.597        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0062822     0.0330816     0.120864      0.0134964     0.0412236       10     
reco:pandora:StandardPandora                            1034.05       1294.94       1630.45       1218.54       186.081        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000251045    0.0235793     0.224938     0.000322876    0.0671614       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000255732   0.00196619     0.0113869    0.000304404   0.00356849       10     
reco:pandoracalo:Calorimetry                          0.000204856   0.00475466     0.0139932    0.00292273     0.0049674       10     
reco:pandorapid:Chi2ParticleID                        4.2674e-05    0.000221978   0.00117295    5.8256e-05    0.000360549      10     
reco:linecluster:LineCluster                           0.930569       1.00806       1.4627       0.954835      0.152669        10     
reco:trajcluster:TrajCluster                           0.347081      0.368184      0.428167      0.363894      0.0220157       10     
reco:pmtracktc:PMAlgTrackMaker                          6.16162       7.0042        7.97204       6.97284      0.582918        10     
reco:emtrkmichelid:EmTrackMichelId                      131.107       153.947       223.444       141.939       28.1297        10     
reco:solarflash:SolarOpFlash                            1.08069       1.22352       1.42172       1.21039      0.0819785       10     
[art]:TriggerResults:TriggerResultInserter            2.1151e-05    3.53838e-05   8.0882e-05    2.7351e-05    1.76913e-05      10     
end_path:out1:RootOutput                               5.468e-06    9.2858e-06     3.288e-05     6.835e-06    7.92186e-06      10     
end_path:out1:RootOutput(write)                        0.204457      0.255878      0.269149      0.261101      0.0175424       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3975.16 MB
  Peak resident set size usage (VmHWM): 2504.63 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco_20250806T160413Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco_20250806T160413Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco_20250806T160413Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T062244Z_gen_007982_supernova_g4stage1_g4stage2_detsim_reco_20250806T160413Z_reco2.root.json
reco2_hist.root