Jobsub ID 21366.13@dunegpschedd01.fnal.gov

Jobsub ID	21366.13@dunegpschedd01.fnal.gov
Workflow ID	545
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-06 16:02:39
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc01
Last heartbeat	2025-08-06 23:17:06
From worker node	Hostname	wn-db-06.gina.surf.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 16:03:22
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (25796s)
	CPU time	7h (25433s = 98%)
	Max RSS bytes	2493501440 (2377 MiB)
Outputting started	2025-08-06 23:13:19
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco_20250806T160329Z_reco2.root
Finished	2025-08-06 23:17:06
Saved logs	justin-logs:21366.13-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ne: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:06 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 01:10:07 CEST run: 82 subRun: 0 event: 58380
Comparing two wires in the same plane: return failure
%MSG
07-Aug-2025 01:13:11 CEST  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco_20250806T160329Z_reco2.root"
07-Aug-2025 01:13:11 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/61/a2/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2057.06       2550.19       3450.17       2324.21       459.27         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00611518     0.0319448     0.122026      0.0113968     0.0432454       10     
reco:pandora:StandardPandora                            1882.79       2347.36       3194.13       2132.88       436.919        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000192436    0.0085196     0.0829195    0.000245532     0.0248         10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000202615   0.000527504   0.00317109    0.000228459   0.000881571      10     
reco:pandoracalo:Calorimetry                          0.00017311    0.00423728     0.0105919    0.00352393    0.00336503       10     
reco:pandorapid:Chi2ParticleID                        3.8653e-05    0.00011355    0.000658132   4.9494e-05    0.000182156      10     
reco:linecluster:LineCluster                           0.968982       1.03855       1.45138      0.983714      0.140085        10     
reco:trajcluster:TrajCluster                           0.282474       0.32593      0.428911      0.316039      0.0404096       10     
reco:pmtracktc:PMAlgTrackMaker                          5.22105       5.65687       6.14338       5.62292      0.328733        10     
reco:emtrkmichelid:EmTrackMichelId                      161.221       194.265       247.06        184.458       27.4925        10     
reco:solarflash:SolarOpFlash                            1.20313       1.2624        1.47798       1.23732      0.0752641       10     
[art]:TriggerResults:TriggerResultInserter            2.0039e-05    2.89402e-05   6.0566e-05    2.7337e-05    1.12527e-05      10     
end_path:out1:RootOutput                               4.799e-06    7.9913e-06    2.5398e-05    5.7765e-06    5.95233e-06      10     
end_path:out1:RootOutput(write)                        0.209573      0.227266       0.26615      0.222888      0.0152273       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3962.59 MB
  Peak resident set size usage (VmHWM): 2493.5 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco_20250806T160329Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco_20250806T160329Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco_20250806T160329Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250727T203718Z_gen_005838_supernova_g4stage1_g4stage2_detsim_reco_20250806T160329Z_reco2.root.json
reco2_hist.root