Jobsub ID 11848.10@dunegpschedd02.fnal.gov

Jobsub ID	11848.10@dunegpschedd02.fnal.gov
Workflow ID	77
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-27 23:14:21
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc03
Last heartbeat	2025-07-28 04:25:07
From worker node	Hostname	wn-da-03.gina.surf.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-27 23:16:12
Input files	fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (18293s)
	CPU time	5h (18074s = 98%)
	Max RSS bytes	2544930816 (2427 MiB)
Outputting started	2025-07-28 04:21:06
Output files	fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco_20250727T231620Z_reco2.root
Finished	2025-07-28 04:25:07
Saved logs	justin-logs:11848.10-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:23 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 06:18:24 CEST run: 5166 subRun: 0 event: 9000
Comparing two wires in the same plane: return failure
%MSG
28-Jul-2025 06:20:58 CEST  Closed output file "prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco_20250727T231620Z_reco2.root"
28-Jul-2025 06:20:58 CEST  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/48/11/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1739.17       1804.64       1877.17       1812.11       49.345         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00333072     0.0309206     0.135509     0.00815054     0.0473529       10     
reco:pandora:StandardPandora                            1572.96       1644.84       1718.25       1650.14       49.6138        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000206922   0.00561714     0.0475238    0.000265143    0.014082        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000188277   0.00129595    0.00804401    0.000231264   0.00234392       10     
reco:pandoracalo:Calorimetry                          0.000169993   0.00287296    0.00521173    0.00323128    0.00178042       10     
reco:pandorapid:Chi2ParticleID                        3.8423e-05    0.000159044   0.000656726   4.4815e-05    0.000222177      10     
reco:linecluster:LineCluster                            0.56172      0.645028      0.857464      0.612823      0.0821359       10     
reco:trajcluster:TrajCluster                           0.279576      0.305671      0.328035      0.309267      0.0130346       10     
reco:pmtracktc:PMAlgTrackMaker                          4.95697       5.57367       6.17207       5.58112      0.331402        10     
reco:emtrkmichelid:EmTrackMichelId                      141.728       152.146       157.747       152.687       4.21296        10     
reco:solarflash:SolarOpFlash                            0.78997      0.862587       1.00768      0.854026      0.0564749       10     
[art]:TriggerResults:TriggerResultInserter            1.6672e-05    2.18454e-05   4.7661e-05    1.92415e-05   8.69931e-06      10     
end_path:out1:RootOutput                               3.857e-06    5.6495e-06    1.6971e-05    4.4835e-06    3.7882e-06       10     
end_path:out1:RootOutput(write)                        0.211117      0.224232      0.234537      0.225113      0.006525        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4011.53 MB
  Peak resident set size usage (VmHWM): 2544.93 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco_20250727T231620Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco_20250727T231620Z_reco2.root
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco_20250727T231620Z_reco2.root.ext.json
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T014951Z_gen_000900_supernova_g4stage1_g4stage2_detsim_reco_20250727T231620Z_reco2.root.json
reco2_hist.root