Jobsub ID 20095.4@dunegpschedd01.fnal.gov

Jobsub ID	20095.4@dunegpschedd01.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-01 05:43:40
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-08-01 12:46:13
From worker node	Hostname	dune001-7491736.0-lcg2535.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-01 05:44:39
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (25161s)
	CPU time	6h (24537s = 97%)
	Max RSS bytes	2574782464 (2455 MiB)
Outputting started	2025-08-01 12:44:01
Output files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco_20250801T054447Z_reco2.root
Finished	2025-08-01 12:46:13
Saved logs	justin-logs:20095.4-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:05 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  01-Aug-2025 12:40:06 UTC run: 5038 subRun: 0 event: 22390
TuneFullTree failed.
%MSG
01-Aug-2025 12:43:53 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco_20250801T054447Z_reco2.root"
01-Aug-2025 12:43:53 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/85/02/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2436.79       2484.43       2517.52       2491.47       26.4581        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00390196     0.0335692     0.140441     0.00851852     0.0522964       10     
reco:pandora:StandardPandora                            2211.15       2250.64       2281.19       2255.98       24.6558        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000364707   0.00826398     0.0583905    0.00305693     0.0168095       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000438029   0.00262892     0.0152685    0.00138687    0.00425191       10     
reco:pandoracalo:Calorimetry                          0.00043544    0.00210759      0.00491     0.000648103   0.00192254       10     
reco:pandorapid:Chi2ParticleID                        8.8442e-05    0.000330017   0.00191049    0.000167203   0.00052919       10     
reco:linecluster:LineCluster                            1.21499       1.36231       1.86597       1.27235      0.204629        10     
reco:trajcluster:TrajCluster                           0.486202      0.510593       0.54887       0.50527      0.0214055       10     
reco:pmtracktc:PMAlgTrackMaker                          9.76844       11.4686       12.6813       11.5311       0.92523        10     
reco:emtrkmichelid:EmTrackMichelId                      210.144       218.162       223.338       218.321       3.4828         10     
reco:solarflash:SolarOpFlash                             1.749        1.85136       2.05338       1.78104      0.122572        10     
[art]:TriggerResults:TriggerResultInserter            4.6661e-05    6.21493e-05   9.3912e-05    5.8941e-05    1.26517e-05      10     
end_path:out1:RootOutput                               9.34e-06     1.41962e-05   4.2421e-05    1.1055e-05    9.43608e-06      10     
end_path:out1:RootOutput(write)                        0.364277      0.390942       0.44575       0.38185      0.0250026       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4117.23 MB
  Peak resident set size usage (VmHWM): 2574.78 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco_20250801T054447Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco_20250801T054447Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco_20250801T054447Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T211847Z_gen_002239_supernova_g4stage1_g4stage2_detsim_reco_20250801T054447Z_reco2.root.json
reco2_hist.root