Jobsub ID 17952.170@dunegpschedd01.fnal.gov

Jobsub ID	17952.170@dunegpschedd01.fnal.gov
Workflow ID	69
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 16:21:33
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod02
Last heartbeat	2025-07-26 11:55:47
From worker node	Hostname	wk04.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-25 17:26:39
Input files	fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (22903s)
	CPU time	5h (21514s = 93%)
	Max RSS bytes	2156957696 (2057 MiB)
Outputting started	2025-07-25 23:48:23
Output files
Finished	2025-07-26 11:55:47
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

t: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:04 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:44:05 BST run: 5182 subRun: 0 event: 1980
Comparing two wires in the same plane: return failure
%MSG
26-Jul-2025 00:48:14 BST  Closed output file "prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_20250725T172644Z_reco2.root"
26-Jul-2025 00:48:14 BST  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/c0/23/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1679.81       2259.46       3147.79       2237.09       385.344        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000923244    0.0192948     0.128851     0.00116229     0.0397211       10     
reco:pandora:StandardPandora                            1453.04       1984.85       2834.2        1975.69       363.012        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000229383   0.000636314   0.00285277    0.000309572   0.000764898      10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000223878   0.000634819    0.0029241    0.000343339   0.00078093       10     
reco:pandoracalo:Calorimetry                          0.00020278    0.00138316    0.00400771    0.000301641   0.00151448       10     
reco:pandorapid:Chi2ParticleID                        4.3146e-05    0.000128594   0.000706178    6.598e-05    0.000193078      10     
reco:linecluster:LineCluster                           0.362162       0.53121      0.667054      0.573986      0.0996878       10     
reco:trajcluster:TrajCluster                           0.299784      0.458701      0.532243      0.481076      0.0689678       10     
reco:pmtracktc:PMAlgTrackMaker                          6.09381       9.98968       12.0115       10.0906       1.58072        10     
reco:emtrkmichelid:EmTrackMichelId                      215.362       262.439       307.32        264.62        32.7207        10     
reco:solarflash:SolarOpFlash                           0.496519      0.804542       1.01834      0.884928      0.179531        10     
[art]:TriggerResults:TriggerResultInserter            2.5467e-05    4.39093e-05   8.6805e-05    4.0437e-05    1.67641e-05      10     
end_path:out1:RootOutput                               6.179e-06    1.33918e-05   3.1583e-05     8.675e-06    8.67654e-06      10     
end_path:out1:RootOutput(write)                        0.240553      0.357092      0.430863       0.40297      0.0743114       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3637.89 MB
  Peak resident set size usage (VmHWM): 2156.96 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_20250725T172644Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_20250725T172644Z_reco2.root
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_20250725T172644Z_reco2.root.ext.json
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T210603Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_20250725T172644Z_reco2.root.json
reco2_hist.root