Jobsub ID 19874.21@dunegpschedd01.fnal.gov

Jobsub ID	19874.21@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 15:34:08
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-07-31 20:24:25
From worker node	Hostname	dune001-7488610.0-lcg2710.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 16:04:08
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 20:04:41
Output files
Finished	2025-07-31 21:10:56
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

acktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:13 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:02:14 UTC run: 4991 subRun: 0 event: 780
Flip, endpoint closer to vStart.
%MSG
31-Jul-2025 20:04:36 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco_20250731T160414Z_reco2.root"
31-Jul-2025 20:04:36 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b9/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1229.91       1416.23       1643.93       1414.02       119.556        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0040584     0.0339034     0.136635      0.0106223     0.0503323       10     
reco:pandora:StandardPandora                            1104.1        1270.42       1481.77       1263.99       114.243        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000296729    0.012762      0.103585      0.0038266     0.030321        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000310639   0.00334848     0.0225391    0.00133097    0.00640699       10     
reco:pandoracalo:Calorimetry                          0.000464251   0.00135924     0.0060045    0.000511618   0.00178425       10     
reco:pandorapid:Chi2ParticleID                        8.3316e-05    0.000356305    0.0024807    0.000126691   0.000708424      10     
reco:linecluster:LineCluster                            1.08195       1.21372       1.58496       1.19279      0.130961        10     
reco:trajcluster:TrajCluster                            0.27077      0.354333      0.391676      0.361282      0.0332119       10     
reco:pmtracktc:PMAlgTrackMaker                          5.0968        7.81407       9.87502       8.0741        1.40391        10     
reco:emtrkmichelid:EmTrackMichelId                      114.361       134.728       152.15        136.682       11.1038        10     
reco:solarflash:SolarOpFlash                             1.262        1.40153       1.47031       1.40981      0.0521141       10     
[art]:TriggerResults:TriggerResultInserter            3.4983e-05    4.81739e-05   7.5484e-05    4.3115e-05    1.45078e-05      10     
end_path:out1:RootOutput                               9.905e-06    1.16315e-05   1.8999e-05    1.07965e-05   2.52793e-06      10     
end_path:out1:RootOutput(write)                        0.213854      0.249483      0.273322      0.250472      0.0161672       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4064.49 MB
  Peak resident set size usage (VmHWM): 2472.9 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco_20250731T160414Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco_20250731T160414Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco_20250731T160414Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T022046Z_gen_000078_supernova_g4stage1_g4stage2_detsim_reco_20250731T160414Z_reco2.root.json
reco2_hist.root