Jobsub ID 19773.91@dunegpschedd01.fnal.gov

Jobsub ID	19773.91@dunegpschedd01.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 09:36:41
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce02
Last heartbeat	2025-07-31 19:37:15
From worker node	Hostname	wn30
	cpuinfo	Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 11:15:59
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 19:11:28
Output files
Finished	2025-07-31 21:10:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:19 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:20 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:20 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:06:20 -03 run: 5003 subRun: 0 event: 1460
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 16:11:20 -03  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco_20250731T111605Z_reco2.root"
31-Jul-2025 16:11:21 -03  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/e1/9c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2491.98       2731.97       2840.3        2763.56       107.097        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00513505     0.0461639     0.179879     0.00871602     0.0658898       10     
reco:pandora:StandardPandora                            2196.31       2426.63       2529.76       2451.93       101.656        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00031249    0.00889625     0.0564263    0.00489278     0.0161026       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000332866   0.00442971     0.0271236    0.00225367    0.00764258       10     
reco:pandoracalo:Calorimetry                          0.00038134    0.00165377    0.00422121    0.000884554   0.00124562       10     
reco:pandorapid:Chi2ParticleID                        7.2757e-05    0.000354864   0.00128677    0.000209585   0.000381232      10     
reco:linecluster:LineCluster                            1.51919       1.66602       2.26131       1.58955      0.209206        10     
reco:trajcluster:TrajCluster                             0.586       0.641383      0.690965      0.643642      0.0272294       10     
reco:pmtracktc:PMAlgTrackMaker                          11.032        12.5277       13.7922       12.648       0.931748        10     
reco:emtrkmichelid:EmTrackMichelId                      262.474       287.365       298.745       291.13        10.1138        10     
reco:solarflash:SolarOpFlash                            2.48196       2.61221       2.89646       2.58686       0.11207        10     
[art]:TriggerResults:TriggerResultInserter            4.7392e-05    5.67636e-05    9.21e-05     5.31975e-05   1.20658e-05      10     
end_path:out1:RootOutput                               9.384e-06    1.26171e-05   3.6058e-05     9.891e-06    7.83499e-06      10     
end_path:out1:RootOutput(write)                        0.431086      0.467611      0.540001      0.457466      0.0336454       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4080.28 MB
  Peak resident set size usage (VmHWM): 2602.51 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco_20250731T111605Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco_20250731T111605Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco_20250731T111605Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T223227Z_gen_000146_supernova_g4stage1_g4stage2_detsim_reco_20250731T111605Z_reco2.root.json
reco2_hist.root