Jobsub ID 19706.40@dunegpschedd01.fnal.gov

Jobsub ID	19706.40@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 05:13:27
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2025-07-31 14:01:22
From worker node	Hostname	lupin33.farm.particle.cz
	cpuinfo	AMD EPYC 7532 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 05:17:40
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (28376s)
	CPU time	7h (26668s = 93%)
	Max RSS bytes	2532163584 (2414 MiB)
Outputting started	2025-07-31 13:10:37
Output files
Finished	2025-07-31 14:01:22
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

res in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:03 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:03 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:03 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 15:06:04 CEST run: 4693 subRun: 0 event: 9990
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 15:10:09 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco_20250731T051748Z_reco2.root"
31-Jul-2025 15:10:09 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d3/26/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2650.41       2790.21       2901.63       2805.45       88.2386        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00324237     0.0583029     0.355467     0.00929083      0.1067         10     
reco:pandora:StandardPandora                            2399.05       2536.42       2648.26       2548.13       87.4903        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000417544   0.00788083     0.0530507    0.00224502     0.0152869       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000401834   0.00250806     0.0118585    0.00167506    0.00322926       10     
reco:pandoracalo:Calorimetry                          0.000293803   0.00140798    0.00556014    0.000609832   0.00162951       10     
reco:pandorapid:Chi2ParticleID                        8.3921e-05    0.000248841   0.00087736    0.000124041   0.000270411      10     
reco:linecluster:LineCluster                            1.35238       1.48989       1.93452       1.4138       0.170532        10     
reco:trajcluster:TrajCluster                           0.478998      0.522874      0.581292      0.526727      0.0292158       10     
reco:pmtracktc:PMAlgTrackMaker                          10.1058       12.2862       13.5375       12.4553       1.00973        10     
reco:emtrkmichelid:EmTrackMichelId                      228.028       237.048       244.243       237.53        4.23443        10     
reco:solarflash:SolarOpFlash                            1.86581       1.95341       2.14493       1.94293      0.0866547       10     
[art]:TriggerResults:TriggerResultInserter             2.853e-05    5.00233e-05   7.7921e-05     4.469e-05    1.54787e-05      10     
end_path:out1:RootOutput                               8.19e-06     1.1553e-05     3.139e-05     9.65e-06     6.67573e-06      10     
end_path:out1:RootOutput(write)                        0.407586      0.425022      0.476025      0.417214      0.0201295       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4038.43 MB
  Peak resident set size usage (VmHWM): 2532.16 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco_20250731T051748Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco_20250731T051748Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco_20250731T051748Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250115T185540Z_gen_000999_supernova_g4stage1_g4stage2_detsim_reco_20250731T051748Z_reco2.root.json
reco2_hist.root