Jobsub ID 22624.0@dunegpschedd01.fnal.gov

Jobsub ID	22624.0@dunegpschedd01.fnal.gov
Workflow ID	685
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-09 21:30:33
Site	IT_CNAF
Entry	CMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat	2025-08-10 04:56:49
From worker node	Hostname	wn-f01-15-06-05.cr.cnaf.infn.it
	cpuinfo	Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-09 21:31:37
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (26220s)
	CPU time	7h (25776s = 98%)
	Max RSS bytes	2502688768 (2386 MiB)
Outputting started	2025-08-10 04:48:38
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco_20250809T213145Z_reco2.root
Finished	2025-08-10 04:56:49
Saved logs	justin-logs:22624.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:38 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:39 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:39 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:39 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:39 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  10-Aug-2025 06:43:39 CEST run: 3811 subRun: 0 event: 4440
Comparing two wires in the same plane: return failure
%MSG
10-Aug-2025 06:48:33 CEST  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco_20250809T213145Z_reco2.root"
10-Aug-2025 06:48:33 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-vd/18/7a/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2099.38       2600.98       2787.43       2667.4        187.223        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00446768     0.0165164     0.0521358    0.00790888     0.0172695       10     
reco:pandora:StandardPandora                            1855.69       2319.08       2448.68       2364.44       164.233        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000231257   0.00804092     0.0771604    0.000384174    0.0230399       10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0002301    0.000774258   0.00465046    0.000367874   0.00129341       10     
reco:pandoracalo:Calorimetry                          0.000224732   0.00326677     0.0075925    0.00271603    0.00286541       10     
reco:pandorapid:Chi2ParticleID                        4.7754e-05    0.000181222   0.00107778    7.97555e-05   0.000300072      10     
reco:linecluster:LineCluster                           0.344327      0.436732      0.586163      0.435868      0.0584609       10     
reco:trajcluster:TrajCluster                           0.366128      0.497509      0.593029      0.506731      0.055557        10     
reco:pmtracktc:PMAlgTrackMaker                          5.48188       9.8391        11.8216       10.1047       1.62417        10     
reco:emtrkmichelid:EmTrackMichelId                      207.262       269.709       324.309       282.13        34.1316        10     
reco:solarflash:SolarOpFlash                           0.695329       1.00372       1.22135       1.04305       0.16412        10     
[art]:TriggerResults:TriggerResultInserter            2.1241e-05    3.97256e-05   9.5396e-05    3.6312e-05    1.96655e-05      10     
end_path:out1:RootOutput                               6.006e-06    1.16221e-05   3.4735e-05     9.625e-06    7.88512e-06      10     
end_path:out1:RootOutput(write)                        0.233782      0.376936      0.467571      0.398937      0.0726888       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3971.92 MB
  Peak resident set size usage (VmHWM): 2502.69 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco_20250809T213145Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco_20250809T213145Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco_20250809T213145Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20241024T144744Z_gen_000444_supernova_g4stage1_g4stage2_detsim_reco_20250809T213145Z_reco2.root.json
reco2_hist.root