Jobsub ID 14713.154@dunegpschedd02.fnal.gov

Jobsub ID	14713.154@dunegpschedd02.fnal.gov
Workflow ID	480
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-06 03:20:02
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-08-07 06:31:54
From worker node	Hostname	wn59
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-06 12:11:24
Input files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	17h (64139s)
	CPU time	14h (53699s = 83%)
	Max RSS bytes	2496131072 (2380 MiB)
Outputting started	2025-08-07 06:00:25
Output files	fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco_20250806T121551Z_reco2.root
Finished	2025-08-07 06:31:07
Saved logs	justin-logs:14713.154-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:15 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:15 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:15 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:15 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:15 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:16 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:17 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:17 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:18 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:18 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:19 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:19 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:19 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:19 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:19 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:19 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:20 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:20 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:21 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:21 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  07-Aug-2025 02:47:21 -03 run: 195 subRun: 0 event: 84850
Comparing two wires in the same plane: return failure
%MSG
07-Aug-2025 02:57:51 -03  Closed output file "prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco_20250806T121551Z_reco2.root"
07-Aug-2025 02:57:52 -03  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/93/3b/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              4768.84       5994.12       8639.93       5722.98       1070.82        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.027618      0.193956       1.14896      0.0451736     0.327408        10     
reco:pandora:StandardPandora                            4222.21       5051.8        6313.21       5067.67       564.567        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00378741     0.017014       0.12897     0.00430204     0.0373269       10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0016939     0.0156959     0.127684     0.00293084     0.0373543       10     
reco:pandoracalo:Calorimetry                          0.00337746     0.0241502     0.0997235     0.0139469     0.0284568       10     
reco:pandorapid:Chi2ParticleID                        0.000486086   0.00251986     0.0145134    0.000993474   0.00407125       10     
reco:linecluster:LineCluster                            1.79783       2.18343       3.38848       2.07008      0.460893        10     
reco:trajcluster:TrajCluster                           0.978989       1.57438       2.28023       1.54521       0.38124        10     
reco:pmtracktc:PMAlgTrackMaker                          15.6675       21.301        28.0407       20.4973       3.33427        10     
reco:emtrkmichelid:EmTrackMichelId                      524.125       910.641       3370.99       628.097       822.107        10     
reco:solarflash:SolarOpFlash                            2.42004       3.10651       3.9002        3.04233      0.495318        10     
[art]:TriggerResults:TriggerResultInserter            0.000192631   0.000514733   0.00157407    0.000307806   0.000436453      10     
end_path:out1:RootOutput                              0.000295615   0.000529929   0.000876334   0.000525314   0.000141373      10     
end_path:out1:RootOutput(write)                        0.971426       3.22429       15.7988       1.7524        4.25555        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3968 MB
  Peak resident set size usage (VmHWM): 2496.13 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco_20250806T121551Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco_20250806T121551Z_reco2.root
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco_20250806T121551Z_reco2.root.ext.json
prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250802T032933Z_gen_008485_supernova_g4stage1_g4stage2_detsim_reco_20250806T121551Z_reco2.root.json
reco2_hist.root