Jobsub ID 12937.16@dunegpschedd02.fnal.gov

Jobsub ID	12937.16@dunegpschedd02.fnal.gov
Workflow ID	111
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-30 17:33:51
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce02
Last heartbeat	2025-07-31 01:54:03
From worker node	Hostname	wn32
	cpuinfo	Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-30 17:34:50
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	8h (29668s)
	CPU time	7h (27284s = 91%)
	Max RSS bytes	2598514688 (2478 MiB)
Outputting started	2025-07-31 01:49:20
Output files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco_20250730T173500Z_reco2.root
Finished	2025-07-31 01:54:03
Saved logs	justin-logs:12937.16-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:10 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:10 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:10 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:11 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:12 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:12 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  30-Jul-2025 22:44:12 -03 run: 5019 subRun: 0 event: 4060
Comparing two wires in the same plane: return failure
%MSG
30-Jul-2025 22:49:09 -03  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco_20250730T173500Z_reco2.root"
30-Jul-2025 22:49:09 -03  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/96/04/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2652.87       2781.08       2916.42       2773.91       77.027         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.005687      0.0447611     0.181343      0.0102671     0.0681089       10     
reco:pandora:StandardPandora                            2378.99       2484.19       2643.44       2478.97       76.9283        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000319654    0.0107371     0.0838679     0.0027648     0.0244561       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00033735    0.00660746     0.0363388     0.0022334     0.0107404       10     
reco:pandoracalo:Calorimetry                          0.000252273   0.00147089    0.00467978    0.000736193   0.00158101       10     
reco:pandorapid:Chi2ParticleID                        7.4953e-05    0.000366056   0.00204003    0.000196147   0.000560023      10     
reco:linecluster:LineCluster                             1.886        2.74097       3.3221        2.78622      0.391379        10     
reco:trajcluster:TrajCluster                           0.589085      0.645764      0.725875      0.652112      0.0393897       10     
reco:pmtracktc:PMAlgTrackMaker                          10.1125       11.5388       12.841        11.6541      0.795562        10     
reco:emtrkmichelid:EmTrackMichelId                      251.642       278.016       303.028       286.676       18.3464        10     
reco:solarflash:SolarOpFlash                            3.07696       3.40813       3.77035       3.43786      0.217602        10     
[art]:TriggerResults:TriggerResultInserter            4.6149e-05    7.10266e-05   0.000105436   6.82185e-05   1.86064e-05      10     
end_path:out1:RootOutput                              1.0277e-05    1.74313e-05   6.2791e-05    1.1903e-05    1.54262e-05      10     
end_path:out1:RootOutput(write)                         0.44096      0.474632      0.637685      0.451364      0.0566172       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4069.69 MB
  Peak resident set size usage (VmHWM): 2598.51 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco_20250730T173500Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco_20250730T173500Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco_20250730T173500Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T005800Z_gen_000406_supernova_g4stage1_g4stage2_detsim_reco_20250730T173500Z_reco2.root.json
reco2_hist.root