Jobsub ID 13156.50@dunegpschedd02.fnal.gov

Jobsub ID	13156.50@dunegpschedd02.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 01:56:17
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce02
Last heartbeat	2025-07-31 10:19:44
From worker node	Hostname	wn32
	cpuinfo	Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-31 01:58:10
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	8h (29824s)
	CPU time	7h (27491s = 92%)
	Max RSS bytes	2565431296 (2446 MiB)
Outputting started	2025-07-31 10:15:16
Output files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco_20250731T015816Z_reco2.root
Finished	2025-07-31 10:19:44
Saved logs	justin-logs:13156.50-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 07:10:25 -03 run: 4991 subRun: 0 event: 39830
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 07:15:05 -03  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco_20250731T015816Z_reco2.root"
31-Jul-2025 07:15:05 -03  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/70/bb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2664.91       2801.55       2940.58       2820.82       78.2431        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00568547     0.0452958     0.196981     0.00876137     0.0660338       10     
reco:pandora:StandardPandora                            2366.64       2497.9        2618.34       2520.28       73.7258        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000457159   0.00742309     0.0539338     0.0010469     0.0156299       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000566434   0.00300447     0.0140566    0.00200051    0.00381746       10     
reco:pandoracalo:Calorimetry                          0.000191789   0.00132325    0.00367126    0.000752578   0.00122253       10     
reco:pandorapid:Chi2ParticleID                        7.8484e-05    0.000333097   0.00102209    0.000204049   0.000316693      10     
reco:linecluster:LineCluster                            1.72437       2.10919       2.53815       2.08385      0.246593        10     
reco:trajcluster:TrajCluster                           0.577823      0.645358      0.706073      0.639249      0.0364676       10     
reco:pmtracktc:PMAlgTrackMaker                          9.63182       11.4278       13.2855       11.5125      0.929859        10     
reco:emtrkmichelid:EmTrackMichelId                      271.999       286.054       303.502       286.689       9.00078        10     
reco:solarflash:SolarOpFlash                            2.62569       2.86254       3.07756       2.85688      0.132156        10     
[art]:TriggerResults:TriggerResultInserter            5.1957e-05    0.000121988   0.000577237   7.04145e-05   0.000152518      10     
end_path:out1:RootOutput                               9.898e-06    1.52759e-05   4.5723e-05    1.08225e-05   1.07234e-05      10     
end_path:out1:RootOutput(write)                        0.431347      0.491634      0.690076      0.473017      0.0722621       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4037.27 MB
  Peak resident set size usage (VmHWM): 2565.43 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco_20250731T015816Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco_20250731T015816Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco_20250731T015816Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T171524Z_gen_003983_supernova_g4stage1_g4stage2_detsim_reco_20250731T015816Z_reco2.root.json
reco2_hist.root