Jobsub ID 17624.39@dunegpschedd01.fnal.gov

Jobsub ID	17624.39@dunegpschedd01.fnal.gov
Workflow ID	55
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 09:29:52
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-07-24 22:41:41
From worker node	Hostname	dunegli-44332-0-cmswn3007.fnal.gov
	cpuinfo	AMD EPYC 7542 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-24 13:04:24
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	9h (34514s)
	CPU time	9h (34111s = 98%)
	Max RSS bytes	1737920512 (1657 MiB)
Outputting started	2025-07-24 22:39:39
Output files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root
Finished	2025-07-24 22:41:41
Saved logs	justin-logs:17624.39-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:26:03 UTC run: 5089 subRun: 0 event: 165699
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:26:03 UTC run: 5089 subRun: 0 event: 165699
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:26:03 UTC run: 5089 subRun: 0 event: 165699
Comparing two wires in the same plane: return failure
%MSG
55240
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 72019!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 5 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 8 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2170 OpFlashes
Begin processing the 50th record. run: 5089 subRun: 0 event: 165700 at 24-Jul-2025 22:30:42 UTC
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 13
Printing V view candidate length: 13
Printing W view candidate length: 9
Printing 2D view candidate length: 35
Printing background CaloHitList length: 655
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Photon identified*
Printing U view candidate length: 13
Printing V view candidate length: 13
Printing W view candidate length: 9
Printing 2D view candidate length: 35
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:35:27 UTC run: 5089 subRun: 0 event: 165700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:35:27 UTC run: 5089 subRun: 0 event: 165700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:35:27 UTC run: 5089 subRun: 0 event: 165700
Comparing two wires in the same plane: return failure
%MSG
54968
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71493!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 5 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2188 OpFlashes
24-Jul-2025 22:39:31 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root"
24-Jul-2025 22:39:31 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/65/94/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              525.748       685.801       786.645       693.997       56.1244        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0049766     0.0224397     0.0579937     0.0205417     0.0118915       50     
reco:linecluster:LineCluster                           0.030213      0.0687134     0.222664      0.0634341     0.0295702       50     
reco:trajcluster:TrajCluster                          0.00958914     0.0202204     0.0558479     0.0173175    0.00935767       50     
reco:pandora:StandardPandora                            277.19        398.785       491.677       410.091       53.2452        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000450311    0.0170935     0.0479871     0.0176583     0.0110123       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000359097   0.00385151     0.0456843    0.00209983    0.00749332       50     
reco:pandoracalo:Calorimetry                          0.000286568   0.00109558     0.0144879    0.000716756   0.00195877       50     
reco:pandorapid:Chi2ParticleID                        6.0626e-05    0.000331672   0.00582421    0.000186185   0.000791897      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00537048     0.0504085      0.38576      0.0284023     0.0610341       50     
reco:pmtracktccalo:Calorimetry                        0.00034527    0.00112095    0.00720333    0.000769907    0.001051        50     
reco:pmtracktcpid:Chi2ParticleID                      7.0134e-05    0.000230012   0.00113712    0.000172545   0.000178086      50     
reco:emtrkmichelid:EmTrackMichelId                      17.9877       28.0978       38.2012       27.5606       4.46242        50     
reco:opdec:Deconvolution                                19.1838        26.87        39.2247       27.0079       4.12164        50     
reco:ophitspe:OpHitFinderDeco                           155.375       177.305       197.125       178.55        10.2151        50     
reco:opflash:OpFlashFinder                              28.3028       53.6616       68.9436       55.4399       10.5638        50     
reco:solarflash:SolarOpFlash                           0.286934      0.604262      0.971497      0.610846      0.136563        50     
reco:rns:RandomNumberSaver                            7.1166e-05    0.000199853   0.00114895    0.000148253   0.000161546      50     
[art]:TriggerResults:TriggerResultInserter            2.0559e-05    6.33902e-05   0.000258845   5.01805e-05   4.10122e-05      50     
end_path:out1:RootOutput                               8.837e-06    2.53438e-05   8.4301e-05    2.12255e-05   1.45397e-05      50     
end_path:out1:RootOutput(write)                        0.161452       0.28802      0.435397      0.299687      0.0508007       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3251.23 MB
  Peak resident set size usage (VmHWM): 1737.92 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
old_md.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T111959Z_gen_003314_supernova_g4_detsim_reco_20250724T130434Z_reco2.root.temp.json
reco2_hist.root