Jobsub ID 17623.166@dunegpschedd01.fnal.gov

Jobsub ID	17623.166@dunegpschedd01.fnal.gov
Workflow ID	55
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 09:21:52
Site	US_Colorado
Entry	CMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat	2025-07-24 16:36:39
From worker node	Hostname	lnxfarm140.colorado.edu
	cpuinfo	Intel(R) Core(TM) i7-4771 CPU @ 3.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-24 13:02:12
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (12808s)
	CPU time	3h (12400s = 96%)
	Max RSS bytes	1778778112 (1696 MiB)
Outputting started	2025-07-24 16:35:41
Output files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root
Finished	2025-07-24 16:36:39
Saved logs	justin-logs:17623.166-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

al Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 36
Printing V view candidate length: 41
Printing W view candidate length: 35
Printing 2D view candidate length: 112
Printing background CaloHitList length: 664
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 32
Printing V view candidate length: 29
Printing W view candidate length: 26
Printing 2D view candidate length: 87
Printing background CaloHitList length: 1
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 10:33:31 MDT run: 5082 subRun: 0 event: 700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 10:33:31 MDT run: 5082 subRun: 0 event: 700
Comparing two wires in the same plane: return failure
%MSG
54761
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71022!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 6 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 6 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2203 OpFlashes
24-Jul-2025 10:35:39 MDT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root"
24-Jul-2025 10:35:39 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e3/88/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              194.421       250.593       286.396       251.859       15.1629        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0260341     0.0470525     0.0992567     0.0401987     0.0172057       50     
reco:linecluster:LineCluster                           0.236733      0.269098      0.408267      0.264372      0.0264045       50     
reco:trajcluster:TrajCluster                           0.0780167     0.0867428     0.381669      0.0796397     0.0422155       50     
reco:pandora:StandardPandora                            66.0931       106.559       137.19        109.328       14.5775        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000175562   0.00504938     0.0110625    0.00527685    0.00307269       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000171766   0.00120833    0.00995008    0.000827723   0.00160614       50     
reco:pandoracalo:Calorimetry                          0.00011367    0.000307575   0.00294675    0.000260832   0.000382901      50     
reco:pandorapid:Chi2ParticleID                        3.3834e-05    9.25458e-05    0.0011635    7.14155e-05   0.000153985      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00354411     0.018949      0.0715433     0.0112141     0.0176493       50     
reco:pmtracktccalo:Calorimetry                        0.00011054    0.000327656   0.000653024   0.000317015   0.000118427      50     
reco:pmtracktcpid:Chi2ParticleID                      3.9736e-05    7.49412e-05   0.000140556   7.1298e-05    2.06849e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      6.53769       9.9067        13.8616       10.2496       1.79089        50     
reco:opdec:Deconvolution                                16.6969       17.6659       22.4048       17.5664      0.806512        50     
reco:ophitspe:OpHitFinderDeco                           94.3415       100.196       108.14        100.069       2.46971        50     
reco:opflash:OpFlashFinder                              8.20793       15.4603       25.1346       14.296        3.84217        50     
reco:solarflash:SolarOpFlash                            0.1952       0.241724       0.28743      0.240641      0.0209703       50     
reco:rns:RandomNumberSaver                             1.712e-05    3.20914e-05   0.000198596   2.76825e-05   2.56906e-05      50     
[art]:TriggerResults:TriggerResultInserter             8.454e-06    1.37443e-05    2.937e-05    1.1852e-05    5.36227e-06      50     
end_path:out1:RootOutput                               2.245e-06    3.53078e-06   1.1447e-05    3.0395e-06    1.48134e-06      50     
end_path:out1:RootOutput(write)                        0.109775      0.132456      0.170386      0.131286      0.0101848       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3257.38 MB
  Peak resident set size usage (VmHWM): 1778.78 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
old_md.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T181334Z_gen_000014_supernova_g4_detsim_reco_20250724T130216Z_reco2.root.temp.json
reco2_hist.root