Jobsub ID 11128.55@dunegpschedd02.fnal.gov

Jobsub ID	11128.55@dunegpschedd02.fnal.gov
Workflow ID	60
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 02:42:48
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-07-25 11:09:05
From worker node	Hostname	iut2-c291.iu.edu
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-25 04:25:33
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (24117s)
	CPU time	6h (23483s = 97%)
	Max RSS bytes	1660854272 (1583 MiB)
Outputting started	2025-07-25 11:07:31
Output files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root
Finished	2025-07-25 11:09:05
Saved logs	justin-logs:11128.55-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

fied*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 32
Printing V view candidate length: 18
Printing W view candidate length: 26
Printing 2D view candidate length: 76
Printing background CaloHitList length: 1
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:54:30 EDT run: 5117 subRun: 0 event: 220249
Comparing two wires in the same plane: return failure
%MSG
32598
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40027!
Genrating OpFlash 0 with 5 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 8 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 3 hits
...
Generated 1178 OpFlashes
Begin processing the 50th record. run: 5117 subRun: 0 event: 220250 at 25-Jul-2025 06:59:42 EDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 7
Printing V view candidate length: 6
Printing W view candidate length: 5
Printing 2D view candidate length: 18
Printing background CaloHitList length: 445
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 7
Printing V view candidate length: 6
Printing W view candidate length: 5
Printing 2D view candidate length: 18
Printing background CaloHitList length: 4
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  25-Jul-2025 07:02:16 EDT run: 5117 subRun: 0 event: 220250
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 07:02:16 EDT run: 5117 subRun: 0 event: 220250
Comparing two wires in the same plane: return failure
%MSG
33537
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 41015!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 11 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 5 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 1207 OpFlashes
25-Jul-2025 07:07:19 EDT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root"
25-Jul-2025 07:07:19 EDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/57/54/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              398.607       477.898       528.441       481.21        25.976         50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00899399     0.0295183      0.26858      0.0200288     0.0404721       50     
reco:linecluster:LineCluster                           0.0776396     0.103933      0.140247      0.103418      0.0119091       50     
reco:trajcluster:TrajCluster                           0.023947      0.0290987     0.0590612     0.0270277     0.0060129       50     
reco:pandora:StandardPandora                            137.053       163.359       189.604       166.617       12.2754        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000401557   0.00790205     0.0218247    0.00945478    0.00467157       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000402761   0.00261721     0.0302057    0.00194654    0.00439152       50     
reco:pandoracalo:Calorimetry                          0.000256485   0.000848207   0.00899654    0.000683723   0.00119012       50     
reco:pandorapid:Chi2ParticleID                         6.38e-05     0.000221797   0.00326502    0.000167872   0.000437684      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00517466     0.0511317     0.238689      0.0283729     0.0528371       50     
reco:pmtracktccalo:Calorimetry                        0.000201948   0.000729297   0.00167924    0.000697659   0.000280404      50     
reco:pmtracktcpid:Chi2ParticleID                      7.3128e-05    0.000180321   0.000326454   0.000170909   5.66802e-05      50     
reco:emtrkmichelid:EmTrackMichelId                       7.793        11.8074       14.7317       11.8749       1.48414        50     
reco:opdec:Deconvolution                                19.0024       21.7895       24.1724       21.9915       1.25319        50     
reco:ophitspe:OpHitFinderDeco                           212.196       270.761       297.249       275.454       17.6987        50     
reco:opflash:OpFlashFinder                              7.61677       9.44062       11.5786       9.53731      0.867351        50     
reco:solarflash:SolarOpFlash                           0.174557      0.316632       0.47827      0.316413      0.0549828       50     
reco:rns:RandomNumberSaver                             7.345e-05    0.000126094   0.000794183   0.000107945   9.91955e-05      50     
[art]:TriggerResults:TriggerResultInserter            2.8087e-05    0.000118219   0.00151174    4.2683e-05    0.000298325      50     
end_path:out1:RootOutput                               7.516e-06    1.14548e-05   5.2468e-05    1.0242e-05    6.14502e-06      50     
end_path:out1:RootOutput(write)                        0.116184      0.194818      0.245981      0.197276      0.0294262       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3130.72 MB
  Peak resident set size usage (VmHWM): 1660.85 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T183903Z_gen_004405_supernova_g4_halfActiveVol_detsim_reco_20250725T042539Z_reco2.root.temp.json
reco2_hist.root