justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 11111.116@dunegpschedd02.fnal.gov

Jobsub ID11111.116@dunegpschedd02.fnal.gov
Workflow ID59
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-25 01:10:43
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-07-25 05:13:35
From worker nodeHostnamecomp19-06
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-07-25 01:44:22
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco.root
JobscriptExit code0
Real time3h (12437s)
CPU time3h (11039s = 88%)
Max RSS bytes1666273280 (1589 MiB)
Outputting started2025-07-25 05:11:40
Output filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root
Finished2025-07-25 05:13:35
Saved logsjustin-logs:11111.116-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 14
Printing V view candidate length: 30
Printing W view candidate length: 18
Printing 2D view candidate length: 62
Printing background CaloHitList length: 466
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Photon identified*
Printing U view candidate length: 17
Printing V view candidate length: 30
Printing W view candidate length: 18
Printing 2D view candidate length: 65
Printing background CaloHitList length: 4
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:09:05 BST run: 5117 subRun: 0 event: 183100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:09:05 BST run: 5117 subRun: 0 event: 183100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:09:05 BST run: 5117 subRun: 0 event: 183100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:09:05 BST run: 5117 subRun: 0 event: 183100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:09:05 BST run: 5117 subRun: 0 event: 183100
TuneFullTree failed.
%MSG
33353
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40871!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 4 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 6 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 3 hits
...
Generated 1132 OpFlashes
25-Jul-2025 06:11:35 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root"
25-Jul-2025 06:11:35 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/8b/17/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              220.74        233.927       255.33        232.116       7.5275         50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.101798      0.135712      0.252459      0.112521      0.0458839       50     
reco:linecluster:LineCluster                           0.927906       1.02501       1.43244      0.981233      0.124836        50     
reco:trajcluster:TrajCluster                           0.304978      0.322481       0.42868       0.31755      0.0206537       50     
reco:pandora:StandardPandora                            78.4081       83.7857       92.9305       83.5586       2.54602        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000282839   0.00441241     0.0100983    0.00490981    0.00178809       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000283235   0.00111284    0.00885924    0.000879585   0.00127892       50     
reco:pandoracalo:Calorimetry                          0.000149879   0.000371266   0.00293481    0.000317958   0.000372947      50     
reco:pandorapid:Chi2ParticleID                        4.8314e-05    0.00011291    0.00106137    9.64075e-05   0.000136334      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00231568     0.0254623     0.182083      0.0172703     0.0304602       50     
reco:pmtracktccalo:Calorimetry                        0.000196997   0.000324462   0.000507627   0.00032581    6.11942e-05      50     
reco:pmtracktcpid:Chi2ParticleID                      6.9351e-05    8.76966e-05   0.000105802   8.7687e-05    8.65714e-06      50     
reco:emtrkmichelid:EmTrackMichelId                      5.16794       5.86494       9.97302       5.70643      0.710336        50     
reco:opdec:Deconvolution                                13.8585       17.0023       36.6136       15.2945       4.92097        50     
reco:ophitspe:OpHitFinderDeco                           115.186       119.818       129.389       119.13        2.73806        50     
reco:opflash:OpFlashFinder                              5.23678       5.71722       6.27476       5.74456      0.210286        50     
reco:solarflash:SolarOpFlash                           0.118911      0.129794      0.148171      0.129879     0.00521465       50     
reco:rns:RandomNumberSaver                            6.1505e-05    7.16822e-05   0.000314846   6.64155e-05   3.48753e-05      50     
[art]:TriggerResults:TriggerResultInserter            3.0652e-05    3.64592e-05   8.1324e-05    3.5126e-05    6.9789e-06       50     
end_path:out1:RootOutput                               7.124e-06    8.2113e-06    2.2415e-05     7.872e-06    2.07081e-06      50     
end_path:out1:RootOutput(write)                        0.0850115     0.0924736     0.112472      0.0923482    0.00427463       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3146.81 MB
  Peak resident set size usage (VmHWM): 1666.27 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T175205Z_gen_003662_supernova_g4_halfActiveVol_detsim_reco_20250725T014425Z_reco2.root.temp.json
reco2_hist.root
justIN time: 2025-08-28 19:50:05 UTC       justIN version: 01.04.01