justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47702.11@dunegpschedd01.fnal.gov

Jobsub ID47702.11@dunegpschedd01.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 18:11:30
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2025-09-17 18:29:35
From worker nodeHostnameiut2-c356.iu.edu
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 18:12:23
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_778_20231207T103826Z_gen_g4_detsim_hitreco__20240510T052136Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 18:29:35
Saved logsjustin-logs:47702.11-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

reation             0.000120887   0.00619137     0.110311      0.0020284     0.0141248       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000167956    0.0124572     0.158593     0.00390143     0.0230808       100    
reco:pandoracalo:Calorimetry                          0.000100609    0.0262817     0.730543     0.00172707     0.0983706       100    
reco:pandorapid:Chi2ParticleID                        3.6068e-05    0.000193581   0.00088215    0.000161634   0.000148643      100    
reco:cvnmap:CVNMapper                                 2.0739e-05     0.0162212     0.0943624     0.0150394     0.0128798       100    
reco:cvneva:CVNEvaluator                              1.8545e-05     0.374247       2.17365      0.435702      0.261707        100    
reco:energyrecnumu:EnergyReco                         0.000394594    0.194409       1.17152      0.0279327     0.240969        100    
reco:energyrecnue:EnergyReco                          0.000189282   0.00204581     0.039094     0.000777243    0.0046075       100    
reco:energyrecnc:EnergyReco                           0.00187552     0.325473      0.847847      0.442949       0.24161        100    
reco:energyrecnumurange:EnergyReco                    0.000299062    0.022443      0.464435     0.00302336     0.0585024       100    
reco:energyrecnumumcs:EnergyReco                      0.000585549    0.0234061     0.470337     0.00368205     0.0586763       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000454553    0.0231453     0.459057      0.0036328     0.0577835       100    
reco:opdec:Deconvolution                               0.369763       0.56188      0.781445      0.587359      0.101141        100    
reco:ophitspe:OpHitFinderDeco                          0.971319      0.976615      0.989859      0.976369     0.00321341       100    
reco:opflash:OpFlashFinder                            0.000456799   0.00154502    0.00449054    0.00147567    0.000679079      100    
reco:opslicer:OpSlicer                                0.00660969     0.292075      0.895195      0.245848      0.225845        100    
reco:rns:RandomNumberSaver                            2.5097e-05    8.80366e-05   0.000750794   6.84685e-05   8.81707e-05      100    
reco:anglereconue:NuAngularReco                       0.000200167   0.00137886     0.0307111    0.000556045   0.00336361       100    
reco:anglereconumu:NuAngularReco                      8.3918e-05    0.00106774     0.0311237    0.000306923   0.00339512       100    
reco:anglereconuepfps:NuAngularReco                   0.000145033   0.00125854     0.0328376    0.000449476   0.00354766       100    
reco:anglereconumupfps:NuAngularReco                  0.000121788   0.00112888     0.0313572    0.000342007   0.00342396       100    
reco:anglerecohits:NuAngularReco                      0.000433244   0.00197505     0.0360879    0.000934647   0.00401365       100    
[art]:TriggerResults:TriggerResultInserter             9.799e-06    1.4599e-05    5.2248e-05    1.34805e-05   4.91002e-06      100    
end_path:cafmaker:CAFMaker                            0.00234852     0.039108       2.01882     0.00885389     0.203265        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3812.9 MB
  Peak resident set size usage (VmHWM): 2152.75 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 14:29:19 EDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Begin processing the 95th record. run: 74550731 subRun: 1 event: 77895 at 17-Sep-2025 14:28:28 EDT
Boundary wire vector sizes: 41, 42, 37
minwire 0: 526
minwire 1: 2260
minwire 2: 135
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 163, 462
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 38, 337
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00663823, 0.00329879, 0.990063, 
Output 1: 0.887483, 0.111583, 0.000823025, 0.000111395, 
Output 2: 0.998585, 0.00137933, 3.23296e-05, 3.37122e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 773!
Begin processing the 96th record. run: 74550731 subRun: 1 event: 77896 at 17-Sep-2025 14:28:34 EDT
Boundary wire vector sizes: 875, 792, 652
minwire 0: 1577
minwire 1: 979
minwire 2: 1480
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00243902, 0.00484228, 0.992719, 
Output 1: 0.978624, 0.0209859, 0.000270107, 0.000120223, 
Output 2: 0.996948, 0.00297886, 6.03336e-05, 1.26251e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3023!
Begin processing the 97th record. run: 74550731 subRun: 1 event: 77897 at 17-Sep-2025 14:28:41 EDT
Boundary wire vector sizes: 227, 229, 189
minwire 0: 2313
minwire 1: 1175
minwire 2: 1985
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00363522, 0.00448664, 0.991878, 
Output 1: 0.0116323, 0.962924, 0.0244432, 0.00100098, 
Output 2: 0.995889, 0.00400702, 0.000101392, 2.92406e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2112!
Begin processing the 98th record. run: 74550731 subRun: 1 event: 77898 at 17-Sep-2025 14:28:48 EDT
Boundary wire vector sizes: 123, 192, 155
minwire 0: 265
minwire 1: 2230
minwire 2: 49
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000308099, 0.998288, 0.00140431, 
Output 1: 0.00543349, 0.989613, 0.0047929, 0.00016101, 
Output 2: 0.997216, 0.00271622, 6.59864e-05, 1.87841e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1347!
Begin processing the 99th record. run: 74550731 subRun: 1 event: 77899 at 17-Sep-2025 14:28:55 EDT
Boundary wire vector sizes: 3504, 4085, 2074
minwire 0: 1199
minwire 1: 583
minwire 2: 1119
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0659556, 0.820298, 0.113746, 
Output 1: 0.401889, 0.445873, 0.103239, 0.0489987, 
Output 2: 0.0481116, 0.18576, 0.342419, 0.42371, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3277!
Begin processing the 100th record. run: 74550731 subRun: 1 event: 77900 at 17-Sep-2025 14:29:11 EDT
Boundary wire vector sizes: 744, 1054, 709
minwire 0: 1749
minwire 1: 430
minwire 2: 1899
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.549672, 0.0797172, 0.370611, 
Output 1: 0.333436, 0.563866, 0.0756891, 0.0270083, 
Output 2: 0.0262957, 0.0873645, 0.270108, 0.616232, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1720!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_778_20231207T103826Z_gen_g4_detsim_hitreco__20240510T052136Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root
.:
total 5336
-rw-r--r--. 1 dune osgvo 2081096 Sep 17 14:29 flatcaf.root
-rw-r--r--. 1 dune osgvo 1610135 Sep 17 14:29 caf.root
-rw-r--r--. 1 dune osgvo 1610135 Sep 17 14:29 caf_fd_hd_atmo_2501_20250917T181228Z.root
-rw-r--r--. 1 dune osgvo   76479 Sep 17 14:29 caf_20250917T181228Z.log
-rw-r--r--. 1 dune osgvo   31767 Sep 17 14:29 jobscript.log
-rw-r--r--. 1 dune osgvo     374 Sep 17 14:29 caf_20250917T181228Z.file
-rw-r--r--. 1 dune osgvo     374 Sep 17 14:29 caf_20250917T181228Z.pfns
-rw-r--r--. 1 dune osgvo     374 Sep 17 14:12 file.list
-rw-r--r--. 1 dune osgvo     374 Sep 17 14:29 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo     276 Sep 17 14:12 all-input-dids.txt
-rw-r--r--. 1 dune osgvo     276 Sep 17 14:29 caf_20250917T181228Z.did
-rw-r--r--. 1 dune osgvo     276 Sep 17 14:12 did.list
-rw-r--r--. 1 dune osgvo       0 Sep 17 14:12 debugprod.log
justIN time: 2025-09-18 18:00:50 UTC       justIN version: 01.05.00