justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40063.64@dunegpschedd02.fnal.gov

Jobsub ID40063.64@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-15 14:50:48
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-09-15 15:40:36
From worker nodeHostnamehd86.dice.priv
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-15 14:52:26
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_305_20231202T100722Z_gen_g4_detsim_hitreco__20240508T070541Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_105_20231204T120339Z_gen_g4_detsim_hitreco__20240509T214623Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_768_20231208T051001Z_gen_g4_detsim_hitreco__20240510T042527Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_791_20231204T045537Z_gen_g4_detsim_hitreco__20240509T211025Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_495_20231122T052753Z_gen_g4_detsim_hitreco__20240507T172809Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_727_20231207T150938Z_gen_g4_detsim_hitreco__20240510T062157Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_54_20231120T134013Z_gen_g4_detsim_hitreco__20240507T173610Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_890_20231204T140420Z_gen_g4_detsim_hitreco__20240509T204024Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_1072_20231205T010656Z_gen_g4_detsim_hitreco__20240510T022621Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_413_20231201T090542Z_gen_g4_detsim_hitreco__20240507T203147Z_reco2.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-15 15:40:36
Saved logsjustin-logs:40063.64-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

300    
reco:energyrecnumumcsllhd:EnergyReco                  0.000241672    0.0294758      1.40944     0.00212447     0.129114        300    
reco:opdec:Deconvolution                               0.071799      0.197561      0.452821      0.199834      0.0659043       300    
reco:ophitspe:OpHitFinderDeco                          0.0977215      0.17967      0.364256      0.217128      0.0562763       300    
reco:opflash:OpFlashFinder                            0.000128285   0.000848775   0.00210226    0.000824222   0.000416315      300    
reco:opslicer:OpSlicer                                4.4205e-05     0.170255      0.799895      0.138883      0.150717        300    
reco:rns:RandomNumberSaver                            1.3225e-05    1.67917e-05   0.000199291   1.5294e-05    1.09302e-05      300    
reco:anglereconue:NuAngularReco                       7.7037e-05    0.00189965     0.0948958    0.00020114    0.00823528       300    
reco:anglereconumu:NuAngularReco                      5.6068e-05    0.00183686     0.0943182    0.000146986   0.00821355       300    
reco:anglereconuepfps:NuAngularReco                    8.333e-05    0.00190447     0.0946743    0.000210889   0.00828566       300    
reco:anglereconumupfps:NuAngularReco                  7.9192e-05    0.00187784     0.0961602    0.000191677    0.0082534       300    
reco:anglerecohits:NuAngularReco                      6.3372e-05    0.00234357     0.101339     0.000459229   0.00885592       300    
[art]:TriggerResults:TriggerResultInserter             6.883e-06    8.39674e-06   3.0088e-05     7.96e-06     2.60405e-06      300    
end_path:cafmaker:CAFMaker                            0.000990294    0.0378388      5.13088      0.0028265     0.306025        300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3885.06 MB
  Peak resident set size usage (VmHWM): 2212.6 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 15:15:24 UTC ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d9/37/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_791_20231204T045537Z_gen_g4_detsim_hitreco__20240509T211025Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 714
minwire 1: 486
minwire 2: 633
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996418, 0.000591396, 0.00299039, 
Output 1: 0.320808, 0.618404, 0.0527969, 0.00799125, 
Output 2: 0.132422, 0.779371, 0.0849246, 0.0032823, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3320!
Begin processing the 295th record. run: 6471483 subRun: 1 event: 76895 at 15-Sep-2025 15:15:09 UTC
Boundary wire vector sizes: 129, 177, 169
minwire 0: 586
minwire 1: 2191
minwire 2: 155
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.526027, 0.00519552, 0.468777, 
Output 1: 0.265802, 0.686569, 0.0462529, 0.00137619, 
Output 2: 0.0987486, 0.774157, 0.123988, 0.00310642, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1401!
Begin processing the 296th record. run: 6471483 subRun: 1 event: 76896 at 15-Sep-2025 15:15:12 UTC
Boundary wire vector sizes: 15, 94, 64
minwire 0: 2492
minwire 1: 1024
minwire 2: 2278
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00488294, 0.00424931, 0.990868, 
Output 1: 0.0043059, 0.991208, 0.00421106, 0.000275284, 
Output 2: 0.995367, 0.00454075, 8.93895e-05, 2.60452e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1226!
Begin processing the 297th record. run: 6471483 subRun: 1 event: 76897 at 15-Sep-2025 15:15:14 UTC
Boundary wire vector sizes: 431, 735, 177
minwire 0: 2247
minwire 1: 98
minwire 2: 2688
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997053, 0.000614623, 0.00233195, 
Output 1: 0.41555, 0.502643, 0.0624817, 0.0193254, 
Output 2: 0.949572, 0.0473481, 0.00282308, 0.000257358, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 987!
Begin processing the 298th record. run: 6471483 subRun: 1 event: 76898 at 15-Sep-2025 15:15:17 UTC
Boundary wire vector sizes: 186, 186, 172
minwire 0: 1825
minwire 1: 771
minwire 2: 1887
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000279706, 0.998098, 0.00162253, 
Output 1: 0.00291473, 0.992635, 0.0043211, 0.000129576, 
Output 2: 0.998016, 0.00193191, 5.05747e-05, 1.6179e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1362!
Begin processing the 299th record. run: 6471483 subRun: 1 event: 76899 at 15-Sep-2025 15:15:19 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 536!
Begin processing the 300th record. run: 6471483 subRun: 1 event: 76900 at 15-Sep-2025 15:15:21 UTC
Boundary wire vector sizes: 360, 175, 230
minwire 0: 2067
minwire 1: 897
minwire 2: 2077
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00032791, 0.998618, 0.00105353, 
Output 1: 0.00538846, 0.990861, 0.00360376, 0.000146729, 
Output 2: 0.99802, 0.00192611, 5.24346e-05, 1.80488e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2050!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/32/65/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_305_20231202T100722Z_gen_g4_detsim_hitreco__20240508T070541Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8d/74/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_105_20231204T120339Z_gen_g4_detsim_hitreco__20240509T214623Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_768_20231208T051001Z_gen_g4_detsim_hitreco__20240510T042527Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d9/37/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_791_20231204T045537Z_gen_g4_detsim_hitreco__20240509T211025Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d8/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_495_20231122T052753Z_gen_g4_detsim_hitreco__20240507T172809Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f9/49/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_727_20231207T150938Z_gen_g4_detsim_hitreco__20240510T062157Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a6/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_54_20231120T134013Z_gen_g4_detsim_hitreco__20240507T173610Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/64/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_890_20231204T140420Z_gen_g4_detsim_hitreco__20240509T204024Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d7/c8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_1072_20231205T010656Z_gen_g4_detsim_hitreco__20240510T022621Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e2/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_413_20231201T090542Z_gen_g4_detsim_hitreco__20240507T203147Z_reco2.root
.:
total 17932
-rw-r--r-- 1 dnpil001 users 6816993 Sep 15 15:15 flatcaf.root
-rw-r--r-- 1 dnpil001 users 5631298 Sep 15 15:15 caf.root
-rw-r--r-- 1 dnpil001 users 5631298 Sep 15 15:15 caf_fd_hd_atmo_2501_20250915T145234Z.root
-rw-r--r-- 1 dnpil001 users  185873 Sep 15 15:15 caf_20250915T145234Z.log
-rw-r--r-- 1 dnpil001 users   82117 Sep 15 15:15 jobscript.log
-rw-r--r-- 1 dnpil001 users    1928 Sep 15 15:15 caf_20250915T145234Z.file
-rw-r--r-- 1 dnpil001 users    1928 Sep 15 15:15 caf_20250915T145234Z.pfns
-rw-r--r-- 1 dnpil001 users    1928 Sep 15 14:52 file.list
-rw-r--r-- 1 dnpil001 users    1928 Sep 15 15:15 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users    1376 Sep 15 14:52 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users    1376 Sep 15 15:15 caf_20250915T145234Z.did
-rw-r--r-- 1 dnpil001 users    1376 Sep 15 14:52 did.list
-rw-r--r-- 1 dnpil001 users       0 Sep 15 14:52 debugprod.log
justIN time: 2025-09-18 21:56:24 UTC       justIN version: 01.05.00