justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 38120.37@dunegpschedd02.fnal.gov

Jobsub ID38120.37@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-11 10:00:34
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-09-11 10:49:17
From worker nodeHostnamee4093
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-11 10:02:14
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_432_20231208T064827Z_gen_g4_detsim_hitreco__20240510T054933Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_644_20231202T203541Z_gen_g4_detsim_hitreco__20240508T070351Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_814_20231202T204638Z_gen_g4_detsim_hitreco__20240508T070332Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_393_20231201T170319Z_gen_g4_detsim_hitreco__20240507T201317Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_106_20231120T133507Z_gen_g4_detsim_hitreco__20240503T052604Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1046_20231204T010258Z_gen_g4_detsim_hitreco__20240509T214345Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_173_20231207T080550Z_gen_g4_detsim_hitreco__20240510T045823Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_840_20231202T085120Z_gen_g4_detsim_hitreco__20240507T205554Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_756_20231203T060756Z_gen_g4_detsim_hitreco__20240508T051716Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_503_20231202T110107Z_gen_g4_detsim_hitreco__20240507T211010Z_reco2.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-11 10:49:17
Saved logsjustin-logs:38120.37-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

059    0.0269797      1.07773     0.00286088     0.0966772       300    
reco:opdec:Deconvolution                                0.35169      0.623268        1.32        0.612441       0.1324         300    
reco:ophitspe:OpHitFinderDeco                          0.985566       1.02199       1.06713       1.02119      0.0159672       300    
reco:opflash:OpFlashFinder                            0.000270607   0.00147701    0.00862545    0.00127805    0.000975315      300    
reco:opslicer:OpSlicer                                0.00021331     0.240808      0.848947      0.200904      0.191318        300    
reco:rns:RandomNumberSaver                            2.0839e-05    8.88806e-05   0.00407704    5.6952e-05    0.000253556      300    
reco:anglereconue:NuAngularReco                       0.000157174   0.00127618     0.0251422    0.000546554   0.00233244       300    
reco:anglereconumu:NuAngularReco                      7.4921e-05    0.00090189     0.023242     0.000251076   0.00224817       300    
reco:anglereconuepfps:NuAngularReco                   0.000111469   0.00101354     0.0218624    0.000376144   0.00211274       300    
reco:anglereconumupfps:NuAngularReco                  0.000103103   0.000900621    0.0246664    0.000299987    0.0021836       300    
reco:anglerecohits:NuAngularReco                      0.000112841    0.0015902     0.0278419    0.000791037   0.00266779       300    
[art]:TriggerResults:TriggerResultInserter             9.638e-06    1.66291e-05   0.000174677   1.3901e-05    1.34835e-05      300    
end_path:cafmaker:CAFMaker                            0.00155073     0.0203282     0.561618     0.00899512     0.0450692       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3652.75 MB
  Peak resident set size usage (VmHWM): 1977.44 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 05:48:11 CDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6c/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_393_20231201T170319Z_gen_g4_detsim_hitreco__20240507T201317Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0361445, 0.0082469, 0.955609, 
Output 1: 0.105743, 0.251192, 0.261615, 0.38145, 
Output 2: 0.507786, 0.410295, 0.0724959, 0.00942324, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2067!
Begin processing the 294th record. run: 66006610 subRun: 1 event: 81494 at 11-Sep-2025 05:47:22 CDT
Boundary wire vector sizes: 500, 678, 492
minwire 0: 1528
minwire 1: 290
minwire 2: 1284
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.733663, 0.00441433, 0.261923, 
Output 1: 0.196037, 0.338066, 0.228276, 0.237621, 
Output 2: 0.0137739, 0.0385476, 0.219869, 0.727809, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3072!
Begin processing the 295th record. run: 66006610 subRun: 1 event: 81495 at 11-Sep-2025 05:47:30 CDT
Boundary wire vector sizes: 496, 444, 154
minwire 0: 650
minwire 1: 2536
minwire 2: 250
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.984977, 0.00716123, 0.00786176, 
Output 1: 0.0417346, 0.585919, 0.328273, 0.0440734, 
Output 2: 0.0922565, 0.880775, 0.0267558, 0.000213066, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1836!
Begin processing the 296th record. run: 66006610 subRun: 1 event: 81496 at 11-Sep-2025 05:47:37 CDT
Boundary wire vector sizes: 1104, 1375, 538
minwire 0: 299
minwire 1: 2562
minwire 2: 39
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.934939, 0.0329996, 0.0320611, 
Output 1: 0.413391, 0.49896, 0.0688328, 0.0188158, 
Output 2: 0.389249, 0.44098, 0.135314, 0.0344578, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2312!
Begin processing the 297th record. run: 66006610 subRun: 1 event: 81497 at 11-Sep-2025 05:47:46 CDT
Boundary wire vector sizes: 145, 121, 73
minwire 0: 1320
minwire 1: 839
minwire 2: 1535
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000461432, 0.997331, 0.00220793, 
Output 1: 0.979175, 0.020561, 0.000151518, 0.000112715, 
Output 2: 0.998971, 0.000996896, 2.45559e-05, 7.75773e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1106!
Begin processing the 298th record. run: 66006610 subRun: 1 event: 81498 at 11-Sep-2025 05:47:52 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 438!
Begin processing the 299th record. run: 66006610 subRun: 1 event: 81499 at 11-Sep-2025 05:47:58 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 742!
Begin processing the 300th record. run: 66006610 subRun: 1 event: 81500 at 11-Sep-2025 05:48:04 CDT
Boundary wire vector sizes: 254, 231, 222
minwire 0: 2698
minwire 1: 797
minwire 2: 2480
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000310845, 0.998344, 0.00134518, 
Output 1: 0.00384348, 0.99218, 0.00385694, 0.000119106, 
Output 2: 0.998265, 0.00168344, 4.97074e-05, 1.65939e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1655!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/20/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_432_20231208T064827Z_gen_g4_detsim_hitreco__20240510T054933Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6f/77/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_644_20231202T203541Z_gen_g4_detsim_hitreco__20240508T070351Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_814_20231202T204638Z_gen_g4_detsim_hitreco__20240508T070332Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6c/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_393_20231201T170319Z_gen_g4_detsim_hitreco__20240507T201317Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_106_20231120T133507Z_gen_g4_detsim_hitreco__20240503T052604Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/c1/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1046_20231204T010258Z_gen_g4_detsim_hitreco__20240509T214345Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_173_20231207T080550Z_gen_g4_detsim_hitreco__20240510T045823Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_840_20231202T085120Z_gen_g4_detsim_hitreco__20240507T205554Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/9d/0c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_756_20231203T060756Z_gen_g4_detsim_hitreco__20240508T051716Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6f/f1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_503_20231202T110107Z_gen_g4_detsim_hitreco__20240507T211010Z_reco2.root
.:
total 13472
-rw-r--r-- 1 slot1_32 slot1_32 5123750 Sep 11 05:48 flatcaf.root
-rw-r--r-- 1 slot1_32 slot1_32 4179736 Sep 11 05:48 caf.root
-rw-r--r-- 1 slot1_32 slot1_32 4179736 Sep 11 05:48 caf_fd_hd_atmo_2501_20250911T100238Z.root
-rw-r--r-- 1 slot1_32 slot1_32  184644 Sep 11 05:48 caf_20250911T100238Z.log
-rw-r--r-- 1 slot1_32 slot1_32   74300 Sep 11 05:48 jobscript.log
-rw-r--r-- 1 slot1_32 slot1_32    1949 Sep 11 05:48 caf_20250911T100238Z.file
-rw-r--r-- 1 slot1_32 slot1_32    1949 Sep 11 05:48 caf_20250911T100238Z.pfns
-rw-r--r-- 1 slot1_32 slot1_32    1949 Sep 11 05:02 file.list
-rw-r--r-- 1 slot1_32 slot1_32    1949 Sep 11 05:48 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_32 slot1_32    1377 Sep 11 05:02 all-input-dids.txt
-rw-r--r-- 1 slot1_32 slot1_32    1377 Sep 11 05:48 caf_20250911T100238Z.did
-rw-r--r-- 1 slot1_32 slot1_32    1377 Sep 11 05:02 did.list
-rw-r--r-- 1 slot1_32 slot1_32       0 Sep 11 05:03 debugprod.log
justIN time: 2025-09-19 03:47:06 UTC       justIN version: 01.05.00