justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40008.98@dunegpschedd02.fnal.gov

Jobsub ID40008.98@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-15 13:33:44
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-09-15 15:25:44
From worker nodeHostnamewn67
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-15 13:35:17
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_419_20231201T123836Z_gen_g4_detsim_hitreco__20240507T190737Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_205_20231207T010905Z_gen_g4_detsim_hitreco__20240510T042206Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_586_20231203T141839Z_gen_g4_detsim_hitreco__20240508T080126Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_1143_20231203T040310Z_gen_g4_detsim_hitreco__20240508T061646Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_818_20231202T112725Z_gen_g4_detsim_hitreco__20240507T211316Z_reco2.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-15 15:25:44
Saved logsjustin-logs:40008.98-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

03774       2.90758      0.592817      0.312564        500    
reco:energyrecnumu:EnergyReco                         0.000338484    0.437025       5.13065      0.0268838      0.58942        500    
reco:energyrecnue:EnergyReco                          0.000216179   0.00256465     0.107685     0.000723032   0.00666891       500    
reco:energyrecnc:EnergyReco                           0.00326817      0.86897       2.27766       1.2368       0.632906        500    
reco:energyrecnumurange:EnergyReco                    0.000330225    0.0450567      4.90381     0.00417343     0.253797        500    
reco:energyrecnumumcs:EnergyReco                      0.000911469    0.0450494      4.8777      0.00478695     0.251784        500    
reco:energyrecnumumcsllhd:EnergyReco                  0.000798555    0.0451946      4.81452     0.00489244     0.248699        500    
reco:opdec:Deconvolution                               0.611884       1.02763       2.10888       1.02492      0.194202        500    
reco:ophitspe:OpHitFinderDeco                           1.82992       1.84447       1.89616       1.84312     0.00795312       500    
reco:opflash:OpFlashFinder                            0.000199538   0.00177106     0.0152944    0.00157107    0.00122222       500    
reco:opslicer:OpSlicer                                7.5474e-05     0.363965       1.44475      0.272576      0.318377        500    
reco:rns:RandomNumberSaver                            2.3761e-05    3.83625e-05   0.000428929   3.6776e-05    1.90333e-05      500    
reco:anglereconue:NuAngularReco                       0.000221029   0.00146508     0.0832301    0.000421963   0.00477088       500    
reco:anglereconumu:NuAngularReco                      0.000111494   0.00134035     0.0810915    0.000286797   0.00469825       500    
reco:anglereconuepfps:NuAngularReco                   0.000165527   0.00144161     0.0856806    0.000394322   0.00483409       500    
reco:anglereconumupfps:NuAngularReco                  0.000160817    0.0014087     0.0848553    0.000382092   0.00479768       500    
reco:anglerecohits:NuAngularReco                      0.000156077   0.00215297     0.0884982    0.000902224   0.00531697       500    
[art]:TriggerResults:TriggerResultInserter            1.1391e-05    1.86174e-05   6.0103e-05    1.7916e-05    5.62287e-06      500    
end_path:cafmaker:CAFMaker                            0.00150019     0.0176392     0.983514     0.00466334     0.0567118       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3649.63 MB
  Peak resident set size usage (VmHWM): 1960.12 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 12:21:07 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d0/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_1143_20231203T040310Z_gen_g4_detsim_hitreco__20240508T061646Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 416, 438, 374
minwire 0: 1762
minwire 1: 470
minwire 2: 1391
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000923562, 0.996158, 0.00291881, 
Output 1: 0.358009, 0.580622, 0.0564901, 0.00487871, 
Output 2: 0.0457318, 0.910923, 0.0425456, 0.000799691, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2903!
Begin processing the 492nd record. run: 74511302 subRun: 1 event: 58692 at 15-Sep-2025 12:19:24 -03
Boundary wire vector sizes: 368, 397, 269
minwire 0: 1061
minwire 1: 604
minwire 2: 1223
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.181092, 0.0359173, 0.782991, 
Output 1: 0.0498359, 0.154133, 0.288988, 0.507043, 
Output 2: 0.0814554, 0.263683, 0.396748, 0.258113, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1966!
Begin processing the 493rd record. run: 74511302 subRun: 1 event: 58693 at 15-Sep-2025 12:19:37 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 853!
Begin processing the 494th record. run: 74511302 subRun: 1 event: 58694 at 15-Sep-2025 12:19:47 -03
Boundary wire vector sizes: 173, 152, 118
minwire 0: 590
minwire 1: 2310
minwire 2: 366
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.987366, 0.000269631, 0.0123642, 
Output 1: 0.0215885, 0.270352, 0.673961, 0.0340986, 
Output 2: 0.825432, 0.165857, 0.00853441, 0.00017667, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1690!
Begin processing the 495th record. run: 74511302 subRun: 1 event: 58695 at 15-Sep-2025 12:19:58 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1320!
Begin processing the 496th record. run: 74511302 subRun: 1 event: 58696 at 15-Sep-2025 12:20:09 -03
Boundary wire vector sizes: 408, 691, 503
minwire 0: 1238
minwire 1: 1017
minwire 2: 1282
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.78573, 0.0326539, 0.181617, 
Output 1: 0.271296, 0.570391, 0.119817, 0.038496, 
Output 2: 0.398923, 0.378586, 0.167927, 0.0545653, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2159!
Begin processing the 497th record. run: 74511302 subRun: 1 event: 58697 at 15-Sep-2025 12:20:21 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 436!
Begin processing the 498th record. run: 74511302 subRun: 1 event: 58698 at 15-Sep-2025 12:20:31 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 257!
Begin processing the 499th record. run: 74511302 subRun: 1 event: 58699 at 15-Sep-2025 12:20:41 -03
Boundary wire vector sizes: 442, 773, 634
minwire 0: 1602
minwire 1: 1457
minwire 2: 1084
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997294, 0.000482187, 0.00222426, 
Output 1: 0.00194045, 0.99416, 0.00382001, 7.98119e-05, 
Output 2: 0.996213, 0.00371752, 6.83806e-05, 1.16252e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2750!
Begin processing the 500th record. run: 74511302 subRun: 1 event: 58700 at 15-Sep-2025 12:20:55 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 201!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/07/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_419_20231201T123836Z_gen_g4_detsim_hitreco__20240507T190737Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ee/63/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_205_20231207T010905Z_gen_g4_detsim_hitreco__20240510T042206Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fa/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_586_20231203T141839Z_gen_g4_detsim_hitreco__20240508T080126Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d0/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_1143_20231203T040310Z_gen_g4_detsim_hitreco__20240508T061646Z_reco2.root
root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/95/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_968_20231203T153447Z_gen_g4_detsim_hitreco__20240508T080007Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/89/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_818_20231202T112725Z_gen_g4_detsim_hitreco__20240507T211316Z_reco2.root
.:
total 17164
-rw-r--r-- 1 nobody nobody 6513408 Sep 15 12:21 flatcaf.root
-rw-r--r-- 1 nobody nobody 5295788 Sep 15 12:21 caf.root
-rw-r--r-- 1 nobody nobody 5295788 Sep 15 12:21 caf_fd_hd_atmo_2501_20250915T133533Z.root
-rw-r--r-- 1 nobody nobody  289997 Sep 15 12:21 caf_20250915T133533Z.log
-rw-r--r-- 1 nobody nobody  118684 Sep 15 12:21 jobscript.log
-rw-r--r-- 1 nobody nobody    1314 Sep 15 12:21 caf_20250915T133533Z.file
-rw-r--r-- 1 nobody nobody    1314 Sep 15 12:21 caf_20250915T133533Z.pfns
-rw-r--r-- 1 nobody nobody    1314 Sep 15 10:35 file.list
-rw-r--r-- 1 nobody nobody    1314 Sep 15 12:21 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     962 Sep 15 10:35 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     962 Sep 15 12:21 caf_20250915T133533Z.did
-rw-r--r-- 1 nobody nobody     962 Sep 15 10:35 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 15 10:35 debugprod.log
justIN time: 2025-09-18 19:14:31 UTC       justIN version: 01.05.00