justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47811.1@dunegpschedd01.fnal.gov

Jobsub ID47811.1@dunegpschedd01.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 22:25:43
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-09-18 00:38:15
From worker nodeHostnamehnode64.pic.es
cpuinfoAMD EPYC 7402P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-17 22:26:19
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_483_20231203T170048Z_gen_g4_detsim_hitreco__20240509T195540Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_339_20231120T134530Z_gen_g4_detsim_hitreco__20240503T061351Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_75_20231201T120934Z_gen_g4_detsim_hitreco__20240507T194738Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_704_20231202T115045Z_gen_g4_detsim_hitreco__20240507T215744Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_306_20231201T170050Z_gen_g4_detsim_hitreco__20240507T195013Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_279_20231202T053402Z_gen_g4_detsim_hitreco__20240508T033907Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_294_20231202T133007Z_gen_g4_detsim_hitreco__20240507T223156Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_178_20231201T075517Z_gen_g4_detsim_hitreco__20240508T070258Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_1093_20231204T074123Z_gen_g4_detsim_hitreco__20240509T202920Z_reco2.root
JobscriptExit code0
Real time2h (7874s)
CPU time1h (5722s = 72%)
Max RSS bytes2130587648 (2031 MiB)
Outputting started2025-09-18 00:37:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T222630Z.root
Finished2025-09-18 00:38:15
Saved logsjustin-logs:47811.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.320868       2.17443      0.406934      0.206294       1000    
reco:energyrecnumu:EnergyReco                         0.000218436    0.116998       9.14054      0.0148615     0.344903       1000    
reco:energyrecnue:EnergyReco                          0.000119143   0.00307084      0.15632     0.000641539    0.0102006      1000    
reco:energyrecnc:EnergyReco                           0.00184483     0.190478      0.839047      0.211202      0.144051       1000    
reco:energyrecnumurange:EnergyReco                    0.000282368    0.0434277      3.69143     0.00299994     0.201866       1000    
reco:energyrecnumumcs:EnergyReco                      0.000439453    0.0424243      3.50308      0.003552      0.189734       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000274108    0.041297       3.47335     0.00358727     0.183922       1000    
reco:opdec:Deconvolution                               0.112641      0.366625       1.02926      0.370717      0.104933       1000    
reco:ophitspe:OpHitFinderDeco                           0.33753      0.429459       0.63787      0.450043      0.0463784      1000    
reco:opflash:OpFlashFinder                            0.000152495    0.0014972    0.00530213    0.00143344    0.000685947     1000    
reco:opslicer:OpSlicer                                3.9901e-05     0.279616       1.24786      0.223323      0.231159       1000    
reco:rns:RandomNumberSaver                            1.9741e-05    3.63916e-05   0.000821344   3.3471e-05    2.85562e-05     1000    
reco:anglereconue:NuAngularReco                       0.000159785   0.00210203     0.117213     0.000428063   0.00784966      1000    
reco:anglereconumu:NuAngularReco                      6.3862e-05    0.00187436     0.115903     0.000220937   0.00760685      1000    
reco:anglereconuepfps:NuAngularReco                   0.000105393   0.00201059     0.119481     0.000331124   0.00787603      1000    
reco:anglereconumupfps:NuAngularReco                  8.8032e-05    0.00197217      0.11775     0.000282994   0.00779624      1000    
reco:anglerecohits:NuAngularReco                      8.0632e-05    0.00266472      0.12566     0.000739927   0.00863944      1000    
[art]:TriggerResults:TriggerResultInserter             8.661e-06    1.43148e-05   7.0682e-05    1.31205e-05   5.45522e-06     1000    
end_path:cafmaker:CAFMaker                            0.00140557     0.0271447      1.42166     0.00525489     0.101983       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3804.43 MB
  Peak resident set size usage (VmHWM): 2130.59 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 0: 0.111033, 0.578681, 0.310285, 
Output 1: 0.0597746, 0.89224, 0.0473287, 0.0006569, 
Output 2: 0.0184465, 0.291227, 0.580337, 0.109989, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1859!
Begin processing the 993rd record. run: 74517585 subRun: 1 event: 109393 at 18-Sep-2025 02:27:11 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 836!
Begin processing the 994th record. run: 74517585 subRun: 1 event: 109394 at 18-Sep-2025 02:27:14 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 413!
Begin processing the 995th record. run: 74517585 subRun: 1 event: 109395 at 18-Sep-2025 02:27:17 CEST
Boundary wire vector sizes: 4369, 5479, 4013
minwire 0: 1196
minwire 1: 439
minwire 2: 1201
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.727509, 0.242137, 0.0303541, 
Output 1: 0.488223, 0.432363, 0.053839, 0.0255748, 
Output 2: 0.0342065, 0.119663, 0.293863, 0.552268, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3222!
Begin processing the 996th record. run: 74517585 subRun: 1 event: 109396 at 18-Sep-2025 02:27:33 CEST
Boundary wire vector sizes: 92, 156, 126
minwire 0: 1302
minwire 1: 1435
minwire 2: 1041
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00203217, 0.799876, 0.198092, 
Output 1: 0.964096, 0.0349578, 0.000763385, 0.000183116, 
Output 2: 0.997198, 0.00267695, 8.95891e-05, 3.54586e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1708!
Begin processing the 997th record. run: 74517585 subRun: 1 event: 109397 at 18-Sep-2025 02:27:37 CEST
Boundary wire vector sizes: 2180, 1781, 1214
minwire 0: 984
minwire 1: 1115
minwire 2: 704
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0408665, 0.761278, 0.197856, 
Output 1: 0.0442455, 0.339663, 0.527701, 0.088391, 
Output 2: 0.138986, 0.633474, 0.212362, 0.0151784, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3128!
Begin processing the 998th record. run: 74517585 subRun: 1 event: 109398 at 18-Sep-2025 02:27:45 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1148!
Begin processing the 999th record. run: 74517585 subRun: 1 event: 109399 at 18-Sep-2025 02:27:49 CEST
Boundary wire vector sizes: 10389, 13112, 7754
minwire 0: 161
minwire 1: 491
minwire 2: 63
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.515275, 0.441452, 0.0432729, 
Output 1: 0.16838, 0.324124, 0.28082, 0.226675, 
Output 2: 0.00828637, 0.0273423, 0.152579, 0.811793, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3505!
Begin processing the 1000th record. run: 74517585 subRun: 1 event: 109400 at 18-Sep-2025 02:37:28 CEST
Boundary wire vector sizes: 358, 618, 510
minwire 0: 1866
minwire 1: 89
minwire 2: 1980
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.351359, 0.0431964, 0.605445, 
Output 1: 0.0090701, 0.0656301, 0.155427, 0.769873, 
Output 2: 0.113527, 0.339975, 0.341154, 0.205344, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1376!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a3/84/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_483_20231203T170048Z_gen_g4_detsim_hitreco__20240509T195540Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/c1/b2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_339_20231120T134530Z_gen_g4_detsim_hitreco__20240503T061351Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f9/11/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_75_20231201T120934Z_gen_g4_detsim_hitreco__20240507T194738Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/80/22/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_704_20231202T115045Z_gen_g4_detsim_hitreco__20240507T215744Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_306_20231201T170050Z_gen_g4_detsim_hitreco__20240507T195013Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bc/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_279_20231202T053402Z_gen_g4_detsim_hitreco__20240508T033907Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/83/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_294_20231202T133007Z_gen_g4_detsim_hitreco__20240507T223156Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/e9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_178_20231201T075517Z_gen_g4_detsim_hitreco__20240508T070258Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a8/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_1093_20231204T074123Z_gen_g4_detsim_hitreco__20240509T202920Z_reco2.root
.:
total 51000
-rw-r--r-- 1 dune001 dune 19147943 Sep 18 02:37 flatcaf.root
-rw-r--r-- 1 dune001 dune 16088260 Sep 18 02:37 caf.root
-rw-r--r-- 1 dune001 dune 16088260 Sep 18 02:37 caf_fd_hd_atmo_2501_20250917T222630Z.root
-rw-r--r-- 1 dune001 dune   608675 Sep 18 02:37 caf_20250917T222630Z.log
-rw-r--r-- 1 dune001 dune   240601 Sep 18 02:37 jobscript.log
-rw-r--r-- 1 dune001 dune     1719 Sep 18 02:37 caf_20250917T222630Z.file
-rw-r--r-- 1 dune001 dune     1719 Sep 18 02:37 caf_20250917T222630Z.pfns
-rw-r--r-- 1 dune001 dune     1719 Sep 18 00:26 file.list
-rw-r--r-- 1 dune001 dune     1719 Sep 18 02:37 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1238 Sep 18 00:26 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1238 Sep 18 02:37 caf_20250917T222630Z.did
-rw-r--r-- 1 dune001 dune     1238 Sep 18 00:26 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 18 00:26 debugprod.log
justIN time: 2025-09-18 19:04:01 UTC       justIN version: 01.05.00