justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 42083.6@dunegpschedd02.fnal.gov

Jobsub ID42083.6@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 13:18:28
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2025-09-18 17:33:17
From worker nodeHostnamemh-7662-12.t2.ucsd.edu
cpuinfoAMD EPYC 7662 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 13:19:08
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_299_20231207T072019Z_gen_g4_detsim_hitreco__20240510T063126Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_395_20231120T135034Z_gen_g4_detsim_hitreco__20240503T061452Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_896_20231203T154247Z_gen_g4_detsim_hitreco__20240508T025412Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_359_20231207T033427Z_gen_g4_detsim_hitreco__20240510T034014Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_740_20231207T140623Z_gen_g4_detsim_hitreco__20240510T045523Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_71_20231027T230010Z_gen_g4_detsim_hitreco__20240503T045603Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_600_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031617Z_reco2.root
JobscriptExit code0
Real time3h (13504s)
CPU time2h (7443s = 55%)
Max RSS bytes2391441408 (2280 MiB)
Outputting started2025-09-18 17:04:13
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250918T131918Z.root
Finished2025-09-18 17:33:17
Saved logsjustin-logs:42083.6-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00157341     0.314761        900    
reco:pandorapid:Chi2ParticleID                        4.6331e-05    0.000274121   0.00296172    0.000210455   0.000257507      900    
reco:cvnmap:CVNMapper                                  2.174e-05     0.0228092     0.086686      0.0167659     0.020083        900    
reco:cvneva:CVNEvaluator                              1.9931e-05     0.470986       3.03195      0.615569      0.310994        900    
reco:energyrecnumu:EnergyReco                         0.000306987     0.28634       12.9556      0.0303178     0.630222        900    
reco:energyrecnue:EnergyReco                          0.000170954   0.00509996     0.289042     0.000936446    0.020892        900    
reco:energyrecnc:EnergyReco                           0.00271019     0.493434       1.73455      0.624477      0.348683        900    
reco:energyrecnumurange:EnergyReco                    0.000294677    0.0824749      7.95757     0.00444884     0.468683        900    
reco:energyrecnumumcs:EnergyReco                      0.000779637    0.0837611      7.82359     0.00529646     0.464689        900    
reco:energyrecnumumcsllhd:EnergyReco                  0.000605713    0.0828133      8.4447      0.00524177     0.467321        900    
reco:opdec:Deconvolution                               0.455332      0.810326       1.44585      0.830276      0.173122        900    
reco:ophitspe:OpHitFinderDeco                           1.34105       1.37297       1.42931       1.37416      0.015259        900    
reco:opflash:OpFlashFinder                            0.000210074   0.00233856     0.0370586    0.00199795    0.00191346       900    
reco:opslicer:OpSlicer                                6.4342e-05     0.378058       2.1602       0.289755      0.320442        900    
reco:rns:RandomNumberSaver                            2.4591e-05    6.90545e-05   0.000757937   6.17815e-05   4.11189e-05      900    
reco:anglereconue:NuAngularReco                       0.000201684   0.00358175     0.248521     0.000566003    0.0170292       900    
reco:anglereconumu:NuAngularReco                      9.0502e-05    0.00323005     0.242955     0.000319362    0.0166019       900    
reco:anglereconuepfps:NuAngularReco                   0.000134863   0.00341428     0.244339     0.000479856    0.0165782       900    
reco:anglereconumupfps:NuAngularReco                  0.000127403   0.00334613     0.245703     0.000420335    0.0165729       900    
reco:anglerecohits:NuAngularReco                      0.000137023   0.00443984     0.259011     0.00104843     0.017937        900    
[art]:TriggerResults:TriggerResultInserter             1.117e-05    2.22755e-05   0.000377008   1.91455e-05   2.07479e-05      900    
end_path:cafmaker:CAFMaker                            0.00164516     0.0461494       2.39       0.00953275     0.188038        900    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3848.36 MB
  Peak resident set size usage (VmHWM): 2391.44 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 709, 818, 578
minwire 0: 1677
minwire 1: 517
minwire 2: 1863
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.031662, 0.380701, 0.587637, 
Output 1: 0.173918, 0.668306, 0.142168, 0.0156081, 
Output 2: 0.0437488, 0.262217, 0.490557, 0.203477, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2163!
Begin processing the 895th record. run: 6470038 subRun: 1 event: 60095 at 18-Sep-2025 10:03:18 PDT
Boundary wire vector sizes: 188, 115, 158
minwire 0: 2252
minwire 1: 609
minwire 2: 2364
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000297589, 0.998514, 0.00118793, 
Output 1: 0.00452307, 0.989869, 0.00542175, 0.0001862, 
Output 2: 0.998154, 0.00179205, 5.20369e-05, 1.65639e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1088!
Begin processing the 896th record. run: 6470038 subRun: 1 event: 60096 at 18-Sep-2025 10:03:27 PDT
Boundary wire vector sizes: 88, 41, 52
minwire 0: 483
minwire 1: 1790
minwire 2: 565
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00108991, 0.994257, 0.00465327, 
Output 1: 0.976315, 0.0233835, 0.000219834, 8.12521e-05, 
Output 2: 0.998833, 0.00113668, 2.46222e-05, 5.44142e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 758!
Begin processing the 897th record. run: 6470038 subRun: 1 event: 60097 at 18-Sep-2025 10:03:35 PDT
Boundary wire vector sizes: 484, 419, 221
minwire 0: 2303
minwire 1: 40
minwire 2: 2479
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.807603, 0.00661528, 0.185782, 
Output 1: 0.49989, 0.480589, 0.0160563, 0.00346469, 
Output 2: 0.0239877, 0.105529, 0.358354, 0.51213, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1035!
Begin processing the 898th record. run: 6470038 subRun: 1 event: 60098 at 18-Sep-2025 10:03:44 PDT
Boundary wire vector sizes: 71, 70, 76
minwire 0: 2554
minwire 1: 826
minwire 2: 2239
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00323584, 0.00147987, 0.995284, 
Output 1: 0.118111, 0.877397, 0.00433944, 0.000152632, 
Output 2: 0.997097, 0.00277163, 0.00012462, 6.2973e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1211!
Begin processing the 899th record. run: 6470038 subRun: 1 event: 60099 at 18-Sep-2025 10:03:53 PDT
Boundary wire vector sizes: 228, 242, 238
minwire 0: 2135
minwire 1: 689
minwire 2: 2155
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000259891, 0.998776, 0.000963773, 
Output 1: 0.0110672, 0.982329, 0.0063752, 0.000228526, 
Output 2: 0.998188, 0.00176464, 4.59703e-05, 1.3538e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1542!
Begin processing the 900th record. run: 6470038 subRun: 1 event: 60100 at 18-Sep-2025 10:04:03 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 294!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1e/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_299_20231207T072019Z_gen_g4_detsim_hitreco__20240510T063126Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_395_20231120T135034Z_gen_g4_detsim_hitreco__20240503T061452Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/61/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_896_20231203T154247Z_gen_g4_detsim_hitreco__20240508T025412Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_359_20231207T033427Z_gen_g4_detsim_hitreco__20240510T034014Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_740_20231207T140623Z_gen_g4_detsim_hitreco__20240510T045523Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_71_20231027T230010Z_gen_g4_detsim_hitreco__20240503T045603Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_600_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031617Z_reco2.root
.:
total 47908
-rw-r--r-- 1 cuser cuser 18005425 Sep 18 10:04 flatcaf.root
-rw-r--r-- 1 cuser cuser 15101596 Sep 18 10:04 caf.root
-rw-r--r-- 1 cuser cuser 15101596 Sep 18 10:04 caf_fd_hd_atmo_2501_20250918T131918Z.root
-rw-r--r-- 1 cuser cuser   540431 Sep 18 10:04 caf_20250918T131918Z.log
-rw-r--r-- 1 cuser cuser   213381 Sep 18 10:04 jobscript.log
-rw-r--r-- 1 cuser cuser     1301 Sep 18 10:04 caf_20250918T131918Z.file
-rw-r--r-- 1 cuser cuser     1301 Sep 18 10:04 caf_20250918T131918Z.pfns
-rw-r--r-- 1 cuser cuser     1301 Sep 18 06:19 file.list
-rw-r--r-- 1 cuser cuser     1301 Sep 18 10:04 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser      958 Sep 18 06:19 all-input-dids.txt
-rw-r--r-- 1 cuser cuser      958 Sep 18 10:04 caf_20250918T131918Z.did
-rw-r--r-- 1 cuser cuser      958 Sep 18 06:19 did.list
-rw-r--r-- 1 cuser cuser        0 Sep 18 06:19 debugprod.log
justIN time: 2025-09-18 19:07:28 UTC       justIN version: 01.05.00