Jobsub ID 39988.101@dunegpschedd02.fnal.gov

Jobsub ID	39988.101@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-09-15 15:16:47
From worker node	Hostname	cit2-c056.mwt2.org
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 12:35:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_1101_20231204T041938Z_gen_g4_detsim_hitreco__20240509T202900Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_440_20231203T081107Z_gen_g4_detsim_hitreco__20240508T033758Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_718_20231208T045411Z_gen_g4_detsim_hitreco__20240510T052035Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_285_20231208T040402Z_gen_g4_detsim_hitreco__20240510T034810Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_71_20231205T115445Z_gen_g4_detsim_hitreco__20240510T043003Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_707_20231202T090227Z_gen_g4_detsim_hitreco__20240507T210603Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_240_20231202T043827Z_gen_g4_detsim_hitreco__20240507T203246Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_412_20231204T051528Z_gen_g4_detsim_hitreco__20240509T213106Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_764_20231203T082425Z_gen_g4_detsim_hitreco__20240508T060603Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_522_20231202T120815Z_gen_g4_detsim_hitreco__20240507T212346Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9658s)
	CPU time	1h (5070s = 52%)
	Max RSS bytes	2014539776 (1921 MiB)
Outputting started	2025-09-15 15:16:33
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T123549Z.root
Finished	2025-09-15 15:16:47
Saved logs	justin-logs:39988.101-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00213516     0.364444       1.14536      0.456025      0.236537       1000    
reco:energyrecnumurange:EnergyReco                    0.000302305    0.075729       4.99096     0.00346215      0.36696       1000    
reco:energyrecnumumcs:EnergyReco                      0.000678132    0.0761481       5.089      0.00416674     0.373217       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000518102    0.0764106      5.23156     0.00412654     0.378564       1000    
reco:opdec:Deconvolution                               0.292985      0.615518       1.39434      0.630394      0.150987       1000    
reco:ophitspe:OpHitFinderDeco                          0.857024      0.959622       1.17978      0.999157      0.0652406      1000    
reco:opflash:OpFlashFinder                            0.000205458   0.00187577    0.00534189    0.00179393    0.000866946     1000    
reco:opslicer:OpSlicer                                6.3238e-05     0.325814       1.20326       0.26026      0.261812       1000    
reco:rns:RandomNumberSaver                            1.9461e-05    5.40484e-05   0.00118638    4.2718e-05    7.78188e-05     1000    
reco:anglereconue:NuAngularReco                       0.000166004   0.00257422     0.174558     0.000553248   0.00967082      1000    
reco:anglereconumu:NuAngularReco                      0.000105681   0.00236906     0.176679     0.000359877   0.00975198      1000    
reco:anglereconuepfps:NuAngularReco                   0.000146989   0.00251564     0.173131     0.000475088   0.00978115      1000    
reco:anglereconumupfps:NuAngularReco                  0.000139525   0.00241019     0.154879      0.000474     0.00900911      1000    
reco:anglerecohits:NuAngularReco                      0.000129253   0.00325187     0.167196     0.00102798     0.0100923      1000    
[art]:TriggerResults:TriggerResultInserter            1.2716e-05    2.05494e-05   9.7139e-05    2.02665e-05   6.68432e-06     1000    
end_path:cafmaker:CAFMaker                            0.00127451     0.0314656      1.65603     0.00721536      0.10972       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3678.23 MB
  Peak resident set size usage (VmHWM): 2014.54 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17!
Begin processing the 990th record. run: 6403947 subRun: 1 event: 52290 at 15-Sep-2025 11:15:21 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1039!
Begin processing the 991st record. run: 6403947 subRun: 1 event: 52291 at 15-Sep-2025 11:15:27 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 49!
Begin processing the 992nd record. run: 6403947 subRun: 1 event: 52292 at 15-Sep-2025 11:15:32 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 212!
Begin processing the 993rd record. run: 6403947 subRun: 1 event: 52293 at 15-Sep-2025 11:15:38 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1732!
Begin processing the 994th record. run: 6403947 subRun: 1 event: 52294 at 15-Sep-2025 11:15:44 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1579!
Begin processing the 995th record. run: 6403947 subRun: 1 event: 52295 at 15-Sep-2025 11:15:50 EDT
Boundary wire vector sizes: 456, 423, 434
minwire 0: 972
minwire 1: 2175
minwire 2: 471
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.923162, 0.00659651, 0.070242, 
Output 1: 0.0379237, 0.427488, 0.499987, 0.0346012, 
Output 2: 0.412189, 0.482156, 0.100156, 0.00549862, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2267!
Begin processing the 996th record. run: 6403947 subRun: 1 event: 52296 at 15-Sep-2025 11:15:58 EDT
Boundary wire vector sizes: 165, 175, 162
minwire 0: 1929
minwire 1: 1339
minwire 2: 1461
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00697006, 0.00913736, 0.983893, 
Output 1: 0.458409, 0.517551, 0.0230701, 0.000970106, 
Output 2: 0.904521, 0.0926248, 0.00278004, 7.46641e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2127!
Begin processing the 997th record. run: 6403947 subRun: 1 event: 52297 at 15-Sep-2025 11:16:06 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 788!
Begin processing the 998th record. run: 6403947 subRun: 1 event: 52298 at 15-Sep-2025 11:16:12 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 705!
Begin processing the 999th record. run: 6403947 subRun: 1 event: 52299 at 15-Sep-2025 11:16:18 EDT
Boundary wire vector sizes: 302, 269, 257
minwire 0: 519
minwire 1: 2713
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.186858, 0.0417672, 0.771375, 
Output 1: 0.121255, 0.283446, 0.356893, 0.238405, 
Output 2: 0.183323, 0.325033, 0.33139, 0.160254, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1456!
Begin processing the 1000th record. run: 6403947 subRun: 1 event: 52300 at 15-Sep-2025 11:16:25 EDT
Boundary wire vector sizes: 254, 227, 93
minwire 0: 2054
minwire 1: 328
minwire 2: 2310
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.654283, 0.216713, 0.129003, 
Output 1: 0.75858, 0.231365, 0.00969862, 0.000357027, 
Output 2: 0.130775, 0.780752, 0.0852379, 0.00323532, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1027!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4e/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_1101_20231204T041938Z_gen_g4_detsim_hitreco__20240509T202900Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e4/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_440_20231203T081107Z_gen_g4_detsim_hitreco__20240508T033758Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_718_20231208T045411Z_gen_g4_detsim_hitreco__20240510T052035Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/82/b2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_285_20231208T040402Z_gen_g4_detsim_hitreco__20240510T034810Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/79/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_71_20231205T115445Z_gen_g4_detsim_hitreco__20240510T043003Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/db/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_707_20231202T090227Z_gen_g4_detsim_hitreco__20240507T210603Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_240_20231202T043827Z_gen_g4_detsim_hitreco__20240507T203246Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/72/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_412_20231204T051528Z_gen_g4_detsim_hitreco__20240509T213106Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3b/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_764_20231203T082425Z_gen_g4_detsim_hitreco__20240508T060603Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/46/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_522_20231202T120815Z_gen_g4_detsim_hitreco__20240507T212346Z_reco2.root
.:
total 42752
-rw-r--r--. 1 dune osgvo 16096822 Sep 15 11:16 flatcaf.root
-rw-r--r--. 1 dune osgvo 13409382 Sep 15 11:16 caf.root
-rw-r--r--. 1 dune osgvo 13409382 Sep 15 11:16 caf_fd_hd_atmo_2501_20250915T123549Z.root
-rw-r--r--. 1 dune osgvo   598919 Sep 15 11:16 caf_20250915T123549Z.log
-rw-r--r--. 1 dune osgvo   216103 Sep 15 11:16 jobscript.log
-rw-r--r--. 1 dune osgvo     1954 Sep 15 11:16 caf_20250915T123549Z.file
-rw-r--r--. 1 dune osgvo     1954 Sep 15 11:16 caf_20250915T123549Z.pfns
-rw-r--r--. 1 dune osgvo     1954 Sep 15 08:35 file.list
-rw-r--r--. 1 dune osgvo     1954 Sep 15 11:16 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo     1374 Sep 15 08:35 all-input-dids.txt
-rw-r--r--. 1 dune osgvo     1374 Sep 15 11:16 caf_20250915T123549Z.did
-rw-r--r--. 1 dune osgvo     1374 Sep 15 08:35 did.list
-rw-r--r--. 1 dune osgvo        0 Sep 15 08:36 debugprod.log