Jobsub ID 39988.143@dunegpschedd02.fnal.gov

Jobsub ID	39988.143@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-15 15:45:48
From worker node	Hostname	mh-7763-3.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 12:35:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_365_20231207T115834Z_gen_g4_detsim_hitreco__20240510T055145Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_285_20231207T082154Z_gen_g4_detsim_hitreco__20240510T032023Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_560_20231204T004037Z_gen_g4_detsim_hitreco__20240509T205932Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_930_20231202T211601Z_gen_g4_detsim_hitreco__20240508T070540Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_250_20231207T200500Z_gen_g4_detsim_hitreco__20240510T041753Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_701_20231202T121750Z_gen_g4_detsim_hitreco__20240507T212424Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_880_20231202T131012Z_gen_g4_detsim_hitreco__20240507T221955Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_13_20231204T080407Z_gen_g4_detsim_hitreco__20240509T213405Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_175_20231201T070953Z_gen_g4_detsim_hitreco__20240507T183852Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2.root
Jobscript	Exit code	0
	Real time	3h (11400s)
	CPU time	55m (3334s = 29%)
	Max RSS bytes	2025144320 (1931 MiB)
Outputting started	2025-09-15 15:45:27
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T123533Z.root
Finished	2025-09-15 15:45:48
Saved logs	justin-logs:39988.143-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

yReco                           0.00153063     0.549465       5.28884      0.605663      0.513218       1000    
reco:energyrecnumurange:EnergyReco                    0.000241898    0.0400594      3.10133     0.00312722      0.18899       1000    
reco:energyrecnumumcs:EnergyReco                      0.000633207    0.039389       3.11083       0.0035       0.179475       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000404357    0.0381171      2.02279     0.00343448     0.161769       1000    
reco:opdec:Deconvolution                               0.425685      0.871296       4.34923      0.769051      0.460179       1000    
reco:ophitspe:OpHitFinderDeco                           1.24007       1.33651       1.76328       1.3351       0.0574605      1000    
reco:opflash:OpFlashFinder                            0.000222508   0.00139812    0.00586097    0.00134719    0.000633145     1000    
reco:opslicer:OpSlicer                                8.8049e-05     0.269624       1.2174       0.215397      0.228393       1000    
reco:rns:RandomNumberSaver                             2.127e-05     4.54e-05      0.0017612    3.82945e-05   5.87131e-05     1000    
reco:anglereconue:NuAngularReco                       0.000150379   0.00155325     0.0759619    0.000455008   0.00468842      1000    
reco:anglereconumu:NuAngularReco                       6.515e-05    0.00132917     0.0741899    0.000238193   0.00460496      1000    
reco:anglereconuepfps:NuAngularReco                   0.000103689   0.00144356     0.0745879    0.000358573   0.00463158      1000    
reco:anglereconumupfps:NuAngularReco                   8.379e-05     0.0013937     0.0749101    0.000306548   0.00457518      1000    
reco:anglerecohits:NuAngularReco                      0.000102769   0.00207597     0.0845217    0.000791675   0.00523202      1000    
[art]:TriggerResults:TriggerResultInserter             8.79e-06     1.56482e-05    7.144e-05    1.4605e-05    6.4944e-06      1000    
end_path:cafmaker:CAFMaker                            0.00115214     0.0190611      1.15033     0.00501665     0.0596697      1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3823.63 MB
  Peak resident set size usage (VmHWM): 2025.14 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00063935, 0.995876, 0.00348483, 
Output 1: 0.0448046, 0.906081, 0.0467968, 0.00231801, 
Output 2: 0.99027, 0.00944871, 0.000272431, 8.57674e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1406!
Begin processing the 994th record. run: 6220954 subRun: 1 event: 20394 at 15-Sep-2025 08:44:18 PDT
Boundary wire vector sizes: 60, 37, 55
minwire 0: 1974
minwire 1: 1618
minwire 2: 1648
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.983416, 0.00100538, 0.0155785, 
Output 1: 0.448268, 0.547943, 0.00366803, 0.000121654, 
Output 2: 0.987405, 0.0123896, 0.000199354, 5.56492e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1429!
Begin processing the 995th record. run: 6220954 subRun: 1 event: 20395 at 15-Sep-2025 08:44:26 PDT
Boundary wire vector sizes: 64, 219, 146
minwire 0: 1379
minwire 1: 1659
minwire 2: 1054
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.979991, 0.000419915, 0.0195891, 
Output 1: 0.993243, 0.006456, 0.000217295, 8.35935e-05, 
Output 2: 0.978915, 0.0206069, 0.000410672, 6.74204e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1251!
Begin processing the 996th record. run: 6220954 subRun: 1 event: 20396 at 15-Sep-2025 08:44:34 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 378!
Begin processing the 997th record. run: 6220954 subRun: 1 event: 20397 at 15-Sep-2025 08:44:41 PDT
Boundary wire vector sizes: 39, 35, 28
minwire 0: 3020
minwire 1: 689
minwire 2: 2807
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.076421, 0.0243422, 0.899237, 
Output 1: 0.0358187, 0.232447, 0.663412, 0.0683223, 
Output 2: 0.905447, 0.0913822, 0.00309871, 7.23794e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1005!
Begin processing the 998th record. run: 6220954 subRun: 1 event: 20398 at 15-Sep-2025 08:44:49 PDT
Boundary wire vector sizes: 2046, 3033, 1993
minwire 0: 412
minwire 1: 1709
minwire 2: 104
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0225278, 0.945687, 0.0317848, 
Output 1: 0.321606, 0.653722, 0.0228187, 0.00185338, 
Output 2: 0.101817, 0.579345, 0.276496, 0.0423418, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3043!
Begin processing the 999th record. run: 6220954 subRun: 1 event: 20399 at 15-Sep-2025 08:45:09 PDT
Boundary wire vector sizes: 314, 358, 207
minwire 0: 1882
minwire 1: 1597
minwire 2: 1461
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.66183, 0.105442, 0.232727, 
Output 1: 0.213157, 0.716592, 0.0663913, 0.00385918, 
Output 2: 0.582819, 0.364607, 0.0495154, 0.00305812, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2653!
Begin processing the 1000th record. run: 6220954 subRun: 1 event: 20400 at 15-Sep-2025 08:45:19 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 226!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7d/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_365_20231207T115834Z_gen_g4_detsim_hitreco__20240510T055145Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/9a/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_285_20231207T082154Z_gen_g4_detsim_hitreco__20240510T032023Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d0/ca/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_560_20231204T004037Z_gen_g4_detsim_hitreco__20240509T205932Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/3c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_930_20231202T211601Z_gen_g4_detsim_hitreco__20240508T070540Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/da/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_250_20231207T200500Z_gen_g4_detsim_hitreco__20240510T041753Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fe/21/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_701_20231202T121750Z_gen_g4_detsim_hitreco__20240507T212424Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/7d/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_880_20231202T131012Z_gen_g4_detsim_hitreco__20240507T221955Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/0f/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_13_20231204T080407Z_gen_g4_detsim_hitreco__20240509T213405Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/9c/11/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_175_20231201T070953Z_gen_g4_detsim_hitreco__20240507T183852Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0e/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_203_20231122T082858Z_gen_g4_detsim_hitreco__20240503T065347Z_reco2.root
.:
total 44740
-rw-r--r--. 1 cuser cuser 16831423 Sep 15 08:45 flatcaf.root
-rw-r--r--. 1 cuser cuser 14045006 Sep 15 08:45 caf.root
-rw-r--r--. 1 cuser cuser 14045006 Sep 15 08:45 caf_fd_hd_atmo_2501_20250915T123533Z.root
-rw-r--r--. 1 cuser cuser   577867 Sep 15 08:45 caf_20250915T123533Z.log
-rw-r--r--. 1 cuser cuser   239477 Sep 15 08:45 jobscript.log
-rw-r--r--. 1 cuser cuser     1952 Sep 15 08:45 caf_20250915T123533Z.file
-rw-r--r--. 1 cuser cuser     1952 Sep 15 08:45 caf_20250915T123533Z.pfns
-rw-r--r--. 1 cuser cuser     1952 Sep 15 05:35 file.list
-rw-r--r--. 1 cuser cuser     1952 Sep 15 08:45 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser     1374 Sep 15 05:35 all-input-dids.txt
-rw-r--r--. 1 cuser cuser     1374 Sep 15 08:45 caf_20250915T123533Z.did
-rw-r--r--. 1 cuser cuser     1374 Sep 15 05:35 did.list
-rw-r--r--. 1 cuser cuser        0 Sep 15 05:35 debugprod.log