Jobsub ID 40343.23@dunegpschedd02.fnal.gov

Jobsub ID	40343.23@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-16 06:34:40
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce510
Last heartbeat	2025-09-16 07:51:25
From worker node	Hostname	b9p10p0963.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-16 06:35:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_201_20231203T072313Z_gen_g4_detsim_hitreco__20240508T054103Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_309_20231203T232046Z_gen_g4_detsim_hitreco__20240509T204947Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_747_20231202T081638Z_gen_g4_detsim_hitreco__20240507T194958Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_459_20231203T175256Z_gen_g4_detsim_hitreco__20240509T215828Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_560_20231208T032025Z_gen_g4_detsim_hitreco__20240510T064635Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_1197_20231205T070526Z_gen_g4_detsim_hitreco__20240509T222730Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_345_20231203T174638Z_gen_g4_detsim_hitreco__20240509T214439Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_286_20231204T001712Z_gen_g4_detsim_hitreco__20240509T205123Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_486_20231203T022233Z_gen_g4_detsim_hitreco__20240508T071428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_32_20231122T061020Z_gen_g4_detsim_hitreco__20240503T070318Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4523s)
	CPU time	1h (3704s = 81%)
	Max RSS bytes	2040877056 (1946 MiB)
Outputting started	2025-09-16 07:51:01
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250916T063547Z.root
Finished	2025-09-16 07:51:25
Saved logs	justin-logs:40343.23-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00056363    0.00699477      1000    
reco:energyrecnc:EnergyReco                            0.0013195     0.0855379     0.411946      0.104534      0.0631292      1000    
reco:energyrecnumurange:EnergyReco                    0.000154892    0.0331096      1.75181     0.00286076     0.138072       1000    
reco:energyrecnumumcs:EnergyReco                      0.000439623    0.0333726      1.79903     0.00327018     0.138287       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000307392    0.0323654      1.77155     0.00325575     0.132011       1000    
reco:opdec:Deconvolution                               0.0507206     0.201807      0.449784      0.199901      0.0683469      1000    
reco:ophitspe:OpHitFinderDeco                          0.140921      0.150004      0.215891      0.148932     0.00523569      1000    
reco:opflash:OpFlashFinder                            0.000172192   0.00127504    0.00353205    0.00119789    0.000601981     1000    
reco:opslicer:OpSlicer                                 3.982e-05     0.231059       1.00204      0.176801       0.19991       1000    
reco:rns:RandomNumberSaver                            1.6471e-05    3.64628e-05   0.000387263   3.3595e-05    1.63922e-05     1000    
reco:anglereconue:NuAngularReco                       0.000115061   0.00153626     0.0669268    0.000373593    0.0047202      1000    
reco:anglereconumu:NuAngularReco                       5.713e-05    0.00132125     0.0520811    0.000196962   0.00441219      1000    
reco:anglereconuepfps:NuAngularReco                   9.6321e-05    0.00141121     0.0522769    0.000294068    0.0043832      1000    
reco:anglereconumupfps:NuAngularReco                   7.937e-05    0.00137787     0.0530979    0.000251737   0.00444585      1000    
reco:anglerecohits:NuAngularReco                      7.4261e-05    0.00194767     0.057789     0.000638806   0.00499937      1000    
[art]:TriggerResults:TriggerResultInserter             7.81e-06     1.28458e-05    4.586e-05     1.237e-05    4.40139e-06     1000    
end_path:cafmaker:CAFMaker                             0.001056      0.0192074     0.870947     0.00418667     0.061555       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3722.78 MB
  Peak resident set size usage (VmHWM): 2040.88 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 125, 112, 97
minwire 0: 363
minwire 1: 2028
minwire 2: 415
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0776822, 0.806396, 0.115922, 
Output 1: 0.297743, 0.647825, 0.0511181, 0.00331424, 
Output 2: 0.245564, 0.743588, 0.0107181, 0.000129829, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 914!
Begin processing the 995th record. run: 74257901 subRun: 1 event: 3295 at 16-Sep-2025 09:50:42 CEST
Boundary wire vector sizes: 204, 99, 129
minwire 0: 1023
minwire 1: 1735
minwire 2: 992
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992299, 0.000642543, 0.00705896, 
Output 1: 0.994575, 0.00483423, 0.000348562, 0.000242263, 
Output 2: 0.992359, 0.00742551, 0.000174068, 4.19228e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1716!
Begin processing the 996th record. run: 74257901 subRun: 1 event: 3296 at 16-Sep-2025 09:50:44 CEST
Boundary wire vector sizes: 2568, 1723, 1919
minwire 0: 448
minwire 1: 1267
minwire 2: 147
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0102812, 0.95824, 0.0314792, 
Output 1: 0.363113, 0.406483, 0.127156, 0.103248, 
Output 2: 0.0407363, 0.12229, 0.326406, 0.510568, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2969!
Begin processing the 997th record. run: 74257901 subRun: 1 event: 3297 at 16-Sep-2025 09:50:49 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 217!
Begin processing the 998th record. run: 74257901 subRun: 1 event: 3298 at 16-Sep-2025 09:50:51 CEST
Boundary wire vector sizes: 575, 156, 293
minwire 0: 469
minwire 1: 1857
minwire 2: 534
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.988878, 0.000367734, 0.0107539, 
Output 1: 0.974986, 0.0217476, 0.00214223, 0.00112466, 
Output 2: 0.978482, 0.0203145, 0.000978356, 0.000225018, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1692!
Begin processing the 999th record. run: 74257901 subRun: 1 event: 3299 at 16-Sep-2025 09:50:53 CEST
Boundary wire vector sizes: 226, 437, 335
minwire 0: 1694
minwire 1: 1518
minwire 2: 1262
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997149, 0.00055163, 0.00229912, 
Output 1: 0.00284262, 0.993433, 0.00364762, 7.65396e-05, 
Output 2: 0.996826, 0.00311072, 6.26149e-05, 1.11208e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2343!
Begin processing the 1000th record. run: 74257901 subRun: 1 event: 3300 at 16-Sep-2025 09:50:56 CEST
Boundary wire vector sizes: 85, 72, 81
minwire 0: 1991
minwire 1: 812
minwire 2: 1594
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.948332, 0.000687825, 0.0509807, 
Output 1: 0.714835, 0.282123, 0.00295108, 9.11885e-05, 
Output 2: 0.979759, 0.0198049, 0.000419418, 1.70688e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1832!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/75/47/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_201_20231203T072313Z_gen_g4_detsim_hitreco__20240508T054103Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6b/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_309_20231203T232046Z_gen_g4_detsim_hitreco__20240509T204947Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c0/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_747_20231202T081638Z_gen_g4_detsim_hitreco__20240507T194958Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c3/de/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_459_20231203T175256Z_gen_g4_detsim_hitreco__20240509T215828Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6e/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_560_20231208T032025Z_gen_g4_detsim_hitreco__20240510T064635Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c4/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_1197_20231205T070526Z_gen_g4_detsim_hitreco__20240509T222730Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/0a/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_345_20231203T174638Z_gen_g4_detsim_hitreco__20240509T214439Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d3/5f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_286_20231204T001712Z_gen_g4_detsim_hitreco__20240509T205123Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3e/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_486_20231203T022233Z_gen_g4_detsim_hitreco__20240508T071428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/75/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_32_20231122T061020Z_gen_g4_detsim_hitreco__20240503T070318Z_reco2.root
.:
total 50076
-rw-r--r--. 1 duneprd np-comp 18779293 Sep 16 09:50 flatcaf.root
-rw-r--r--. 1 duneprd np-comp 15795426 Sep 16 09:50 caf.root
-rw-r--r--. 1 duneprd np-comp 15795426 Sep 16 09:51 caf_fd_hd_atmo_2501_20250916T063547Z.root
-rw-r--r--. 1 duneprd np-comp   592949 Sep 16 09:50 caf_20250916T063547Z.log
-rw-r--r--. 1 duneprd np-comp   243146 Sep 16 09:51 jobscript.log
-rw-r--r--. 1 duneprd np-comp     1896 Sep 16 09:51 caf_20250916T063547Z.file
-rw-r--r--. 1 duneprd np-comp     1896 Sep 16 09:51 caf_20250916T063547Z.pfns
-rw-r--r--. 1 duneprd np-comp     1896 Sep 16 08:35 file.list
-rw-r--r--. 1 duneprd np-comp     1896 Sep 16 09:51 justin-processed-pfns.txt
-rw-r--r--. 1 duneprd np-comp     1376 Sep 16 08:35 all-input-dids.txt
-rw-r--r--. 1 duneprd np-comp     1376 Sep 16 09:51 caf_20250916T063547Z.did
-rw-r--r--. 1 duneprd np-comp     1376 Sep 16 08:35 did.list
-rw-r--r--. 1 duneprd np-comp        0 Sep 16 08:35 debugprod.log