Jobsub ID 40191.16@dunegpschedd02.fnal.gov

Jobsub ID	40191.16@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 17:19:55
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-15 19:15:02
From worker node	Hostname	wn-sate-038.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 17:21:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_737_20231203T025145Z_gen_g4_detsim_hitreco__20240508T072128Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_1172_20231203T171622Z_gen_g4_detsim_hitreco__20240508T024850Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_302_20231202T053421Z_gen_g4_detsim_hitreco__20240508T031649Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_867_20231203T082649Z_gen_g4_detsim_hitreco__20240508T060838Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_143_20231028T022235Z_gen_g4_detsim_hitreco__20240503T042820Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_161_20231118T173415Z_gen_g4_detsim_hitreco__20240503T051734Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_452_20231208T051830Z_gen_g4_detsim_hitreco__20240510T065432Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_744_20231207T090004Z_gen_g4_detsim_hitreco__20240510T035929Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_663_20231202T162501Z_gen_g4_detsim_hitreco__20240508T045751Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6804s)
	CPU time	1h (5812s = 85%)
	Max RSS bytes	2039930880 (1945 MiB)
Outputting started	2025-09-15 19:14:41
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T172125Z.root
Finished	2025-09-15 19:15:02
Saved logs	justin-logs:40191.16-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000183105   0.00275939     0.240025     0.000608576    0.0109525      1300    
reco:energyrecnc:EnergyReco                           0.00252704     0.0854136     0.576236      0.0965815     0.0673585      1300    
reco:energyrecnumurange:EnergyReco                    0.000287842    0.0405295      8.02726      0.0029543     0.269787       1300    
reco:energyrecnumumcs:EnergyReco                       0.0005684     0.041414       7.9889      0.00329737     0.268604       1300    
reco:energyrecnumumcsllhd:EnergyReco                  0.000422816    0.0417016      8.0535      0.00337312     0.270688       1300    
reco:opdec:Deconvolution                               0.0366021     0.235527      0.576836      0.232995      0.0821922      1300    
reco:ophitspe:OpHitFinderDeco                          0.0806088     0.139488      0.361933      0.135403      0.0273514      1300    
reco:opflash:OpFlashFinder                            0.00013281    0.00123943    0.00377223    0.00119694    0.000558782     1300    
reco:opslicer:OpSlicer                                4.3552e-05     0.229484       0.9266       0.179242      0.192046       1300    
reco:rns:RandomNumberSaver                            1.9036e-05    2.77525e-05   0.00029221     2.686e-05    8.9535e-06      1300    
reco:anglereconue:NuAngularReco                       0.000154661   0.00178959     0.173791     0.000358675    0.0079478      1300    
reco:anglereconumu:NuAngularReco                      9.3807e-05    0.00166001     0.175213     0.000228005   0.00797294      1300    
reco:anglereconuepfps:NuAngularReco                   0.000138792   0.00176152     0.177407     0.000321666   0.00802539      1300    
reco:anglereconumupfps:NuAngularReco                  0.000134303   0.00173713     0.174579     0.00029737    0.00795159      1300    
reco:anglerecohits:NuAngularReco                      0.000104987   0.00231996     0.181789     0.000675658   0.00851758      1300    
[art]:TriggerResults:TriggerResultInserter            1.0449e-05    1.24223e-05   4.9633e-05    1.1952e-05    2.00875e-06     1300    
end_path:cafmaker:CAFMaker                            0.00108891     0.022043       2.30934     0.00390579     0.0985451      1300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3787.72 MB
  Peak resident set size usage (VmHWM): 2039.93 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 389, 183, 214
minwire 0: 1836
minwire 1: 1627
minwire 2: 1570
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996062, 0.000446723, 0.00349183, 
Output 1: 0.0412339, 0.916822, 0.0408724, 0.00107166, 
Output 2: 0.992725, 0.00713418, 0.000137772, 2.52982e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2247!
Begin processing the 1292nd record. run: 74505925 subRun: 1 event: 66392 at 15-Sep-2025 21:14:07 CEST
Boundary wire vector sizes: 85, 86, 78
minwire 0: 806
minwire 1: 1728
minwire 2: 112
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00993455, 0.00209269, 0.987973, 
Output 1: 0.0516972, 0.213483, 0.71475, 0.0200701, 
Output 2: 0.956469, 0.0413631, 0.00212395, 4.3701e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1897!
Begin processing the 1293rd record. run: 74505925 subRun: 1 event: 66393 at 15-Sep-2025 21:14:11 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 552!
Begin processing the 1294th record. run: 74505925 subRun: 1 event: 66394 at 15-Sep-2025 21:14:13 CEST
Boundary wire vector sizes: 76, 79, 69
minwire 0: 1136
minwire 1: 2445
minwire 2: 570
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000245506, 0.998777, 0.000977884, 
Output 1: 0.0045314, 0.991122, 0.00421512, 0.000131333, 
Output 2: 0.998653, 0.00130884, 3.66838e-05, 1.07125e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1444!
Begin processing the 1295th record. run: 74505925 subRun: 1 event: 66395 at 15-Sep-2025 21:14:17 CEST
Boundary wire vector sizes: 2893, 2405, 1961
minwire 0: 1725
minwire 1: 15
minwire 2: 2058
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00245218, 0.927981, 0.0695668, 
Output 1: 0.539362, 0.436327, 0.0189596, 0.00535115, 
Output 2: 0.176867, 0.504593, 0.227573, 0.0909664, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2209!
Begin processing the 1296th record. run: 74505925 subRun: 1 event: 66396 at 15-Sep-2025 21:14:26 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 963!
Begin processing the 1297th record. run: 74505925 subRun: 1 event: 66397 at 15-Sep-2025 21:14:28 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1125!
Begin processing the 1298th record. run: 74505925 subRun: 1 event: 66398 at 15-Sep-2025 21:14:31 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 454!
Begin processing the 1299th record. run: 74505925 subRun: 1 event: 66399 at 15-Sep-2025 21:14:33 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 442!
Begin processing the 1300th record. run: 74505925 subRun: 1 event: 66400 at 15-Sep-2025 21:14:36 CEST
Boundary wire vector sizes: 360, 488, 419
minwire 0: 1025
minwire 1: 1836
minwire 2: 717
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00367762, 0.0312885, 0.965034, 
Output 1: 0.988346, 0.0112411, 0.000285705, 0.000126993, 
Output 2: 0.984834, 0.0148956, 0.000230281, 4.02967e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1996!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/1c/41/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_737_20231203T025145Z_gen_g4_detsim_hitreco__20240508T072128Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/70/77/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_1172_20231203T171622Z_gen_g4_detsim_hitreco__20240508T024850Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/35/24/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_302_20231202T053421Z_gen_g4_detsim_hitreco__20240508T031649Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/35/e9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_867_20231203T082649Z_gen_g4_detsim_hitreco__20240508T060838Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5b/76/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_143_20231028T022235Z_gen_g4_detsim_hitreco__20240503T042820Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/aa/5d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_161_20231118T173415Z_gen_g4_detsim_hitreco__20240503T051734Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/cd/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_452_20231208T051830Z_gen_g4_detsim_hitreco__20240510T065432Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_744_20231207T090004Z_gen_g4_detsim_hitreco__20240510T035929Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/64/cb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_663_20231202T162501Z_gen_g4_detsim_hitreco__20240508T045751Z_reco2.root
.:
total 60276
-rw-r--r--. 1 dune012 dune 22637627 Sep 15 21:14 flatcaf.root
-rw-r--r--. 1 dune012 dune 18985760 Sep 15 21:14 caf.root
-rw-r--r--. 1 dune012 dune 18985760 Sep 15 21:14 caf_fd_hd_atmo_2501_20250915T172125Z.root
-rw-r--r--. 1 dune012 dune   764026 Sep 15 21:14 caf_20250915T172125Z.log
-rw-r--r--. 1 dune012 dune   292196 Sep 15 21:14 jobscript.log
-rw-r--r--. 1 dune012 dune     1911 Sep 15 21:14 caf_20250915T172125Z.file
-rw-r--r--. 1 dune012 dune     1911 Sep 15 21:14 caf_20250915T172125Z.pfns
-rw-r--r--. 1 dune012 dune     1911 Sep 15 19:21 file.list
-rw-r--r--. 1 dune012 dune     1911 Sep 15 21:14 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     1375 Sep 15 19:21 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     1375 Sep 15 21:14 caf_20250915T172125Z.did
-rw-r--r--. 1 dune012 dune     1375 Sep 15 19:21 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 15 19:21 debugprod.log