Jobsub ID 95864.21@dunegpschedd02.fnal.gov

Jobsub ID	95864.21@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 08:52:37
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 09:20:48
From worker node	Hostname	wn56
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 08:53:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_897_20231202T120907Z_gen_g4_detsim_hitreco__20240507T220531Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_82_20231201T101500Z_gen_g4_detsim_hitreco__20240507T185846Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_280_20231207T023402Z_gen_g4_detsim_hitreco__20240510T043717Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_4_20231201T042237Z_gen_g4_detsim_hitreco__20240507T202933Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_771_20231207T130249Z_gen_g4_detsim_hitreco__20240510T062850Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_223_20231202T202051Z_gen_g4_detsim_hitreco__20240508T070614Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_111_20231202T092133Z_gen_g4_detsim_hitreco__20240507T212456Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_1183_20231203T040420Z_gen_g4_detsim_hitreco__20240508T061631Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_88_20231205T110432Z_gen_g4_detsim_hitreco__20240510T063633Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_374_20231204T034420Z_gen_g4_detsim_hitreco__20240509T204917Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 09:20:48
Saved logs	justin-logs:95864.21-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0023     0.114782        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000870151    0.0343609     0.859411     0.00380963     0.114628        100    
reco:opdec:Deconvolution                                0.51979      0.887592       2.18894      0.813652      0.277916        100    
reco:ophitspe:OpHitFinderDeco                           1.38343       1.39707       1.41494       1.39821     0.00783509       100    
reco:opflash:OpFlashFinder                            0.000199779   0.00187367    0.00677904    0.00171859    0.000946577      100    
reco:opslicer:OpSlicer                                7.2276e-05     0.305551       1.04177      0.219327      0.263499        100    
reco:rns:RandomNumberSaver                            2.5488e-05    9.39427e-05   0.00116787    5.19475e-05   0.000166148      100    
reco:anglereconue:NuAngularReco                       0.000224996   0.00163782     0.018528     0.000627492   0.00324939       100    
reco:anglereconumu:NuAngularReco                      0.000125447   0.00132046     0.0184039    0.000369628   0.00316281       100    
reco:anglereconuepfps:NuAngularReco                   0.000183959   0.00154871      0.01864     0.000531174   0.00323384       100    
reco:anglereconumupfps:NuAngularReco                  0.000154586   0.00142259     0.0185644    0.000482997   0.00319148       100    
reco:anglerecohits:NuAngularReco                      0.000190051   0.00232662     0.0226089    0.00114983    0.00379912       100    
[art]:TriggerResults:TriggerResultInserter            1.3354e-05    2.37094e-05   7.0829e-05    2.16735e-05   7.74134e-06      100    
end_path:cafmaker:CAFMaker                            0.00181746     0.0266719     0.471176     0.00935871     0.0622469       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3674.42 MB
  Peak resident set size usage (VmHWM): 1998.91 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 06:16:31 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/92/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_82_20231201T101500Z_gen_g4_detsim_hitreco__20240507T185846Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 1: 0.00252776, 0.993821, 0.0035778, 7.36554e-05, 
Output 2: 0.996338, 0.00358775, 7.34045e-05, 1.3801e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1979!
Begin processing the 93rd record. run: 6405486 subRun: 1 event: 89793 at 23-Sep-2025 06:15:12 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 282!
Begin processing the 94th record. run: 6405486 subRun: 1 event: 89794 at 23-Sep-2025 06:15:21 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 629!
Begin processing the 95th record. run: 6405486 subRun: 1 event: 89795 at 23-Sep-2025 06:15:30 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 478!
Begin processing the 96th record. run: 6405486 subRun: 1 event: 89796 at 23-Sep-2025 06:15:38 -03
Boundary wire vector sizes: 104, 141, 125
minwire 0: 590
minwire 1: 2249
minwire 2: 236
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00206204, 0.980184, 0.0177544, 
Output 1: 0.985223, 0.014455, 0.000251646, 7.00469e-05, 
Output 2: 0.997219, 0.0027092, 5.97459e-05, 1.18112e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1336!
Begin processing the 97th record. run: 6405486 subRun: 1 event: 89797 at 23-Sep-2025 06:15:48 -03
Boundary wire vector sizes: 46, 155, 87
minwire 0: 1003
minwire 1: 2262
minwire 2: 652
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.993228, 0.0015599, 0.00521207, 
Output 1: 0.0127405, 0.984047, 0.00310456, 0.000107688, 
Output 2: 0.985586, 0.0141584, 0.000251251, 4.59207e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1636!
Begin processing the 98th record. run: 6405486 subRun: 1 event: 89798 at 23-Sep-2025 06:15:58 -03
Boundary wire vector sizes: 312, 326, 221
minwire 0: 988
minwire 1: 1745
minwire 2: 907
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000331344, 0.998076, 0.00159235, 
Output 1: 0.877282, 0.12192, 0.000655169, 0.000142509, 
Output 2: 0.998664, 0.00130107, 3.11353e-05, 3.962e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1868!
Begin processing the 99th record. run: 6405486 subRun: 1 event: 89799 at 23-Sep-2025 06:16:10 -03
Boundary wire vector sizes: 107, 97, 78
minwire 0: 2088
minwire 1: 797
minwire 2: 2081
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000579874, 0.992735, 0.00668531, 
Output 1: 0.754538, 0.243946, 0.00137193, 0.00014477, 
Output 2: 0.998607, 0.00135748, 3.25475e-05, 3.11965e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1364!
Begin processing the 100th record. run: 6405486 subRun: 1 event: 89800 at 23-Sep-2025 06:16:19 -03
Boundary wire vector sizes: 149, 176, 155
minwire 0: 88
minwire 1: 2181
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00277683, 0.991678, 0.00554476, 
Output 1: 0.0137398, 0.0473596, 0.258081, 0.680819, 
Output 2: 0.945657, 0.0500539, 0.00379432, 0.000494981, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 857!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/2c/a9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_897_20231202T120907Z_gen_g4_detsim_hitreco__20240507T220531Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/92/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_82_20231201T101500Z_gen_g4_detsim_hitreco__20240507T185846Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2b/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_280_20231207T023402Z_gen_g4_detsim_hitreco__20240510T043717Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_4_20231201T042237Z_gen_g4_detsim_hitreco__20240507T202933Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/ce/7f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_771_20231207T130249Z_gen_g4_detsim_hitreco__20240510T062850Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/76/26/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_223_20231202T202051Z_gen_g4_detsim_hitreco__20240508T070614Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/2e/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_111_20231202T092133Z_gen_g4_detsim_hitreco__20240507T212456Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/00/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_1183_20231203T040420Z_gen_g4_detsim_hitreco__20240508T061631Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/13/ca/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_88_20231205T110432Z_gen_g4_detsim_hitreco__20240510T063633Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e0/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_374_20231204T034420Z_gen_g4_detsim_hitreco__20240509T204917Z_reco2.root
.:
total 3664
-rw-r--r-- 1 nobody nobody 1458538 Sep 23 06:16 flatcaf.root
-rw-r--r-- 1 nobody nobody 1065990 Sep 23 06:16 caf.root
-rw-r--r-- 1 nobody nobody 1065990 Sep 23 06:16 caf_fd_hd_atmo_3302_20250923T085344Z.root
-rw-r--r-- 1 nobody nobody   71680 Sep 23 06:16 caf_20250923T085344Z.log
-rw-r--r-- 1 nobody nobody   34675 Sep 23 06:16 jobscript.log
-rw-r--r-- 1 nobody nobody    1977 Sep 23 06:16 caf_20250923T085344Z.file
-rw-r--r-- 1 nobody nobody    1977 Sep 23 06:16 caf_20250923T085344Z.pfns
-rw-r--r-- 1 nobody nobody    1977 Sep 23 05:53 file.list
-rw-r--r-- 1 nobody nobody    1977 Sep 23 06:16 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1371 Sep 23 05:53 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1371 Sep 23 06:16 caf_20250923T085344Z.did
-rw-r--r-- 1 nobody nobody    1371 Sep 23 05:53 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 05:54 debugprod.log