Jobsub ID 47404.27@dunegpschedd01.fnal.gov

Jobsub ID	47404.27@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 07:52:57
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-09-17 10:23:15
From worker node	Hostname	dunegli-6878159-0-fnpc17131.fnal.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 07:53:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_769_20231202T160955Z_gen_g4_detsim_hitreco__20240508T043543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_629_20231208T064612Z_gen_g4_detsim_hitreco__20240510T051108Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_634_20231208T010247Z_gen_g4_detsim_hitreco__20240510T031450Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_528_20231203T102001Z_gen_g4_detsim_hitreco__20240509T195535Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_432_20231205T025934Z_gen_g4_detsim_hitreco__20240510T030429Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_1197_20231203T071659Z_gen_g4_detsim_hitreco__20240508T073751Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_408_20231201T130135Z_gen_g4_detsim_hitreco__20240507T194801Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_404_20231202T180520Z_gen_g4_detsim_hitreco__20240508T063751Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6447068_175_20231205T101654Z_gen_g4_detsim_hitreco__20240510T055034Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_531_20231208T064450Z_gen_g4_detsim_hitreco__20240510T050645Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8971s)
	CPU time	1h (4351s = 48%)
	Max RSS bytes	2096197632 (1999 MiB)
Outputting started	2025-09-17 10:23:07
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T075339Z.root
Finished	2025-09-17 10:23:15
Saved logs	justin-logs:47404.27-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.0417842      5.50656     0.00322744     0.216491       1000    
reco:opdec:Deconvolution                               0.310562       0.61327       1.36153      0.608534      0.142013       1000    
reco:ophitspe:OpHitFinderDeco                          0.920303      0.992983       1.38048      0.980137      0.0696977      1000    
reco:opflash:OpFlashFinder                            0.000134255   0.00131618    0.00784195    0.00121129    0.00068699      1000    
reco:opslicer:OpSlicer                                3.7928e-05     0.254882       1.35568      0.189976      0.226074       1000    
reco:rns:RandomNumberSaver                            1.7262e-05    2.79472e-05   0.000656382   2.51095e-05   2.0859e-05      1000    
reco:anglereconue:NuAngularReco                       0.000139568   0.00202853     0.161006     0.00037286    0.00810591      1000    
reco:anglereconumu:NuAngularReco                       6.576e-05    0.00185655     0.160032     0.000212541   0.00808774      1000    
reco:anglereconuepfps:NuAngularReco                   0.000105354   0.00197413     0.159497     0.000313322   0.00822543      1000    
reco:anglereconumupfps:NuAngularReco                  8.9908e-05    0.00195466     0.161705     0.000294292   0.00827744      1000    
reco:anglerecohits:NuAngularReco                      7.6575e-05     0.0025379     0.168102     0.000683275   0.00890348      1000    
[art]:TriggerResults:TriggerResultInserter             9.413e-06    1.32358e-05   8.8184e-05    1.2102e-05    3.46804e-06     1000    
end_path:cafmaker:CAFMaker                            0.00120571     0.0236509      1.56055     0.00429444     0.0915703      1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3754.08 MB
  Peak resident set size usage (VmHWM): 2096.2 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 2: 0.876825, 0.120571, 0.00239788, 0.000206174, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1074!
Begin processing the 995th record. run: 6472619 subRun: 1 event: 53195 at 17-Sep-2025 10:22:11 UTC
Boundary wire vector sizes: 833, 435, 555
minwire 0: 1058
minwire 1: 1265
minwire 2: 1254
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00679209, 0.937926, 0.0552822, 
Output 1: 0.0503931, 0.515003, 0.392134, 0.0424703, 
Output 2: 0.79908, 0.185856, 0.0144527, 0.000610592, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2238!
Begin processing the 996th record. run: 6472619 subRun: 1 event: 53196 at 17-Sep-2025 10:22:19 UTC
Boundary wire vector sizes: 965, 513, 696
minwire 0: 923
minwire 1: 1936
minwire 2: 699
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.721234, 0.121104, 0.157662, 
Output 1: 0.0464891, 0.761184, 0.184519, 0.00780856, 
Output 2: 0.222619, 0.639774, 0.131908, 0.0056982, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2598!
Begin processing the 997th record. run: 6472619 subRun: 1 event: 53197 at 17-Sep-2025 10:22:28 UTC
Boundary wire vector sizes: 2767, 3059, 2288
minwire 0: 792
minwire 1: 910
minwire 2: 656
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.21054, 0.377544, 0.411916, 
Output 1: 0.329531, 0.647518, 0.0188117, 0.00413931, 
Output 2: 0.0627367, 0.279166, 0.346488, 0.311609, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2830!
Begin processing the 998th record. run: 6472619 subRun: 1 event: 53198 at 17-Sep-2025 10:22:44 UTC
Boundary wire vector sizes: 1088, 1239, 1234
minwire 0: 452
minwire 1: 1564
minwire 2: 7
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0816879, 0.00638516, 0.911927, 
Output 1: 0.281735, 0.578998, 0.128469, 0.0107975, 
Output 2: 0.0612095, 0.413175, 0.448476, 0.07714, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2533!
Begin processing the 999th record. run: 6472619 subRun: 1 event: 53199 at 17-Sep-2025 10:22:55 UTC
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 0!
Begin processing the 1000th record. run: 6472619 subRun: 1 event: 53200 at 17-Sep-2025 10:22:59 UTC
Boundary wire vector sizes: 142, 164, 116
minwire 0: 2865
minwire 1: 729
minwire 2: 2581
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000446381, 0.99763, 0.00192395, 
Output 1: 0.0101963, 0.915526, 0.0717151, 0.0025623, 
Output 2: 0.997674, 0.00223167, 9.18233e-05, 2.73952e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1222!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/da/5f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_769_20231202T160955Z_gen_g4_detsim_hitreco__20240508T043543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cb/3d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_629_20231208T064612Z_gen_g4_detsim_hitreco__20240510T051108Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e2/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_634_20231208T010247Z_gen_g4_detsim_hitreco__20240510T031450Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/52/92/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_528_20231203T102001Z_gen_g4_detsim_hitreco__20240509T195535Z_reco2.root
root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/38/39/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_432_20231205T025934Z_gen_g4_detsim_hitreco__20240510T030429Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c6/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_1197_20231203T071659Z_gen_g4_detsim_hitreco__20240508T073751Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/62/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_408_20231201T130135Z_gen_g4_detsim_hitreco__20240507T194801Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5b/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_404_20231202T180520Z_gen_g4_detsim_hitreco__20240508T063751Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/1c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6447068_175_20231205T101654Z_gen_g4_detsim_hitreco__20240510T055034Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cc/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_531_20231208T064450Z_gen_g4_detsim_hitreco__20240510T050645Z_reco2.root
.:
total 45224
-rw-r--r-- 1 dunegli fnalgrid 17003528 Sep 17 10:23 flatcaf.root
-rw-r--r-- 1 dunegli fnalgrid 14217988 Sep 17 10:23 caf.root
-rw-r--r-- 1 dunegli fnalgrid 14217988 Sep 17 10:23 caf_fd_hd_atmo_2501_20250917T075339Z.root
-rw-r--r-- 1 dunegli fnalgrid   594382 Sep 17 10:23 caf_20250917T075339Z.log
-rw-r--r-- 1 dunegli fnalgrid   216139 Sep 17 10:23 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid     1876 Sep 17 10:23 caf_20250917T075339Z.file
-rw-r--r-- 1 dunegli fnalgrid     1876 Sep 17 10:23 caf_20250917T075339Z.pfns
-rw-r--r-- 1 dunegli fnalgrid     1876 Sep 17 07:53 file.list
-rw-r--r-- 1 dunegli fnalgrid     1876 Sep 17 10:23 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid     1377 Sep 17 07:53 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid     1377 Sep 17 10:23 caf_20250917T075339Z.did
-rw-r--r-- 1 dunegli fnalgrid     1377 Sep 17 07:53 did.list
-rw-r--r-- 1 dunegli fnalgrid        0 Sep 17 07:54 debugprod.log