Jobsub ID 44838.1@dunegpschedd02.fnal.gov

Jobsub ID	44838.1@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 09:26:02
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-21 09:38:44
From worker node	Hostname	wn-snel-030.farm.nikhef.nl
	cpuinfo	AMD EPYC 7H12 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 09:27:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_711_20231204T225152Z_gen_g4_detsim_hitreco__20240509T221952Z_reco2.root
Jobscript	Exit code	0
	Real time	11m (671s)
	CPU time	9m (588s = 87%)
	Max RSS bytes	2165862400 (2065 MiB)
Outputting started	2025-09-21 09:38:26
Output files
Finished	2025-09-21 09:38:44
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

=============================
Full event                                             0.368575       5.79547       171.649       2.6537        18.3686        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0151043     0.0373086     0.0663018     0.0347702     0.0116282       100    
reco:pandora:StandardPandora                           0.0952085      4.5439        159.121       1.6071        17.1446        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000104817   0.00668137     0.243727     0.00103642     0.0254251       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00014956     0.0110584      0.11522     0.00246073     0.0234404       100    
reco:pandoracalo:Calorimetry                           8.544e-05     0.0375939      2.0109      0.000715006    0.211423        100    
reco:pandorapid:Chi2ParticleID                         3.238e-05    0.000132487   0.00112672    0.000100198   0.000141434      100    
reco:cvnmap:CVNMapper                                 1.7142e-05    0.00777918     0.0341844    0.00916583     0.0060194       100    
reco:cvneva:CVNEvaluator                              1.5449e-05     0.246569       1.65225      0.317129      0.200075        100    
reco:energyrecnumu:EnergyReco                         0.000277911    0.0734213      1.80616      0.0174866     0.226041        100    
reco:energyrecnue:EnergyReco                          0.000146545   0.00241582     0.0821372    0.000508629   0.00847768       100    
reco:energyrecnc:EnergyReco                           0.00192814     0.0871516     0.615622      0.0949135     0.0800584       100    
reco:energyrecnumurange:EnergyReco                    0.000285565    0.0415871      1.63143     0.00234042     0.208008        100    
reco:energyrecnumumcs:EnergyReco                      0.00047947     0.042056       1.63042     0.00257426     0.207244        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000318237    0.0419153      1.62701     0.00255003     0.206683        100    
reco:opdec:Deconvolution                               0.0480879      0.22048      0.458717      0.219711      0.0740575       100    
reco:ophitspe:OpHitFinderDeco                          0.129526      0.134287      0.140384      0.134102      0.0023126       100    
reco:opflash:OpFlashFinder                            0.000214422   0.00118193    0.00400329    0.00106139    0.000609176      100    
reco:opslicer:OpSlicer                                7.0793e-05     0.228718      0.901007      0.186091      0.184193        100    
reco:rns:RandomNumberSaver                            1.8705e-05    2.96047e-05   0.000297348   2.5893e-05    2.71726e-05      100    
reco:anglereconue:NuAngularReco                       0.000167814   0.00162854     0.060842     0.000348488   0.00626026       100    
reco:anglereconumu:NuAngularReco                      7.6805e-05    0.00147686     0.0605641    0.000194951   0.00624023       100    
reco:anglereconuepfps:NuAngularReco                   0.000126747   0.00161402     0.0636343    0.000306465   0.00652018       100    
reco:anglereconumupfps:NuAngularReco                  0.000109756   0.00159061     0.0657689    0.000264035   0.00673273       100    
reco:anglerecohits:NuAngularReco                      0.000132148   0.00227221     0.0716236    0.000629762   0.00743565       100    
[art]:TriggerResults:TriggerResultInserter             8.896e-06    1.19305e-05    4.237e-05    1.18675e-05   3.28658e-06      100    
end_path:cafmaker:CAFMaker                            0.00175491     0.0215475     0.794464     0.00392447     0.082043        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3826.84 MB
  Peak resident set size usage (VmHWM): 2165.86 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1078!
Begin processing the 93rd record. run: 50598687 subRun: 1 event: 71193 at 21-Sep-2025 11:35:12 CEST
Boundary wire vector sizes: 105, 64, 91
minwire 0: 669
minwire 1: 2088
minwire 2: 575
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995517, 0.000548676, 0.00393391, 
Output 1: 0.00311271, 0.991167, 0.00558221, 0.000137861, 
Output 2: 0.994392, 0.00548757, 0.000118371, 2.32508e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 872!
Begin processing the 94th record. run: 50598687 subRun: 1 event: 71194 at 21-Sep-2025 11:35:14 CEST
Boundary wire vector sizes: 37, 38, 35
minwire 0: 54
minwire 1: 2277
minwire 2: 38
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00719817, 0.00184885, 0.990953, 
Output 1: 0.445882, 0.538537, 0.0151331, 0.000447218, 
Output 2: 0.998169, 0.00173627, 9.00642e-05, 4.59547e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 623!
Begin processing the 95th record. run: 50598687 subRun: 1 event: 71195 at 21-Sep-2025 11:35:17 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 746!
Begin processing the 96th record. run: 50598687 subRun: 1 event: 71196 at 21-Sep-2025 11:35:19 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 408!
Begin processing the 97th record. run: 50598687 subRun: 1 event: 71197 at 21-Sep-2025 11:35:22 CEST
Boundary wire vector sizes: 18136, 14243, 7868
minwire 0: 747
minwire 1: 230
minwire 2: 10
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.389282, 0.56994, 0.0407778, 
Output 1: 0.258047, 0.40549, 0.186408, 0.150055, 
Output 2: 0.0136618, 0.0465502, 0.186065, 0.753723, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 4075!
Begin processing the 98th record. run: 50598687 subRun: 1 event: 71198 at 21-Sep-2025 11:38:18 CEST
Boundary wire vector sizes: 93, 101, 20
minwire 0: 2765
minwire 1: 655
minwire 2: 2532
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0043925, 0.00219185, 0.993416, 
Output 1: 0.0331135, 0.957067, 0.0091642, 0.000655568, 
Output 2: 0.997802, 0.00211467, 7.891e-05, 4.64e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1267!
Begin processing the 99th record. run: 50598687 subRun: 1 event: 71199 at 21-Sep-2025 11:38:20 CEST
Boundary wire vector sizes: 86, 107, 91
minwire 0: 826
minwire 1: 1915
minwire 2: 528
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000571905, 0.995054, 0.00437429, 
Output 1: 0.995839, 0.00392883, 0.000115294, 0.000117274, 
Output 2: 0.997766, 0.00214336, 6.11913e-05, 2.89538e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1302!
Begin processing the 100th record. run: 50598687 subRun: 1 event: 71200 at 21-Sep-2025 11:38:23 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 475!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_711_20231204T225152Z_gen_g4_detsim_hitreco__20240509T221952Z_reco2.root
.:
total 6808
-rw-r--r--. 1 dune012 dune 2634006 Sep 21 11:38 flatcaf.root
-rw-r--r--. 1 dune012 dune 2088088 Sep 21 11:38 caf.root
-rw-r--r--. 1 dune012 dune 2088088 Sep 21 11:38 caf_fd_hd_atmo_2501_20250921T092723Z.root
-rw-r--r--. 1 dune012 dune   75154 Sep 21 11:38 caf_20250921T092723Z.log
-rw-r--r--. 1 dune012 dune   37269 Sep 21 11:38 jobscript.log
-rw-r--r--. 1 dune012 dune     187 Sep 21 11:38 caf_20250921T092723Z.file
-rw-r--r--. 1 dune012 dune     187 Sep 21 11:38 caf_20250921T092723Z.pfns
-rw-r--r--. 1 dune012 dune     187 Sep 21 11:27 file.list
-rw-r--r--. 1 dune012 dune     187 Sep 21 11:38 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     138 Sep 21 11:27 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     138 Sep 21 11:38 caf_20250921T092723Z.did
-rw-r--r--. 1 dune012 dune     138 Sep 21 11:27 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 21 11:27 debugprod.log