justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 50475.6@dunegpschedd01.fnal.gov

Jobsub ID50475.6@dunegpschedd01.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-20 03:32:31
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-20 04:17:14
From worker nodeHostnamewn-lot-027.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-20 03:33:02
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_196_20231121T152719Z_gen_g4_detsim_hitreco__20240503T070917Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_641_20231204T072243Z_gen_g4_detsim_hitreco__20240509T210550Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_772_20231202T162755Z_gen_g4_detsim_hitreco__20240508T051028Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_817_20231202T122334Z_gen_g4_detsim_hitreco__20240507T212745Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_735_20231202T122043Z_gen_g4_detsim_hitreco__20240507T212425Z_reco2.root
JobscriptExit code0
Real time43m (2635s)
CPU time37m (2234s = 84%)
Max RSS bytes2194497536 (2092 MiB)
Outputting started2025-09-20 04:16:58
Output files
Finished2025-09-20 04:17:14
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5.7519        600    
reco:pandoraTrack:LArPandoraTrackCreation             0.000114304    0.0064421     0.376915      0.0013009     0.0229177       600    
reco:pandoraShower:LArPandoraModularShowerCreation     0.0001605     0.0115283     0.348554     0.00284645     0.0318124       600    
reco:pandoracalo:Calorimetry                          9.2042e-05     0.018877       1.7626      0.000926284    0.103982        600    
reco:pandorapid:Chi2ParticleID                        3.2922e-05    0.000124051   0.00147498    9.54635e-05   0.00011421       600    
reco:cvnmap:CVNMapper                                 1.6481e-05    0.00734526     0.0386178    0.00888168    0.00511439       600    
reco:cvneva:CVNEvaluator                              1.5018e-05     0.273521       2.08607      0.366693      0.179319        600    
reco:energyrecnumu:EnergyReco                         0.000238586    0.0596418      3.3835       0.0122885     0.177184        600    
reco:energyrecnue:EnergyReco                          0.000139681   0.00229472     0.153781     0.000532953   0.00870683       600    
reco:energyrecnc:EnergyReco                           0.00217575     0.0765215     0.493972      0.0945698     0.0575519       600    
reco:energyrecnumurange:EnergyReco                    0.000179125    0.0307656      3.15585     0.00231754     0.166225        600    
reco:energyrecnumumcs:EnergyReco                      0.000456565    0.0317625      3.30936     0.00269851     0.169369        600    
reco:energyrecnumumcsllhd:EnergyReco                  0.000327403    0.0313562      3.1518      0.00264387     0.163916        600    
reco:opdec:Deconvolution                                0.02884      0.207712      0.429041      0.210882      0.0710712       600    
reco:ophitspe:OpHitFinderDeco                          0.0785132      0.12497      0.157101      0.132483      0.0187673       600    
reco:opflash:OpFlashFinder                            0.00011732    0.00106639     0.0036041    0.00101957    0.000505144      600    
reco:opslicer:OpSlicer                                3.4264e-05     0.218615       0.88214      0.176031      0.181548        600    
reco:rns:RandomNumberSaver                             1.64e-05     2.35573e-05   0.000296956   2.36845e-05   1.14431e-05      600    
reco:anglereconue:NuAngularReco                       0.000136195   0.00158069     0.126344     0.000313572   0.00698875       600    
reco:anglereconumu:NuAngularReco                      7.0081e-05    0.00147086     0.126441     0.00019516    0.00703554       600    
reco:anglereconuepfps:NuAngularReco                   0.000106349   0.00156214     0.122337     0.000274273   0.00691167       600    
reco:anglereconumupfps:NuAngularReco                  0.00010136    0.00153168     0.129926     0.000248079   0.00714348       600    
reco:anglerecohits:NuAngularReco                      8.1543e-05    0.00202622     0.134906     0.000596657   0.00755279       600    
[art]:TriggerResults:TriggerResultInserter             8.686e-06    1.03535e-05   4.2279e-05    1.0129e-05    1.71012e-06      600    
end_path:cafmaker:CAFMaker                             0.0011258     0.0200841      1.59743     0.00360286     0.0885211       600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3862.27 MB
  Peak resident set size usage (VmHWM): 2194.5 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 585!
Begin processing the 589th record. run: 6403947 subRun: 1 event: 73589 at 20-Sep-2025 06:16:32 CEST
Boundary wire vector sizes: 127, 129, 104
minwire 0: 1124
minwire 1: 1658
minwire 2: 683
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00134074, 0.988423, 0.010236, 
Output 1: 0.154781, 0.831739, 0.0128888, 0.000590854, 
Output 2: 0.948661, 0.0504684, 0.000850514, 1.98166e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1645!
Begin processing the 590th record. run: 6403947 subRun: 1 event: 73590 at 20-Sep-2025 06:16:35 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 527!
Begin processing the 591st record. run: 6403947 subRun: 1 event: 73591 at 20-Sep-2025 06:16:37 CEST
Boundary wire vector sizes: 395, 407, 282
minwire 0: 2360
minwire 1: 9
minwire 2: 2733
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0890615, 0.859344, 0.0515944, 
Output 1: 0.0197869, 0.407185, 0.49274, 0.0802878, 
Output 2: 0.566256, 0.352709, 0.0742302, 0.00680471, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1088!
Begin processing the 592nd record. run: 6403947 subRun: 1 event: 73592 at 20-Sep-2025 06:16:39 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 399!
Begin processing the 593rd record. run: 6403947 subRun: 1 event: 73593 at 20-Sep-2025 06:16:41 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 340!
Begin processing the 594th record. run: 6403947 subRun: 1 event: 73594 at 20-Sep-2025 06:16:43 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 298!
Begin processing the 595th record. run: 6403947 subRun: 1 event: 73595 at 20-Sep-2025 06:16:44 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 456!
Begin processing the 596th record. run: 6403947 subRun: 1 event: 73596 at 20-Sep-2025 06:16:46 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1324!
Begin processing the 597th record. run: 6403947 subRun: 1 event: 73597 at 20-Sep-2025 06:16:48 CEST
Boundary wire vector sizes: 175, 193, 138
minwire 0: 1636
minwire 1: 766
minwire 2: 1924
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00314508, 0.0024594, 0.994395, 
Output 1: 0.988244, 0.011217, 0.000329934, 0.000209038, 
Output 2: 0.997052, 0.00288163, 5.41263e-05, 1.26783e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1119!
Begin processing the 598th record. run: 6403947 subRun: 1 event: 73598 at 20-Sep-2025 06:16:51 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 446!
Begin processing the 599th record. run: 6403947 subRun: 1 event: 73599 at 20-Sep-2025 06:16:52 CEST
Boundary wire vector sizes: 153, 210, 131
minwire 0: 2025
minwire 1: 679
minwire 2: 2193
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996615, 0.000517185, 0.00286747, 
Output 1: 0.00222305, 0.994616, 0.00308887, 7.20068e-05, 
Output 2: 0.994455, 0.00543811, 0.000105295, 1.74924e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1349!
Begin processing the 600th record. run: 6403947 subRun: 1 event: 73600 at 20-Sep-2025 06:16:55 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 646!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a0/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_196_20231121T152719Z_gen_g4_detsim_hitreco__20240503T070917Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/16/75/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_641_20231204T072243Z_gen_g4_detsim_hitreco__20240509T210550Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a1/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_772_20231202T162755Z_gen_g4_detsim_hitreco__20240508T051028Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d4/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_817_20231202T122334Z_gen_g4_detsim_hitreco__20240507T212745Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/90/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_735_20231202T122043Z_gen_g4_detsim_hitreco__20240507T212425Z_reco2.root
.:
total 27664
-rw-r--r--. 1 dune012 dune 10416666 Sep 20 06:16 flatcaf.root
-rw-r--r--. 1 dune012 dune  8670983 Sep 20 06:16 caf.root
-rw-r--r--. 1 dune012 dune  8670983 Sep 20 06:16 caf_fd_hd_atmo_2501_20250920T033310Z.root
-rw-r--r--. 1 dune012 dune   367986 Sep 20 06:16 caf_20250920T033310Z.log
-rw-r--r--. 1 dune012 dune   151769 Sep 20 06:16 jobscript.log
-rw-r--r--. 1 dune012 dune      947 Sep 20 06:16 caf_20250920T033310Z.file
-rw-r--r--. 1 dune012 dune      947 Sep 20 06:16 caf_20250920T033310Z.pfns
-rw-r--r--. 1 dune012 dune      947 Sep 20 05:33 file.list
-rw-r--r--. 1 dune012 dune      947 Sep 20 06:16 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune      688 Sep 20 05:33 all-input-dids.txt
-rw-r--r--. 1 dune012 dune      688 Sep 20 06:16 caf_20250920T033310Z.did
-rw-r--r--. 1 dune012 dune      688 Sep 20 05:33 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 20 05:33 debugprod.log
justIN time: 2025-11-05 07:30:17 UTC       justIN version: 01.05.01