justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 48443.4@dunegpschedd01.fnal.gov

Jobsub ID48443.4@dunegpschedd01.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 11:50:24
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-09-18 12:52:13
From worker nodeHostnamewn-sate-050.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-18 11:50:59
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_173_20231204T060235Z_gen_g4_detsim_hitreco__20240509T214745Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_931_20231202T113802Z_gen_g4_detsim_hitreco__20240507T212327Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_602_20231203T141918Z_gen_g4_detsim_hitreco__20240508T080128Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_918_20231202T191330Z_gen_g4_detsim_hitreco__20240508T064428Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_56_20231206T230207Z_gen_g4_detsim_hitreco__20240510T031902Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_24_20231208T040205Z_gen_g4_detsim_hitreco__20240510T065328Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_77_20231204T080606Z_gen_g4_detsim_hitreco__20240509T220853Z_reco2.root
JobscriptExit code0
Real time1h (3655s)
CPU time55m (3304s = 90%)
Max RSS bytes2038583296 (1944 MiB)
Outputting started2025-09-18 12:51:54
Output files
Finished2025-09-18 12:52:13
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

icleID                        4.0356e-05    0.000141108   0.00109257    0.000111624   0.000116386      700    
reco:cvnmap:CVNMapper                                 1.9767e-05    0.00883244     0.0388605     0.0105084    0.00604181       700    
reco:cvneva:CVNEvaluator                              1.8915e-05     0.450781       2.13351      0.604206      0.276617        700    
reco:energyrecnumu:EnergyReco                         0.000259908    0.0791195      7.59308      0.0127681     0.375142        700    
reco:energyrecnue:EnergyReco                          0.000181932   0.00273718     0.110894     0.000636742   0.00847665       700    
reco:energyrecnc:EnergyReco                           0.00245951     0.0557827     0.304657      0.0223316     0.0547388       700    
reco:energyrecnumurange:EnergyReco                    0.000226626    0.0577994      7.10922     0.00278535      0.3524         700    
reco:energyrecnumumcs:EnergyReco                      0.000472859    0.0585942      7.08629     0.00328179     0.351446        700    
reco:energyrecnumumcsllhd:EnergyReco                  0.000394822    0.0588531      7.28034     0.00327081      0.35698        700    
reco:opdec:Deconvolution                               0.0139922     0.208602      0.475242      0.211325      0.0801169       700    
reco:ophitspe:OpHitFinderDeco                         0.00299717     0.0799666     0.157591      0.129791      0.0609869       700    
reco:opflash:OpFlashFinder                            0.000134583   0.00120284     0.0151213    0.00114959    0.000753917      700    
reco:opslicer:OpSlicer                                3.9945e-05     0.244448      0.907498      0.200727      0.192142        700    
reco:rns:RandomNumberSaver                            1.6501e-05    1.96917e-05   0.000219172    1.863e-05    8.02493e-06      700    
reco:anglereconue:NuAngularReco                       0.000120647   0.00184233     0.0904444    0.00032755    0.00670626       700    
reco:anglereconumu:NuAngularReco                      9.0871e-05    0.00176013     0.0905124    0.000258742   0.00667331       700    
reco:anglereconuepfps:NuAngularReco                   0.000134433   0.00183301     0.0911447    0.000337088    0.0066822       700    
reco:anglereconumupfps:NuAngularReco                  0.000133531    0.0018208     0.0909221    0.000328523   0.00667878       700    
reco:anglerecohits:NuAngularReco                      0.000101621   0.00238515     0.0962591    0.000695364   0.00720959       700    
[art]:TriggerResults:TriggerResultInserter             9.979e-06     1.102e-05    3.8953e-05    1.07105e-05   1.55578e-06      700    
end_path:cafmaker:CAFMaker                            0.00101105     0.0229104      1.09251     0.00401352     0.0817399       700    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3695.2 MB
  Peak resident set size usage (VmHWM): 2038.58 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 0: 0.00643008, 0.843179, 0.150391, 
Output 1: 0.0486422, 0.925606, 0.0247286, 0.00102352, 
Output 2: 0.932085, 0.0666211, 0.00126495, 2.85548e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1044!
Begin processing the 693rd record. run: 6422672 subRun: 1 event: 7793 at 18-Sep-2025 14:51:11 CEST
Boundary wire vector sizes: 337, 187, 151
minwire 0: 804
minwire 1: 2232
minwire 2: 593
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995366, 0.000556195, 0.00407743, 
Output 1: 0.535468, 0.453054, 0.00957194, 0.00190573, 
Output 2: 0.988781, 0.0107217, 0.00045286, 4.41831e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1687!
Begin processing the 694th record. run: 6422672 subRun: 1 event: 7794 at 18-Sep-2025 14:51:14 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 417!
Begin processing the 695th record. run: 6422672 subRun: 1 event: 7795 at 18-Sep-2025 14:51:16 CEST
Boundary wire vector sizes: 4303, 5071, 3048
minwire 0: 1747
minwire 1: 554
minwire 2: 1806
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.691507, 0.0883504, 0.220143, 
Output 1: 0.288333, 0.631541, 0.0574243, 0.0227015, 
Output 2: 0.0152994, 0.066855, 0.236409, 0.681436, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3002!
Begin processing the 696th record. run: 6422672 subRun: 1 event: 7796 at 18-Sep-2025 14:51:35 CEST
Boundary wire vector sizes: 74, 59, 68
minwire 0: 2253
minwire 1: 1209
minwire 2: 1960
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0384551, 0.938809, 0.0227361, 
Output 1: 0.168055, 0.812781, 0.0184984, 0.000665135, 
Output 2: 0.984687, 0.014993, 0.000311112, 8.62163e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1358!
Begin processing the 697th record. run: 6422672 subRun: 1 event: 7797 at 18-Sep-2025 14:51:38 CEST
Boundary wire vector sizes: 2026, 1850, 1875
minwire 0: 131
minwire 1: 1335
minwire 2: 48
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0248805, 0.808028, 0.167092, 
Output 1: 0.0391167, 0.111122, 0.164648, 0.685114, 
Output 2: 0.0665593, 0.140395, 0.291526, 0.501519, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2359!
Begin processing the 698th record. run: 6422672 subRun: 1 event: 7798 at 18-Sep-2025 14:51:45 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 933!
Begin processing the 699th record. run: 6422672 subRun: 1 event: 7799 at 18-Sep-2025 14:51:47 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1219!
Begin processing the 700th record. run: 6422672 subRun: 1 event: 7800 at 18-Sep-2025 14:51:50 CEST
Boundary wire vector sizes: 156, 171, 158
minwire 0: 42
minwire 1: 2277
minwire 2: 43
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000488623, 0.998071, 0.00144083, 
Output 1: 0.00293053, 0.993194, 0.00376611, 0.000109377, 
Output 2: 0.9981, 0.00184664, 5.18538e-05, 1.63991e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 705!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/d1/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_173_20231204T060235Z_gen_g4_detsim_hitreco__20240509T214745Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/79/b0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_931_20231202T113802Z_gen_g4_detsim_hitreco__20240507T212327Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/39/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_602_20231203T141918Z_gen_g4_detsim_hitreco__20240508T080128Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/72/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_918_20231202T191330Z_gen_g4_detsim_hitreco__20240508T064428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ca/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_56_20231206T230207Z_gen_g4_detsim_hitreco__20240510T031902Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/30/9d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_24_20231208T040205Z_gen_g4_detsim_hitreco__20240510T065328Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_77_20231204T080606Z_gen_g4_detsim_hitreco__20240509T220853Z_reco2.root
.:
total 37848
-rw-r--r--. 1 dune012 dune 14229993 Sep 18 14:51 flatcaf.root
-rw-r--r--. 1 dune012 dune 11934248 Sep 18 14:51 caf.root
-rw-r--r--. 1 dune012 dune 11934248 Sep 18 14:51 caf_fd_hd_atmo_2501_20250918T115107Z.root
-rw-r--r--. 1 dune012 dune   423328 Sep 18 14:51 caf_20250918T115107Z.log
-rw-r--r--. 1 dune012 dune   176689 Sep 18 14:51 jobscript.log
-rw-r--r--. 1 dune012 dune     1397 Sep 18 14:51 caf_20250918T115107Z.file
-rw-r--r--. 1 dune012 dune     1397 Sep 18 14:51 caf_20250918T115107Z.pfns
-rw-r--r--. 1 dune012 dune     1397 Sep 18 13:51 file.list
-rw-r--r--. 1 dune012 dune     1397 Sep 18 14:51 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune      960 Sep 18 13:51 all-input-dids.txt
-rw-r--r--. 1 dune012 dune      960 Sep 18 14:51 caf_20250918T115107Z.did
-rw-r--r--. 1 dune012 dune      960 Sep 18 13:51 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 18 13:51 debugprod.log
justIN time: 2025-09-18 20:55:45 UTC       justIN version: 01.05.00