justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.25@dunegpschedd02.fnal.gov

Jobsub ID234970.25@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:46:27
From worker nodeHostnamedunegli-7568339-0-fnpc23022.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:46
Input filesusertests:pdhd_prod_beam__231691_84_1_20251008T183352Z_gen_g4_IonScintPDExt.root_227038_148_1_20251010T165331Z_PDInt.root
JobscriptExit code0
Real time1h (3716s)
CPU time1h (3660s = 98%)
Max RSS bytes7394557952 (7052 MiB)
Outputting started2025-10-29 20:39:42
Output files
Finished2025-10-29 20:46:27
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:36:34 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15386!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 100581 traces, 100581 summary
Retagger: tagging trace set: gauss with 77756 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77756 traces tagged "gauss"
FrameSaver: q=2.30373e+06 n=1235182 tag=gauss
wclsFrameSaver: saving 100581 traces tagged "wiener"
FrameSaver: q=2.80973e+06 n=1169032 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9966 XUs and 7927 XVs -> 74 XUVs
C:0 T:1 57581 XUs and 45951 XVs -> 1342 XUVs
C:0 T:2 57110 XUs and 51400 XVs -> 1869 XUVs
C:0 T:3 73 XUs and 61 XVs -> 8 XUVs
C:0 T:4 5 XUs and 9 XVs -> 0 XUVs
C:0 T:5 28072 XUs and 14518 XVs -> 258 XUVs
C:0 T:6 29178 XUs and 27871 XVs -> 560 XUVs
C:0 T:7 0 XUs and 3 XVs -> 0 XUVs
4111 XUVs total
2399 collection wire objects
4111 potential space points
Neighbour search...
70187 tests to find 38508 neighbours
Iterating with no regularization...
Begin: 2.56362e+07
0 2.4023e+07
1 2.38959e+07
2 2.38857e+07
Now with regularization...
Begin: 2.26405e+07
0 2.26381e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:38:03 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15896!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 103170 traces, 103170 summary
Retagger: tagging trace set: gauss with 79655 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 79655 traces tagged "gauss"
FrameSaver: q=2.73123e+06 n=1225290 tag=gauss
wclsFrameSaver: saving 103170 traces tagged "wiener"
FrameSaver: q=3.38845e+06 n=1206218 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11320 XUs and 7081 XVs -> 64 XUVs
C:0 T:1 64770 XUs and 45671 XVs -> 2452 XUVs
C:0 T:2 14213 XUs and 12898 XVs -> 1121 XUVs
C:0 T:4 10 XUs and 7 XVs -> 0 XUVs
C:0 T:5 20151 XUs and 25024 XVs -> 1663 XUVs
C:0 T:6 58806 XUs and 45836 XVs -> 2745 XUVs
C:0 T:7 44 XUs and 33 XVs -> 0 XUVs
8045 XUVs total
3165 collection wire objects
8045 potential space points
Neighbour search...
309845 tests to find 140402 neighbours
Iterating with no regularization...
Begin: 5.47712e+07
0 4.77375e+07
1 4.7148e+07
2 4.70852e+07
3 4.70714e+07
Now with regularization...
Begin: 4.36604e+07
0 4.36452e+07
29-Oct-2025 20:39:41 UTC  Closed output file "pdhd_prod_beam__231691_84_1_20251008T183352Z_gen_g4_IonScintPDExt.root_227038_148_1_20251010T165331Z_PDInt.root_234970_25_1_20251029T193748Z_detsim_reco1.root"
29-Oct-2025 20:39:41 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            67.3144       88.425        123.836       86.6582       14.6949        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000378702   0.00586278     0.022049     0.00502088    0.00570863       10     
reco:opdec:Deconvolution                              0.48133      0.581868      0.741674      0.586859      0.0698839       10     
reco:ophitspe:OpHitFinderDeco                        0.0984093     0.107264      0.144355      0.103871      0.0128911       10     
reco:opflash:OpFlashFinder                           0.0424361     0.0670476     0.108696      0.0633793     0.018998        10     
reco:opslicer:OpSlicer                               0.130788      0.165527      0.214491      0.165593      0.0248561       10     
reco:wclsdatahd:WireCellToolkit                       31.0614       31.8114       32.1334       31.9024      0.313852        10     
reco:gaushit:GausHitFinder                           0.443559      0.503956      0.636941       0.49336      0.052392        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00018313    0.000268342   0.000739791   0.000220941   0.000158535      10     
reco:reco3d:SpacePointSolver                         0.215114      0.594435      0.932387      0.518397       0.22488        10     
reco:hitpdune:DisambigFromSpacePoints                0.240706      0.491434      0.774242      0.447956      0.178041        10     
reco:pandora:StandardPandora                          6.57537       15.7534       34.9981       14.5076       7.33456        10     
reco:pandoraWriter:StandardPandora                   0.136317      0.168214      0.224734      0.159796      0.0248428       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.657092       1.18237       2.00701       1.17017      0.391176        10     
reco:pandoraShower:LArPandoraShowerCreation          0.730096       1.11955       1.68009       1.05479      0.301573        10     
reco:pandoracalo:Calorimetry                         0.367516      0.640943       1.08869      0.626243      0.205369        10     
reco:pandoracalonosce:Calorimetry                    0.365245      0.629124       1.0687       0.610548      0.202776        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.737763       1.17145       1.82566       1.05252      0.339011        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.709468       1.12351       1.73114       1.01447      0.318493        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0280088     0.0331653     0.0403732     0.0334879    0.00378178       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.027043      0.031952      0.0357577     0.0323915    0.00273697       10     
reco:emtrkmichelid:EmTrackMichelId                    16.9138       21.5267       27.8119       20.9757       3.00326        10     
reco:crttag:SingleCRTMatchingProducer               0.00307996    0.00373107    0.00594303     0.0033679    0.000819972      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00242525    0.00288596    0.00333678    0.00287445    0.000242574      10     
reco:pandora2Track:LArPandoraTrackCreation            1.75985       2.8682        4.46178       2.76903      0.860021        10     
reco:pandora2calo:ShowerCalorimetry                  0.641968       1.03063       1.62887      0.942838       0.30725        10     
reco:pandora2calonosce:ShowerCalorimetry             0.623478       1.01241       1.5784       0.929992      0.304306        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0270992     0.0321373     0.0362036     0.0342059    0.00346029       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0260564     0.0312413      0.03537      0.0324028    0.00332423       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.75728       1.28797       1.92653       1.25971      0.373197        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.816023       1.42764       2.33844       1.3258       0.469902        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.798834       1.40428       2.30864       1.29583      0.469014        10     
[art]:TriggerResults:TriggerResultInserter          2.0319e-05    3.38937e-05   9.8668e-05    2.7357e-05    2.18209e-05      10     
end_path:out1:RootOutput                             9.438e-06    1.30038e-05   3.0749e-05    1.1026e-05    5.96919e-06      10     
end_path:out1:RootOutput(write)                       1.33103       1.54279       1.86539       1.50934      0.170967        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6447.01 MB
  Peak resident set size usage (VmHWM): 4242.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12130428
-rw-r--r-- 1 dunegli fnalgrid      39879 Oct 29 20:39 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:39 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:39 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:39 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:39 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:39 time.db
-rw-r--r-- 1 dunegli fnalgrid 6358847092 Oct 29 20:39 pdhd_prod_beam__231691_84_1_20251008T183352Z_gen_g4_IonScintPDExt.root_227038_148_1_20251010T165331Z_PDInt.root_234970_25_1_20251029T193748Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   53598400 Oct 29 20:38 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:26 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:23 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6008806995 Oct 29 20:23 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-03 18:32:32 UTC       justIN version: 01.05.01