justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242475.38@dunegpschedd01.fnal.gov

Jobsub ID242475.38@dunegpschedd01.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 18:03:07
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-03 20:06:54
From worker nodeHostnamee2589
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 18:12:17
Input filesusertests:pdhd_prod_beam__226470_86_1_20251008T212529Z_gen_g4_IonScintPDExt.root_227040_1_1_20251010T165940Z_PDInt.root
JobscriptExit code0
Real time1h (6656s)
CPU time1h (6271s = 94%)
Max RSS bytes6928355328 (6607 MiB)
Outputting started2025-11-03 20:03:13
Output filesusertests:pdhd_prod_beam__226470_86_1_20251008T212529Z_gen_g4_IonScintPDExt.root_227040_1_1_20251010T165940Z_PDInt.root_242475_38_1_20251103T181227Z_detsim_reco1.root
Finished2025-11-03 20:06:54
Saved logsjustin-logs:242475.38-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

> 251 XUVs
4956 XUVs total
1914 collection wire objects
4956 potential space points
Neighbour search...
508704 tests to find 137844 neighbours
Iterating with no regularization...
Begin: 2.77059e+07
0 2.38993e+07
1 2.35941e+07
2 2.35707e+07
Now with regularization...
Begin: 2.29553e+07
0 2.29489e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 13:56:58 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13847!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 94958 traces, 94958 summary
Retagger: tagging trace set: gauss with 71723 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71723 traces tagged "gauss"
FrameSaver: q=2.3252e+06 n=1112821 tag=gauss
wclsFrameSaver: saving 94958 traces tagged "wiener"
FrameSaver: q=2.85328e+06 n=1068670 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 5597 XUs and 6801 XVs -> 62 XUVs
C:0 T:1 24381 XUs and 25981 XVs -> 903 XUVs
C:0 T:2 24543 XUs and 21998 XVs -> 440 XUVs
C:0 T:3 1 XUs and 4 XVs -> 0 XUVs
C:0 T:4 0 XUs and 4 XVs -> 0 XUVs
C:0 T:5 38058 XUs and 32265 XVs -> 2243 XUVs
C:0 T:6 50544 XUs and 43326 XVs -> 2605 XUVs
C:0 T:7 14 XUs and 12 XVs -> 2 XUVs
6255 XUVs total
2468 collection wire objects
6255 potential space points
Neighbour search...
244033 tests to find 109902 neighbours
Iterating with no regularization...
Begin: 5.9961e+07
0 5.3278e+07
1 5.26017e+07
2 5.2518e+07
3 5.24946e+07
Now with regularization...
Begin: 4.95825e+07
0 4.95654e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 14:00:01 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14010!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 92691 traces, 92691 summary
Retagger: tagging trace set: gauss with 74081 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74081 traces tagged "gauss"
FrameSaver: q=2.01153e+06 n=1110149 tag=gauss
wclsFrameSaver: saving 92691 traces tagged "wiener"
FrameSaver: q=2.61145e+06 n=1074164 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 16959 XUs and 11889 XVs -> 136 XUVs
C:0 T:1 95830 XUs and 60804 XVs -> 2387 XUVs
C:0 T:2 13601 XUs and 24132 XVs -> 1558 XUVs
C:0 T:3 1 XUs and 11 XVs -> 0 XUVs
C:0 T:4 1 XUs and 3 XVs -> 0 XUVs
C:0 T:5 35682 XUs and 22463 XVs -> 1111 XUVs
C:0 T:6 11504 XUs and 31439 XVs -> 347 XUVs
5539 XUVs total
2256 collection wire objects
5539 potential space points
Neighbour search...
251893 tests to find 128336 neighbours
Iterating with no regularization...
Begin: 3.36776e+07
0 3.09854e+07
1 3.07781e+07
2 3.07575e+07
Now with regularization...
Begin: 2.94684e+07
0 2.94601e+07
03-Nov-2025 14:03:10 CST  Closed output file "pdhd_prod_beam__226470_86_1_20251008T212529Z_gen_g4_IonScintPDExt.root_227040_1_1_20251010T165940Z_PDInt.root_242475_38_1_20251103T181227Z_detsim_reco1.root"
03-Nov-2025 14:03:10 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            175.535       245.077       316.596       260.108       52.4805        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00270387     0.0130343     0.0237597     0.0118578    0.00566625       10     
reco:opdec:Deconvolution                             0.966447       1.09606       1.30817       1.05313      0.115369        10     
reco:ophitspe:OpHitFinderDeco                        0.236128      0.288684      0.343581      0.283502      0.038579        10     
reco:opflash:OpFlashFinder                           0.0778345     0.106268       0.15653      0.0938356     0.0264849       10     
reco:opslicer:OpSlicer                               0.255927      0.319794      0.441104      0.301774      0.0542155       10     
reco:wclsdatahd:WireCellToolkit                       64.1273       88.2651       165.491       84.4561       28.215         10     
reco:gaushit:GausHitFinder                           0.815198       1.4498        3.83253       1.21891      0.818709        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000489554   0.00337597      0.01272     0.00235477    0.00338538       10     
reco:reco3d:SpacePointSolver                         0.843081       2.56323       6.20827       1.99957       1.75065        10     
reco:hitpdune:DisambigFromSpacePoints                0.583377       1.00833       1.5359        0.95243      0.287368        10     
reco:pandora:StandardPandora                          16.6058       37.0824       59.6075       33.3787       13.6028        10     
reco:pandoraWriter:StandardPandora                   0.326842      0.798292       2.4665       0.642585       0.59827        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.12318       6.64018       18.775        3.92623       5.38365        10     
reco:pandoraShower:LArPandoraShowerCreation           1.38566       5.98884       14.5449       3.8133        4.67223        10     
reco:pandoracalo:Calorimetry                          1.03488       2.1425        4.96674       1.77659       1.20083        10     
reco:pandoracalonosce:Calorimetry                    0.817891       2.15839       5.1515        1.75573       1.26148        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.23583       3.73399       10.456        2.57275       2.79732        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.45807       3.71001       9.66102       2.50006       2.67799        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0837124     0.105636      0.148796      0.0968991      0.02181        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0745153     0.100354      0.148909      0.0898034     0.0224279       10     
reco:emtrkmichelid:EmTrackMichelId                    31.4632       51.8373       63.8794       53.7944       10.0423        10     
reco:crttag:SingleCRTMatchingProducer               0.00677734     0.0184569     0.0351216     0.017161     0.00869306       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00475597     0.0138905     0.0327049     0.0108588    0.00918564       10     
reco:pandora2Track:LArPandoraTrackCreation            5.4772        13.5385       27.4116       11.264        7.65352        10     
reco:pandora2calo:ShowerCalorimetry                   1.29563       3.1424        7.30395       2.01778       2.08335        10     
reco:pandora2calonosce:ShowerCalorimetry              1.22308       3.03161       6.48537       1.86765        1.928         10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0775383     0.107604      0.144184      0.102107      0.0220256       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0648581     0.0974818     0.122798      0.0940774     0.018474        10     
reco:pandora2Shower:LArPandoraShowerCreation          2.05294       4.67596       11.402        3.19725       3.1941         10     
reco:pandora2Showercalo:ShowerCalorimetry             2.02748       3.6481        7.86975       2.79275       1.85832        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.8655        3.33044       7.06066       2.75711       1.62232        10     
[art]:TriggerResults:TriggerResultInserter          7.3258e-05    0.000143092   0.000516925   0.00010349    0.000126041      10     
end_path:out1:RootOutput                            2.0428e-05    4.85015e-05   0.000175261   2.97815e-05   4.38601e-05      10     
end_path:out1:RootOutput(write)                       2.99781       3.72483       4.9871        3.53964      0.684044        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6142.94 MB
  Peak resident set size usage (VmHWM): 3930.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11959544
-rw-r--r-- 1 slot1_12 slot1_12      42837 Nov  3 14:03 jobscript.log
-rw-r--r-- 1 slot1_12 slot1_12        199 Nov  3 14:03 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_12 slot1_12        519 Nov  3 14:03 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_12 slot1_12       3574 Nov  3 14:03 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_12 slot1_12     200704 Nov  3 14:03 mem.db
-rw-r--r-- 1 slot1_12 slot1_12      36864 Nov  3 14:03 time.db
-rw-r--r-- 1 slot1_12 slot1_12 6265193751 Nov  3 14:03 pdhd_prod_beam__226470_86_1_20251008T212529Z_gen_g4_IonScintPDExt.root_227040_1_1_20251010T165940Z_PDInt.root_242475_38_1_20251103T181227Z_detsim_reco1.root
-rw-r--r-- 1 slot1_12 slot1_12   50479020 Nov  3 14:01 Pandora_Events.pndr
-rw-r--r-- 1 slot1_12 slot1_12          0 Nov  3 13:22 ProtoDUNE.log
-rw-r--r-- 1 slot1_12 slot1_12          0 Nov  3 13:17 debugprod.log
-rw-r--r-- 1 slot1_12 slot1_12        519 Nov  3 13:16 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_12 slot1_12 5930576153 Nov  3 13:16 temp_detsim.root
-rw-r--r-- 1 slot1_12 slot1_12        120 Nov  3 12:12 all-input-dids.txt
justIN time: 2025-11-04 01:57:41 UTC       justIN version: 01.05.01