justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236450.0@dunegpschedd02.fnal.gov

Jobsub ID236450.0@dunegpschedd02.fnal.gov
Workflow ID9535
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-02 23:14:07
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-03 03:22:00
From worker nodeHostnamewn-da-14.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-02 23:14:48
Input filesmonte-carlo-009535-000001
JobscriptExit code0
Real time4h (14740s)
CPU time4h (14608s = 99%)
Max RSS bytes4612845568 (4399 MiB)
Outputting started2025-11-03 03:20:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09535/1/001/reco_stage1.root
Finished2025-11-03 03:22:00
Saved logsjustin-logs:236450.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 156675 traces, 0 summary
Retagger: tagging trace set: gauss with 117118 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 117118 traces tagged "gauss"
FrameSaver: q=3.97998e+06 n=2054794 tag=gauss
wclsFrameSaver: saving 156675 traces tagged "wiener"
FrameSaver: q=4.21829e+06 n=1803724 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 36722 XUs and 38037 XVs -> 522 XUVs
C:0 T:1 86074 XUs and 77952 XVs -> 6513 XUVs
C:0 T:2 82488 XUs and 93745 XVs -> 3428 XUVs
C:0 T:3 41185 XUs and 40201 XVs -> 258 XUVs
C:0 T:4 18495 XUs and 24071 XVs -> 335 XUVs
C:0 T:5 108777 XUs and 157294 XVs -> 14511 XUVs
C:0 T:6 69571 XUs and 71180 XVs -> 2050 XUVs
C:0 T:7 31040 XUs and 31188 XVs -> 199 XUVs
C:0 T:8 12092 XUs and 9710 XVs -> 125 XUVs
C:0 T:9 77262 XUs and 98138 XVs -> 7036 XUVs
C:0 T:10 41752 XUs and 35284 XVs -> 3425 XUVs
C:0 T:11 256 XUs and 186 XVs -> 21 XUVs
38423 XUVs total
14659 collection wire objects
35595 potential space points
Neighbour search...
2488427 tests to find 1248446 neighbours
Iterating with no regularization...
Begin: 8.40129e+07
0 6.463e+07
1 6.36062e+07
2 6.34505e+07
3 6.34025e+07
Now with regularization...
Begin: 5.23042e+07
0 5.22586e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410276660
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410276660
fLifetime: 30000 [us]
03-Nov-2025 04:20:26 CET  Closed output file "reco_stage1.root"
03-Nov-2025 04:20:26 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          463.462       498.782       540.047       491.08        24.9668        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000992077    0.0056863     0.0101761     0.0055276    0.00288512       10     
reco:ophit:OpHitFinder                              0.27118      0.315329      0.346322      0.319604      0.0226279       10     
reco:opflash:OpFlashFinder                         0.171394      0.221205      0.256284      0.226569      0.0234248       10     
reco:opslicer:OpSlicer                             0.636706      0.766844      0.841734      0.793751      0.0701992       10     
reco:caldata:DataPrepModule                         4.98763       5.14365       5.35186       5.13705      0.100552        10     
reco:wclsdatasp:WireCellToolkit                     37.2319       53.5204       59.3095       56.9793       8.1973         10     
reco:gaushit:GausHitFinder                          1.27206       1.48385       1.63383       1.47426      0.0942159       10     
reco:nhitsfilter:NumberOfHitsFilter               0.000658092   0.000950083   0.00147698    0.000858308   0.000231484      10     
reco:reco3d:SpacePointSolver                        10.8267       20.1282       34.7802        18.06        7.5124         10     
reco:hitpdune:DisambigFromSpacePoints               1.33964       1.87925       2.13927       1.90682      0.219966        10     
reco:pandora:StandardPandora                        37.1505       49.6671       99.5475       43.9964       17.365         10     
reco:pandoraWriter:StandardPandora                 0.449972      0.544438      0.594152      0.545982      0.0387655       10     
reco:pandoraTrack:LArPandoraTrackCreation           6.74731       8.14884       10.4974       8.04347      0.962128        10     
reco:pandoraShower:LArPandoraShowerCreation         5.58384       9.62362       13.7837       9.46517       1.91483        10     
reco:pandoracalo:Calorimetry                        3.36829       4.30782       4.71187       4.42911      0.353186        10     
reco:pandoracalonosce:Calorimetry                   3.34134       4.0584        4.81814       3.99587      0.451211        10     
reco:pandorapid:Chi2ParticleID                    0.00658147    0.00738039    0.00866962    0.00723342    0.000598523      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0508279     0.0592095     0.0658929     0.0602943    0.00515858       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00610577    0.00673414    0.00775025    0.00673481    0.000438059      10     
reco:pandoraShowercalo:ShowerCalorimetry            5.5777        9.03776       11.624        8.8216        1.6019         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.24837       8.91453       11.0247       8.85922       1.5875         10     
reco:emtrkmichelid:EmTrackMichelId                  207.158       240.895       258.582       241.306       13.1758        10     
reco:crttag:SingleCRTMatchingProducer              0.0169924     0.0253485     0.0451609     0.0232682    0.00856211       10     
reco:crtreco:TwoCRTMatchingProducer               0.00926836     0.0166381     0.0251501     0.015949     0.00511418       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.862376      0.954159       1.09175      0.922036      0.0735597       10     
reco:pandora2Track:LArPandoraTrackCreation          16.1432       24.9252       32.5181       25.8864       4.2285         10     
reco:pandora2calo:Calorimetry                       7.89913       12.387        15.1938       12.5152       2.04953        10     
reco:pandora2calonosce:Calorimetry                  7.60909       11.3258       14.6847       11.2437       1.97514        10     
reco:pandora2pid:Chi2ParticleID                     0.01482      0.0172514     0.019551      0.0173416    0.00142151       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0603141     0.0762747     0.0870017     0.0766205    0.00723423       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0142616     0.0170432     0.0193888     0.0171138    0.00129705       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.8057        8.8291        10.9374       9.05262       1.49085        10     
reco:pandora2Showercalo:ShowerCalorimetry            6.007        9.51248       11.8305       9.59571       1.45243        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.14471        8.327        10.2549       8.77613       1.29239        10     
[art]:TriggerResults:TriggerResultInserter        4.0587e-05    6.14379e-05   0.000182858   4.70545e-05   4.08194e-05      10     
end_path:out1:RootOutput                           6.313e-06     8.494e-06    1.3545e-05     8.28e-06     1.8169e-06       10     
end_path:out1:RootOutput(write)                     2.73231       3.40052       3.72923       3.47347      0.276695        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5418.66 MB
  Peak resident set size usage (VmHWM): 4612.85 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.19 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11216356
-rw-r--r--. 1 dune009 dune 2601201299 Nov  3 02:54 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2466987347 Nov  3 02:52 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2330887460 Nov  3 04:20 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2040871154 Nov  3 00:36 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1959518070 Nov  3 00:21 g4_stage1.root
-rw-r--r--. 1 dune009 dune   85474530 Nov  3 04:13 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242847 Nov  3 04:20 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  3 04:20 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  3 04:20 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  3 04:20 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  3 02:54 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  3 00:36 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  3 04:20 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  3 00:14 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  3 04:20 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  3 03:02 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  3 02:55 debugprod.log
justIN time: 2025-11-04 01:37:14 UTC       justIN version: 01.05.01