justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249056.0@dunegpschedd02.fnal.gov

Jobsub ID249056.0@dunegpschedd02.fnal.gov
Workflow ID10439
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-21 00:17:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-21 03:21:21
From worker nodeHostnamewn-da-11.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-21 00:18:45
Input filesmonte-carlo-010439-000001
JobscriptExit code0
Real time3h (10877s)
CPU time2h (10738s = 98%)
Max RSS bytes4217376768 (4022 MiB)
Outputting started2025-11-21 03:20:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10439/1/001/reco_stage1.root
Finished2025-11-21 03:21:21
Saved logsjustin-logs:249056.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 109948 traces, 0 summary
Retagger: tagging trace set: gauss with 83386 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 83386 traces tagged "gauss"
FrameSaver: q=2.89225e+06 n=1466703 tag=gauss
wclsFrameSaver: saving 109948 traces tagged "wiener"
FrameSaver: q=3.07366e+06 n=1293560 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 28270 XUs and 22158 XVs -> 309 XUVs
C:0 T:1 43247 XUs and 36598 XVs -> 3749 XUVs
C:0 T:2 70523 XUs and 91298 XVs -> 2827 XUVs
C:0 T:3 35239 XUs and 41246 XVs -> 218 XUVs
C:0 T:4 12482 XUs and 13282 XVs -> 295 XUVs
C:0 T:5 59914 XUs and 47717 XVs -> 4894 XUVs
C:0 T:6 65193 XUs and 79415 XVs -> 3491 XUVs
C:0 T:7 26761 XUs and 31944 XVs -> 191 XUVs
C:0 T:8 6342 XUs and 6161 XVs -> 155 XUVs
C:0 T:9 39280 XUs and 57084 XVs -> 4237 XUVs
C:0 T:10 31571 XUs and 86557 XVs -> 5583 XUVs
C:0 T:11 118 XUs and 97 XVs -> 8 XUVs
25957 XUVs total
10265 collection wire objects
24115 potential space points
Neighbour search...
1611721 tests to find 752526 neighbours
Iterating with no regularization...
Begin: 7.3048e+07
0 5.69259e+07
1 5.59956e+07
2 5.58709e+07
3 5.58354e+07
Now with regularization...
Begin: 4.62686e+07
0 4.62475e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410639694
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410639694
fLifetime: 30000 [us]
21-Nov-2025 04:20:02 CET  Closed output file "reco_stage1.root"
21-Nov-2025 04:20:02 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          222.168       333.315       485.557       335.457       62.7963        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000774866   0.00468299    0.00799541    0.00530469    0.00226441       10     
reco:ophit:OpHitFinder                             0.214448      0.239797       0.29902      0.231532      0.0251987       10     
reco:opflash:OpFlashFinder                         0.118096      0.140451      0.192099      0.131797      0.021319        10     
reco:opslicer:OpSlicer                              0.49524      0.604675      0.784399      0.580558      0.0796295       10     
reco:caldata:DataPrepModule                         4.26448       4.43847       4.78496       4.40914      0.150081        10     
reco:wclsdatasp:WireCellToolkit                     33.9282       52.6543       62.6552       56.4752       9.92393        10     
reco:gaushit:GausHitFinder                         0.728464       1.01658       1.41329      0.964868       0.18489        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000418217   0.000649916   0.000852143   0.000641793   0.000120887      10     
reco:reco3d:SpacePointSolver                        6.76967       12.7771       31.2903       10.3895       6.87725        10     
reco:hitpdune:DisambigFromSpacePoints              0.643025       1.20856       2.19736       1.17452      0.391019        10     
reco:pandora:StandardPandora                        11.425        25.7504       53.5661       22.6918       11.1373        10     
reco:pandoraWriter:StandardPandora                  0.21522      0.360722       0.49213      0.356673      0.0691397       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.64923       3.47313       5.0375        3.4866       0.837647        10     
reco:pandoraShower:LArPandoraShowerCreation         1.09666       4.30789       8.14536       4.4386        1.7848         10     
reco:pandoracalo:Calorimetry                       0.886459       1.88324       2.53817       1.95463      0.453336        10     
reco:pandoracalonosce:Calorimetry                  0.793081       1.7589        2.48343       1.80431      0.447752        10     
reco:pandorapid:Chi2ParticleID                    0.00378257    0.00479059    0.00642275    0.00462009    0.000668128      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0262142     0.0416343     0.0530377     0.0415007    0.00674239       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00364839    0.00461741    0.00510113    0.00463695    0.00043785       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.02899       4.28161       7.18098       4.58976       1.71383        10     
reco:pandoraShowercalonosce:ShowerCalorimetry      0.995135       3.82702       7.03585       3.99985       1.53863        10     
reco:emtrkmichelid:EmTrackMichelId                  119.646       175.89        230.253       178.524       27.2887        10     
reco:crttag:SingleCRTMatchingProducer             0.00553009     0.0123604     0.0176365     0.0122867    0.00409607       10     
reco:crtreco:TwoCRTMatchingProducer                0.0046365    0.00831569     0.0107105    0.00829597    0.00171677       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.537871      0.713361      0.913442      0.748345      0.108883        10     
reco:pandora2Track:LArPandoraTrackCreation          3.80307       11.5679       20.821        10.7852       4.29183        10     
reco:pandora2calo:Calorimetry                       1.80305       5.44752       9.54011       4.95892       1.99554        10     
reco:pandora2calonosce:Calorimetry                  1.67601       5.14313       9.05457       4.74562       1.88393        10     
reco:pandora2pid:Chi2ParticleID                   0.00761263     0.011884      0.0151257     0.0117985    0.00188719       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0318141     0.0523288     0.0698789     0.0532057    0.00945645       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00749002     0.0118097     0.014771      0.0117139    0.00193589       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.48765       4.20052       7.59679       3.98805       1.47469        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.53473       4.62672       7.39855       4.45977       1.50726        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.41017       4.21813       6.5176        4.30774       1.32002        10     
[art]:TriggerResults:TriggerResultInserter        2.8975e-05    3.89112e-05   0.000104118   3.26075e-05   2.18272e-05      10     
end_path:out1:RootOutput                           5.881e-06    6.9101e-06    1.1471e-05    6.3125e-06    1.57263e-06      10     
end_path:out1:RootOutput(write)                     1.7058        2.49866       3.39082       2.50804      0.413646        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5019.32 MB
  Peak resident set size usage (VmHWM): 4217.38 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.74 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9850132
-rw-r--r--. 1 dune009 dune 2323181049 Nov 21 03:22 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2206880908 Nov 21 03:20 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 1995270291 Nov 21 04:20 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1782708530 Nov 21 01:35 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1706649265 Nov 21 01:24 g4_stage1.root
-rw-r--r--. 1 dune009 dune   71227550 Nov 21 04:16 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242862 Nov 21 04:20 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 21 04:20 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 21 04:20 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 21 04:20 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 21 03:22 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 21 01:35 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 21 04:20 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 21 01:18 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 21 04:20 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 21 03:28 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 21 03:22 debugprod.log
justIN time: 2026-02-09 01:58:26 UTC       justIN version: 01.06.00