justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257949.1@dunegpschedd02.fnal.gov

Jobsub ID257949.1@dunegpschedd02.fnal.gov
Workflow ID11035
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-05 12:33:43
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-05 16:56:32
From worker nodeHostnamewn1904241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-05 12:35:12
Input filesmonte-carlo-011035-000001
JobscriptExit code0
Real time4h (15356s)
CPU time4h (14878s = 96%)
Max RSS bytes4068237312 (3879 MiB)
Outputting started2025-12-05 16:51:09
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11035/1/001/reco_stage1.root
Finished2025-12-05 16:56:32
Saved logsjustin-logs:257949.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

unning tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 113220 traces, 0 summary
Retagger: tagging trace set: gauss with 89368 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 89368 traces tagged "gauss"
FrameSaver: q=3.18117e+06 n=1624602 tag=gauss
wclsFrameSaver: saving 113220 traces tagged "wiener"
FrameSaver: q=3.3321e+06 n=1454897 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 24279 XUs and 30440 XVs -> 613 XUVs
C:0 T:1 42200 XUs and 52028 XVs -> 6479 XUVs
C:0 T:2 62152 XUs and 75271 XVs -> 1996 XUVs
C:0 T:3 35490 XUs and 37216 XVs -> 224 XUVs
C:0 T:4 8392 XUs and 10535 XVs -> 38 XUVs
C:0 T:5 25688 XUs and 21740 XVs -> 2273 XUVs
C:0 T:6 78200 XUs and 94180 XVs -> 2421 XUVs
C:0 T:7 32728 XUs and 38699 XVs -> 340 XUVs
C:0 T:8 8454 XUs and 8332 XVs -> 66 XUVs
C:0 T:9 50374 XUs and 65480 XVs -> 3975 XUVs
C:0 T:10 30486 XUs and 46696 XVs -> 4058 XUVs
C:0 T:11 67 XUs and 61 XVs -> 6 XUVs
22489 XUVs total
9702 collection wire objects
20347 potential space points
Neighbour search...
921311 tests to find 505324 neighbours
Iterating with no regularization...
Begin: 7.81121e+07
0 6.65005e+07
1 6.58321e+07
2 6.57256e+07
3 6.56964e+07
Now with regularization...
Begin: 5.54241e+07
0 5.54035e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410931563
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410931563
fLifetime: 30000 [us]
05-Dec-2025 16:51:06 GMT  Closed output file "reco_stage1.root"
05-Dec-2025 16:51:06 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          303.878       470.791       639.474       463.474       82.8514        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00186933    0.00591217    0.00979385    0.00621077    0.00252975       10     
reco:ophit:OpHitFinder                             0.281498      0.403227      0.504164      0.402144      0.0531117       10     
reco:opflash:OpFlashFinder                         0.147531      0.209995      0.243223      0.221971      0.0298764       10     
reco:opslicer:OpSlicer                             0.780597      0.964978       1.15067      0.944121      0.100254        10     
reco:caldata:DataPrepModule                         7.42033       7.82077       8.14692       7.84591      0.228547        10     
reco:wclsdatasp:WireCellToolkit                     38.3335       46.3963       57.9312       45.9545       5.10434        10     
reco:gaushit:GausHitFinder                          1.07324       1.62684       2.32982       1.74498      0.387294        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000496615   0.000826895   0.00116119    0.000751675   0.000212589      10     
reco:reco3d:SpacePointSolver                        9.55036       23.0765       49.4861       22.8393       10.5962        10     
reco:hitpdune:DisambigFromSpacePoints              0.980298       2.24482       3.77089       2.21208       0.78058        10     
reco:pandora:StandardPandora                        19.5872       45.3329       78.5036       45.5529       15.3193        10     
reco:pandoraWriter:StandardPandora                 0.480831      0.643748        0.847       0.623617      0.0973795       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.69133       6.05944       10.0161       5.80011       1.90086        10     
reco:pandoraShower:LArPandoraShowerCreation         2.23381       6.60314       11.7607       5.96997       2.48245        10     
reco:pandoracalo:Calorimetry                        1.52961       3.49938       5.72995       3.59537       1.11024        10     
reco:pandoracalonosce:Calorimetry                   1.3291        3.28296       5.51586       3.20576       1.05762        10     
reco:pandorapid:Chi2ParticleID                    0.00569888    0.00834314     0.0100229    0.00897167    0.00151611       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.056746      0.0771251     0.101929      0.073372      0.0155596       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00647666    0.00792908    0.00969736     0.0080187    0.00108081       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.20816       6.91087       12.8309       6.64172       2.7945         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.17901       6.75296       12.6441       6.83878       2.78784        10     
reco:emtrkmichelid:EmTrackMichelId                  178.358       248.341       316.762       251.82        35.4844        10     
reco:crttag:SingleCRTMatchingProducer               0.01414      0.0268974     0.0653775     0.0222251     0.0142805       10     
reco:crtreco:TwoCRTMatchingProducer               0.00821567     0.0127686     0.0173704     0.0124512    0.00296844       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.708008       1.05349       1.32065       1.1065       0.175031        10     
reco:pandora2Track:LArPandoraTrackCreation          7.2052        17.2435       23.0629       17.2087       4.26781        10     
reco:pandora2calo:Calorimetry                       3.57305       9.04882       13.7853       8.47598       2.84284        10     
reco:pandora2calonosce:Calorimetry                  3.40976       8.54758       12.7776       7.95456       2.50572        10     
reco:pandora2pid:Chi2ParticleID                    0.0116918     0.0187334     0.0237413     0.0187623    0.00399625       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0696294     0.0987189     0.127549      0.0931716     0.020187        10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0108987     0.0188885     0.0241932     0.0194764    0.00415519       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.59754       6.10188       9.11225       6.1431        1.86546        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.03178       7.70466       11.6988       8.0552        2.4025         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.79854       6.70998       9.67441       6.79163       1.90708        10     
[art]:TriggerResults:TriggerResultInserter        6.2769e-05    0.000106568   0.000255441   9.09365e-05   5.57185e-05      10     
end_path:out1:RootOutput                           9.759e-06    0.000187048   0.00107673    1.6116e-05    0.000352406      10     
end_path:out1:RootOutput(write)                     2.54486       3.75943       5.32776       3.6745       0.792298        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5058.1 MB
  Peak resident set size usage (VmHWM): 4068.24 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.80 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10107424
-rw-r--r-- 1 dune01 dune01 2385032711 Dec  5 15:30 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2262283987 Dec  5 15:26 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2072269256 Dec  5 16:51 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1814201565 Dec  5 13:00 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1740402479 Dec  5 12:43 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   75181060 Dec  5 16:45 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242872 Dec  5 16:51 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  5 16:51 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  5 16:51 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  5 16:51 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  5 15:30 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  5 13:00 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  5 16:51 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  5 12:35 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  5 16:51 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  5 15:37 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  5 15:30 debugprod.log
justIN time: 2026-02-07 19:09:43 UTC       justIN version: 01.06.00