justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 248319.0@dunegpschedd02.fnal.gov

Jobsub ID248319.0@dunegpschedd02.fnal.gov
Workflow ID10362
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-18 23:12:15
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-19 03:45:15
From worker nodeHostnamewn1905251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-18 23:12:54
Input filesmonte-carlo-010362-000001
JobscriptExit code0
Real time3h (14016s)
CPU time3h (13524s = 96%)
Max RSS bytes4148658176 (3956 MiB)
Outputting started2025-11-19 03:06:31
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10362/1/001/reco_stage1.root
Finished2025-11-19 03:45:15
Saved logsjustin-logs:248319.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 142027 traces, 0 summary
Retagger: tagging trace set: gauss with 112394 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 112394 traces tagged "gauss"
FrameSaver: q=4.13649e+06 n=2117343 tag=gauss
wclsFrameSaver: saving 142027 traces tagged "wiener"
FrameSaver: q=4.35472e+06 n=1914442 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 52169 XUs and 40992 XVs -> 616 XUVs
C:0 T:1 107972 XUs and 81043 XVs -> 5905 XUVs
C:0 T:2 42910 XUs and 77552 XVs -> 4546 XUVs
C:0 T:3 21635 XUs and 27330 XVs -> 90 XUVs
C:0 T:4 17001 XUs and 16598 XVs -> 184 XUVs
C:0 T:5 124526 XUs and 45577 XVs -> 7063 XUVs
C:0 T:6 135872 XUs and 169433 XVs -> 6357 XUVs
C:0 T:7 50937 XUs and 65967 XVs -> 687 XUVs
C:0 T:8 12258 XUs and 10214 XVs -> 121 XUVs
C:0 T:9 82226 XUs and 86582 XVs -> 4651 XUVs
C:0 T:10 44065 XUs and 25947 XVs -> 4387 XUVs
C:0 T:11 154 XUs and 116 XVs -> 7 XUVs
34614 XUVs total
12643 collection wire objects
31317 potential space points
Neighbour search...
1517005 tests to find 788954 neighbours
Iterating with no regularization...
Begin: 9.25464e+07
0 7.41128e+07
1 7.29931e+07
2 7.28355e+07
3 7.27892e+07
Now with regularization...
Begin: 6.12619e+07
0 6.12358e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410598558
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410598558
fLifetime: 30000 [us]
19-Nov-2025 03:06:28 GMT  Closed output file "reco_stage1.root"
19-Nov-2025 03:06:28 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          188.646       385.456       561.287       379.224       110.607        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00204459    0.00426786    0.00669816    0.00360978    0.00143029       10     
reco:ophit:OpHitFinder                             0.266056      0.310296      0.368934      0.303272      0.0340487       10     
reco:opflash:OpFlashFinder                         0.109251      0.176063      0.214552      0.181231      0.0301795       10     
reco:opslicer:OpSlicer                             0.511295      0.723473       1.02586      0.717241      0.158319        10     
reco:caldata:DataPrepModule                         4.86192       6.14766       7.34681       6.34198      0.851988        10     
reco:wclsdatasp:WireCellToolkit                     34.824        45.5219       64.4053       42.9193       9.40441        10     
reco:gaushit:GausHitFinder                         0.683821       1.22932       2.14669       1.11153      0.439011        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000291396   0.000767028   0.00132171    0.000747317   0.000316084      10     
reco:reco3d:SpacePointSolver                        4.25504       15.587        31.2234       12.825        8.27568        10     
reco:hitpdune:DisambigFromSpacePoints              0.689985        1.709        2.78066       1.59529      0.776433        10     
reco:pandora:StandardPandora                        9.86304       35.3921       66.5709       30.3455       16.6169        10     
reco:pandoraWriter:StandardPandora                  0.23921      0.503341      0.745378      0.517661      0.161482        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.59993       5.38271       9.74794       4.16791       2.77219        10     
reco:pandoraShower:LArPandoraShowerCreation         1.3495        5.92481       12.6966       4.85582       3.56413        10     
reco:pandoracalo:Calorimetry                       0.895547       2.92184       6.20738       2.13106       1.63716        10     
reco:pandoracalonosce:Calorimetry                  0.819218       2.59856       6.30202       2.01584       1.52396        10     
reco:pandorapid:Chi2ParticleID                     0.0035918    0.00571705     0.0116416    0.00528717    0.00217004       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0303443     0.057004      0.122072      0.0544761     0.0262891       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0036145    0.00556963    0.00984815    0.00527257    0.00174674       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.4539        5.98721       13.7523       4.95725       3.6799         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.30619       6.32006       13.3938       6.06161       3.98345        10     
reco:emtrkmichelid:EmTrackMichelId                  106.587       197.127       260.277       194.951       43.6418        10     
reco:crttag:SingleCRTMatchingProducer             0.00872748      0.02201      0.0429009     0.018947      0.0112288       10     
reco:crtreco:TwoCRTMatchingProducer               0.00606238     0.0155763     0.0328943     0.0110684    0.00902228       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.690319      0.993757       1.44194      0.923968      0.238533        10     
reco:pandora2Track:LArPandoraTrackCreation          4.40509       15.9724       27.1327       14.9378       7.77771        10     
reco:pandora2calo:Calorimetry                       2.14289       7.75108       14.3196       7.45178       3.86745        10     
reco:pandora2calonosce:Calorimetry                  1.98801       7.06881       12.8619       7.23621       3.42809        10     
reco:pandora2pid:Chi2ParticleID                   0.00711738     0.0133062     0.0197846     0.0120528    0.00375143       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0357962     0.0618997     0.0967433     0.0624667     0.0184156       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00740807     0.0120674     0.0165815     0.0121414    0.00282754       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.63502       5.00218       8.71304       5.28236       2.0501         10     
reco:pandora2Showercalo:ShowerCalorimetry           1.99948       6.66609       12.4968       7.40486       3.13227        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.00037       5.33684       9.60881       5.39759       2.58963        10     
[art]:TriggerResults:TriggerResultInserter        6.0274e-05    9.14918e-05   0.000239389   7.0377e-05    5.26822e-05      10     
end_path:out1:RootOutput                           1.048e-05    1.41895e-05   2.5407e-05    1.2433e-05    4.34986e-06      10     
end_path:out1:RootOutput(write)                     1.59308       2.72406       4.15427       2.66618       0.71357        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4975.37 MB
  Peak resident set size usage (VmHWM): 4148.66 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.65 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9868204
-rw-r--r-- 1 dune01 dune01 2322215429 Nov 19 01:59 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2205996936 Nov 19 01:56 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1986338380 Nov 19 03:06 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1797024247 Nov 18 23:34 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1722925064 Nov 18 23:20 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   69908320 Nov 19 03:00 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242777 Nov 19 03:06 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 19 03:06 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 19 03:06 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 19 03:06 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 19 01:59 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 23:34 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 19 03:06 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 18 23:12 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 19 03:06 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 19 02:06 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 19 01:59 debugprod.log
justIN time: 2026-02-09 05:00:03 UTC       justIN version: 01.06.00