justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 248109.1@dunegpschedd02.fnal.gov

Jobsub ID248109.1@dunegpschedd02.fnal.gov
Workflow ID10331
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-18 15:27:51
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-18 20:08:10
From worker nodeHostnamewn1905251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-18 15:29:10
Input filesmonte-carlo-010331-000001
JobscriptExit code0
Real time4h (16646s)
CPU time4h (16156s = 97%)
Max RSS bytes4292882432 (4094 MiB)
Outputting started2025-11-18 20:06:37
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10331/1/001/reco_stage1.root
Finished2025-11-18 20:08:10
Saved logsjustin-logs:248109.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

awDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 104880 traces, 0 summary
Retagger: tagging trace set: gauss with 77880 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77880 traces tagged "gauss"
FrameSaver: q=2.67264e+06 n=1331862 tag=gauss
wclsFrameSaver: saving 104880 traces tagged "wiener"
FrameSaver: q=2.86148e+06 n=1172601 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 24518 XUs and 26538 XVs -> 210 XUVs
C:0 T:1 44805 XUs and 43252 XVs -> 3090 XUVs
C:0 T:2 27777 XUs and 46992 XVs -> 1546 XUVs
C:0 T:3 15716 XUs and 21638 XVs -> 96 XUVs
C:0 T:4 9804 XUs and 8843 XVs -> 45 XUVs
C:0 T:5 34846 XUs and 22507 XVs -> 2522 XUVs
C:0 T:6 150438 XUs and 184685 XVs -> 16840 XUVs
C:0 T:7 44934 XUs and 60937 XVs -> 1135 XUVs
C:0 T:8 4833 XUs and 6625 XVs -> 143 XUVs
C:0 T:9 37714 XUs and 58700 XVs -> 7058 XUVs
C:0 T:10 40406 XUs and 19058 XVs -> 5946 XUVs
C:0 T:11 84 XUs and 34 XVs -> 18 XUVs
38649 XUVs total
9385 collection wire objects
35102 potential space points
Neighbour search...
4094052 tests to find 1932476 neighbours
Iterating with no regularization...
Begin: 3.99661e+07
0 3.02743e+07
1 2.94704e+07
2 2.93468e+07
3 2.93076e+07
4 2.92907e+07
Now with regularization...
Begin: 2.1702e+07
0 2.16704e+07
1 2.1647e+07
2 2.16303e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410591970
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410591970
fLifetime: 30000 [us]
18-Nov-2025 20:06:34 GMT  Closed output file "reco_stage1.root"
18-Nov-2025 20:06:34 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          320.889       542.299       800.524       538.796       157.937        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00216003    0.00564701     0.0101304    0.00560391    0.00207241       10     
reco:ophit:OpHitFinder                             0.264933       0.35886      0.507157      0.339134      0.0880611       10     
reco:opflash:OpFlashFinder                         0.136521      0.225117      0.343642      0.218336      0.0649375       10     
reco:opslicer:OpSlicer                             0.507417       0.89623        1.51        0.765347      0.334943        10     
reco:caldata:DataPrepModule                         4.85543       7.14333       8.85261       7.1395        1.50091        10     
reco:wclsdatasp:WireCellToolkit                     36.6171       48.8428       65.0169       48.6902       8.71484        10     
reco:gaushit:GausHitFinder                          0.8594        1.47568       2.2549        1.45022      0.374593        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000424827   0.00130918    0.00231933    0.00124583    0.000575585      10     
reco:reco3d:SpacePointSolver                        12.7041       26.5566       65.083        19.6356       15.0095        10     
reco:hitpdune:DisambigFromSpacePoints              0.895406       2.31324       4.05468       2.10061      0.965797        10     
reco:pandora:StandardPandora                        16.5662       44.4413       83.4892       38.7644       18.6249        10     
reco:pandoraWriter:StandardPandora                 0.315011      0.776314       1.2079       0.727979      0.279382        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.99693       9.94362       16.3013       10.8175       4.29838        10     
reco:pandoraShower:LArPandoraShowerCreation         2.74947       12.4235       23.0713       13.9798       6.2126         10     
reco:pandoracalo:Calorimetry                        1.4641        5.39065       8.42635       6.09448       2.20326        10     
reco:pandoracalonosce:Calorimetry                   1.23252       5.24888       8.4515        5.08157       2.36438        10     
reco:pandorapid:Chi2ParticleID                    0.00398369    0.00902824     0.0175368    0.00879991    0.00451784       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0361238     0.0808651      0.1492       0.0794233     0.0361362       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00403324    0.00777614     0.0126096    0.00825092    0.00290228       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.79934       12.4099       22.713        13.7594       6.75774        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.66938       11.8176       23.3483       12.0499       6.71778        10     
reco:emtrkmichelid:EmTrackMichelId                  185.867       254.998       323.223       254.668       52.6802        10     
reco:crttag:SingleCRTMatchingProducer              0.0130238     0.0315316     0.0609394     0.0267755     0.0172027       10     
reco:crtreco:TwoCRTMatchingProducer               0.00950549     0.0254453     0.0525928     0.0211759     0.0157147       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.792515       1.26548       1.81136       1.15821      0.391482        10     
reco:pandora2Track:LArPandoraTrackCreation          11.8875       30.7784       54.072        33.3781       14.1635        10     
reco:pandora2calo:Calorimetry                       5.1839        15.2185       29.7189       17.5138       7.23319        10     
reco:pandora2calonosce:Calorimetry                  4.2123        13.0788       24.298        11.0135       6.90213        10     
reco:pandora2pid:Chi2ParticleID                   0.00883864     0.0164661     0.0257753     0.0150904    0.00609295       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0463841     0.0836545     0.119293      0.0808716     0.0250031       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00952744     0.0179749     0.0285551     0.0155366    0.00651255       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.47294       10.1327       19.6301       8.57511       5.14939        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.58192       11.6621       26.3475       10.392        6.25322        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.57498       10.7799       25.5652       7.86317       6.44631        10     
[art]:TriggerResults:TriggerResultInserter        6.0233e-05    0.000119139   0.000248036   9.29645e-05   6.03024e-05      10     
end_path:out1:RootOutput                          1.1452e-05    2.25312e-05   4.4914e-05    1.9887e-05    1.04295e-05      10     
end_path:out1:RootOutput(write)                     2.41658       3.57724       5.5594        3.28337       1.00703        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5114.67 MB
  Peak resident set size usage (VmHWM): 4292.88 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.40 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10497716
-rw-r--r-- 1 dune01 dune01 2453869941 Nov 18 18:33 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2330022343 Nov 18 18:30 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2158797668 Nov 18 20:06 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1906336227 Nov 18 15:59 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1821551257 Nov 18 15:38 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   78450590 Nov 18 20:02 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243009 Nov 18 20:06 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 18 20:06 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 18 20:06 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 18 20:06 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 18 18:33 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 15:59 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 20:06 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 18 15:29 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 18 20:06 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 18 18:44 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 18 18:33 debugprod.log
justIN time: 2026-02-09 06:12:28 UTC       justIN version: 01.06.00