justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267296.1@dunegpschedd01.fnal.gov

Jobsub ID267296.1@dunegpschedd01.fnal.gov
Workflow ID11157
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-12 09:47:15
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-12-12 16:33:25
From worker nodeHostnameheplnc151.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-12 11:37:39
Input filesmonte-carlo-011157-000001
JobscriptExit code0
Real time4h (17658s)
CPU time4h (16820s = 95%)
Max RSS bytes4318937088 (4118 MiB)
Outputting started2025-12-12 16:31:58
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11157/1/001/reco_stage1.root
Finished2025-12-12 16:33:25
Saved logsjustin-logs:267296.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

nels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 148872 traces, 0 summary
Retagger: tagging trace set: gauss with 117122 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 117122 traces tagged "gauss"
FrameSaver: q=4.21756e+06 n=2146498 tag=gauss
wclsFrameSaver: saving 148872 traces tagged "wiener"
FrameSaver: q=4.46147e+06 n=1927102 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 53285 XUs and 39441 XVs -> 671 XUVs
C:0 T:1 103552 XUs and 84400 XVs -> 8500 XUVs
C:0 T:2 116219 XUs and 158061 XVs -> 10795 XUVs
C:0 T:3 52849 XUs and 62262 XVs -> 453 XUVs
C:0 T:4 18504 XUs and 19272 XVs -> 203 XUVs
C:0 T:5 77588 XUs and 59263 XVs -> 8182 XUVs
C:0 T:6 79454 XUs and 78870 XVs -> 2729 XUVs
C:0 T:7 34766 XUs and 34973 XVs -> 190 XUVs
C:0 T:8 8763 XUs and 5136 XVs -> 73 XUVs
C:0 T:9 70289 XUs and 53442 XVs -> 4920 XUVs
C:0 T:10 68865 XUs and 44324 XVs -> 10626 XUVs
C:0 T:11 340 XUs and 241 XVs -> 57 XUVs
47399 XUVs total
13275 collection wire objects
43518 potential space points
Neighbour search...
4591298 tests to find 2349512 neighbours
Iterating with no regularization...
Begin: 1.00365e+08
0 8.46668e+07
1 8.35817e+07
2 8.34189e+07
3 8.33701e+07
Now with regularization...
Begin: 6.94244e+07
0 6.93525e+07
1 6.93117e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411058630
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411058630
fLifetime: 30000 [us]
12-Dec-2025 16:31:55 GMT  Closed output file "reco_stage1.root"
12-Dec-2025 16:31:55 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          364.228       592.734       713.707       590.862       99.748         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00179762    0.00454808    0.00683374    0.00457949    0.00162094       10     
reco:ophit:OpHitFinder                             0.328414      0.402246      0.458205      0.419086      0.0412649       10     
reco:opflash:OpFlashFinder                         0.214629      0.262872      0.308786      0.262571      0.0265241       10     
reco:opslicer:OpSlicer                             0.823034       1.09447       1.22494       1.12022      0.127386        10     
reco:caldata:DataPrepModule                         5.52525       6.92241       7.75289       7.20199      0.654571        10     
reco:wclsdatasp:WireCellToolkit                     40.8239       44.847        49.5235       44.4693       3.24812        10     
reco:gaushit:GausHitFinder                          1.10307       1.6543        2.22409       1.52969      0.408058        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000664635   0.000884091   0.00132586    0.000836139   0.000204664      10     
reco:reco3d:SpacePointSolver                        9.12043       20.261        25.894        22.0679       4.74684        10     
reco:hitpdune:DisambigFromSpacePoints               1.37674       2.4948         3.209        2.48637      0.557165        10     
reco:pandora:StandardPandora                        22.7783       63.5528       90.1813       65.6409       17.484         10     
reco:pandoraWriter:StandardPandora                 0.502781      0.675921      0.857449      0.679769      0.117038        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.73649       8.09161       13.4171       8.33357       2.4332         10     
reco:pandoraShower:LArPandoraShowerCreation         3.5545        9.73539       15.0649       9.82318       3.00282        10     
reco:pandoracalo:Calorimetry                        2.48923       4.56413       6.16357       4.62987       1.1199         10     
reco:pandoracalonosce:Calorimetry                   1.70897       4.10222       6.96345       4.14884       1.38214        10     
reco:pandorapid:Chi2ParticleID                    0.00610955    0.00891736     0.0118862     0.0089448    0.00181009       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0527592     0.0801723     0.106688      0.0770506     0.0205407       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00588302    0.00831196     0.0115179    0.00852689     0.0019147       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.02801       9.62845       14.3373       9.18373       3.11058        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.88922       9.21407       12.7997       9.91299       2.72296        10     
reco:emtrkmichelid:EmTrackMichelId                  224.01        320.597       370.401       321.498       42.1883        10     
reco:crttag:SingleCRTMatchingProducer              0.0175322     0.0256672     0.0375836     0.0232869    0.00642295       10     
reco:crtreco:TwoCRTMatchingProducer                0.0101872     0.015794      0.0220362     0.0163093    0.00366487       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.87963       1.18169       1.42926       1.23781      0.184529        10     
reco:pandora2Track:LArPandoraTrackCreation          12.3947       27.3496       39.0137       26.5877       7.00527        10     
reco:pandora2calo:Calorimetry                       5.46392       13.0846       19.5982       12.9868       4.22762        10     
reco:pandora2calonosce:Calorimetry                  4.97133       11.3044       16.0049       13.0332       3.59511        10     
reco:pandora2pid:Chi2ParticleID                    0.0157215     0.0234446     0.031579      0.0242635    0.00575872       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0714234     0.111649      0.144972      0.113595      0.0238149       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0160643     0.024721      0.0318838     0.0248143    0.00501742       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.48768       9.02829       12.2302       9.82241       2.35645        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.26212       9.78556       14.2983       10.0647       2.99456        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.06262       8.15161       13.0265       7.89379       2.94783        10     
[art]:TriggerResults:TriggerResultInserter        5.4102e-05    8.29194e-05   0.000234342   7.00675e-05   5.1029e-05       10     
end_path:out1:RootOutput                           9.678e-06    4.75868e-05   0.000366553   1.1957e-05    0.000106344      10     
end_path:out1:RootOutput(write)                     2.6334        4.26444       5.30794       4.40325      0.821323        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5234.19 MB
  Peak resident set size usage (VmHWM): 4318.94 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.30 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.31 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11346932
-rw-r--r-- 1 pltdune04 pltdune 2629174696 Dec 12 14:50 detsim_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 2491683429 Dec 12 14:46 detsim_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 2348515875 Dec 12 16:31 reco_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 2073515982 Dec 12 12:04 g4_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 1992225858 Dec 12 11:46 g4_stage1.root
-rw-r--r-- 1 pltdune04 pltdune   83522800 Dec 12 16:23 Pandora_Events.pndr
-rw-r--r-- 1 pltdune04 pltdune     243213 Dec 12 16:31 jobscript.log
-rw-r--r-- 1 pltdune04 pltdune     217088 Dec 12 16:31 mem.db
-rw-r--r-- 1 pltdune04 pltdune      62376 Dec 12 16:31 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune04 pltdune      36864 Dec 12 16:31 time.db
-rw-r--r-- 1 pltdune04 pltdune        519 Dec 12 14:50 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Dec 12 12:04 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Dec 12 16:31 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune04 pltdune         52 Dec 12 11:37 all-input-dids.txt
-rw-r--r-- 1 pltdune04 pltdune          7 Dec 12 16:31 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune04 pltdune          0 Dec 12 14:59 ProtoDUNE.log
-rw-r--r-- 1 pltdune04 pltdune          0 Dec 12 14:50 debugprod.log
justIN time: 2026-02-07 13:30:27 UTC       justIN version: 01.06.00