justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271355.1@dunegpschedd01.fnal.gov

Jobsub ID271355.1@dunegpschedd01.fnal.gov
Workflow ID11476
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-02 19:14:47
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-01-02 23:07:10
From worker nodeHostnameheplnc159.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-02 19:15:41
Input filesmonte-carlo-011476-000001
JobscriptExit code0
Real time3h (13804s)
CPU time3h (13229s = 95%)
Max RSS bytes4064673792 (3876 MiB)
Outputting started2026-01-02 23:05:45
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11476/1/001/reco_stage1.root
Finished2026-01-02 23:07:10
Saved logsjustin-logs:271355.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 144822 traces, 0 summary
Retagger: tagging trace set: gauss with 104263 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 104263 traces tagged "gauss"
FrameSaver: q=3.4869e+06 n=1731284 tag=gauss
wclsFrameSaver: saving 144822 traces tagged "wiener"
FrameSaver: q=3.73004e+06 n=1492303 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 42211 XUs and 29463 XVs -> 715 XUVs
C:0 T:1 127645 XUs and 76996 XVs -> 9684 XUVs
C:0 T:2 117797 XUs and 123186 XVs -> 11085 XUVs
C:0 T:3 46378 XUs and 45311 XVs -> 665 XUVs
C:0 T:5 59068 XUs and 39919 XVs -> 5429 XUVs
C:0 T:6 96976 XUs and 98183 XVs -> 5365 XUVs
C:0 T:7 38587 XUs and 35315 XVs -> 340 XUVs
C:0 T:8 6299 XUs and 9329 XVs -> 118 XUVs
C:0 T:9 50510 XUs and 75214 XVs -> 9019 XUVs
C:0 T:10 38815 XUs and 114288 XVs -> 15824 XUVs
C:0 T:11 289 XUs and 261 XVs -> 28 XUVs
58272 XUVs total
12657 collection wire objects
53172 potential space points
Neighbour search...
9357626 tests to find 4285636 neighbours
Iterating with no regularization...
Begin: 6.03317e+07
0 5.00963e+07
1 4.93828e+07
2 4.92661e+07
3 4.92276e+07
Now with regularization...
Begin: 3.98234e+07
0 3.9769e+07
1 3.97271e+07
2 3.96933e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411500438
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411500438
fLifetime: 30000 [us]
02-Jan-2026 23:05:43 GMT  Closed output file "reco_stage1.root"
02-Jan-2026 23:05:43 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          270.014       418.791       514.582        446.2        76.198         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000895966   0.00407087    0.00703045    0.00401185    0.00184606       10     
reco:ophit:OpHitFinder                             0.366171      0.401118      0.481403      0.390914      0.0309242       10     
reco:opflash:OpFlashFinder                          0.17232      0.210801      0.289301      0.196264      0.0339087       10     
reco:opslicer:OpSlicer                             0.734577      0.961903       1.13694       0.96873      0.121128        10     
reco:caldata:DataPrepModule                         7.43499       7.79935       8.03592       7.8171       0.161552        10     
reco:wclsdatasp:WireCellToolkit                     38.099        41.8887       44.3306       42.7841       2.13641        10     
reco:gaushit:GausHitFinder                          1.08562       1.63948       2.09641       1.69133      0.331977        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000482538   0.000680541   0.000808522   0.000710808   0.000105519      10     
reco:reco3d:SpacePointSolver                        7.85713       23.7072       37.7329       22.8209       8.77993        10     
reco:hitpdune:DisambigFromSpacePoints               1.19022       2.20134       3.02873       2.30458      0.561483        10     
reco:pandora:StandardPandora                        18.5356       41.2383       73.9289       41.1346       15.757         10     
reco:pandoraWriter:StandardPandora                 0.417521      0.604409      0.749772      0.642524      0.0984791       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.5875        4.67222       6.2522        4.81309       1.06567        10     
reco:pandoraShower:LArPandoraShowerCreation         2.18687       5.07772       7.5287        5.49553       1.57227        10     
reco:pandoracalo:Calorimetry                        1.56096       2.68028       4.07127       2.71157      0.696102        10     
reco:pandoracalonosce:Calorimetry                   1.39051       2.46125       3.44769       2.6232       0.622816        10     
reco:pandorapid:Chi2ParticleID                    0.00469054    0.00802872    0.00984276    0.00796252    0.00133738       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0450015     0.0742029     0.0945477     0.0786373     0.0152513       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00480067    0.00752626    0.00906183    0.00770494    0.00128879       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.15661       5.35219       8.75642       5.71774       1.97649        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.90985       5.33582       8.75031       5.98415       2.1484         10     
reco:emtrkmichelid:EmTrackMichelId                  154.195       220.526       269.654       227.092       34.9963        10     
reco:crttag:SingleCRTMatchingProducer              0.0107326     0.017254      0.0224373     0.0178442    0.00297104       10     
reco:crtreco:TwoCRTMatchingProducer                0.0084051     0.0104796     0.013272      0.0107105     0.0015501       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.750163       1.07151       1.46529       1.13707      0.225899        10     
reco:pandora2Track:LArPandoraTrackCreation          6.19664       14.9459       22.123        15.6548       4.49664        10     
reco:pandora2calo:Calorimetry                       3.24641       7.84028       12.3036       7.90319       2.62021        10     
reco:pandora2calonosce:Calorimetry                  2.93276       6.92794       9.79244       7.64873       2.02953        10     
reco:pandora2pid:Chi2ParticleID                   0.00924465     0.0163062     0.0220935     0.017435       0.00442        10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0475747     0.0787802      0.1118       0.0822587     0.0207841       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00934211     0.014792      0.0203354     0.0145485    0.00384571       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.64001       5.02587       6.75897       5.44813       1.46045        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.38488       6.60173       9.37708        7.09         2.0327         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.61054       5.74333       7.51757       6.21231       1.50102        10     
[art]:TriggerResults:TriggerResultInserter        5.5324e-05    8.97029e-05   0.000203163   8.09075e-05   3.92113e-05      10     
end_path:out1:RootOutput                          1.1281e-05    1.42759e-05   2.0869e-05    1.3981e-05    2.43579e-06      10     
end_path:out1:RootOutput(write)                     2.44962       3.50023       4.33973       3.72054       0.65637        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4917.68 MB
  Peak resident set size usage (VmHWM): 4064.67 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.36 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.91 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9994708
-rw-r--r-- 1 pltdune04 pltdune 2343574788 Jan  2 21:53 detsim_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 2220724757 Jan  2 21:51 detsim_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 2025254424 Jan  2 23:05 reco_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 1830584642 Jan  2 19:38 g4_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 1740314117 Jan  2 19:23 g4_stage1.root
-rw-r--r-- 1 pltdune04 pltdune   73511920 Jan  2 23:00 Pandora_Events.pndr
-rw-r--r-- 1 pltdune04 pltdune     243162 Jan  2 23:05 jobscript.log
-rw-r--r-- 1 pltdune04 pltdune     217088 Jan  2 23:05 mem.db
-rw-r--r-- 1 pltdune04 pltdune      62376 Jan  2 23:05 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune04 pltdune      36864 Jan  2 23:05 time.db
-rw-r--r-- 1 pltdune04 pltdune        519 Jan  2 21:53 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Jan  2 19:38 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Jan  2 23:05 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune04 pltdune         52 Jan  2 19:15 all-input-dids.txt
-rw-r--r-- 1 pltdune04 pltdune          7 Jan  2 23:05 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune04 pltdune          0 Jan  2 21:58 ProtoDUNE.log
-rw-r--r-- 1 pltdune04 pltdune          0 Jan  2 21:53 debugprod.log
justIN time: 2026-02-06 22:22:29 UTC       justIN version: 01.06.00