justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296363.0@dunegpschedd02.fnal.gov

Jobsub ID296363.0@dunegpschedd02.fnal.gov
Workflow ID12493
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-28 16:18:52
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-28 22:06:30
From worker nodeHostnamewn5917291.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-28 16:19:43
Input filesmonte-carlo-012493-000001
JobscriptExit code0
Real time5h (20700s)
CPU time5h (19947s = 96%)
Max RSS bytes4193894400 (3999 MiB)
Outputting started2026-01-28 22:04:44
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12493/1/001/reco_stage1.root
Finished2026-01-28 22:06:30
Saved logsjustin-logs:296363.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 169212 traces, 0 summary
Retagger: tagging trace set: gauss with 120356 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 120356 traces tagged "gauss"
FrameSaver: q=4.21198e+06 n=2014819 tag=gauss
wclsFrameSaver: saving 169212 traces tagged "wiener"
FrameSaver: q=4.51086e+06 n=1744752 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 35167 XUs and 33947 XVs -> 394 XUVs
C:0 T:1 68584 XUs and 64328 XVs -> 5441 XUVs
C:0 T:2 134889 XUs and 293695 XVs -> 23690 XUVs
C:0 T:3 53845 XUs and 69265 XVs -> 802 XUVs
C:0 T:4 15573 XUs and 15967 XVs -> 161 XUVs
C:0 T:5 76436 XUs and 58577 XVs -> 5169 XUVs
C:0 T:6 114649 XUs and 170914 XVs -> 11363 XUVs
C:0 T:7 34712 XUs and 48202 XVs -> 583 XUVs
C:0 T:8 7026 XUs and 6956 XVs -> 131 XUVs
C:0 T:9 46469 XUs and 59622 XVs -> 5529 XUVs
C:0 T:10 76846 XUs and 198633 XVs -> 30689 XUVs
C:0 T:11 240 XUs and 416 XVs -> 41 XUVs
83993 XUVs total
15662 collection wire objects
78173 potential space points
Neighbour search...
21039145 tests to find 8208698 neighbours
Iterating with no regularization...
Begin: 1.13995e+08
0 8.35788e+07
1 8.16768e+07
2 8.14696e+07
3 8.14051e+07
Now with regularization...
Begin: 6.89502e+07
0 6.88591e+07
1 6.88085e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 412020974
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412020974
fLifetime: 30000 [us]
28-Jan-2026 22:04:41 GMT  Closed output file "reco_stage1.root"
28-Jan-2026 22:04:41 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          604.019       743.522       820.42        770.964       64.4157        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0016011    0.00540847    0.00816661    0.00543573    0.00217188       10     
reco:ophit:OpHitFinder                             0.412539      0.469713      0.532464      0.465201      0.0361101       10     
reco:opflash:OpFlashFinder                          0.22222      0.268847      0.340828      0.259808      0.0403646       10     
reco:opslicer:OpSlicer                              1.02503       1.33943       1.67885       1.2929       0.200622        10     
reco:caldata:DataPrepModule                         6.77697       8.88518       9.74128       9.06313      0.751413        10     
reco:wclsdatasp:WireCellToolkit                     52.3772       61.9014       66.5444       63.5433       4.42588        10     
reco:gaushit:GausHitFinder                          1.8094        2.24568       2.57868       2.31588      0.207931        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000832057   0.00101244    0.00116449     0.0010049    0.000112751      10     
reco:reco3d:SpacePointSolver                        15.9264       29.3792       47.8912       29.2452       7.80878        10     
reco:hitpdune:DisambigFromSpacePoints               2.58523       3.20201       3.62913       3.21259      0.301172        10     
reco:pandora:StandardPandora                        56.9611        76.09        101.059       76.0387       13.7362        10     
reco:pandoraWriter:StandardPandora                  0.69777      0.841083      0.931768      0.835084      0.0734593       10     
reco:pandoraTrack:LArPandoraTrackCreation           7.43215       9.39127       13.0549        9.345        1.53813        10     
reco:pandoraShower:LArPandoraShowerCreation         8.24824       11.147        14.2158       11.4877       2.11934        10     
reco:pandoracalo:Calorimetry                        3.6691        5.00524       6.22457       5.07927      0.782915        10     
reco:pandoracalonosce:Calorimetry                   3.41075       4.70083       5.62566       4.90382       0.72079        10     
reco:pandorapid:Chi2ParticleID                    0.00919029     0.010929      0.0141908     0.0106014    0.00142807       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0644035     0.100407      0.112451      0.103682      0.013322        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00708303     0.0102984     0.0135633     0.010307     0.00158247       10     
reco:pandoraShowercalo:ShowerCalorimetry            8.36784       11.3845       14.7214       11.9636       2.50247        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.98581       10.6997       13.6475       11.0657       2.05867        10     
reco:emtrkmichelid:EmTrackMichelId                  341.923       400.314       429.043       402.291       25.1796        10     
reco:crttag:SingleCRTMatchingProducer              0.0141738     0.0232175     0.0325018     0.0227318    0.00516643       10     
reco:crtreco:TwoCRTMatchingProducer                0.0104449     0.0185914     0.0242735     0.0190854    0.00368042       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.997629       1.51538       1.78814       1.56065      0.215876        10     
reco:pandora2Track:LArPandoraTrackCreation          20.9487       31.1636       38.9204       32.4675       5.75044        10     
reco:pandora2calo:Calorimetry                       10.8577       16.5222       20.3386       18.2927       3.36615        10     
reco:pandora2calonosce:Calorimetry                  10.5024       15.8899       20.5659       16.8442       3.54711        10     
reco:pandora2pid:Chi2ParticleID                    0.0253273     0.029773      0.0361499     0.0287032    0.00310041       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.125634      0.146258       0.19503      0.144529      0.0183448       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0248782     0.0293089     0.0367878     0.0286174    0.00316524       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.85766       11.0121       14.7306       10.9779       2.30343        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.64032       12.8575       16.6138       14.0073       2.59522        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.83438       11.4605       15.0701       11.9003       2.92634        10     
[art]:TriggerResults:TriggerResultInserter        4.1919e-05    6.07973e-05   0.00015439    4.4895e-05    3.41538e-05      10     
end_path:out1:RootOutput                           6.001e-06    7.8379e-06    1.3336e-05     7.269e-06    1.93594e-06      10     
end_path:out1:RootOutput(write)                     3.78671       5.23898       5.96641       5.26553      0.573459        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5050.73 MB
  Peak resident set size usage (VmHWM): 4193.89 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.65 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.23 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10386628
-rw-r--r-- 1 dune01 dune01 2435618947 Jan 28 19:57 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2309378375 Jan 28 19:53 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2142192481 Jan 28 22:04 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1874520042 Jan 28 16:51 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1793638458 Jan 28 16:29 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   79922280 Jan 28 21:54 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242892 Jan 28 22:04 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 28 22:04 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 28 22:04 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 28 22:04 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 28 19:57 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 28 16:51 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 28 22:04 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 28 16:19 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 28 22:04 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 28 20:08 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 28 19:57 debugprod.log
justIN time: 2026-02-04 16:37:59 UTC       justIN version: 01.06.00