justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20577.0@dunegpschedd01.fnal.gov

Jobsub ID20577.0@dunegpschedd01.fnal.gov
Workflow ID265
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-03 07:30:14
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-08-03 11:54:04
From worker nodeHostnamewn2204241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-03 07:31:16
Input filesmonte-carlo-000265-000001
JobscriptExit code0
Real time4h (15677s)
CPU time3h (14090s = 89%)
Max RSS bytes4037513216 (3850 MiB)
Outputting started2025-08-03 11:52:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00265/1/001/reco_stage1.root
Finished2025-08-03 11:54:04
Saved logsjustin-logs:20577.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

unning tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 101307 traces, 0 summary
Retagger: tagging trace set: gauss with 76656 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76656 traces tagged "gauss"
FrameSaver: q=2.47672e+06 n=1312695 tag=gauss
wclsFrameSaver: saving 101307 traces tagged "wiener"
FrameSaver: q=2.6171e+06 n=1148490 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 42527 XUs and 28327 XVs -> 479 XUVs
C:0 T:1 94763 XUs and 64485 XVs -> 5912 XUVs
C:0 T:2 42171 XUs and 66697 XVs -> 1873 XUVs
C:0 T:3 25342 XUs and 32734 XVs -> 172 XUVs
C:0 T:4 11015 XUs and 8216 XVs -> 41 XUVs
C:0 T:5 39492 XUs and 22645 XVs -> 4484 XUVs
C:0 T:6 71923 XUs and 71395 XVs -> 3979 XUVs
C:0 T:7 32945 XUs and 30991 XVs -> 243 XUVs
C:0 T:8 5112 XUs and 3436 XVs -> 41 XUVs
C:0 T:9 25235 XUs and 24825 XVs -> 1905 XUVs
C:0 T:10 18517 XUs and 12080 XVs -> 2449 XUVs
C:0 T:11 132 XUs and 61 XVs -> 20 XUVs
21598 XUVs total
8958 collection wire objects
19844 potential space points
Neighbour search...
1170654 tests to find 610234 neighbours
Iterating with no regularization...
Begin: 4.46974e+07
0 3.83481e+07
1 3.79009e+07
2 3.78287e+07
3 3.78068e+07
Now with regularization...
Begin: 3.10556e+07
0 3.10421e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408432848
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408432848
fLifetime: 30000 [us]
03-Aug-2025 12:52:32 BST  Closed output file "reco_stage1.root"
03-Aug-2025 12:52:32 BST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          280.618       405.268       450.558       424.742       48.9526        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00227783    0.00419208    0.00695339    0.00427609    0.00141031       10     
reco:ophit:OpHitFinder                             0.237065      0.340335      0.432702      0.364834      0.0661382       10     
reco:opflash:OpFlashFinder                         0.142764      0.191735      0.287981      0.172887      0.0466958       10     
reco:opslicer:OpSlicer                             0.465243      0.796377      0.991754      0.860294      0.176793        10     
reco:caldata:DataPrepModule                         4.9526        6.97066       9.27797       7.00439       1.40512        10     
reco:wclsdatasp:WireCellToolkit                     33.9624       42.1805       60.1224       40.2204       7.04924        10     
reco:gaushit:GausHitFinder                          1.14572       1.54342       1.95804       1.60095      0.248786        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00044804    0.000761858   0.00180149    0.000696346   0.000360828      10     
reco:reco3d:SpacePointSolver                        11.1681       22.9423       35.8917       24.4783       7.50942        10     
reco:hitpdune:DisambigFromSpacePoints               1.30236       1.8447        2.59454       1.81275       0.38433        10     
reco:pandora:StandardPandora                        18.4645       34.7894       42.7905       37.0499       6.76869        10     
reco:pandoraWriter:StandardPandora                 0.358069      0.561134      0.666826       0.61688      0.106912        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.67196       4.86954       5.92965       4.95037      0.995221        10     
reco:pandoraShower:LArPandoraShowerCreation         2.25156       5.15788       6.61555        5.412        1.2234         10     
reco:pandoracalo:Calorimetry                        1.64015       2.99887       3.76631       3.16504      0.665589        10     
reco:pandoracalonosce:Calorimetry                   1.36551       2.82581       3.62116       2.99999      0.686429        10     
reco:pandorapid:Chi2ParticleID                    0.00626457    0.00835745     0.0108434     0.0084041    0.00114665       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0570338     0.0785036     0.0945433     0.0795059     0.0100494       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00539743    0.00755853    0.00902268    0.00779326    0.00103549       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.21585       5.67703       7.65308       6.01788       1.54174        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.22126       5.38209       7.24058       5.93111       1.42434        10     
reco:emtrkmichelid:EmTrackMichelId                  168.381       217.052       242.486       229.375       25.7326        10     
reco:crttag:SingleCRTMatchingProducer             0.00979142     0.0175034     0.0283395     0.0164868    0.00477965       10     
reco:crtreco:TwoCRTMatchingProducer               0.00593144     0.0102714     0.0136767     0.0110001    0.00256145       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.573436      0.977733       1.27581       1.07777      0.233943        10     
reco:pandora2Track:LArPandoraTrackCreation          7.18933       13.8135       18.8269       14.5191       3.35085        10     
reco:pandora2calo:Calorimetry                       3.60691       7.43102       10.118        7.94861       1.92785        10     
reco:pandora2calonosce:Calorimetry                  3.20672       6.94772       10.037        6.95805       2.18569        10     
reco:pandora2pid:Chi2ParticleID                    0.0082652     0.0170035     0.0213809     0.0185737     0.0042894       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0464077     0.0829888     0.128784      0.0900014     0.0249203       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00815366     0.0150911     0.0215001     0.0148105    0.00473482       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.63193       4.49546       6.38719       4.54296       1.25266        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.88199       5.82563       8.26689       6.02464       1.86215        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       2.876        5.71622       7.81519       6.03243       1.54848        10     
[art]:TriggerResults:TriggerResultInserter        5.5564e-05    0.000102681   0.000199855   9.27995e-05   3.63549e-05      10     
end_path:out1:RootOutput                          1.2093e-05    1.68159e-05   2.1781e-05    1.6977e-05    2.72588e-06      10     
end_path:out1:RootOutput(write)                     2.60314       3.55421       4.08231       3.68017      0.437768        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4868.18 MB
  Peak resident set size usage (VmHWM): 4037.51 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.31 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9695308
-rw-r--r-- 1 dune01 dune01 2295512245 Aug  3 11:42 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2184123653 Aug  3 11:40 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1968562864 Aug  3 12:52 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1735086213 Aug  3 09:19 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1672702568 Aug  3 08:41 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   71372860 Aug  3 12:49 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242999 Aug  3 12:52 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Aug  3 12:52 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Aug  3 12:52 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Aug  3 12:52 time.db
-rw-r--r-- 1 dune01 dune01        519 Aug  3 11:42 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Aug  3 09:19 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Aug  3 12:52 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Aug  3 08:31 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Aug  3 12:52 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Aug  3 11:49 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Aug  3 11:42 debugprod.log
justIN time: 2025-08-04 14:16:58 UTC       justIN version: 01.04.00