justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233495.0@dunegpschedd02.fnal.gov

Jobsub ID233495.0@dunegpschedd02.fnal.gov
Workflow ID9400
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-28 06:21:51
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-10-28 11:06:26
From worker nodeHostnamewn2209251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 06:23:01
Input filesmonte-carlo-009400-000001
JobscriptExit code0
Real time4h (16898s)
CPU time4h (16110s = 95%)
Max RSS bytes4412936192 (4208 MiB)
Outputting started2025-10-28 11:04:40
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09400/1/001/reco_stage1.root
Finished2025-10-28 11:06:26
Saved logsjustin-logs:233495.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

pAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 171443 traces, 0 summary
Retagger: tagging trace set: gauss with 125879 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 125879 traces tagged "gauss"
FrameSaver: q=4.16484e+06 n=2148602 tag=gauss
wclsFrameSaver: saving 171443 traces tagged "wiener"
FrameSaver: q=4.45494e+06 n=1874084 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 53366 XUs and 40836 XVs -> 576 XUVs
C:0 T:1 110558 XUs and 87211 XVs -> 5727 XUVs
C:0 T:2 105448 XUs and 112026 XVs -> 13022 XUVs
C:0 T:3 44735 XUs and 38683 XVs -> 473 XUVs
C:0 T:4 16416 XUs and 11354 XVs -> 246 XUVs
C:0 T:5 77065 XUs and 41498 XVs -> 5007 XUVs
C:0 T:6 181434 XUs and 121949 XVs -> 9797 XUVs
C:0 T:7 65706 XUs and 38278 XVs -> 661 XUVs
C:0 T:8 11470 XUs and 9697 XVs -> 153 XUVs
C:0 T:9 73410 XUs and 77590 XVs -> 5429 XUVs
C:0 T:10 83101 XUs and 41330 XVs -> 10326 XUVs
C:0 T:11 209 XUs and 140 XVs -> 48 XUVs
51465 XUVs total
15742 collection wire objects
46540 potential space points
Neighbour search...
3900116 tests to find 2057654 neighbours
Iterating with no regularization...
Begin: 7.53716e+07
0 6.14087e+07
1 6.03897e+07
2 6.02491e+07
3 6.02097e+07
Now with regularization...
Begin: 4.86263e+07
0 4.85903e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410161926
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410161926
fLifetime: 30000 [us]
28-Oct-2025 11:04:38 GMT  Closed output file "reco_stage1.root"
28-Oct-2025 11:04:38 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          382.686       552.003       839.014       532.09        130.263        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00152402    0.00581556    0.00844662    0.00570059    0.00214732       10     
reco:ophit:OpHitFinder                             0.339428      0.399233       0.52913      0.388834      0.0521543       10     
reco:opflash:OpFlashFinder                         0.173844      0.242383      0.390356      0.240347      0.0558886       10     
reco:opslicer:OpSlicer                             0.818935      0.948511       1.1347       0.936643      0.0869352       10     
reco:caldata:DataPrepModule                         6.3921        7.26148       8.37007       7.2169       0.516733        10     
reco:wclsdatasp:WireCellToolkit                     39.0716       45.7379       62.8737       44.5626       6.21926        10     
reco:gaushit:GausHitFinder                          1.51507       1.9728        2.40247       1.99675      0.273684        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000630817   0.00104023    0.00156773    0.000989797   0.000330383      10     
reco:reco3d:SpacePointSolver                        16.9882       24.3323       34.8131       22.9601       5.30835        10     
reco:hitpdune:DisambigFromSpacePoints               1.68825       2.57294       3.94391       2.34212      0.706627        10     
reco:pandora:StandardPandora                        29.423        64.4755       131.063       56.1753       30.9437        10     
reco:pandoraWriter:StandardPandora                 0.433363       0.71985       1.12988      0.687322      0.198384        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.29514       9.50852       15.5043       9.49246       3.4841         10     
reco:pandoraShower:LArPandoraShowerCreation         5.70424       11.9967       22.3939       11.2804       4.80947        10     
reco:pandoracalo:Calorimetry                        2.97458       5.63893       9.4735        5.57876       1.91914        10     
reco:pandoracalonosce:Calorimetry                   2.46089       5.09846       8.17979       5.35159       1.59489        10     
reco:pandorapid:Chi2ParticleID                    0.00709928    0.00940484     0.0146847    0.00874072    0.00212169       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.055653      0.0839723      0.11536      0.0835082     0.0184244       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00602378    0.00807882     0.0104736    0.00820161    0.00125648       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.02056       11.8633       20.6846       11.9283       4.3638         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.86741       11.922        22.0976       11.2005       4.68099        10     
reco:emtrkmichelid:EmTrackMichelId                  193.971       248.211       329.971       248.605       40.1372        10     
reco:crttag:SingleCRTMatchingProducer              0.0181096     0.027896      0.042746      0.0264316    0.00772576       10     
reco:crtreco:TwoCRTMatchingProducer                 0.0136       0.0208694     0.033343      0.0207253    0.00557586       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.703265       1.08182       1.43342       1.1027       0.206319        10     
reco:pandora2Track:LArPandoraTrackCreation          16.1843       29.4106       49.7572       25.8627       9.99607        10     
reco:pandora2calo:Calorimetry                       7.96861       16.119        28.5736       14.2141       5.90696        10     
reco:pandora2calonosce:Calorimetry                  7.22489       15.0105       24.7247       14.4509       4.99895        10     
reco:pandora2pid:Chi2ParticleID                    0.0131815     0.0213287     0.0302745     0.0212369    0.00600047       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0596338     0.101653       0.13275      0.110859      0.0259809       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0111155     0.0194237     0.0263916     0.0197518    0.00508661       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.6005        10.1285       16.6955       9.66032       3.18815        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.42624       12.5141       20.4337       12.1456       3.79876        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.94379       10.3584       16.3142       10.4165       2.7475         10     
[art]:TriggerResults:TriggerResultInserter        6.7517e-05    0.00010798    0.00022318    9.6251e-05    4.02793e-05      10     
end_path:out1:RootOutput                          1.0871e-05    1.40675e-05   2.4365e-05     1.316e-05    3.76723e-06      10     
end_path:out1:RootOutput(write)                     3.00699       3.93291       5.37929       3.99757      0.731566        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5252.82 MB
  Peak resident set size usage (VmHWM): 4412.94 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.54 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10993104
-rw-r--r-- 1 dune01 dune01 2556627238 Oct 28 09:29 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2429009571 Oct 28 09:26 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2268703809 Oct 28 11:04 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1991528255 Oct 28 06:51 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1928900546 Oct 28 06:32 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   81544780 Oct 28 10:56 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242864 Oct 28 11:04 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Oct 28 11:04 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Oct 28 11:04 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Oct 28 11:04 time.db
-rw-r--r-- 1 dune01 dune01        519 Oct 28 09:29 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Oct 28 06:51 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Oct 28 11:04 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Oct 28 06:23 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Oct 28 11:04 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Oct 28 09:42 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Oct 28 09:29 debugprod.log
justIN time: 2025-11-03 18:41:10 UTC       justIN version: 01.05.01