justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239341.1@dunegpschedd02.fnal.gov

Jobsub ID239341.1@dunegpschedd02.fnal.gov
Workflow ID9850
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-07 05:13:29
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-07 10:16:43
From worker nodeHostnamewn2204251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-07 05:14:46
Input filesmonte-carlo-009850-000001
JobscriptExit code0
Real time5h (18011s)
CPU time4h (17458s = 96%)
Max RSS bytes4797034496 (4574 MiB)
Outputting started2025-11-07 10:14:58
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09850/1/001/reco_stage1.root
Finished2025-11-07 10:16:43
Saved logsjustin-logs:239341.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 180433 traces, 0 summary
Retagger: tagging trace set: gauss with 139341 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 139341 traces tagged "gauss"
FrameSaver: q=5.02145e+06 n=2501724 tag=gauss
wclsFrameSaver: saving 180433 traces tagged "wiener"
FrameSaver: q=5.31877e+06 n=2235001 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 11077 XUs and 15598 XVs -> 54 XUVs
C:0 T:1 16978 XUs and 19552 XVs -> 1506 XUVs
C:0 T:2 135156 XUs and 208021 XVs -> 11085 XUVs
C:0 T:3 53584 XUs and 64388 XVs -> 663 XUVs
C:0 T:4 16530 XUs and 20286 XVs -> 208 XUVs
C:0 T:5 93025 XUs and 115614 XVs -> 16506 XUVs
C:0 T:6 98937 XUs and 110929 XVs -> 5064 XUVs
C:0 T:7 40024 XUs and 40946 XVs -> 373 XUVs
C:0 T:8 17268 XUs and 19485 XVs -> 343 XUVs
C:0 T:9 161548 XUs and 301573 XVs -> 22165 XUVs
C:0 T:10 81799 XUs and 59920 XVs -> 6062 XUVs
C:0 T:11 349 XUs and 246 XVs -> 21 XUVs
64050 XUVs total
17203 collection wire objects
59540 potential space points
Neighbour search...
8655056 tests to find 3860634 neighbours
Iterating with no regularization...
Begin: 1.10749e+08
0 9.12985e+07
1 9.00007e+07
2 8.98092e+07
3 8.97547e+07
Now with regularization...
Begin: 7.45685e+07
0 7.44814e+07
1 7.44257e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410352503
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410352503
fLifetime: 30000 [us]
07-Nov-2025 10:14:55 GMT  Closed output file "reco_stage1.root"
07-Nov-2025 10:14:55 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          372.068       630.958       880.825        639.6        169.204        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00227333    0.00681134     0.0110804    0.00659357    0.00260238       10     
reco:ophit:OpHitFinder                             0.294987      0.440687      0.549408      0.446735      0.0705233       10     
reco:opflash:OpFlashFinder                         0.203646      0.284255      0.400121      0.272463      0.0618869       10     
reco:opslicer:OpSlicer                             0.698065       1.02803       1.28316       1.02966      0.187176        10     
reco:caldata:DataPrepModule                         4.70645       7.56508       8.44667       7.79451      0.992034        10     
reco:wclsdatasp:WireCellToolkit                     43.7633       50.051        62.623        49.2862       5.53006        10     
reco:gaushit:GausHitFinder                          1.16073       1.9747        2.59977       2.13604      0.478499        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000615276   0.00108595    0.00161566    0.00102966    0.000309461      10     
reco:reco3d:SpacePointSolver                        12.461        28.4906       41.9533       30.9572       10.143         10     
reco:hitpdune:DisambigFromSpacePoints               1.54338       2.96233       4.46735       3.06172       1.09168        10     
reco:pandora:StandardPandora                        29.3979       68.0537       127.67        63.6774       35.4534        10     
reco:pandoraWriter:StandardPandora                 0.521979      0.746191       1.0262       0.701068      0.169716        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.49881       10.3285       15.819        10.2087       3.78142        10     
reco:pandoraShower:LArPandoraShowerCreation         4.13261       13.7413       21.2594       14.1351       6.0041         10     
reco:pandoracalo:Calorimetry                        2.56538       6.04128       8.72763       6.43613       2.12225        10     
reco:pandoracalonosce:Calorimetry                   2.30785       5.37569       8.43848       5.5781        1.87396        10     
reco:pandorapid:Chi2ParticleID                    0.00643388    0.00843829     0.0106411    0.00794159    0.00151086       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0589673     0.0786575     0.101891      0.0772345     0.0152569       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00512943    0.00834903     0.0106563    0.00871186    0.00175814       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.88385       13.5585       22.4873       14.9094       6.07637        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.01792       13.7572       21.8397       15.2706       6.17411        10     
reco:emtrkmichelid:EmTrackMichelId                   193.4        285.487       370.73        290.417       56.4193        10     
reco:crttag:SingleCRTMatchingProducer              0.0180115     0.0321822     0.0593072     0.0285681     0.0118443       10     
reco:crtreco:TwoCRTMatchingProducer                0.0103705     0.0235928     0.0471832     0.0217843     0.0101685       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.951287       1.30267       1.94976       1.26348      0.267014        10     
reco:pandora2Track:LArPandoraTrackCreation          15.1698       36.6397       61.6857       34.4795       14.5721        10     
reco:pandora2calo:Calorimetry                       7.14006       19.5433       32.1969       17.8674       8.16715        10     
reco:pandora2calonosce:Calorimetry                  6.70235       18.6404       30.6461       17.4508       7.83687        10     
reco:pandora2pid:Chi2ParticleID                    0.016746      0.0246498     0.0372345     0.0232864     0.006349        10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.082716      0.125081       0.16834      0.117982      0.0306725       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0161728     0.0235276     0.0337455     0.0227354    0.00550577       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.19068       12.5732       21.2214       11.5966       4.68488        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.88287       14.3427       25.2229       12.3525       5.99594        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.21603       13.0516       20.5503        12.72        4.94876        10     
[art]:TriggerResults:TriggerResultInserter         5.335e-05    0.000108758   0.000266782   9.14465e-05   5.76114e-05      10     
end_path:out1:RootOutput                          1.1832e-05    1.95899e-05   4.2951e-05    1.8029e-05    8.33851e-06      10     
end_path:out1:RootOutput(write)                     3.04967       4.39563       5.50338       4.63561      0.891292        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5638.87 MB
  Peak resident set size usage (VmHWM): 4797.03 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.36 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.72 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 12025736
-rw-r--r-- 1 dune01 dune01 2754807488 Nov  7 08:26 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2616333046 Nov  7 08:23 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2485830861 Nov  7 10:14 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2222208495 Nov  7 05:46 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 2148957768 Nov  7 05:24 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   85584750 Nov  7 10:06 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242911 Nov  7 10:14 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  7 10:14 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  7 10:14 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  7 10:14 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  7 08:26 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  7 05:46 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  7 10:14 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  7 05:14 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  7 10:14 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  7 08:36 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  7 08:26 debugprod.log
justIN time: 2026-02-10 13:29:03 UTC       justIN version: 01.06.00