justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264372.1@dunegpschedd01.fnal.gov

Jobsub ID264372.1@dunegpschedd01.fnal.gov
Workflow ID11061
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-06 11:33:57
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-06 16:42:20
From worker nodeHostnamewn1206240.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-06 11:34:51
Input filesmonte-carlo-011061-000001
JobscriptExit code0
Real time5h (18354s)
CPU time4h (17535s = 95%)
Max RSS bytes4136841216 (3945 MiB)
Outputting started2025-12-06 16:40:46
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11061/1/001/reco_stage1.root
Finished2025-12-06 16:42:20
Saved logsjustin-logs:264372.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 90836 traces, 0 summary
Retagger: tagging trace set: gauss with 69015 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 69015 traces tagged "gauss"
FrameSaver: q=2.25413e+06 n=1208450 tag=gauss
wclsFrameSaver: saving 90836 traces tagged "wiener"
FrameSaver: q=2.38209e+06 n=1061342 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 14983 XUs and 19270 XVs -> 183 XUVs
C:0 T:1 28591 XUs and 30984 XVs -> 2002 XUVs
C:0 T:2 81933 XUs and 86896 XVs -> 3230 XUVs
C:0 T:3 45015 XUs and 39497 XVs -> 239 XUVs
C:0 T:4 11542 XUs and 11631 XVs -> 68 XUVs
C:0 T:5 40101 XUs and 30728 XVs -> 3282 XUVs
C:0 T:6 42654 XUs and 37425 XVs -> 2021 XUVs
C:0 T:7 18648 XUs and 15415 XVs -> 109 XUVs
C:0 T:8 6870 XUs and 8388 XVs -> 203 XUVs
C:0 T:9 56135 XUs and 79097 XVs -> 5696 XUVs
C:0 T:10 13105 XUs and 5333 XVs -> 1899 XUVs
C:0 T:11 251 XUs and 126 XVs -> 36 XUVs
18968 XUVs total
9096 collection wire objects
17701 potential space points
Neighbour search...
724315 tests to find 386156 neighbours
Iterating with no regularization...
Begin: 4.58146e+07
0 3.87737e+07
1 3.82908e+07
2 3.82233e+07
3 3.82054e+07
Now with regularization...
Begin: 3.1602e+07
0 3.15878e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410949912
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410949912
fLifetime: 30000 [us]
06-Dec-2025 16:40:44 GMT  Closed output file "reco_stage1.root"
06-Dec-2025 16:40:44 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          431.784       605.618       761.273       628.114       113.814        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0015586    0.00535675    0.00876364    0.00571878    0.00217733       10     
reco:ophit:OpHitFinder                             0.365652      0.418828      0.462462      0.430542      0.0334357       10     
reco:opflash:OpFlashFinder                         0.199448      0.240432      0.311631      0.224812      0.0342097       10     
reco:opslicer:OpSlicer                             0.872173       1.16219       1.46946       1.16222      0.191088        10     
reco:caldata:DataPrepModule                         8.45805       8.99022       9.6738        8.97658      0.378854        10     
reco:wclsdatasp:WireCellToolkit                     54.0798       60.4347       67.3971       60.4005       4.59731        10     
reco:gaushit:GausHitFinder                          1.59042       1.96454       2.5555        2.01561      0.293896        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000638971   0.000875635    0.0010588    0.000879004   0.000148093      10     
reco:reco3d:SpacePointSolver                        11.997        19.8176       40.5278       16.6519       8.46189        10     
reco:hitpdune:DisambigFromSpacePoints               1.60684       2.4496        3.49781       2.45007      0.668695        10     
reco:pandora:StandardPandora                        28.0736       47.6288        70.79        49.5197       14.4995        10     
reco:pandoraWriter:StandardPandora                 0.490416      0.704643      0.866991      0.710894      0.115748        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.72143       7.20066       9.4074        7.43676       1.4636         10     
reco:pandoraShower:LArPandoraShowerCreation         4.09179       7.81034       13.8088       7.98196       3.03036        10     
reco:pandoracalo:Calorimetry                        2.55544       4.21434       6.14851       3.73056       1.32515        10     
reco:pandoracalonosce:Calorimetry                   2.36253       3.83491       5.37899       3.54069       1.24186        10     
reco:pandorapid:Chi2ParticleID                     0.0083529     0.0102079     0.0116593     0.0102604     0.0010662       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.070129      0.100802      0.118144       0.10482      0.015982        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00808238    0.00965683     0.0113462    0.00984157    0.00111854       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.67789       8.37114       15.4398       8.7078        3.81401        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.61405       7.96274       14.0222       8.26462       3.53064        10     
reco:emtrkmichelid:EmTrackMichelId                  260.492       347.504       426.902       360.759       53.7056        10     
reco:crttag:SingleCRTMatchingProducer              0.0126696     0.0279773     0.0399288     0.027363      0.0076263       10     
reco:crtreco:TwoCRTMatchingProducer                0.010572      0.0181312     0.0232384     0.0188212    0.00453771       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.00297       1.40673       1.70526       1.4237       0.231945        10     
reco:pandora2Track:LArPandoraTrackCreation          12.1326       21.4661       32.9768       20.1426       7.27182        10     
reco:pandora2calo:Calorimetry                       5.79307       10.7868       18.8872       9.56564       4.35759        10     
reco:pandora2calonosce:Calorimetry                  4.99265       10.4167       17.1349       9.44099       4.37288        10     
reco:pandora2pid:Chi2ParticleID                    0.0182211     0.0229241     0.0304385     0.0214864    0.00394658       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0845221     0.126293      0.156845      0.128096      0.0208797       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0182035     0.0240488     0.0307298     0.0229335    0.00413675       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.26936       8.20755       11.9683       7.89695       2.72182        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.0315        9.37586       14.176        9.36005       3.38135        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.95442       8.22157       12.7803       8.16578       3.05966        10     
[art]:TriggerResults:TriggerResultInserter        3.7841e-05    5.95958e-05   0.000152296   5.19525e-05   3.13179e-05      10     
end_path:out1:RootOutput                           6.573e-06    8.3016e-06    1.3616e-05     8.065e-06    1.87245e-06      10     
end_path:out1:RootOutput(write)                     3.24703       4.48096       5.75362       4.5479        0.8016         10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4990.98 MB
  Peak resident set size usage (VmHWM): 4136.84 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.38 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.72 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9470076
-rw-r--r-- 1 dune01 dune01 2249091458 Dec  6 14:56 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2136457226 Dec  6 14:52 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1907551059 Dec  6 16:40 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1697415345 Dec  6 12:04 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1638115915 Dec  6 11:44 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   68093980 Dec  6 16:35 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242594 Dec  6 16:40 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  6 16:40 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  6 16:40 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  6 16:40 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  6 14:56 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  6 12:04 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  6 16:40 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  6 11:34 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  6 16:40 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  6 15:06 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  6 14:56 debugprod.log
justIN time: 2026-02-07 17:40:55 UTC       justIN version: 01.06.00