justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 302183.0@dunegpschedd01.fnal.gov

Jobsub ID302183.0@dunegpschedd01.fnal.gov
Workflow ID12542
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-29 21:26:33
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-30 02:52:53
From worker nodeHostnamewn1208091.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-29 21:42:05
Input filesmonte-carlo-012542-000001
JobscriptExit code0
Real time5h (18548s)
CPU time4h (17905s = 96%)
Max RSS bytes4113674240 (3923 MiB)
Outputting started2026-01-30 02:51:15
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12542/1/001/reco_stage1.root
Finished2026-01-30 02:52:53
Saved logsjustin-logs:302183.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

KeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 114570 traces, 0 summary
Retagger: tagging trace set: gauss with 84677 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84677 traces tagged "gauss"
FrameSaver: q=3.08593e+06 n=1521100 tag=gauss
wclsFrameSaver: saving 114570 traces tagged "wiener"
FrameSaver: q=3.28123e+06 n=1343411 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21584 XUs and 23165 XVs -> 130 XUVs
C:0 T:1 40651 XUs and 35035 XVs -> 2766 XUVs
C:0 T:2 205194 XUs and 183186 XVs -> 13613 XUVs
C:0 T:3 57794 XUs and 63748 XVs -> 404 XUVs
C:0 T:4 3258 XUs and 2661 XVs -> 26 XUVs
C:0 T:5 19200 XUs and 7659 XVs -> 1308 XUVs
C:0 T:6 126540 XUs and 178426 XVs -> 7460 XUVs
C:0 T:7 46715 XUs and 64525 XVs -> 453 XUVs
C:0 T:8 5427 XUs and 5414 XVs -> 97 XUVs
C:0 T:9 27585 XUs and 31771 XVs -> 2533 XUVs
C:0 T:10 44628 XUs and 40050 XVs -> 6206 XUVs
C:0 T:11 49 XUs and 30 XVs -> 4 XUVs
35000 XUVs total
10480 collection wire objects
31997 potential space points
Neighbour search...
2433849 tests to find 1257568 neighbours
Iterating with no regularization...
Begin: 7.0059e+07
0 5.16671e+07
1 5.06995e+07
2 5.0553e+07
3 5.0504e+07
Now with regularization...
Begin: 4.19107e+07
0 4.18575e+07
1 4.18333e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412045458
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412045458
fLifetime: 30000 [us]
30-Jan-2026 02:51:11 GMT  Closed output file "reco_stage1.root"
30-Jan-2026 02:51:11 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          409.278       681.711       1906.17       561.325       416.132        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00133524    0.00479585    0.00829766    0.00515677     0.001901        10     
reco:ophit:OpHitFinder                             0.275822      0.384832      0.489947      0.373992      0.0716035       10     
reco:opflash:OpFlashFinder                         0.165029      0.212896      0.313682      0.193452      0.0502564       10     
reco:opslicer:OpSlicer                             0.677916       0.99778       1.5183       0.942381      0.248998        10     
reco:caldata:DataPrepModule                         5.54766       7.9171        9.45229       8.21587       1.0424         10     
reco:wclsdatasp:WireCellToolkit                     47.914        55.5906       72.9868       55.3853       6.68711        10     
reco:gaushit:GausHitFinder                          1.43455       1.8508        2.70832       1.80533      0.406345        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000632131   0.000844313   0.00133521    0.000823592   0.000189785      10     
reco:reco3d:SpacePointSolver                        10.1389        26.69        74.6093       22.5548       17.5168        10     
reco:hitpdune:DisambigFromSpacePoints               1.4344        2.48308       4.84944       2.52117      0.922611        10     
reco:pandora:StandardPandora                        23.4852       139.172       970.767       43.3853       277.647        10     
reco:pandoraWriter:StandardPandora                 0.388839       0.64132       1.1701       0.640368      0.210363        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.8406        7.4055        20.8475       6.33215       4.68814        10     
reco:pandoraShower:LArPandoraShowerCreation         4.05746       7.78673       18.9201       6.50625       4.07534        10     
reco:pandoracalo:Calorimetry                        2.12271       4.06951       10.6957       3.58417       2.3014         10     
reco:pandoracalonosce:Calorimetry                   1.77917       3.5796        9.72075       3.13305       2.15345        10     
reco:pandorapid:Chi2ParticleID                    0.00643435    0.00925497     0.0182365    0.00865424    0.00322335       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0527863     0.0833008     0.117236      0.0821656     0.0201782       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00596052    0.00892669     0.015583     0.00902902    0.00254531       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.34876       7.43597       19.1521       6.42856       4.30712        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.20986       7.2011        16.8204       6.58623       3.8246         10     
reco:emtrkmichelid:EmTrackMichelId                  251.188       336.598       519.848       323.075       73.0987        10     
reco:crttag:SingleCRTMatchingProducer              0.0110717     0.0198898     0.0371374     0.0172941     0.0072261       10     
reco:crtreco:TwoCRTMatchingProducer               0.00737826     0.0143654     0.0240917     0.0153242    0.00484507       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.758048       1.25077       1.88287       1.19861      0.298689        10     
reco:pandora2Track:LArPandoraTrackCreation          9.41077       22.4282       51.8758       21.1101       10.905         10     
reco:pandora2calo:Calorimetry                       3.87543       10.236        24.342        9.10802       5.46719        10     
reco:pandora2calonosce:Calorimetry                  4.60392       9.47078       20.7801       9.3607        4.31361        10     
reco:pandora2pid:Chi2ParticleID                    0.0145312     0.0210559     0.0264138     0.0218812    0.00419774       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0651287     0.117756      0.208907      0.115408      0.0426365       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0144878     0.0230994     0.0410192     0.0220633    0.00699607       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.09463       7.73845       14.7556       7.41504       2.81371        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.24874       8.37885       15.3333       8.49687       3.01997        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.9814        7.34212       14.9005       6.83297       3.20484        10     
[art]:TriggerResults:TriggerResultInserter        3.2651e-05    5.19537e-05   0.000112342   4.69935e-05   2.24165e-05      10     
end_path:out1:RootOutput                           5.621e-06    6.9692e-06     1.042e-05    6.7125e-06    1.31688e-06      10     
end_path:out1:RootOutput(write)                     2.9962        4.37259       7.18905       4.10464        1.071         10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4960.95 MB
  Peak resident set size usage (VmHWM): 4113.67 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.06 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9609472
-rw-r--r-- 1 dune01 dune01 2272429533 Jan 30 00:54 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2159410478 Jan 30 00:51 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1946979042 Jan 30 02:51 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1728502093 Jan 29 22:09 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1659003456 Jan 29 21:50 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73143420 Jan 30 02:44 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243061 Jan 30 02:51 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 30 02:51 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 30 02:51 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 30 02:51 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 30 00:54 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 29 22:09 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 30 02:51 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 29 21:42 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 30 02:51 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 30 01:24 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 30 00:54 debugprod.log
justIN time: 2026-02-04 14:04:27 UTC       justIN version: 01.06.00