justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301203.0@dunegpschedd01.fnal.gov

Jobsub ID301203.0@dunegpschedd01.fnal.gov
Workflow ID12405
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-26 18:52:27
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-26 23:41:38
From worker nodeHostnamewn2203301.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-26 18:53:47
Input filesmonte-carlo-012405-000001
JobscriptExit code0
Real time4h (17175s)
CPU time4h (16767s = 97%)
Max RSS bytes3946098688 (3763 MiB)
Outputting started2026-01-26 23:40:03
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12405/1/001/reco_stage1.root
Finished2026-01-26 23:41:38
Saved logsjustin-logs:301203.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 120801 traces, 0 summary
Retagger: tagging trace set: gauss with 89570 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 89570 traces tagged "gauss"
FrameSaver: q=3.17195e+06 n=1538603 tag=gauss
wclsFrameSaver: saving 120801 traces tagged "wiener"
FrameSaver: q=3.37719e+06 n=1357258 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 15669 XUs and 22701 XVs -> 161 XUVs
C:0 T:1 32859 XUs and 39097 XVs -> 4304 XUVs
C:0 T:2 69896 XUs and 81135 XVs -> 4156 XUVs
C:0 T:3 39779 XUs and 33346 XVs -> 218 XUVs
C:0 T:4 14145 XUs and 13547 XVs -> 112 XUVs
C:0 T:5 53693 XUs and 44047 XVs -> 6240 XUVs
C:0 T:6 124511 XUs and 94585 XVs -> 9026 XUVs
C:0 T:7 50670 XUs and 38314 XVs -> 774 XUVs
C:0 T:8 9490 XUs and 7622 XVs -> 82 XUVs
C:0 T:9 60529 XUs and 64505 XVs -> 7495 XUVs
C:0 T:10 35233 XUs and 17134 XVs -> 3587 XUVs
C:0 T:11 105 XUs and 78 XVs -> 10 XUVs
36165 XUVs total
10354 collection wire objects
33201 potential space points
Neighbour search...
3243977 tests to find 1697354 neighbours
Iterating with no regularization...
Begin: 6.89014e+07
0 6.08409e+07
1 6.01796e+07
2 6.0069e+07
3 6.00363e+07
Now with regularization...
Begin: 4.97662e+07
0 4.97346e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411982933
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411982933
fLifetime: 30000 [us]
26-Jan-2026 23:39:59 GMT  Closed output file "reco_stage1.root"
26-Jan-2026 23:39:59 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          424.196       569.313       730.557       590.79        90.5814        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00167865    0.00419969    0.00694122    0.00461615    0.00166909       10     
reco:ophit:OpHitFinder                             0.293095      0.395888      0.464397      0.408453      0.0533302       10     
reco:opflash:OpFlashFinder                         0.182518      0.225151      0.275068      0.230407      0.0289877       10     
reco:opslicer:OpSlicer                             0.630508      0.956167       1.17584      0.967601       0.15519        10     
reco:caldata:DataPrepModule                         7.49891       8.87073       10.2418       8.62885      0.961842        10     
reco:wclsdatasp:WireCellToolkit                     58.0046       62.5096       68.6674       61.6799        3.383         10     
reco:gaushit:GausHitFinder                          1.2245        1.9854        2.50106       2.08362      0.381767        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000578572   0.000793617   0.000982418   0.000821966   0.000112769      10     
reco:reco3d:SpacePointSolver                        9.78337       17.9783       25.3689       18.7235       4.64128        10     
reco:hitpdune:DisambigFromSpacePoints               1.51238       2.83066       3.80976       2.93741      0.751501        10     
reco:pandora:StandardPandora                        26.3323       45.5936       85.4813       43.7217       14.714         10     
reco:pandoraWriter:StandardPandora                 0.412918      0.646447      0.809806      0.653821      0.119178        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.46152       9.36239       11.8824       9.6462        2.00209        10     
reco:pandoraShower:LArPandoraShowerCreation         4.63596       9.69864       16.1305       9.80722       2.98343        10     
reco:pandoracalo:Calorimetry                        1.97686       4.25112       6.22758       4.50955       1.21801        10     
reco:pandoracalonosce:Calorimetry                   2.36691       4.15644       5.79819       4.24783       1.01695        10     
reco:pandorapid:Chi2ParticleID                    0.00476182     0.0095411     0.0115416    0.00964847    0.00180361       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0815523     0.0968271      0.11216      0.0973181     0.0106396       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00567558    0.00882176     0.0112686    0.00906054    0.00148715       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.31043       9.1283        14.6325       9.69066       3.10846        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.4495        8.78582       14.3875       9.11046       2.63402        10     
reco:emtrkmichelid:EmTrackMichelId                  231.945       292.086       360.903       302.839       39.2808        10     
reco:crttag:SingleCRTMatchingProducer              0.0273972     0.0339906     0.0480763     0.0326131    0.00575642       10     
reco:crtreco:TwoCRTMatchingProducer                0.0197461     0.026267      0.0343944     0.0258172    0.00435722       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.05098       1.57983       1.97125       1.73271      0.328236        10     
reco:pandora2Track:LArPandoraTrackCreation          13.3811       30.2308       45.0649       30.3425       8.75556        10     
reco:pandora2calo:Calorimetry                       4.28832        12.65        17.2905       13.9868       4.29365        10     
reco:pandora2calonosce:Calorimetry                  4.80848       12.2481       17.7887       12.6413       3.80093        10     
reco:pandora2pid:Chi2ParticleID                   0.00988489      0.02041      0.0293904     0.0211927     0.005681        10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP         0.06574      0.110364      0.138964      0.119046      0.0249064       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0098589     0.0197627     0.0293883     0.0204126    0.00565035       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.17407       9.43204       12.6445       10.0634       2.81992        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.72566       10.1494       13.4806       11.5235       3.11163        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.86302       9.37212       13.0387       10.4706       2.8639         10     
[art]:TriggerResults:TriggerResultInserter        7.3506e-05    0.000103654   0.000204544   9.8923e-05    3.69447e-05      10     
end_path:out1:RootOutput                          1.0417e-05    1.44004e-05   2.0966e-05    1.45065e-05   3.14918e-06      10     
end_path:out1:RootOutput(write)                     3.08073       3.6928        4.55491       3.37154      0.530497        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4753.05 MB
  Peak resident set size usage (VmHWM): 3946.1 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.39 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.30 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9115596
-rw-r--r-- 1 dune01 dune01 2182044354 Jan 26 22:01 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2072665290 Jan 26 21:58 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1831066675 Jan 26 23:39 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1629393775 Jan 26 19:17 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1552678406 Jan 26 19:01 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   65893320 Jan 26 23:32 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242810 Jan 26 23:40 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 26 23:40 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 26 23:40 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 26 23:40 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 26 22:01 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 26 19:17 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 26 23:40 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 26 18:53 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 26 23:40 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 26 22:09 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 26 22:01 debugprod.log
justIN time: 2026-02-04 20:38:31 UTC       justIN version: 01.06.00