justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264902.0@dunegpschedd01.fnal.gov

Jobsub ID264902.0@dunegpschedd01.fnal.gov
Workflow ID11079
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-08 13:28:30
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-08 17:58:36
From worker nodeHostnamewn2207290.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-08 13:30:12
Input filesmonte-carlo-011079-000001
JobscriptExit code0
Real time4h (16018s)
CPU time4h (15149s = 94%)
Max RSS bytes4212469760 (4017 MiB)
Outputting started2025-12-08 17:57:11
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11079/1/001/reco_stage1.root
Finished2025-12-08 17:58:36
Saved logsjustin-logs:264902.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 126407 traces, 0 summary
Retagger: tagging trace set: gauss with 96911 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96911 traces tagged "gauss"
FrameSaver: q=3.29775e+06 n=1770850 tag=gauss
wclsFrameSaver: saving 126407 traces tagged "wiener"
FrameSaver: q=3.49273e+06 n=1580281 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 32142 XUs and 28637 XVs -> 289 XUVs
C:0 T:1 61794 XUs and 53957 XVs -> 3854 XUVs
C:0 T:2 57145 XUs and 71214 XVs -> 2455 XUVs
C:0 T:3 31509 XUs and 33827 XVs -> 263 XUVs
C:0 T:4 12026 XUs and 9584 XVs -> 55 XUVs
C:0 T:5 53573 XUs and 30367 XVs -> 8449 XUVs
C:0 T:6 116678 XUs and 208067 XVs -> 14192 XUVs
C:0 T:7 44916 XUs and 47034 XVs -> 697 XUVs
C:0 T:8 7288 XUs and 7904 XVs -> 147 XUVs
C:0 T:9 73612 XUs and 75762 XVs -> 10124 XUVs
C:0 T:10 70614 XUs and 108922 XVs -> 12515 XUVs
C:0 T:11 85 XUs and 74 XVs -> 7 XUVs
53047 XUVs total
12339 collection wire objects
49290 potential space points
Neighbour search...
6862084 tests to find 3339752 neighbours
Iterating with no regularization...
Begin: 6.94863e+07
0 5.40781e+07
1 5.31662e+07
2 5.30024e+07
3 5.29449e+07
4 5.29165e+07
Now with regularization...
Begin: 4.4156e+07
0 4.40889e+07
1 4.40455e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410992629
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410992629
fLifetime: 30000 [us]
08-Dec-2025 17:57:08 GMT  Closed output file "reco_stage1.root"
08-Dec-2025 17:57:08 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          201.113       471.894       706.801       451.615       153.607        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00141763    0.00723577     0.012458     0.00602204     0.0036438       10     
reco:ophit:OpHitFinder                             0.298574      0.388038      0.610713       0.37902      0.085335        10     
reco:opflash:OpFlashFinder                         0.137246      0.222725      0.415194      0.201139      0.0791181       10     
reco:opslicer:OpSlicer                             0.560085      0.925246       1.50035      0.917783      0.260757        10     
reco:caldata:DataPrepModule                         5.83795       7.97606       8.67809       8.15561      0.785597        10     
reco:wclsdatasp:WireCellToolkit                     38.2119       53.5905       62.5151       54.9215       7.54774        10     
reco:gaushit:GausHitFinder                         0.715543       1.5825        2.38534       1.63455      0.471684        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000451148   0.00112878     0.0018722    0.00118177    0.000389373      10     
reco:reco3d:SpacePointSolver                        6.14359       21.678        50.1858       19.9681       11.8279        10     
reco:hitpdune:DisambigFromSpacePoints              0.675173       2.04683       3.73229       2.00171      0.917215        10     
reco:pandora:StandardPandora                        14.4588       37.5337       69.0121       32.4744       16.283         10     
reco:pandoraWriter:StandardPandora                 0.294788      0.670968       1.11801      0.667273      0.229812        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.68133       7.81445       14.3613       7.12915       4.00232        10     
reco:pandoraShower:LArPandoraShowerCreation          1.424        8.66233       19.817        8.69455       5.87148        10     
reco:pandoracalo:Calorimetry                       0.968914       4.48469       9.50601       4.13232       2.85164        10     
reco:pandoracalonosce:Calorimetry                  0.857016       3.99924       9.18898       3.90963       2.62719        10     
reco:pandorapid:Chi2ParticleID                    0.00364684    0.00779806     0.0124946     0.0075167    0.00282215       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0324633     0.068511      0.119422      0.0559039     0.0281426       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00351016    0.00678415     0.0100955    0.00615479    0.00234605       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.47947       8.92007        20.88        8.55698       5.9531         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.3714        7.91954       17.3151       6.93834       4.95993        10     
reco:emtrkmichelid:EmTrackMichelId                  101.621       225.946       314.185       228.634       59.7078        10     
reco:crttag:SingleCRTMatchingProducer              0.0106535     0.0329869     0.078481      0.0282386     0.0178404       10     
reco:crtreco:TwoCRTMatchingProducer               0.00475342     0.0236706     0.0580765     0.0182646     0.0151918       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.523246       1.0515        1.58911       1.04394      0.300767        10     
reco:pandora2Track:LArPandoraTrackCreation          4.20625       23.7758       43.1687       23.1477       13.0359        10     
reco:pandora2calo:Calorimetry                       2.16393       11.5147       22.9028       9.91716       6.99526        10     
reco:pandora2calonosce:Calorimetry                  2.02293       10.4717       20.0154       7.53744       6.52806        10     
reco:pandora2pid:Chi2ParticleID                   0.00740096     0.0188158     0.0266812     0.0206368    0.00621819       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.035377      0.0983774     0.167706      0.102364      0.0375586       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00724258     0.017867      0.0252093     0.018126     0.00570002       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.02739       7.78075       15.5774       5.47341       4.49219        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.80581       9.07549       15.8318       8.62735       4.69027        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.52714       9.71257       16.9436       8.38942       5.28426        10     
[art]:TriggerResults:TriggerResultInserter        0.000111199   0.000199444   0.00040447    0.000139302   0.000104465      10     
end_path:out1:RootOutput                          1.5078e-05    2.27447e-05   3.2521e-05    2.1896e-05    5.4925e-06       10     
end_path:out1:RootOutput(write)                     2.30009       3.62557       4.74902       3.75145      0.818649        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5058.13 MB
  Peak resident set size usage (VmHWM): 4212.47 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.40 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.76 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10051364
-rw-r--r-- 1 dune01 dune01 2364360305 Dec  8 16:35 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2246937328 Dec  8 16:32 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2032021685 Dec  8 17:57 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1828780660 Dec  8 14:05 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1750084823 Dec  8 13:40 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   69780280 Dec  8 17:52 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242722 Dec  8 17:57 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  8 17:57 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  8 17:57 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  8 17:57 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  8 16:35 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  8 14:05 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  8 17:57 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  8 13:30 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  8 17:57 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  8 16:42 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  8 16:35 debugprod.log
justIN time: 2026-02-07 16:25:18 UTC       justIN version: 01.06.00